Airflow Component Introduction

Airflow has only one set of code, but it contains several components:

  • Webserver: WebServer is a Web service that, when started, provides a visual interface to view and manage daGs.
  • Scheduler: The scheduler is a scheduler that does not rely on the WebServer and schedules tasks to be executed

Airflow is available in both Running Airflow locally and Running Airflow in Docker, both of which are described on the official website

Local deployment

Environmental requirements

Airflow is developed in Python and therefore requires a Python environment regardless of platform. The version required is python3.8.12 and airflow2.1.3.

  1. Airflow requires a home directory:
    • The new directory~/airflow
    • performexport AIRFLOW_HOME=~/airflow
  2. Install the airflow

Well, if you can’t access it, you might need to do something about it

PIP install apache - airflow = = 2.1.3 - the constraint "Https://raw.githubusercontent.com/apache/airflow/constraints-2.1.3/constraints-3.8.txt"Copy the code
  1. Initializing the database

Sqllite as requested by default. What configuration do we need to do? Simply run the following command:

airflow db init airflow users create \ --username admin \ --firstname Peter \ --lastname Parker \ --role Admin \ --email  [email protected]Copy the code

Actually, we have installed airflow at this point, so if we need the ability to perform tasks manually without the need for automatic scheduling or visualization, we will launch the two services

  1. Start the webserver
airflow webserver --port 8080
Copy the code

Access localhost: 8080

  1. Start the scheduler
airflow scheduler
Copy the code

Docker deployment

Download the docker – compose. Yml

$ curl -LO https://raw.githubusercontent.com/bitnami/bitnami-docker-airflow/master/docker-compose.yml
Copy the code

Create the required directory & environment variable configuration

$ mkdir -p ./dags ./logs ./plugins
$ echo -e "AIRFLOW_UID=$(id -u)\nAIRFLOW_GID=0" > .env
Copy the code

Data initialization

$ docker-compose up airflow-init
Copy the code

Start the

$ docker-compose up
Copy the code

Stop the service

$ docker-compose down --volumes --remove-orphans
Copy the code

Visit localhos: 8080

Matters needing attention:

There are many instances of the startup container. Note whether the memory is sufficient

recruitment

Little red book, one of the few reliable Internet unicorn at present, want to get on the bus to look at it ~

Urgent recruitment of big data development, data platform development, other departments can help internal promotion, looking for talents.

Interested message plus wechat chat!!