Airflow Detailed construction process （ Personal test + summary ）

Official website :Apache airflow
Airflow It is a platform created by the community , Used to write... Programmatically , Schedule and monitor workflows .
I took a ride half a month ago , Now make a note , Don't push , Started to build , Whole process multi map ：

Environmental preparation

System : cent os 7
conda edition : 4.8.2
airflow edition 1.10.11

Begin to build

I will use one conda Create a apache airflow Environment :

conda create -n airflow_env python=3.7

Switch to the current environment ：

conda activate airflow_env

build airflow

build airflow Words , The official website has a set of detailed documents Airflow build
Then follow this way

# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
export AIRFLOW_HOME=~/airflow

# install from pypi using pip
pip install apache-airflow

# initialize the database
airflow initdb

# start the web server, default port is 8080
# airflow webserver -p 8080     Make a change here , Followed by -D Parameters let it run in the background 
airflow webserver -p 8080 -D	

# start the scheduler
airflow scheduler

# visit localhost:8080 in the browser and enable the example dag in the home page

After the above steps , You can open this page in the browser :

Build here , You can see a basic appearance , But for now , I can find the following problems ：

• airflow The metadata storage information of is used by default sqlite For storage .
• The time in the upper right corner of the page is later than normal 8 Hour
• airflow The default is to schedule tasks with a single thread , Here's the picture （ Then I know ）
  
  Now the problem has been exposed , Then start processing .

airflow Metadata storage of the default is sqlite, Now switch to mysql

sqlite Multithreading not supported , So I plan to switch to mysql,mysql For installation, please refer to :Linux centos install mysql
airflow There are many other options available in , So in airflow Inside, we can choose some required components , For example, now we need to use mysql Storage airflow Information about , The list of optional functions is here airlfow Other options for
Need pip install 'apache-airflow[mysql]'
here , I won't choose , Come directly to the full set of plug-ins , This will encounter more pits ：

yum install mysql-devel gcc gcc-devel python-devel krb5-devel.x86_64 cyrus-sasl-devel  -y
pip install 'apache-airflow[all]' -i https://pypi.tuna.tsinghua.edu.cn/simple/

At this time, you may encounter many mistakes , For example, the following :

The mistakes in this should be handled with patience , There are usually errors in the version , Uninstall the version appearing in the scarlet letter , Just re specify the version number and install .

To configure airflow

• To configure mysql As airflow Storage of metadata information ：
  • vim /etc/my.cnf stay [mysqld] Add... Below explicit_defaults_for_timestamp=1
  • systemctl restart mysqld restart mysql
  • stay mysql Create a database , The name of my database is airflow
    
  • Get into airflow Under folder , modify airflow.cfg.
    
• solve airflow The problem that only one task can be performed at a time :
  
  Please refer to airflow practitioners
• solve airflow The problem of being eight hours late
  modify airflow.cfg The content of :
  
  This part is for reference airflow Modify source code
  The specific contents are as follows :

  # find airflow Installation position of 
  find / -name airflow
  

  

1. modify /root/anaconda3/envs/airflow_env/lib/python3.7/site-packages/airflow/utils/timezone.py
   In the 27 Add below line :
   

   from airflow import configuration as conf
   try:
   	tz = conf.get("core", "default_timezone")
   	if tz == "system":
   		utc = pendulum.local_timezone()
   	else:
   		utc = pendulum.timezone(tz)
   except Exception:
   	pass
   

   modify utcnow() function :
   
2. modify /root/anaconda3/envs/airflow_env/lib/python3.7/site-packages/airflow/utils/sqlalchemy.py
   In the 38 Add below line : utc = pendulum.timezone('UTC')

from airflow import configuration as conf
try:
	tz = conf.get("core", "default_timezone")
	if tz == "system":
		utc = pendulum.local_timezone()
	else:
		utc = pendulum.timezone(tz)
except Exception:
	pass

Comment this out :

3. modify /root/anaconda3/envs/airflow_env/lib/python3.7/site-packages/airflow/www/templates/admin/master.html

Modify the line indicated by the down arrow , The modified content is under the comment
At this time, the modification is completed , Restart at this time airflow Just fine . Modify the operation reference of the source code :airflow Modify China time zone ( Change airflow Source code )

The final result is as follows ：

Used mysql As the storage of metadata information , stay mysql You can see it in it :

Sign in airflow When :

Solved the three problems mentioned before

summary

This airflow As a scheduling tool , This time, I just mentioned how to install and build , When you are free in the future , I want to add an article on how to write airlfow Task script for . The current level is only the writing of simple task scripts , More work is needed .
Last , If the boss reads it , Find out what's wrong with this document , Or what better advice , Leave a comment below . thank you ！