🚀 原文链接：https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html
Airflow allows you to use your own Python modules in the DAG and in the Airflow configuration. The following article will describe how you can create your own module so that Airflow can load it correctly, as well as diagnose problems when modules are not loaded properly.
1. Packages Loading in Python
The list of directories from which Python tries to load the module is given by the variable sys.path. Python really tries to intelligently determine the contents of of this variable, including depending on the operating system and how Python is installed and which Python version is used.
You can check the contents of this variable for the current Python environment by running an interactive terminal as in the example below:
>>> import sys>>> from pprint import pprint>>> pprint(sys.path)['','/home/arch/.pyenv/versions/3.7.4/lib/python37.zip','/home/arch/.pyenv/versions/3.7.4/lib/python3.7','/home/arch/.pyenv/versions/3.7.4/lib/python3.7/lib-dynload','/home/arch/venvs/airflow/lib/python3.7/site-packages']
sys.path is initialized during program startup. The first precedence is given to the current directory, i.e, path[0] is the directory containing the current script that was used to invoke or an empty string in case it was an interactive shell. Second precedence is given to the PYTHONPATH if provided, followed by installation-dependent default paths which is managed by site module.
sys.path can also be modified during a Python session by simply using append (for example, sys.path.append("/path/to/custom/package")). Python will start searching for packages in the newer paths once they’re added. Airflow makes use of this feature as described in the section Additional modules in Airflow.
In the variable sys.path there is a directory site-packages which contains the installed external packages, which means you can install packages with pip or anaconda and you can use them in Airflow. In the next section, you will learn how to create your own simple installable package and how to specify additional directories to be added to sys.path using the environment variable PYTHONPATH.
2. Creating a package in Python
- Before starting, install the following packages:
setuptools: setuptools is a package development process library designed for creating and distributing Python packages.wheel: The wheel package provides a bdist_wheel command for setuptools. It creates .whl file which is directly installable through thepip installcommand. We can then upload the same file to PyPI.
$ pip install --upgrade pip setuptools wheel
Create the package directory - in our case, we will call it
airflow_operators.$ mkdir airflow_operators
Create the file
__init__.pyinside the package and add following code:print("Hello from airflow_operators")
When we import this package, it should print the above message.
- Create
setup.py:
```bash import setuptools
setuptools.setup( name=’airflow_operators’, )
5. Build the wheel:<br />```bash$ python setup.py bdist_wheel
This will create a few directories in the project and the overall structure will look like following:
.├── airflow_operators│ ├── __init__.py├── airflow_operators.egg-info│ ├── PKG-INFO│ ├── SOURCES.txt│ ├── dependency_links.txt│ └── top_level.txt├── build│ └── bdist.macosx-10.15-x86_64├── dist│ └── airflow_operators-0.0.0-py3-none-any.whl└── setup.py
Install the .whl file using pip:
$ pip install dist/airflow_operators-0.0.0-py3-none-any.whl
The package is now ready to use!
>>> import airflow_operatorsHello from airflow_operators
The package can be removed using pip command:
$ pip uninstall airflow_operators
For more details on how to create to create and publish python packages, see Packaging Python Projects.
3. Adding directories to the path
You can specify additional directories to be added to sys.path using the environment variable PYTHONPATH. Start the python shell by providing the path to root of your project using the following command:
PYTHONPATH=/home/arch/projects/airflow_operators python
The sys.path variable will look like below:
>>> import sys>>> from pprint import pprint>>> pprint(sys.path)['','/home/arch/projects/airflow_operators''/home/arch/.pyenv/versions/3.7.4/lib/python37.zip','/home/arch/.pyenv/versions/3.7.4/lib/python3.7','/home/arch/.pyenv/versions/3.7.4/lib/python3.7/lib-dynload','/home/arch/venvs/airflow/lib/python3.7/site-packages']
As we can see that our provided directory is now added to the path, let’s try to import the package now:
>>> import airflow_operatorsHello from airflow_operators
We can also use PYTHONPATH variable with the airflow commands. For example, if we run the following airflow command:
PYTHONPATH=/home/arch/projects/airflow_operators airflow info
We’ll see the Python PATH updated with our mentioned PYTHONPATH value as shown below:
Python PATH: [/home/arch/venv/bin:/home/arch/projects/airflow_operators:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/home/arch/venv/lib/python3.8/site-packages:/home/arch/airflow/dags:/home/arch/airflow/config:/home/arch/airflow/plugins]
4. Additional modules in Airflow
Airflow adds three additional directories to the sys.path:
- DAGS folder: It is configured with option
dags_folderin section[core]. - Config folder: It is configured by setting
AIRFLOW_HOMEvariable ({AIRFLOW_HOME}/config) by default. - Plugins Folder: It is configured with option
plugins_folderin section[core].
You can also see the exact paths using the airflow info command, and use them similar to directories specified with the environment variable PYTHONPATH. An example of the contents of the sys.path variable specified by this command may be as follows:
Python PATH: [/home/rootcss/venvs/airflow/bin:/usr/lib/python38.zip:/usr/lib/python3.8:/usr/lib/python3.8/lib-dynload:/home/rootcss/venvs/airflow/lib/python3.8/site-packages:/home/rootcss/airflow/dags:/home/rootcss/airflow/config:/home/rootcss/airflow/plugins]
Below is the sample output of the airflow info command:
:::tips
👀 See also
:::
:::info
When are plugins (re)loaded?
:::
➜ workspace airflow infoApache Airflow: 2.0.2System infoOS | Linuxarchitecture | x86_64uname | uname_result(system='Linux', node='wd-ws-prod-env-yumingmin-ubuntu18-gpu-649c5857fd-nxbx7',| release='5.8.14-1.el7.elrepo.x86_64', version='#1 SMP Mon Oct 5 19:02:22 EDT 2020', machine='x86_64',| processor='x86_64')locale | ('en_US', 'UTF-8')python_version | 3.6.10 | packaged by conda-forge | (default, Apr 24 2020, 16:44:11) [GCC 7.3.0]python_location | /opt/conda/bin/python3.6Tools infogit | git version 2.17.1ssh | OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017kubectl | NOT AVAILABLEgcloud | NOT AVAILABLEcloud_sql_proxy | NOT AVAILABLEmysql | mysql Ver 14.14 Distrib 5.7.33, for Linux (x86_64) using EditLine wrappersqlite3 | 3.34.0 2020-12-01 16:14:00 a26b6597e3ae272231b96f9982c3bcc17ddec2f2b6eb4df06a224b91089fed5bpsql | NOT AVAILABLEPaths infoairflow_home | /opt/workspace/yumingmin_shared_drive/bigdata/airflowsystem_path | /opt/conda/bin:/opt/conda/condabin:/opt/code-server:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/| usr/bin:/sbin:/binpython_path | /opt/conda/bin:/opt/conda/lib/python36.zip:/opt/conda/lib/python3.6:/opt/conda/lib/python3.6/lib-dynload:/opt/conda/lib/python3.6/site-packages:/opt/wo| rkspace/yumingmin_shared_drive/bigdata/airflow/dags:/opt/workspace/yumingmin_shared_drive/bigdata/airflow/config:/opt/workspace/yumingmin_shared_drive/| bigdata/airflow/pluginsairflow_on_path | TrueConfig infoexecutor | SequentialExecutortask_logging_handler | airflow.utils.log.file_task_handler.FileTaskHandlersql_alchemy_conn | mysql://root:root@localhost:3306/airflow?unix_socket=/var/run/mysqld/mysqld.sockdags_folder | /opt/workspace/yumingmin_shared_drive/bigdata/airflow/dagsplugins_folder | /opt/workspace/yumingmin_shared_drive/bigdata/airflow/pluginsbase_log_folder | /opt/workspace/yumingmin_shared_drive/bigdata/airflow/logsProviders infoapache-airflow-providers-amazon | 1.4.0apache-airflow-providers-apache-cas | 1.0.1sandra |apache-airflow-providers-apache-dru | 1.1.0id |apache-airflow-providers-apache-hdf | 1.0.1s |apache-airflow-providers-apache-hiv | 1.0.3e |apache-airflow-providers-apache-kyl | 1.0.1in |apache-airflow-providers-apache-liv | 1.1.0y |apache-airflow-providers-apache-pig | 1.0.1apache-airflow-providers-apache-pin | 1.0.1ot |apache-airflow-providers-apache-spa | 1.0.3rk |apache-airflow-providers-apache-sqo | 1.0.1op |apache-airflow-providers-celery | 1.0.1apache-airflow-providers-cloudant | 1.0.1apache-airflow-providers-cncf-kuber | 1.2.0netes |apache-airflow-providers-databricks | 1.0.1apache-airflow-providers-datadog | 1.0.1apache-airflow-providers-dingding | 1.0.2apache-airflow-providers-discord | 1.0.1apache-airflow-providers-docker | 1.2.0apache-airflow-providers-elasticsea | 1.0.4rch |apache-airflow-providers-exasol | 1.1.1apache-airflow-providers-facebook | 1.1.0apache-airflow-providers-ftp | 1.1.0apache-airflow-providers-grpc | 1.1.0apache-airflow-providers-hashicorp | 1.0.2apache-airflow-providers-http | 1.1.1apache-airflow-providers-imap | 1.0.1apache-airflow-providers-jdbc | 1.0.1apache-airflow-providers-jenkins | 1.1.0apache-airflow-providers-jira | 1.0.2apache-airflow-providers-microsoft- | 2.0.0azure |apache-airflow-providers-microsoft- | 1.1.0mssql |apache-airflow-providers-microsoft- | 1.2.0winrm |apache-airflow-providers-mongo | 1.0.1apache-airflow-providers-mysql | 1.1.0apache-airflow-providers-odbc | 1.0.1apache-airflow-providers-openfaas | 1.1.1apache-airflow-providers-opsgenie | 1.0.2apache-airflow-providers-oracle | 1.1.0apache-airflow-providers-pagerduty | 1.0.1apache-airflow-providers-papermill | 1.0.2apache-airflow-providers-plexus | 1.0.1apache-airflow-providers-postgres | 1.0.2apache-airflow-providers-presto | 1.0.2apache-airflow-providers-qubole | 1.0.2apache-airflow-providers-redis | 1.0.1apache-airflow-providers-salesforce | 2.0.0apache-airflow-providers-samba | 1.0.1apache-airflow-providers-segment | 1.0.1apache-airflow-providers-sendgrid | 1.0.2apache-airflow-providers-sftp | 1.2.0apache-airflow-providers-singularit | 1.1.0y |apache-airflow-providers-slack | 3.0.0apache-airflow-providers-snowflake | 1.3.0apache-airflow-providers-sqlite | 1.0.2apache-airflow-providers-ssh | 1.3.0apache-airflow-providers-tableau | 1.0.0apache-airflow-providers-telegram | 1.0.2apache-airflow-providers-vertica | 1.0.1apache-airflow-providers-yandex | 1.0.1apache-airflow-providers-zendesk | 1.0.1
若有收获，就点个赞吧
0 人点赞
