airflow - Meltano Hub

Airflow

airflow from apache

The airflow Meltano orchestrator pulls data from Airflow that can then be sent to a destination using a loader.

Other Available Variants

apache (default)

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

Installation and configuration

Add the airflow orchestrator to your project using
```
meltano add
```
:

meltano add orchestrator airflow

Configure the airflow settings using
```
meltano config
```
:

meltano config airflow set --interactive

Next steps

Use the meltano schedule command to create pipeline schedules in your project, to be run by Airflow.

If you're running Airflow for the first time in a new environment, create an admin user:

meltano invoke airflow:create-admin
# This is equivalent to `airflow users create` with some arguments in the Airflow documentation

Launch the Airflow UI and log in using the username/password you created:
```
meltano invoke airflow:ui
```
By default, the UI will be available at at http://localhost:8080. You can change this using the webserver.web_server_port setting documented below.
Start Scheduler or execute Airflow commands directly using the instructions in the Meltano docs.

If you run into any issues, learn how to get help.

Capabilities

The current capabilities for

airflow

may have been automatically set when originally added to the Hub. Please review the capabilities when using this orchestrator. If you find they are out of date, please consider updating them by making a pull request to the YAML file that defines the capabilities for this orchestrator.

This plugin has the following capabilities:

You can override these capabilities or specify additional ones in your meltano.yml by adding the capabilities key.

Settings

The airflow settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

core.dags_folder
core.plugins_folder
core.sql_alchemy_conn
core.load_examples
core.dags_are_paused_at_creation
webserver.web_server_port

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

DAGs Folder (core.dags_folder)

Environment variable: AIRFLOW_CORE_DAGS_FOLDER
Default Value: $MELTANO_PROJECT_ROOT/orchestrate/dags

[No description provided.]

Plugins Folder (core.plugins_folder)

Environment variable: AIRFLOW_CORE_PLUGINS_FOLDER
Default Value: $MELTANO_PROJECT_ROOT/orchestrate/plugins

[No description provided.]

SQL Alchemy Connection (core.sql_alchemy_conn)

Environment variable: AIRFLOW_CORE_SQL_ALCHEMY_CONN
Default Value: sqlite:///$MELTANO_PROJECT_ROOT/.meltano/orchestrators/airflow/airflow.db

[No description provided.]

Load Examples (core.load_examples)

Environment variable: AIRFLOW_CORE_LOAD_EXAMPLES
Default Value: false

[No description provided.]

Pause DAGs at Creation (core.dags_are_paused_at_creation)

Environment variable: AIRFLOW_CORE_DAGS_ARE_PAUSED_AT_CREATION
Default Value: false

[No description provided.]

Webserver Port (webserver.web_server_port)

Environment variable: AIRFLOW_WEBSERVER_WEB_SERVER_PORT
Default Value: 8080

[No description provided.]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the airflow orchestrator to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the

#plugins-general

channel.

Install

meltano add orchestrator airflow

Maintenance Status

Repo

https://github.com/apache/airflow

Maintainer

Meltano Stats

Keywords