Apache Airflow Pros and Cons

Use of cron tab expression, importing various modules, importing user defined operators. Review collected by and hosted on G2.com.

declaring dag in a fixed pattern or else the scheduler won't pick up ypur dag and show import error Review collected by and hosted on G2.com.

Airflow is very scalable

Dynamic Pipeline integration

We can easily define our own operator by extending pre defined libraries

We can connect Airflow with so many applications and Data Warehouses like Databricks, MySQL and so on Review collected by and hosted on G2.com.

User Interface Struggles

It is sometime hectic to manage the metadata database of Airflow

Performance Struggles sometimes when we create numerous tasks

Limited built in features Review collected by and hosted on G2.com.

- Best open source software to get started on

- Great material online to troubleshoot and community Review collected by and hosted on G2.com.

- Needs a dedicated data engineer and devops

- Maintianence could take lot of time

- needs another tool for data quality measurement Review collected by and hosted on G2.com.

The best thing about Airflow is how versatile of a tool it is. Airflow can be used to build workflow on just about every database and tool and the sheer wealth of integrations it has is brilliant and just all-round useful Review collected by and hosted on G2.com.

Learning Airflow is a seriously complicated task. And even that is not often enough to become truly good at it. The scheduling system is hardly intuitive. Versioning for the Dags and reverting them a very simple task in a competitor Prefect is not a part of Airflow at all Review collected by and hosted on G2.com.

Python is my favorite coding language, the DAGs in Airflow are written in Python. There are several built-in operators in Airflow to execute the Python function, call Databricks job and execute bash commands. I love to build the pipeline in Airflow Review collected by and hosted on G2.com.

The Airflow orchestration tool is a bit complicated in developing compared to other pipeline tools. While other tools have drag-and-drop options, coding in Airflow takes more time. Review collected by and hosted on G2.com.

It is easy configure

it is easy to handle script over UI

It shows error on UI where your script got error

Can run script easily Review collected by and hosted on G2.com.

For multiple file triggering it is a bit difficult Review collected by and hosted on G2.com.

- easy scheduling

- python framework, so easy to learn

- can be dockerized easily with some tutorials

- easy to learn even for beginners.

- better than other scheduling tools Review collected by and hosted on G2.com.

- the UI can be made more customizable.

- concurrency is low if you have a small system.

- the UI can be daunting for a few people like managerial positions.

- security can be improved Review collected by and hosted on G2.com.

I really like the ability to view the failure and status of each step in a workflow quickly. I like that it has the ability to retry only what fails and gives a lot of control. Review collected by and hosted on G2.com.

I do not like that there is a good deal of latency between starting tasks with the default settings. I might be able to reduce it but it will require a decent effort to do so. Review collected by and hosted on G2.com.

Apache Airflow provides a very good user interface and it is very easy to work on this tool. It also provides various features for the representation of pipelines. It provides several stages for the DAG like running, failed, etc. There are different colors for different stages in this tool. Review collected by and hosted on G2.com.

When we run any DAG in the Apache Airflow the DAG failed when it will not get the desired file from upstream but it does not make proper logs of the successful stage. We need to check the logs in the EC2 logs whether our data is successfully loaded or not. Most of the time Apache Airflow shows success on the DAG but actually, the job failed. Review collected by and hosted on G2.com.

It is something many use and feel comfortable with. That huge ecosystem provides alot of benefits. Review collected by and hosted on G2.com.

It is fundamentally a flawed design. They are making progress in overcoming some of the scaling issues, and it is improving Review collected by and hosted on G2.com.