Apache Airflow Pros and Cons

I like the airflow features such as dags/scripts monitoring. We can even track past 365 days history and logs.

TriggerDagrun operator is best for dependency management across all dags. Review collected by and hosted on G2.com.

I am missing connections feature across multiple sources in apache Airflow which can help us to manage end to end ETL system. Currentely it is taking sometime to Start dag/pod, it would be great if they can make it quick. Review collected by and hosted on G2.com.

Easy to use the tool. Contains all kinds of processors which support all requirements. Support isolation of projects using processor group. Also helps to send or receive data from the remote cluster Review collected by and hosted on G2.com.

If we have less memory in the cluster, it does not support auto-scaling. If one processor fails, we need to restart everything from scratch. Sometimes we have seen the flow file corruption Review collected by and hosted on G2.com.

It is an open source platform for scheduling, monitoring and developing data and compute pipelines very easily. It uses python to create workflows and it can run anything in the world. It is easy to use. Open source community is good. Good set of ready to use operators. Easy coding with python. And good graphical UI Review collected by and hosted on G2.com.

Although it is really good platform for managing workflows still it has some issues like there is no true way to monitor the quality of your data. Learning airflow requires a time investment. No guide on best practices on developing DAGs. No versioning in the airflow scheduler. Debugging is very time consuming. Review collected by and hosted on G2.com.

With a gradual learning curve, Airflow helped schedule our workflows and trigger airflow DAGs from other GCP products like cloud function, allowing for event-based workflows dependent on multiple GCP products and data sources from various teams. Review collected by and hosted on G2.com.

Apache airflow is just an orchestrator, and it's unreliable with jobs with higher run-time, like ML model training using Vertex Custom Job or some BigQuery SQL scripts that run for a long time. We have also faced situations where the BQ job ran successfully, but Airflow had lost connection. There are also connection issues with dataproc pyspark jobs. Review collected by and hosted on G2.com.

It is a very great tool for job orchestration in the data processing pipelines. It helps me sleep well at night as i can scheduled my job prior before the time needed. Review collected by and hosted on G2.com.

It requires a little complexity and technical background for optimal use . someone without the knowledge of programming cannot be confidently use this tool Review collected by and hosted on G2.com.

The ease of managing Data pipelines , ML workflows has helped developers focus on other aspects.

The DAGS are pretty straightforward to implement and efficient

The Scheudler is far more ahead when it comes to peers Review collected by and hosted on G2.com.

The Metadata DB management on a long term perspective seems to be tedious for Airflow developer

The Airflow starts to show slowness in dag updates of the metadata sb is populated to extent

In Airflow 2 , there is no Option for ad hoc query , which is kinda trouble when we need check metadata DB Review collected by and hosted on G2.com.

flexibility and ease in writing dags. multiple operator and executors to run the job. Review collected by and hosted on G2.com.

somtime shceduler with too many jobs takes too much CPU of DB. so it need to have bigger db most of the time for high scale. Review collected by and hosted on G2.com.

Airflow is way better orchestration tool than any other tool in the market.it is based on python so it is super easy to create dags and schedule it.there are plenty of inbuilt operators which performs Swift operation.creating cluster and replying dags in GCP is super easy Review collected by and hosted on G2.com.

But I still feels there are some capabilities still needs to be built like dataflow. Plus installation in your local system should be more easy and documentation should be there . Also dependency between dags should also be improved. Review collected by and hosted on G2.com.

The simplicity of use and how it facilitates its use for teams. Review collected by and hosted on G2.com.

The lack of community and use cases or best practices. Review collected by and hosted on G2.com.

Airflow provides all the features like DAG like workflows and action and scripts running features.

It also integrate with running environments like celery or kubernetes that helps to run multiple job parallel. Review collected by and hosted on G2.com.

It takes times to setup environment and for enty lavel personal it is difficult to setup it.

I think if airflow provides cloud tool integration and easy setup it will help many users Review collected by and hosted on G2.com.