When assessing the two solutions, reviewers found Apache Airflow easier to use and administer. However, reviewers preferred the ease of set up, and doing business with AWS Step Functions overall.
the timeline it provides and functional hooks (lambdas) which can be triggered during every step transition.
Step functions can only submit one spark streaming job in an EMR. It should be enhanced to be able to submit multiple spark streaming jobs in the same EMR in parallel.
Easy to configure a new pipeline once you create a codebase. I currently use airflow to manage our extraction, transformation, enrichment and quality checks. Because most of our pipelines follow a similar format we can create a standard pipeline swapping...
Need documentation, writeups, blogs and awareness about the product
the timeline it provides and functional hooks (lambdas) which can be triggered during every step transition.
Easy to configure a new pipeline once you create a codebase. I currently use airflow to manage our extraction, transformation, enrichment and quality checks. Because most of our pipelines follow a similar format we can create a standard pipeline swapping...
Step functions can only submit one spark streaming job in an EMR. It should be enhanced to be able to submit multiple spark streaming jobs in the same EMR in parallel.
Need documentation, writeups, blogs and awareness about the product