התחל במצב לא מקוון עם האפליקציה Player FM !
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
Manage episode 465365556 series 2948506
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
58 פרקים
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 465365556 series 2948506
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
58 פרקים
כל הפרקים
×
1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 Airflow’s Role in the Rise of DataOps with Andy Byron 26:15

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27

1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46

1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40

1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54

1 Leveraging Airflow To Build Scalable and Reliable Data Platforms at 99acres.com with Samyak Jain 25:08
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.