התחל במצב לא מקוון עם האפליקציה Player FM !
פודקאסטים ששווה להאזין
בחסות


1 The Road Back to Country With Blake Shelton 25:13
How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari
Manage episode 450104898 series 2948506
When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.
Key Takeaways:
(02:03) Airflow as a service streamlines Uber’s data workflows.
(06:16) Serialization boosts security and reduces errors.
(10:05) Java-based scheduler improves system reliability.
(13:40) Custom recovery model supports emergency pipeline switching.
(15:58) No-code UI allows easy pipeline creation for non-coders.
(18:12) Backfill feature enables historical data processing.
(22:06) Regular updates keep Uber aligned with Airflow advancements.
(26:07) Plans to leverage Airflow’s latest features.
Resources Mentioned:
https://www.linkedin.com/in/shahshobhit/
https://www.linkedin.com/in/maheshwarisumit/
Uber -
https://www.linkedin.com/company/uber-com/
https://airflow.apache.org/
https://airflowsummit.org/
Uber -
https://www.uber.com/tw/en/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
61 פרקים
How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 450104898 series 2948506
When data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari, both Staff Software Engineers at Uber, we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs.
Key Takeaways:
(02:03) Airflow as a service streamlines Uber’s data workflows.
(06:16) Serialization boosts security and reduces errors.
(10:05) Java-based scheduler improves system reliability.
(13:40) Custom recovery model supports emergency pipeline switching.
(15:58) No-code UI allows easy pipeline creation for non-coders.
(18:12) Backfill feature enables historical data processing.
(22:06) Regular updates keep Uber aligned with Airflow advancements.
(26:07) Plans to leverage Airflow’s latest features.
Resources Mentioned:
https://www.linkedin.com/in/shahshobhit/
https://www.linkedin.com/in/maheshwarisumit/
Uber -
https://www.linkedin.com/company/uber-com/
https://airflow.apache.org/
https://airflowsummit.org/
Uber -
https://www.uber.com/tw/en/
https://astronomer.typeform.com/airflowsurvey24
Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
61 פרקים
כל הפרקים
×
1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 Airflow’s Role in the Rise of DataOps with Andy Byron 26:15

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27

1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.