38 subscribers
התחל במצב לא מקוון עם האפליקציה Player FM !
Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall
Manage episode 461366533 series 2053958
Data engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall, Data and ML Ops Engineer at Ford Motor Company. Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment.
Key Takeaways:
(03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford.
(05:42) Early adoption of Airflow to orchestrate complex data pipelines.
(09:29) Ford's move from on-premise data solutions to Google Cloud Platform.
(12:03) The importance of Airflow's scheduling capabilities for efficient data management.
(16:12) Using Kubernetes to scale Airflow for large-scale data processing.
(19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools.
(22:22) Integration of data engineering and data science pipelines at Ford.
(28:03) How deferrable operators in Airflow improve performance and save costs.
(32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs.
(36:09) The significance of monitoring and observability in managing Airflow instances.
Resources Mentioned:
https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/
https://airflow.apache.org/
https://cloud.google.com/
Ford Motor Company | LinkedIn -
https://www.linkedin.com/company/ford-motor-company/
Ford Motor Company | Website -
https://www.ford.com/
https://www.astronomer.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
59 פרקים
Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 461366533 series 2053958
Data engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall, Data and ML Ops Engineer at Ford Motor Company. Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment.
Key Takeaways:
(03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford.
(05:42) Early adoption of Airflow to orchestrate complex data pipelines.
(09:29) Ford's move from on-premise data solutions to Google Cloud Platform.
(12:03) The importance of Airflow's scheduling capabilities for efficient data management.
(16:12) Using Kubernetes to scale Airflow for large-scale data processing.
(19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools.
(22:22) Integration of data engineering and data science pipelines at Ford.
(28:03) How deferrable operators in Airflow improve performance and save costs.
(32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs.
(36:09) The significance of monitoring and observability in managing Airflow instances.
Resources Mentioned:
https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/
https://airflow.apache.org/
https://cloud.google.com/
Ford Motor Company | LinkedIn -
https://www.linkedin.com/company/ford-motor-company/
Ford Motor Company | Website -
https://www.ford.com/
https://www.astronomer.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
59 פרקים
כל הפרקים
×
1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 Airflow’s Role in the Rise of DataOps with Andy Byron 26:15

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27

1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46

1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40

1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.