38 subscribers
התחל במצב לא מקוון עם האפליקציה Player FM !
פודקאסטים ששווה להאזין
בחסות


1 TED Tech
Airflow in Action: Powering Instacart's Complex Ecosystem
Manage episode 477511103 series 2053958
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
https://airflow.apache.org/
https://aws.amazon.com/ecs/
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 פרקים
Airflow in Action: Powering Instacart's Complex Ecosystem
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 477511103 series 2053958
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
https://airflow.apache.org/
https://aws.amazon.com/ecs/
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 פרקים
כל הפרקים
×
1 The Future of Airflow Telemetry at Metyis with Bolke de Bruin 21:55

1 Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28

1 Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34

1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27

1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46

1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40

1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54

1 Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling 33:45

1 Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer 43:39

1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00

1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19

1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54

1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35

1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24

1 Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta 24:11

1 Maximizing Business Impact Through Data at GlossGenius with Katie Bauer 25:49

1 Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade 27:47
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.