The award-winning WIRED UK Podcast with James Temperton and the rest of the team. Listen every week for the an informed and entertaining rundown of latest technology, science, business and culture news. New episodes every Friday.
…
continue reading
Player FM - Internet Radio Done Right
33 subscribers
Checked 6d ago
הוסף לפני eight שנים
תוכן מסופק על ידי The Data Flowcast. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The Data Flowcast או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.
דומה לThe Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Thanks for visiting The Cell Phone Junkie! I will be taking the time each week to discuss my favorite topic, cell phones. Any feedback is appreciated and welcome. You can email me at: questions (AT) thecellphonejunkie (DOT) com or call: 206-203-3734 Thanks and welcome!
…
continue reading
T
The Ticket: Discover the Future of Customer Service, Support, and Experience, with Intercom
1
The Ticket: Discover the Future of Customer Service, Support, and Experience, with Intercom
Intercom
1k
480
On The Ticket, you'll hear Intercom's Customer Support team in conversation with the customer service leaders, renowned customer experience thinkers, and influential authors who are shaping the field of customer support. Follow The Ticket to get the weekly episodes and sign up for our twice-monthly newsletter bursting with all the insights, trends, tips, and assets your team needs to embrace the future of customer service. https://www.intercom.com/blog/newsletter 🏠 www.intercom.com
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
T
The Talk Show With John Gruber
1
The Talk Show With John Gruber
Daring Fireball / John Gruber
2k353
The director’s commentary track for Daring Fireball. Long digressions on Apple, technology, design, movies, and more.
…
continue reading
Download This Show is your weekly guide to the world of media, culture, and technology. From social media to gadgets, streaming services to privacy issues. Each week Rae Johnston and guests take a fun, deep dive into how technology is reshaping our lives.
…
continue reading
Hosts Ben Brock Johnson and Amory Sivertson dig into the internet's vast and curious ecosystem of online communities to find untold histories, unsolved mysteries, and other jaw-dropping stories online and IRL.
…
continue reading
Media and tech aren’t just intersecting — they’re fully intertwined. And to understand how those worlds work, and what they mean for you, veteran journalist Peter Kafka talks to industry leaders, upstarts and observers - and gets them to spell it out in plain, BS-free English. Part of the Vox Media Podcast Network.
…
continue reading
The 21st Century began with the rise of the Internet and social media. The next decade will mark the rise of the Intelligent Machines. AI will inhabit all our devices from cars and appliances to smart phones and robots. The Intelligent Machines podcast explores the most exciting revolution humanity has ever seen, filled with promise and peril. More than ever we need to understand what these new devices will bring to our lives and how to make best use of them as the 21st century unfolds. On t ...
…
continue reading
WSJ’s Bold Names brings you conversations with the leaders of the bold-named companies featured in the pages of The Wall Street Journal. Hosts Tim Higgins and Christopher Mims speak to CEOs and business leaders in interviews that challenge conventional wisdom and take you inside the decisions being made in the C-suite and beyond.
…
continue reading
Player FM - אפליקציית פודקאסט
התחל במצב לא מקוון עם האפליקציה Player FM !
התחל במצב לא מקוון עם האפליקציה Player FM !
))
Daily Deals
פודקאסטים ששווה להאזין
בחסות
S
Smart Travel: Upgrade Your Getaways
1 Cash or Miles? The 2025 Points Valuations That Could Change How You Travel 42:36
42:36 
הפעל מאוחר יותר 
הפעל מאוחר יותר 
רשימות 
לייק 
אהבתי42:36
הפעל מאוחר יותר
הפעל מאוחר יותר
רשימות
לייק
אהבתיGet fresh NerdWallet data on what your points and miles are really worth in 2025 — and why the answer might change how you book travel. Should you take a $650 flight voucher or 32,500 miles? How much are your points and miles actually worth? Hosts Sally French and Meghan Coyle break down the latest NerdWallet valuations to help you make smarter redemption choices. But first, they cover the week’s biggest travel headlines, including Southwest Airlines’ new partnership with EVA Air, Frontier’s companion certificate promotion and status match, and JetBlue opening up award bookings with Condor Airlines. Then, travel Nerd Craig Joseph joins Meghan to discuss NerdWallet’s latest airline, hotel, and credit card point valuations, with tips and tricks on comparing loyalty programs, maximizing transfer partners, and deciding when cash is more valuable than points. They also discuss the impact of devaluations, how close-in bookings can save you points, and why premium cabins can sometimes offer outsized redemption value. Plus: Craig’s hot take on airport lounges. Card benefits, terms and fees can change. For the most up-to-date information about cards mentioned in this episode, read our reviews: Is the Frontier Airlines World Mastercard Worth Its Annual Fee? https://www.nerdwallet.com/article/travel/is-the-frontier-airlines-world-mastercard-worth-its-annual-fee Citi Strata Credit Card Review: Solid Rewards for No Annual Fee https://www.nerdwallet.com/reviews/credit-cards/citi-strata Citi Double Cash Review: A Solid Choice for Everyday Spending https://www.nerdwallet.com/reviews/credit-cards/citi-double-cash Citi Custom Cash Card Review: Low-Maintenance 5% Cash Back https://www.nerdwallet.com/reviews/credit-cards/citi-custom-cash Citi Strata Premier: Big Rewards Across Top Spending Categories https://www.nerdwallet.com/reviews/credit-cards/citi-strata-premier Is the New Alaska Atmos Summit Card Worth a $395 Annual Fee? https://www.nerdwallet.com/article/travel/is-the-alaska-airlines-atmos-summit-card-worth-its-annual-fee Resources discussed in this episode: Airline Miles vs. Cash Calculator https://www.nerdwallet.com/article/travel/calculator-should-you-book-a-flight-with-cash-or-miles How Much Are Travel Points and Miles Worth in 2025? https://www.nerdwallet.com/article/travel/airline-miles-and-hotel-points-valuations Want even more tips and tricks to get the most out of your travel dollars? Subscribe to TravelNerd , our free newsletter designed to help you crack the code on spending less on your travel. In this episode, the Nerds discuss: points and miles valuation, airline miles value, hotel points value, credit card points value, Southwest EVA Air partnership, Frontier Companion Certificate, JetBlue Condor award booking, Citi American Airlines transfer, Amex Membership Rewards value, Capital One points value, Bilt points value, Hyatt points value, Hilton points value, Marriott points value, Wyndham points value, IHG points value, Alaska miles value, JetBlue points value, American Airlines miles value, United miles value, Southwest points value, Virgin Atlantic miles value, ANA miles value, Avianca LifeMiles value, best way to use Amex points, best way to use Citi points, best way to use Capital One points, use cash or points for flights, last minute award flight value, premium cabin redemption value, economy flight points value, airline devaluation, hotel point devaluation, cash vs points travel booking, when to transfer credit card points, how to maximize travel rewards, and NerdWallet points and miles calculator. Learn more about your ad choices. Visit megaphone.fm/adchoices…
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
Manage episode 465328064 series 2053958
תוכן מסופק על ידי The Data Flowcast. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The Data Flowcast או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
77 פרקים
Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
שתפוManage episode 465328064 series 2053958
תוכן מסופק על ידי The Data Flowcast. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The Data Flowcast או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.
Scaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer, Former Platform Engineer at Monzo Bank, joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios.
Key Takeaways:
(03:11) Using Airflow to schedule computation in BigQuery.
(07:02) How DAGs with 8,000+ tasks were managed nightly.
(08:18) Ensuring accuracy in regulatory reporting for banking.
(11:35) Handling task inconsistency and DAG failures with automation.
(16:09) Building a service to resolve DAG consistency issues in Airflow.
(25:05) Challenges with scaling the Airflow UI for thousands of tasks.
(27:03) The role of upstream and downstream task management in Airflow.
(37:33) The importance of operational metrics for monitoring Airflow health.
(39:19) Balancing new tools with root cause analysis to address scaling issues.
(41:35) Why scaling solutions require both technical and leadership buy-in
Resources Mentioned:
https://www.linkedin.com/in/jonathan-rainer/
https://www.linkedin.com/company/monzo-bank/
https://airflow.apache.org/
BigQuery -
https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html
https://kubernetes.io/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
77 פרקים
כל הפרקים
×
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Airflow and AI Power Investigative Journalism at the Financial Times with Zdravko Hvarlingov 24:28
24:28 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:28
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe Financial Times leverages Airflow and AI to uncover powerful stories hidden within vast, unstructured data. In this episode, Zdravko Hvarlingov , Senior Software Engineer at the Financial Times , discusses building multi-tenant Airflow systems and AI-driven pipelines that surface stories that might otherwise be missed. Zdravko walks through entity extraction and fuzzy matching, linking the UK Register of Members’ Financial Interests with Companies House, and how this work cuts weeks of manual analysis to minutes. Key Takeaways: 00:00 Introduction. 02:12 What computational journalism means for day-to-day newsroom work. 05:22 Why a shared orchestration platform supports consistent, scalable workflows. 08:30 Tradeoffs of one centralized platform versus many separate instances. 11:52 Using pipelines to structure messy sources for faster analysis. 14:14 Turning recurring disclosures into usable data for investigations. 16:03 Applying lightweight ML and matching to reveal entities and links. 18:46 How automation reduces manual effort and shortens time to insight. 20:41 Practical improvements that make backfilling and reliability easier. Resources Mentioned: Zdravko Hvarlingov https://www.linkedin.com/in/zdravko-hvarlingov-3aa36016b/ Financial Times | LinkedIn https://www.linkedin.com/company/financial-times/ Financial Times | Website https://www.ft.com/ Apache Airflow https://airflow.apache.org/ UK Register of Members’ Financial Interests https://www.parliament.uk/mps-lords-and-offices/standards-and-financial-interests/parliamentary-commissioner-for-standards/registers-of-interests/register-of-members-financial-interests/ UK Companies House https://www.gov.uk/government/organisations/companies-house Doppler https://www.doppler.com/ Kubernetes https://kubernetes.io/ Airflow Kubernetes Executor https://airflow.apache.org/docs/apache-airflow/stable/executor/kubernetes.html GitHub https://github.com/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo 29:36
29:36 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי29:36
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines. In this episode, we feature Oscar Ligthart , Lead Data Engineer, and Rodrigo Loredo , Lead Analytics Engineer, both at Vinted , as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration. Key Takeaways: 00:00 Introduction. 05:28 Challenges of decentralization. 06:45 YAML-based generator standardizes pipelines and dependencies. 12:28 Declarative assets and sensors align cross-DAG dependencies. 17:29 Task-level callbacks enable auto-recovery and clear ownership. 21:39 Standardized building blocks simplify upgrades and maintenance. 24:52 Platform focus frees domain work. 26:49 Container-only standardization prevents sprawl. Resources Mentioned: Oscar Ligthart https://www.linkedin.com/in/oscar-ligthart/ Rodrigo Loredo https://www.linkedin.com/in/rodrigo-loredo-410a16134/ Vinted | LinkedIn https://www.linkedin.com/company/vinted/ Vinted | Website https://www.vinted.com/?srsltid=AfmBOor87MGR_eLOauCO93V9A-aLDaAhGYx9cnu_oN8s1SAXMlCRuhW7 Apache Airflow https://airflow.apache.org/ Kubernetes https://kubernetes.io/ dbt https://www.getdbt.com/ Google Cloud Vertex AI https://cloud.google.com/vertex-ai Airflow Datasets & Assets (concepts) https://www.astronomer.io/docs/learn/airflow-datasets Airflow Summit https://airflowsummit.org/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Transforming Data Pipelines at XENA Intelligence with Naseem Shah 28:32
28:32 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי28:32
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe shift from simple cron jobs to orchestrated AI-powered workflows is reshaping how startups scale. For a small team, these transitions come with unique challenges and big opportunities. In this episode, Naseem Shah , Head of Engineering at Xena Intelligence , shares how he built data pipelines from scratch, adopted Apache Airflow and transformed Amazon review analysis with LLMs. Key Takeaways: 00:00 Introduction. 03:28 The importance of building initial products that support growth and investment. 06:16 The process of adopting new tools to improve reliability and efficiency. 09:29 Approaches to learning complex technologies through practice and fundamentals. 13:57 Trade-offs small teams face when balancing performance and costs. 18:40 Using AI-driven approaches to generate insights from large datasets. 22:38 How unstructured data can be transformed into actionable information. 25:55 Moving from manual tasks to fully automated workflows. 28:05 Orchestration as a foundation for scaling advanced use cases. Resources Mentioned: Naseem Shah https://www.linkedin.com/in/naseemshah/ Xena Intelligence | LinkedIn https://www.linkedin.com/company/xena-intelligence/ Xena Intelligence | Website https://xenaintelligence.com/ Apache Airflow https://airflow.apache.org/ Google Cloud Composer https://cloud.google.com/composer Techstars https://www.techstars.com/ Docker https://www.docker.com/ AWS SQS https://aws.amazon.com/sqs/ PostgreSQL https://www.postgresql.org/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith 24:03
24:03 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:03
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיUsing Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed. In this episode, Alex Iannicelli , Staff Software Engineer at Overture Maps Foundation , and Daniel Smith , Senior Solutions Architect at Wherobots , join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms. Key Takeaways: 00:00 Introduction. 03:22 How merging multiple data sources supports comprehensive datasets. 04:20 The value of flexible configurations for running pipelines on different platforms. 06:35 Why orchestration tools are essential for handling continuous data streams. 09:45 The importance of observability for monitoring progress and troubleshooting issues. 11:30 Strategies for processing large, complex datasets efficiently. 13:27 Expanding orchestration beyond core pipelines to automate frequent tasks. 17:02 Advantages of using open-source operators to simplify integration and deployment. 20:32 Desired improvements in orchestration tools for usability and workflow management. Resources Mentioned: Alex Iannicelli https://www.linkedin.com/in/atiannicelli/ Overture Maps Foundation | LinkedIn https://www.linkedin.com/company/overture-maps-foundation/ Overture Maps Foundation | Website https://overturemaps.org Daniel Smith https://www.linkedin.com/in/daniel-smith-analyst/ Wherobots | LinkedIn https://www.linkedin.com/company/wherobots Wherobots | Website https://www.wherobots.com Apache Airflow https://airflow.apache.org/ Apache Sedona https://sedona.apache.org/ Github repo https://github.com/wherobots/airflow-providers-wherobots Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya 19:04
19:04 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי19:04
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיPepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible. In this episode, Kunal Bhattacharya , Senior Manager of Data Platform Engineering at PepsiCo , shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency. Key Takeaways: 00:00 Introduction. 02:31 Enabling developer delight by extending platform capabilities. 03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack. 06:10 Local developer environments built using official Airflow Helm charts. 07:13 Pre-staging and PR environments as testing playgrounds. 08:08 Automating labeling and resource allocation via DAG factories. 12:16 Cost optimization through pod labeling and Datadog insights. 14:01 Isolating dbt engines to improve performance across teams. 16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling. Resources Mentioned: Kunal Bhattacharya https://www.linkedin.com/in/kunaljubce/ PepsiCo | LinkedIn https://www.linkedin.com/company/pepsico/ PepsiCo | Website https://www.pepsico.com Apache Airflow https://airflow.apache.org/ Snowflake https://www.snowflake.com dbt https://www.getdbt.com Kubernetes https://kubernetes.io Great Expectations https://greatexpectations.io Monte Carlo https://www.montecarlodata.com Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building a Unified Data Platform at Pattern with William Graham 24:09
24:09 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:09
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines. In this episode, we are joined by William Graham , Senior Data Engineer at Pattern , who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management. Key Takeaways: 00:00 Introduction. 02:44 Structure of Pattern’s data teams across acquisition, engineering and platform. 04:27 How Airflow became the central scheduler for batch jobs. 08:57 Credential management challenges that led to decoupling scheduling and orchestration. 12:21 Heimdall simplifies multi-application access through a unified interface. 13:15 Standardized operators in Airflow using Heimdall integration. 17:13 Open-source contributions and early adoption of Heimdall within Pattern. 21:01 Community support for Airflow and satisfaction with scheduling flexibility. Resources Mentioned: William Graham https://www.linkedin.com/in/willgraham2/ Pattern | LinkedIn https://www.linkedin.com/company/pattern-hq/ Pattern | Website https://pattern.com Apache Airflow https://airflow.apache.org Heimdall on GitHub https://github.com/patterninc/heimdall Netflix Genie https://netflix.github.io/genie/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty 25:34
25:34 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי25:34
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations. In this episode, we are joined by Collin McNulty , Sr. Director of Global Support at Astronomer , who shares insights from his journey into data engineering and the lessons learned from leading Astronomer’s Customer Reliability Engineering (CRE) team. Key Takeaways: 00:00 Introduction. 03:07 Lessons learned in adapting to major platform transitions. 05:18 How proactive monitoring improves reliability and customer experience. 08:10 Using automation to enhance internal support processes. 12:09 Why keeping systems current helps avoid unnecessary issues. 15:14 Approaches that strengthen system reliability and efficiency. 18:46 Best practices for simplifying complex orchestration dependencies. 23:24 Anticipated innovations that expand orchestration capabilities. Resources Mentioned: Collin McNulty https://www.linkedin.com/in/collin-mcnulty/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ Prometheus https://prometheus.io/ Splunk https://www.splunk.com/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov 19:26
19:26 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי19:26
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics. In this episode, Evgenii Prusov , Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH , joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer. Key Takeaways: 00:00 Introduction. 02:49 Building a centralized data platform for 15 European countries. 05:19 Adopting SaaS to manage Airflow from day one. 07:01 Leveraging Airflow for data orchestration across products. 08:16 Teaching non-Python users how to work with Airflow is challenging. 12:25 Creating a global data community across Europe, the US and Japan. 14:04 Monthly calls help share knowledge and align regional teams. 15:47 Contributing to the open-source Airflow project as a way to deepen expertise. 16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow. Resources Mentioned: Evgenii Prusov https://www.linkedin.com/in/prusov/ Daiichi Sankyo Europe GmbH | LinkedIn https://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/ Daiichi Sankyo Europe GmbH | Website https://www.daiichi-sankyo.eu Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Snowflake https://www.snowflake.com/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah 23:05
23:05 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:05
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיStyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base. In this episode, Paschal Onuorah , Senior Data Engineer at StyleSeat , shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals. Key Takeaways: 00:00 Introduction. 05:44 The role of the data engineering team in driving business success. 08:52 Leveraging technology for real-time business intelligence. 10:52 Data-driven strategies for improving marketing outcomes. 13:05 How adopting the right tools can increase revenue growth. 14:25 Advantages of simplifying and integrating technical workflows. 18:45 Benefits of multi-environment configurations for development and production. 20:17 Foundational skills and best practices for learning Airflow effectively. 22:33 Opportunities for deeper tool integration and improved data visualization. Resources Mentioned: Paschal Onuorah https://www.linkedin.com/in/onuorah-paschal/ StyleSeat | LinkedIn https://www.linkedin.com/company/styleseat/ StyleSeat | Website https://www.styleseat.com Apache Airflow https://airflow.apache.org/ dbt https://www.getdbt.com/ Astronomer Cosmos https://www.astronomer.io/cosmos/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak 22:25
22:25 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:25
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines. In this episode, we’re joined by Ian Buss , Principal Software Engineer at Astronomer, and Piotr Chomiak , Principal Product Manager at Astronomer , who share insights into the Astro Executor and remote execution. Key Takeaways: 00:00 Introduction. 04:13 How product leadership drives scalability for enterprise needs. 08:23 Architectural changes that improve reliability and remove bottlenecks. 10:15 Metrics that enhance visibility into system performance. 12:54 The role of remote execution in addressing security requirements. 15:56 Differences between open-source solutions and managed offerings. 19:04 Broad industry adoption and applicability of remote execution. 20:39 Future advancements in language support and multi-tenancy. Resources Mentioned: Ian Buss https://www.linkedin.com/in/ian-buss/ Piotr Chomiak https://www.linkedin.com/in/piotr-chomiak-b1955624/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ Airflow Slack Community https://airflow.apache.org/community/ Beyond Analytics conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille 24:17
24:17 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:17
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיScaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable. In this episode, Sébastien Crocquevieille , Data Engineer at Numberly , unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features. Key Takeaways: 00:00 Introduction. 02:13 Overview of the company’s operations and global presence. 04:00 The tech stack and structure of the data engineering team. 04:24 Running nearly 2,000 DAGs in production using Airflow. 05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot. 07:05 Details on the Kubernetes-based Airflow setup using Helm charts. 09:31 Transition from GitSync to NFS for DAG syncing due to performance issues. 14:11 Making every team member Airflow-literate through local installation. 17:56 Using custom libraries and plugins to extend Airflow functionality. Resources Mentioned: Sébastien Crocquevieille https://www.linkedin.com/in/scroc/ Numberly | LinkedIn https://www.linkedin.com/company/numberly/ Numberly | Website https://numberly.com/ Apache Airflow https://airflow.apache.org/ Grafana https://grafana.com/ Apache Kafka https://kafka.apache.org/ Helm Chart for Apache Airflow https://airflow.apache.org/docs/helm-chart/stable/index.html Kubernetes https://kubernetes.io/ GitLab https://about.gitlab.com/ KubernetesPodOperator – Airflow https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye 21:32
21:32 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי21:32
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיManaging financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency. In this episode, we are joined by Adeolu Adegboye , Data Engineer at Moniepoint Group , who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business. Key Takeaways: (00:00) Introduction. (02:48) The role of data engineering in supporting all business operations. (04:17) Leveraging workflow orchestration to manage daily processes. (05:20) Proactively monitoring for anomalies to prevent potential issues. (08:12) Simplifying complex insights for non-technical teams. (13:01) Improving efficiency through dynamic and parallel workflows. (14:19) Optimizing system performance to handle large-scale operations. (17:19) Exploring creative and innovative uses for workflow automation. Resources Mentioned: Adeolu Adegboye https://www.linkedin.com/in/adeolu-adegboye/ Moniepoint Group | LinkedIn https://www.linkedin.com/company/moniepoint-inc/ Moniepoint Group | Website https://www.moniepoint.com Apache Airflow https://airflow.apache.org/ ClickHouse https://clickhouse.com/ Grafana https://grafana.com/ Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler 28:02
28:02 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי28:02
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments. In this episode, Jens Scheffler , Test Execution Cluster Technical Architect at Bosch , shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release. Key Takeaways: (02:39) The role of remote execution in supporting large-scale testing needs. (04:44) How community support contributed to the Edge Executor’s development. (08:41) Navigating network and infrastructure limitations within secure environments. (13:25) Transitioning from database-heavy processes to an API-driven model. (14:16) How the new task SDK in Airflow 3 improves distributed task execution. (16:54) What is required to set up and configure the Edge Executor. (19:36) Managing multiple queues to optimize tasks across different environments. (23:30) Examples of extreme distance use cases for edge execution. Resources Mentioned: Jens Scheffler https://www.linkedin.com/in/jens-scheffler/ Bosch | LinkedIn https://www.linkedin.com/company/bosch/ Bosch | Website https://www.bosch.com/ Apache Airflow https://airflow.apache.org/ Edge Executor (Edge3 Provider Package) https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html Astronomer’s Astro Executor https://www.astronomer.io/docs/astro/astro-executor/ Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero 31:24
31:24 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי31:24
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיManaging modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly thrive, infrastructure must become both accessible and compliant without sacrificing velocity or reliability. In this episode, we’re joined by Cory O’Daniel , CEO and Co-Founder at Massdriver , and Jacob Ferriero , Senior Software Engineer at Astronomer , to unpack what it takes to make data platform engineering scalable, sustainable and secure. They share lessons from years of experience working with DevOps, ML teams and platform engineers and discuss how Airflow fits into the orchestration layer of today’s data stacks. Key Takeaways: (03:27) Making infrastructure accessible without deep ops knowledge. (07:23) Distinct personas and responsibilities across data teams. (09:53) Infrastructure hurdles specific to ML workloads. (11:13) Compliance and governance shaping platform design. (13:27) Tooling mismatches between teams cause friction. (15:13) Airflow’s orchestration role within broader system architecture. (22:10) Creating reusable infrastructure patterns for consistency. (24:13) Enabling secure access without slowing down development. (26:55) Opportunities to improve Airflow with event-driven and reliability tooling. Resources Mentioned: Cory O’Daniel https://www.linkedin.com/in/coryodaniel/ Massdriver | LinkedIn https://www.linkedin.com/company/massdriver/ Massdriver | Website https://www.massdriver.cloud/ Jacob Ferriero https://www.linkedin.com/in/jacob-ferriero/ Astronomer https://www.linkedin.com/company/astronomer/ Apache Airflow https://airflow.apache.org/ Prequel https://www.prequel.co/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Future of Airflow Telemetry with Bolke de Bruin 21:55
21:55 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי21:55
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיTelemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. In this episode, we’re joined by Bolke de Bruin , Director at Metyis and a long-time Airflow PMC member. Bolke discusses how telemetry has been handled in the past, why it matters now and what it will take to get it right. Key Takeaways: (03:20) The role of foundations in establishing credibility and sustainability. (04:52) Why data collection is critical to open-source project direction. (07:24) Lessons learned from previous approaches to user data collection. (10:23) The current state of telemetry in the project. (10:53) Community trust as a prerequisite for technical implementation. (12:54) The importance of managing sensitive data within trusted ecosystems. (16:37) Ethical considerations in balancing participation and access. (18:45) Forward-looking ideas for improving workflow design and usability. Resources Mentioned: Bolke de Bruin https://www.linkedin.com/in/bolke/ Metyis | LinkedIn https://www.linkedin.com/company/metyis/ Metyis | Website http://www.metyis.com Apache Airflow https://airflow.apache.org/ Airflow Summit https://airflowsummit.org/ Airflow Dev List https://lists.apache.org/list.html?dev@airflow.apache.org https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28
22:28 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:28
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיContributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj , Software Engineer II at Cloudera , shares how his team built a drag-and-drop DAG editor for non-coders, contributions which helped shape the Airflow 3.0 Ul and introduced features like external XCom control and bulk APls. Key Takeaways: (02:30) Day-to-day responsibilities building platforms that simplify orchestration. (05:27) Factors that make onboarding into large open-source projects accessible. (07:35) The value of improved user interfaces for task state visibility and control. (09:49) Enabling faster debugging by exposing internal data through APIs. (13:00) Balancing frontend design goals with backend functionality. (14:19) Creating workflow editors that lower the barrier to entry. (16:54) Supporting a variety of task types within a visual DAG builder. (19:32) Common infrastructure challenges faced by orchestration users. (20:37) Addressing dependency management across distributed environments. Resources Mentioned: Shubham Raj https://www.linkedin.com/in/shubhamrajofficial/ Cloudera | LinkedIn https://www.linkedin.com/company/cloudera/ Cloudera | Website https://www.cloudera.com/ Apache Airflow https://airflow.apache.org/ 2023 Airflow Summit https://airflowsummit.org/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34
19:34 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי19:34
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיManaging data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows. In this episode, we speak with Yunhao Qing , Software Engineer at Lyft , about building a governed data-engineering platform powered by Airflow that balances flexibility, standardization and scale. Key Takeaways: (03:17) Supporting internal teams with a centralized orchestration platform. (04:54) Migrating to a managed service to reduce infrastructure overhead. (06:04) Embedding platform-level governance into custom components. (08:02) Consolidating and regulating the creation of custom code. (09:48) Identifying and correcting inefficient workflow patterns. (11:17) Replacing manual workarounds with native platform features. (14:32) Preparing teams for major version upgrades. (16:03) Leveraging asset-based scheduling for smarter triggers. (18:13) Envisioning GenAI and semantic search for future productivity. Resources Mentioned: Yunhao Qing https://www.linkedin.com/in/yunhao-qing Lyft | LinkedIn https://www.linkedin.com/company/lyft/ Lyft | Website https://www.lyft.com/ Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Kubernetes https://kubernetes.io/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19
22:19 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:19
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיUnderstanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone. In this episode, Marc Lamberti , Head of Customer Education at Astronomer , joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners demystify Airflow. Key Takeaways: (02:36) Early exposure to Airflow while addressing inefficiencies in data workflows. (04:10) Common barriers to implementing open source tools in enterprise settings. (06:18) The shift from part-time teaching to a full-time focus on Airflow education. (07:53) A modular, guided approach to structuring educational content. (09:57) The value of highlighting underused Airflow features for broader adoption. (12:35) Certifications as a method to assess readiness and uncover knowledge gaps. (13:25) Coverage of essential Airflow concepts in the Fundamentals exam. (16:07) The DAG Authoring exam’s emphasis on practical, advanced features. (20:08) A call for more visible integration of Airflow with AI workflows. Resources Mentioned: Marc Lamberti https://www.linkedin.com/in/marclamberti/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer Academy https://academy.astronomer.io/ Airflow Fundamentals Certification https://www.astronomer.io/certification/ DAG Authoring Certification https://academy.astronomer.io/plan/astronomer-certification-dag-authoring-for-apache-airflow-exam The Complete Hands-On Introduction to Airflow https://www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_Beta_Prof_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Beta&utm_content=deal4584&utm_term=_._ag_162511579404_._ad_696197165418_._kw__._de_c_._dm__._pl__._ti_dsa-1677053911088_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21168154305&gbraid=0AAAAADROdO3MpljfP-gssiYSmDEPdhZV9&gclid=Cj0KCQjw097CBhDIARIsAJ3-nxdjZA6G5-Y0-akk6Huksy2PLb04t92J4iNfUSIbMdrSAla_tb-o2N8aArOeEALw_wcB&couponCode=PMNVD3025 https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09
30:09 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי30:09
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams. In this episode, we speak with Alberto Crespi , Data Architect at lastminute.com , who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach. Key Takeaways: (02:17) Defining responsibilities within data architecture teams. (04:15) Consolidating multiple orchestrators into a single solution. (07:00) Scaling Airflow environments with shared infrastructure and DevOps practices. (10:59) Managing dependencies and readiness using SQL sensors. (14:23) Enhancing visibility and response through Slack-integrated monitoring. (19:28) Extending Airflow’s flexibility to run legacy systems. (22:28) Integrating transformation tools into orchestrated pipelines. (25:54) Enabling non-engineers to contribute to pipeline development. (27:33) Fostering adoption through collaboration and communication. Resources Mentioned: Alberto Crespi https://www.linkedin.com/in/crespialberto/ lastminute.com | Website https://lastminute.com Apache Airflow https://airflow.apache.org/ dbt Labs https://www.getdbt.com/ Astronomer Cosmos https://github.com/astronomer/astronomer-cosmos GitLab Slack https://slack.com/ Kubernetes https://kubernetes.io/ Confluence https://www.atlassian.com/software/confluence Slack https://slack.com/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21
23:21 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:21
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיInnovation in orchestration is redefining how engineers approach both traditional ETL pipelines and emerging AI workloads. Understanding how to harness Airflow’s flexibility and observability is essential for teams navigating today’s evolving data landscape. In this episode, Anu Pabla , Principal Engineer at The ODP Corporation , joins us to discuss her journey from legacy orchestration patterns to AI-native pipelines and why she sees Airflow as the future of AI workload orchestration. Key Takeaways: (03:43) Engaging with external technology communities fosters innovation. (05:05) Mentoring early-career engineers builds confidence in a complex tech landscape. (07:51) Orchestration patterns continue to evolve with modern data needs. (08:41) Managing AI workflows requires structured and flexible orchestration. (10:35) High-quality, meaningful data remains foundational across use cases. (15:08) Community-driven open source tools offer lasting value. (16:59) Self-healing systems support both legacy and AI pipelines. (20:20) Orchestration platforms can drive future AI-native workloads. Resources Mentioned: Anu Pabla https://www.linkedin.com/in/atomicap/ The ODP Corporation https://www.linkedin.com/company/the-odp-corporation/ The ODP Corporation | Website https://www.theodpcorp.com/homepage Apache Airflow https://airflow.apache.org/ LlamaIndex https://www.llamaindex.ai/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44
24:44 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:44
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe orchestration layer is foundational to building robust AI- and ML-powered data pipelines, especially in complex hybrid enterprise environments. IBM’s partnership with Astronomer reflects a strategic alignment to simplify and scale Airflow-based workflows across industries. In this episode, we’re joined by IBM ’s Senior Product Manager, BJ Adesoji , and GTM PM and Growth Leader , Ryan Yackel . We discuss how IBM customers are using Airflow in production, the challenges they face at scale and what the new IBM–Astronomer collaboration unlocks. Key Takeaways: (03:09) The growing importance of orchestration tools in enterprise environments. (04:48) How organizations are expanding orchestration beyond traditional use cases. (05:24) Common patterns across industries adopting orchestration platforms. (07:16) Why orchestration is essential for supporting business-critical workloads. (10:00) The role of orchestration in compliance and regulatory processes. (13:02) Challenges enterprises face when managing orchestration infrastructure. (14:58) Opportunities to simplify and centralize orchestration at scale. (19:11) The value of integrating orchestration with broader data toolchains. (20:54) How AI is shaping the future of orchestrated data workflows. Resources Mentioned: BJ Adesoji https://www.linkedin.com/in/bj-soji/ Ryan Yackel https://www.linkedin.com/in/ryanyackel/ IBM | LinkedIn https://www.linkedin.com/company/databand-ai/ IBM Databand https://www.ibm.com/products/databand IBM DataStage https://www.ibm.com/products/datastage IBM watsonx.governance https://www.ibm.com/products/watsonx-governance IBM Knowledge Catalog https://www.ibm.com/products/knowledge-catalog Apache Airflow https://airflow.apache.org/ watsonx Orchestrate https://www.ibm.com/products/watsonx-orchestrate Domino https://domino.ai/ Astronomer https://www.astronomer.io/ Snowflake https://www.snowflake.com/en/ dbt Labs https://www.getdbt.com/ Amazon SageMaker https://aws.amazon.com/sagemaker/ Cloudera https://www.cloudera.com/ MongoDB https://www.mongodb.com/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02
31:02 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי31:02
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיEfficient orchestration and maintainability are crucial for data engineering at scale. Gil Reich , Data Developer for Data Science at Wix , shares how his team reduced code duplication, standardized pipelines, and improved Airflow task orchestration using a Python-based framework built within the data science team. In this episode, Gil explains how this internal framework simplifies DAG creation, improves documentation accuracy, and enables consistent task generation for machine learning pipelines. He also shares lessons from complex DAG optimization and maintaining testable code. Key Takeaways: (03:23) Code duplication creates long-term problems. (08:16) Frameworks bring order to complex pipelines. (09:41) Shared functions cut down repetitive code. (17:18) Auto-generated docs stay accurate by design. (22:40) On-demand DAGs support real-time workflows. (25:08) Task-level sensors improve run efficiency. (27:40) Combine local runs with automated tests. (30:09) Clean code helps teams scale faster. Resources Mentioned: Gil Reich https://www.linkedin.com/in/gilreich/ Wix | LinkedIn https://www.linkedin.com/company/wix-com/ Wix | Website https://www.wix.com/ DS DAG Framework https://airflowsummit.org/slides/2024/92-refactoring-dags.pdf Apache Airflow https://airflow.apache.org/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13
22:13 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:13
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיLegacy architecture and AI workloads pose unique challenges at scale, especially in a global enterprise with complex data systems. In this episode, we explore strategies to proactively monitor and optimize pipelines while minimizing downstream failures. Adonis Castillo Cordero , Senior Automation Manager at Procter & Gamble , joins us to share actionable best practices for dependency mapping, anomaly detection and architecture simplification using Apache Airflow. Key Takeaways: (03:13) Integrating legacy data systems into modern architecture. (05:51) Designing workflows for real-time data processing. (07:57) Mapping dependencies early to avoid pipeline failures. (09:02) Building automated monitoring into orchestration frameworks. (12:09) Detecting anomalies to prevent performance bottlenecks. (15:24) Monitoring data quality to catch silent failures. (17:02) Prioritizing responses based on impact severity. (18:55) Simplifying dashboards to highlight critical metrics. Resources Mentioned: Adonis Castillo Cordero https://www.linkedin.com/in/adoniscc/ Procter & Gamble | LinkedIn https://www.linkedin.com/company/procter-and-gamble/ Procter & Gamble | Website http://www.pg.com Apache Airflow https://airflow.apache.org/ OpenLineage https://openlineage.io/ Azure Monitor https://azure.microsoft.com/en-us/products/monitor/ AWS Lookout for Metrics https://aws.amazon.com/lookout-for-metrics/ Monte Carlo https://www.montecarlodata.com/ Great Expectations https://greatexpectations.io/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54
38:54 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי38:54
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיBuilding reliable data pipelines starts with maintaining strong data quality standards and creating efficient systems for auditing, publishing and monitoring. In this episode, we explore the real-world patterns and best practices for ensuring data pipelines stay accurate, scalable and trustworthy. Joseph Machado , Senior Data Engineer at Netflix , joins us to share practical insights gleaned from supporting Netflix’s Ads business as well as over a decade of experience in the data engineering space. He discusses implementing audit publish patterns, building observability dashboards, defining in-band and separate data quality checks, and optimizing data validation across large-scale systems. Key Takeaways: . (03:14) Supporting data privacy and engineering efficiency within data systems. (10:41) Validating outputs with reconciliation checks to catch transformation issues. (16:06) Applying standardized patterns for auditing, validating and publishing data. (19:28) Capturing historical check results to monitor system health and improvements. (21:29) Treating data quality and availability as separate monitoring concerns. (26:26) Using containerization strategies to streamline pipeline executions. (29:47) Leveraging orchestration platforms for better visibility and retry capability. (31:59) Managing business pressure without sacrificing data quality practices. (35:46) Starting simple with quality checks and evolving toward more complex frameworks. Resources Mentioned: Joseph Machado https://www.linkedin.com/in/josephmachado1991/ Netflix | LinkedIn https://www.linkedin.com/company/netflix/ Netflix | Website https://www.netflix.com/browse Start Data Engineering https://www.startdataengineering.com/ Apache Airflow https://airflow.apache.org/ dbt Labs https://www.getdbt.com/ Great Expectations https://greatexpectations.io/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28
30:28 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי30:28
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיCreating consistency across data pipelines is critical for scaling engineering teams and ensuring long-term maintainability. In this episode, Snir Israeli , Senior Data Engineer at Next Insurance , shares how enforcing coding standards and investing in developer experience transformed their approach to data engineering. He explains how implementing automated code checks, clear documentation practices and a scoring system helped drive alignment across teams, improve collaboration and reduce technical debt in a fast-growing data environment. Key Takeaways: (02:59) Inconsistencies in code style create challenges for collaboration and maintenance. (04:22) Programmatically enforcing rules helps teams scale their best practices. (08:55) Performance improvements in data pipelines lead to infrastructure cost savings. (13:22) Developer experience is essential for driving adoption of internal tools. (19:44) Dashboards can operationalize standards enforcement and track progress over time. (22:49) Standardization accelerates onboarding and reduces friction in code reviews. (25:39) Linting rules require ongoing maintenance as tools and platforms evolve. (27:47) Starting small and involving the team leads to better adoption and long-term success. Resources Mentioned: Snir Israeli https://www.linkedin.com/in/snir-israeli/ Next Insurance | LinkedIn https://www.linkedin.com/company/nextinsurance/ Next Insurance | Website https://www.nextinsurance.com/ Apache Airflow https://airflow.apache.org/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49
22:49 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:49
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיAirflow’s adaptability is driving Tekmetric’s ability to unify complex data workflows, deliver accurate insights and support both internal operations and customer-facing services — all within a rapidly growing startup environment. In this episode, Ipsa Trivedi , Lead Data Engineer at Tekmetric , shares how her team is standardizing pipelines while supporting unique customer needs. She explains how Airflow enables end-to-end data services, simplifies orchestration across varied sources and supports scalable customization. Ipsa also highlights early wins with Airflow, its intuitive UI and the team's roadmap toward data quality, observability and a future self-serve data platform. Key Takeaways: (02:26) Powering auto shops nationwide with a unified platform. (05:17) A new data team was formed to centralize and scale insights. (07:23) Flexible, open source and made to fit — Airflow wins. (10:42) Pipelines handle anything from email to AWS. (12:15) Custom DAGs fit every team’s unique needs. (17:01) Data quality checks are built into the plan. (18:17) Self-serve data mesh is the end goal. (19:59) Airflow now fits so well, there's nothing left on the wishlist. Resources Mentioned: Ipsa Trivedi https://www.linkedin.com/in/ipsatrivedi/ Tekmetric | LinkedIn https://www.linkedin.com/company/tekmetric/ Tekmetric | Website https://www.tekmetric.com/ Apache Airflow https://airflow.apache.org/ AWS RDS https://aws.amazon.com/free/database/?trk=fc551e06-56b0-418c-9ddd-5c9dba18569b&sc_channel=ps&ef_id=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE:G:s&s_kwcid=AL!4422!3!548989592596!e!!g!!amazon%20sql%20database!11543056228!112002958549&gclid=CjwKCAjwzMi_BhACEiwAX4YZULS4jV2Xpnpcac_Q3eS9BAg-klKUDyCt6XSdOul8BLHkmWzFFh4NXRoCGhQQAvD_BwE Astro by Astronomer https://www.astronomer.io/product/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28
27:28 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי27:28
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe Airflow 3.0 release marks a significant leap forward in modern data orchestration, introducing architectural upgrades that improve scalability, flexibility and long-term maintainability. In this episode, we welcome Vikram Koka , Chief Strategy Officer at Astronomer , and Jed Cunningham , Principal Software Engineer at Astronomer , to discuss the architectural foundations, new features and future implications of this milestone release. They unpack the rationale behind DAG versioning and task execution interface, explain how Airflow now integrates more seamlessly within broader data ecosystems and share how these changes lay the groundwork for multi-cloud deployments, language-agnostic workflows and stronger enterprise security. Key Takeaways: (02:28) Modern orchestration demands new infrastructure approaches. (05:02) Removing legacy components strengthens system stability. (06:26) Major releases provide the opportunity to reduce technical debt. (08:31) Frontend and API modernization enable long-term adaptability. (09:36) Event-based triggers expand integration possibilities. (11:54) Version control improves visibility and execution reliability. (14:57) Centralized access to workflow definitions increases flexibility. (21:49) Decoupled architecture supports distributed and secure deployments. (26:17) Community collaboration is essential for sustainable growth. Resources Mentioned: Astronomer Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ Git Bundle https://git-scm.com/book/en/v2/Git-Tools-Bundling FastAPI https://fastapi.tiangolo.com/ React https://react.dev/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14
25:14 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי25:14
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas. In this episode, we’re joined by Anant Agarwal , Software Engineer at Instacart , who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows. Key Takeaways: (03:49) The impact of external events on business growth and technological evolution. (04:31) Challenges of managing decentralized systems across multiple teams. (06:14) The importance of standardizing infrastructure and processes for scalability. (09:51) Strategies for implementing efficient and repeatable deployment practices. (12:17) Addressing diverse user personas with tailored solutions. (14:47) Leveraging remote execution to enhance flexibility and scalability. (18:36) Benefits of transitioning to a centralized system for organization-wide use. (20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements. (23:35) Anticipation for new features and improvements in upcoming software versions. Resources Mentioned: Anant Agarwal https://www.linkedin.com/in/anantag/ Instacart | LinkedIn https://www.linkedin.com/company/instacart/ Instacart | Website https://www.instacart.com Apache Airflow https://airflow.apache.org/ AWS Amazon https://aws.amazon.com/ecs/ Terraform https://www.terraform.io/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42
27:42 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי27:42
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData orchestration at scale presents unique challenges, especially when aiming for flexibility and efficiency across cloud environments. Choosing the right tools and frameworks can make all the difference. In this episode, Raviteja Tholupunoori, Senior Engineer at Deloitte Digital , joins us to explore how Airflow enhances orchestration, scalability and cost efficiency in enterprise data workflows. Key Takeaways: (01:45) Early challenges in data orchestration before implementing Airflow. (02:42) Comparing Airflow with ETL tools like Talend and why flexibility matters. (04:24) The role of Airflow in enabling cloud-agnostic data processing. (05:45) Key lessons from managing dynamic DAGs at scale. (13:15) How hybrid executors improve performance and efficiency. (14:13) Best practices for testing and monitoring workflows with Airflow. (15:13) The importance of mocking mechanisms when testing DAGs. (17:57) How Prometheus, Grafana and Loki support Airflow monitoring. (22:03) Cost considerations when running Airflow on self-managed infrastructure. (23:14) Airflow’s latest features, including hybrid executors and dark mode. Resources Mentioned: Raviteja Tholupunoori https://www.linkedin.com/in/raviteja0096/?originalSubdomain=in Deloitte Digital https://www.linkedin.com/company/deloitte-digital/ Apache Airflow https://airflow.apache.org/ Grafana https://grafana.com/solutions/apache-airflow/monitor/ Astronomer Presents: Exploring Apache Airflow® 3 Roadshows https://www.astronomer.io/events/roadshow/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26
23:26 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:26
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe 2025 State of Airflow report sheds light on how global users are adopting, evolving and innovating with Apache Airflow. With over 5,000 responses from 116 countries, the survey reveals critical insights into Airflows’ role in business operations, new use cases and what’s ahead for the community. In this episode, Tamara Fingerlin , Developer Advocate at Astronomer , walks us through her process of analyzing survey data, key trends from the report and what to expect from Airflow 3.0. Key Takeaways: (02:14) The State of Airflow report combines anonymized telemetry and survey results. (03:25) The survey received thousands of responses from many countries, showcasing global reach. (04:49) The survey process involves multiple steps, from question selection to report creation. (09:00) Many users expect to increase Airflow usage for revenue-generating or external use cases. (11:04) Experienced users tend to utilize Airflow more for advanced use cases like MLOps. (15:13) UI improvements offer enhanced navigation and error visibility. (18:15) Architectural changes enable new capabilities like remote execution and language support. (19:40) Long-requested features will be available in the new major release. (21:00) Future aspirations include integrating data visualization capabilities into the UI. Resources Mentioned: Tamara Fingerlin https://www.linkedin.com/in/tamara-janina-fingerlin/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ 2025 State of Airflow Webinar https://www.astronomer.io/airflow/state-of-airflow/ Airflow Slack https://apache-airflow-slack.herokuapp.com/ Astronomer Presents: Exploring Apache Airflow® 3 Roadshows https://www.astronomer.io/events/roadshow/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27
28:27 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי28:27
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe security of open-source software is a growing concern, especially as dependencies and regulations become more complex, making it essential to understand how to manage software supply chains effectively. In this episode, we sit down with Michael Winser , Co-Founder at Alpha-Omega and Security Strategy Ambassador at Eclipse Foundation , and Jarek Potiuk , Member of the Security Committee at the Apache Software Foundation , to discuss the challenges of securing Airflow’s dependencies, the evolving landscape of open-source security and how contributors can help strengthen the ecosystem. Key Takeaways: (02:43) Jarek quit his full-time engineer position and uses Airflow as a freelancer. (04:32) Michael finds happiness in having meaningful work with open-source security. (07:01) Software supply chain security focuses on correctness, integrity and availability. (08:44) Airflow’s 790 dependencies present a unique security challenge. (09:43) Airflow’s security team has significantly improved its vulnerability response. (10:22) The transition to Airflow 3 emphasizes enterprise security readiness. (16:20) The ‘Three Fs’ approach: fix it, fork it, or forget it. (18:45) Dependency health is often more critical than fixing known vulnerabilities. (23:32) The ‘Three Fs’ in action. (26:26) Open-source contributors play a key role in supply chain security. Resources Mentioned: Michael Winser - https://www.linkedin.com/in/michaelw/ Jarek Potiuk - https://www.linkedin.com/in/jarekpotiuk/ Apache Airflow - https://airflow.apache.org/ Apache Software Foundation | LinkedIn - https://www.linkedin.com/company/the-apache-software-foundation/ Apache Software Foundation | Website - https://www.apache.org/ Eclipse Foundation | LinkedIn - https://www.linkedin.com/company/eclipse-foundation/ Eclipse Foundation | Website - https://www.eclipse.org/org/foundation/ OpenSSF Working Groups - https://openssf.org/community/openssf-working-groups/ Astronomer Roadshow: Exploring Apache Airflow 3 | London https://www.astronomer.io/events/roadshow/london/ Astronomer Roadshow: Exploring Apache Airflow 3 | New York https://www.astronomer.io/events/roadshow/new-york/ Astronomer Roadshow: Exploring Apache Airflow 3 | Sydney https://www.astronomer.io/events/roadshow/sydney/ Astronomer Roadshow: Exploring Apache Airflow 3 | San Francisco https://www.astronomer.io/events/roadshow/san-francisco/ Astronomer Roadshow: Exploring Apache Airflow 3 | Chicago https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building Scalable ML Infrastructure at Outerbounds with Savin Goyal 36:46
36:46 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי36:46
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיMachine learning is changing fast, and companies need better tools to handle AI workloads. The right infrastructure helps data scientists focus on solving problems instead of managing complex systems. In this episode, we talk with Savin Goyal , Co-Founder and CTO at Outerbounds , about building ML infrastructure, how orchestration makes workflows easier and how Metaflow and Airflow work together to simplify data science. Key Takeaways: (02:02) Savin spent years building AI and ML infrastructure, including at Netflix. (04:05) ML engineering was not a defined role a decade ago. (08:17) Modernizing AI and ML requires balancing new tools with existing strengths. (10:28) ML workloads can be long-running or require heavy computation. (15:29) Different teams at Netflix used multiple orchestration systems for specific needs. (20:10) Stable APIs prevent rework and keep projects moving. (21:07) Metaflow simplifies ML workflows by optimizing data and compute interactions. (25:53) Limited local computing power makes running ML workloads challenging. (27:43) Airflow UI monitors pipelines, while Metaflow UI gives ML insights. (33:13) The most successful data professionals focus on business impact, not just technology. Resources Mentioned: Savin Goyal - https://www.linkedin.com/in/savingoyal/ Outerbounds - https://www.linkedin.com/company/outerbounds/ Apache Airflow - https://airflow.apache.org/ Metaflow - https://metaflow.org/ Netflix’s Maestro Orchestration System - https://netflixtechblog.com/maestro-netflixs-workflow-orchestrator-ee13a06f9c78?gi=8e6a067a92e9#:~:text=Maestro%20is%20a%20fully%20managed,data%20between%20different%20storages%2C%20etc. TensorFlow - https://www.tensorflow.org/ PyTorch - https://pytorch.org/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy 27:40
27:40 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי27:40
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיKeeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov , Senior Data Engineer at Stripe , and Sharadh Krishnamurthy , Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience. Key Takeaways: (02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data. (05:08) 80% of Stripe engineers use data orchestration, making scalability critical. (06:06) Airflow powers business reports, regulatory needs and ML workflows. (08:02) Custom task frameworks improve dependencies and validation. (08:50) "User scope mode" enables local testing without production impact. (10:39) Migrating to Airflow 2 improves isolation, safety and scalability. (16:40) Monolithic DAGs caused database issues, prompting a service-based shift. (19:24) Frequent Airflow upgrades ensure stability and access to new features. (21:38) DAG versioning and backfill improvements enhance developer experience. (23:38) Greater UI customization would offer more flexibility. Resources Mentioned: Nick Bilozerov - https://www.linkedin.com/in/nick-bilozerov/ Sharadh Krishnamurthy - https://www.linkedin.com/in/sharadhk/ Apache Airflow - https://airflow.apache.org/ Stripe | LinkedIn - https://www.linkedin.com/company/stripe/ Stripe | Website - https://stripe.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot 17:54
17:54 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי17:54
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיTurning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot , Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets. Key Takeaways: (02:04) CSET provides data-driven analysis to inform government decision-makers. (03:54) ETL pipelines merge multiple data sources for more comprehensive insights. (04:20) Airflow is central to automating and streamlining large-scale data ingestion. (05:11) Larger-scale databases create challenges that require scalable solutions. (07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers. (12:13) DAG Factory and dynamic task mapping can improve workflow efficiency. (15:46) Tracking data lineage helps teams understand dependencies across DAGs. (16:14) New Airflow features enhance visibility and debugging for complex pipelines. Resources Mentioned: Jennifer Melot - https://www.linkedin.com/in/jennifer-melot-aa710144/ Center for Security and Emerging Technology (CSET) - https://www.linkedin.com/company/georgetown-cset/ Apache Airflow - https://airflow.apache.org/ Zenodo - https://zenodo.org/ OpenLineage - https://openlineage.io/ Cloud Dataplex - https://cloud.google.com/dataplex Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Hybrid Testing Solutions for Autonomous Driving at Bosch with Jens Scheffler and Christian Schilling 33:45
33:45 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי33:45
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיTesting autonomous vehicles demands precision, scalability and powerful orchestration tools — enter Apache Airflow, a key component of Bosch’s cutting-edge testing framework. In this episode, we sit down with Jens Scheffler , Test Execution Cluster Technical Architect, and Christian Schilling , Product Owner Open Loop Testing Automated Driving, both at Bosch , to explore how Bosch harnesses Airflow to streamline complex testing scenarios. They share insights on scaling workflows, integrating hybrid infrastructures and ensuring vehicle safety through rigorous automated testing. Key Takeaways: (01:35) Airflow orchestrates millions of test hours for autonomous systems. (03:15) Jens scales distributed systems with Kubernetes for job orchestration. (06:02) Airflow runs hundreds of tests simultaneously. (06:44) Virtual testing reduces costs and on-road trials. (12:19) Unified APIs and GUIs streamline operations. (15:05) Self-service setups empower Bosch teams. (18:00) Physical hardware integration ensures real-world timing. (20:30) Dynamic task mapping scales workflows efficiently. (25:22) Open-source contributions improve stability. (31:06) Edge and Celery executors power Bosch's hybrid scheduling. Resources Mentioned: Jens Scheffler - https://www.linkedin.com/in/jens-scheffler/ Christian Schilling - https://www.linkedin.com/in/christian-schilling-a5078831a/ Bosch - https://www.linkedin.com/company/bosch/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io GitHub - https://github.com Edge Executor - https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Overcoming Airflow Scaling Challenges at Monzo Bank with Jonathan Rainer 43:39
43:39 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי43:39
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיScaling a data orchestration platform to manage thousands of tasks daily demands innovative solutions and strategic problem-solving. In this episode, we explore the complexities of scaling Airflow and the challenges of orchestrating thousands of tasks in dynamic data environments. Jonathan Rainer , Former Platform Engineer at Monzo Bank , joins us to share his journey optimizing data pipelines, overcoming UI limitations and ensuring DAG consistency in high-stakes scenarios. Key Takeaways: (03:11) Using Airflow to schedule computation in BigQuery. (07:02) How DAGs with 8,000+ tasks were managed nightly. (08:18) Ensuring accuracy in regulatory reporting for banking. (11:35) Handling task inconsistency and DAG failures with automation. (16:09) Building a service to resolve DAG consistency issues in Airflow. (25:05) Challenges with scaling the Airflow UI for thousands of tasks. (27:03) The role of upstream and downstream task management in Airflow. (37:33) The importance of operational metrics for monitoring Airflow health. (39:19) Balancing new tools with root cause analysis to address scaling issues. (41:35) Why scaling solutions require both technical and leadership buy-in Resources Mentioned: Jonathan Rainer - https://www.linkedin.com/in/jonathan-rainer/ Monzo Bank - https://www.linkedin.com/company/monzo-bank/ Apache Airflow - https://airflow.apache.org/ BigQuery - https://airflow.apache.org/docs/apache-airflow-providers-google/stable/operators/cloud/bigquery.html Kubernetes - https://kubernetes.io/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Orchestrating Analytics and AI Workflows at Telia with Arjun Anandkumar 26:00
26:00 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי26:00
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיT he future of data engineering lies in seamless orchestration and automation. In this episode, Arjun Anandkumar , Data Engineer at Telia , shares how his team uses Airflow to drive analytics and AI workflows. He highlights the challenges of scaling data platforms and how adopting best practices can simplify complex processes for teams across the organization. Arjun also discusses the transformative role of tools like Cosmos and Terraform in enhancing efficiency and collaboration. Key Takeaways: (02:16) Telia operates across the Nordics and Baltics, focusing on telecom and energy services. (03:45) Airflow runs dbt models seamlessly with Cosmos on AWS MWAA. (05:47) Cosmos improves visibility and orchestration in Airflow. (07:00) Medallion Architecture organizes data into bronze, silver and gold layers. (08:34) Task group challenges highlight the need for adaptable workflows. (15:04) Scaling managed services requires trial, error and tailored tweaks. (19:46) Terraform scales infrastructure, while YAML templates manage DAGs efficiently. (20:00) Templated DAGs and robust testing enhance platform management. (24:15) Open-source resources drive innovation in Airflow practices. Resources Mentioned: Arjun Anandkumar - https://www.linkedin.com/in/arjunanand1/?originalSubdomain=dk Telia - https://www.linkedin.com/company/teliacompany/ Apache Airflow - https://airflow.apache.org/ Cosmos by Astronomer - https://www.astronomer.io/cosmos/ Terraform - https://www.terraform.io/ Medallion Architecture by Databricks - https://www.databricks.com/glossary/medallion-architecture Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Role of Airflow in Finance Transformation at Etraveli Group with Mihir Samant 21:19
21:19 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי21:19
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיTransforming bottlenecked finance processes into streamlined, automated systems requires the right tools and a forward-thinking approach. In this episode, Mihir Samant , Senior Data Analyst at Etraveli Group , joins us to share how his team leverages Airflow to revolutionize finance automation. With extensive experience in data workflows and a passion for open-source tools, Mihir provides valuable insights into building efficient, scalable systems. We explore the transformative power of Airflow in automating workflows and enhancing data orchestration within the finance domain. Key Takeaways: (02:14) Etraveli Group specializes in selling affordable flight tickets and ancillary services. (03:56) Mihir’s finance automation team uses Airflow to tackle month-end bottlenecks. (06:00) Airflow's flexibility enables end-to-end automation for finance workflows. (07:00) Open-source Airflow tools offer cost-effective solutions for new teams. (08:46) Sensors and dynamic DAGs are pivotal features for optimizing tasks. (13:30) GitSync simplifies development by syncing environments seamlessly. (16:27) Plans include integrating Databricks for more advanced data handling. (17:58) Airflow and Databricks offer multiple flexible methods to trigger workflows and execute SQL queries seamlessly. Resources Mentioned: Mihir Samant - https://www.linkedin.com/in/misamant/?originalSubdomain=ca Etraveli Group - https://www.linkedin.com/company/etraveli-group/ Apache Airflow - https://airflow.apache.org/ Docker - https://www.docker.com/ Databricks - https://www.databricks.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside Ford’s Data Transformation: Advanced Orchestration Strategies with Vasantha Kosuri-Marshall 38:54
38:54 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי38:54
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData engineering is entering a new era, where orchestration and automation are redefining how large-scale projects operate. This episode features Vasantha Kosuri-Marshall , Data and ML Ops Engineer at Ford Motor Company . Vasantha shares her expertise in managing complex data pipelines. She takes us through Ford's transition to cloud platforms, the adoption of Airflow and the intricate challenges of orchestrating data in a diverse environment. Key Takeaways: (03:10) Vasantha’s transition to the Advanced Driving Assist Systems team at Ford. (05:42) Early adoption of Airflow to orchestrate complex data pipelines. (09:29) Ford's move from on-premise data solutions to Google Cloud Platform. (12:03) The importance of Airflow's scheduling capabilities for efficient data management. (16:12) Using Kubernetes to scale Airflow for large-scale data processing. (19:59) Vasantha’s experience in overcoming challenges with legacy orchestration tools. (22:22) Integration of data engineering and data science pipelines at Ford. (28:03) How deferrable operators in Airflow improve performance and save costs. (32:12) Vasantha’s insights into tuning Airflow properties for thousands of DAGs. (36:09) The significance of monitoring and observability in managing Airflow instances. Resources Mentioned: Vasantha Kosuri-Marshall - https://www.linkedin.com/in/vasantha-kosuri-marshall-0b0aab188/ Apache Airflow - https://airflow.apache.org/ Google Cloud Platform (GCP) - https://cloud.google.com/ Ford Motor Company | LinkedIn - https://www.linkedin.com/company/ford-motor-company/ Ford Motor Company | Website - https://www.ford.com/ Astronomer - https://www.astronomer.io/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Powering Finance With Advanced Data Solutions at Ramp with Ryan Delgado 24:35
24:35 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:35
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData is the backbone of every modern business, but unlocking its full potential requires the right tools and strategies. In this episode, Ryan Delgado , Director of Engineering at Ramp , joins us to explore how innovative data platforms can transform business operations and fuel growth. He shares insights on integrating Apache Airflow, optimizing data workflows and leveraging analytics to enhance customer experiences. Key Takeaways: (01:52) Data is the lifeblood of Ramp, touching every vertical in the company. (03:18) Ramp’s data platform team enables high-velocity scaling through tailored tools. (05:27) Airflow powers Ramp’s enterprise data warehouse integrations for advanced analytics. (07:55) Centralizing data in Snowflake simplifies storage and analytics pipelines. (12:08) Machine learning models at Ramp integrate seamlessly with Airflow for operational excellence. (14:11) Leveraging Airflow datasets eliminates inefficiencies in DAG dependencies. (17:22) Platforms evolve from solving narrow business problems to scaling organizationally. (18:55) ClickHouse enhances Ramp’s OLAP capabilities with 100x performance improvements. (19:47) Ramp’s OLAP platform improves performance by reducing joins and leveraging ClickHouse. (21:46) Ryan envisions a lighter-weight, more Python-native future for Airflow. Resources Mentioned: Ryan Delgado - https://www.linkedin.com/in/ryan-delgado-69544568/ Ramp - https://www.linkedin.com/company/ramp/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ ClickHouse - https://clickhouse.com/ dbt - https://www.getdbt.com/ Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Exploring the Power of Airflow 3 at Astronomer with Amogh Desai 30:24
30:24 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי30:24
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיWhat does it take to go from fixing a broken link to becoming a committer for one of the world’s leading open-source projects? Amogh Desai , Senior Software Engineer at Astronomer , takes us through his journey with Apache Airflow. From small contributions to building meaningful connections in the open-source community, Amogh’s story provides actionable insights for anyone on the cusp of their open-source journey. Key Takeaways: (02:09) Building data engineering platforms at Cloudera with Kubernetes. (04:00) Brainstorming led to contributing to Apache Airflow. (05:17) Starting small with link fixes, progressing to Breeze development. (07:00) Becoming a committer for Apache Airflow in September 2023. (09:51) The steep learning curve for contributing to Airflow. (16:30) Using GitHub’s “good-first-issue” label to get started. (18:15) Setting up a development environment with Breeze. (22:00) Open-source contributions enhance your resume and career. (24:51) Amogh’s advice: Start small and stay consistent. (28:12) Engage with the community via Slack, email lists and meetups. Resources Mentioned: Amogh Desai - https://www.linkedin.com/in/amogh-desai-385141157/?originalSubdomain=in%20%20https://www.linkedin.com/company/astronomer/ Astronomer - https://www.linkedin.com/company/astronomer/ Apache Airflow GitHub Repository - https://github.com/apache/airflow Contributors Quick Guide - https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst Breeze Development Tool - https://github.com/apache/airflow/tree/main/dev/breeze Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Using Airflow To Power Machine Learning Pipelines at Optimove with Vasyl Vasyuta 24:11
24:11 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:11
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData orchestration and machine learning are shaping how organizations handle massive datasets and drive customer-focused strategies. Tools like Apache Airflow are central to this transformation. In this episode, Vasyl Vasyuta , R&D Team Leader at Optimove , joins us to discuss how his team leverages Airflow to optimize data processing, orchestrate machine learning models and create personalized customer experiences. Key Takeaways: (01:59) Optimove tailors marketing notifications with personalized customer journeys. (04:25) Airflow orchestrates Snowflake procedures for massive datasets. (05:11) DAGs manage workflows with branching and replay plugins. (05:41) The "Joystick" plugin enables seamless data replays. (09:33) Airflow supports MLOps for customer data grouping. (11:15) Machine learning predicts customer behavior for better campaigns. (13:20) Thousands of DAGs run every five minutes for data processing. (15:36) Custom versioning allows rollbacks and gradual rollouts. (18:00) Airflow logs enhance operational observability. (23:00) DAG versioning in Airflow 3.0 could boost efficiency. Resources Mentioned: Vasyl Vasyuta - https://www.linkedin.com/in/vasyl-vasyuta-3270b54a/ Optimove - https://www.linkedin.com/company/optimove/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Datadog - https://www.datadoghq.com/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Maximizing Business Impact Through Data at GlossGenius with Katie Bauer 25:49
25:49 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי25:49
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיBridging the gap between data teams and business priorities is essential for maximizing impact and building value-driven workflows. Katie Bauer , Senior Director of Data at GlossGenius , joins us to share her principles for creating effective, aligned data teams. In this episode, Katie draws from her experience at GlossGenius, Reddit and Twitter to highlight the common pitfalls data teams face and how to overcome them. She offers practical strategies for aligning team efforts with organizational goals and fostering collaboration with stakeholders. Key Takeaways: (02:36) GlossGenius provides an all-in-one platform for beauty professionals. (03:59) Airflow orchestrates data and MLOps workflows at GlossGenius. (04:41) Focusing on value helps data teams achieve greater impact. (06:23) Aligning team priorities with company goals minimizes friction. (08:44) Building strong stakeholder relationships requires curiosity. (12:46) Treating roles as flexible fosters team innovation. (13:21) Adapting to new technologies improves effectiveness. (18:28) Acting like your time is valuable earns respect. (23:38) Proactive data initiatives drive strategic value. (24:20) Usage data offers critical insights into tool effectiveness. Resources Mentioned: Katie Bauer - https://www.linkedin.com/in/mkatiebauer/ GlossGenius - https://www.linkedin.com/company/glossgenius/ Apache Airflow - https://airflow.apache.org/ DBT - https://www.getdbt.com/ Cosmos - https://cosmos.apache.org/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade 27:47
27:47 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי27:47
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיScaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade , Staff Software Engineer at LinkedIn , shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers. Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility. Key Takeaways: (01:36) LinkedIn minimizes human involvement in production to reduce errors. (02:00) Airflow powers LinkedIn’s Continuous Deployment platform. (05:43) Continuous deployment adoption grew from 8% to a targeted 80%. (11:25) Kubernetes ensures scalability and flexibility for deployments. (12:04) A custom UI offers real-time deployment transparency. (16:23) No-code YAML workflows simplify deployment tasks. (17:18) Canaries and metrics ensure safe deployments across fabrics. (20:45) A gateway service ensures redundancy across Airflow clusters. (24:22) Abstractions let engineers focus on development, not logistics. (25:20) Multi-language support in Airflow 3.0 simplifies adoption. Resources Mentioned: Rahul Gade - https://www.linkedin.com/in/rahul-gade-68666818/ LinkedIn - https://www.linkedin.com/company/linkedin/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io/ Open Policy Agent (OPA) - https://www.openpolicyagent.org/ Backstage - https://backstage.io/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Uber Manages 1 Million Daily Tasks Using Airflow, with Shobhit Shah and Sumit Maheshwari 28:44
28:44 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי28:44
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיWhen data orchestration reaches Uber’s scale, innovation becomes a necessity, not a luxury. In this episode, we discuss the innovations behind Uber’s unique Airflow setup. With our guests Shobhit Shah and Sumit Maheshwari , both Staff Software Engineers at Uber , we explore how their team manages one of the largest data workflow systems in the world. Shobhit and Sumit walk us through the evolution of Uber’s Airflow implementation, detailing the custom solutions that support 200,000 daily pipelines. They discuss Uber's approach to tackling complex challenges in data orchestration, disaster recovery and scaling to meet the company’s extensive data needs. Key Takeaways: (02:03) Airflow as a service streamlines Uber’s data workflows. (06:16) Serialization boosts security and reduces errors. (10:05) Java-based scheduler improves system reliability. (13:40) Custom recovery model supports emergency pipeline switching. (15:58) No-code UI allows easy pipeline creation for non-coders. (18:12) Backfill feature enables historical data processing. (22:06) Regular updates keep Uber aligned with Airflow advancements. (26:07) Plans to leverage Airflow’s latest features. Resources Mentioned: Shobhit Shah - https://www.linkedin.com/in/shahshobhit/ Sumit Maheshwar - https://www.linkedin.com/in/maheshwarisumit/ Uber - https://www.linkedin.com/company/uber-com/ Apache Airflow - https://airflow.apache.org/ Airflow Summit - https://airflowsummit.org/ Uber - https://www.uber.com/tw/en/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building Resilient Data Systems for Modern Enterprises at Astrafy with Andrea Bombino 28:29
28:29 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי28:29
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיEfficient data orchestration is the backbone of modern analytics and AI-driven workflows. Without the right tools, even the best data can fall short of its potential. In this episode, Andrea Bombino , Co-Founder and Head of Analytics Engineering at Astrafy , shares insights into his team’s approach to optimizing data transformation and orchestration using tools like datasets and Pub/Sub to drive real-time processing. Andrea explains how they leverage Apache Airflow and Google Cloud to power dynamic data workflows. Key Takeaways: (01:55) Astrafy helps companies manage data using Google Cloud. (04:36) Airflow is central to Astrafy’s data engineering efforts. (07:17) Datasets and Pub/Sub are used for real-time workflows. (09:59) Pub/Sub links multiple Airflow environments. (12:40) Datasets eliminate the need for constant monitoring. (15:22) Airflow updates have improved large-scale data operations. (18:03) New Airflow API features make dataset updates easier. (20:45) Real-time orchestration speeds up data processing for clients. (23:26) Pub/Sub enhances flexibility across cloud environments. (26:08) Future Airflow features will offer more control over data workflows. Resources Mentioned: Andrea Bombino - https://www.linkedin.com/in/andrea-bombino/ Astrafy - https://www.linkedin.com/company/astrafy/ Apache Airflow - https://airflow.apache.org/ Google Cloud - https://cloud.google.com/ dbt - https://www.getdbt.com/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Inside Airflow 3: Redefining Data Engineering with Vikram Koka 30:08
30:08 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי30:08
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData orchestration is evolving faster than ever and Apache Airflow 3 is set to revolutionize how enterprises handle complex workflows. In this episode, we dive into the exciting advancements with Vikram Koka , Chief Strategy Officer at Astronomer and PMC Member at The Apache Software Foundation . Vikram shares his insights on the evolution of Airflow and its pivotal role in shaping modern data-driven workflows, particularly with the upcoming release of Airflow 3. Key Takeaways: (02:36) Vikram leads Astronomer’s engineering and open-source teams for Airflow. (05:26) Airflow enables reliable data ingestion and curation. (08:17) Enterprises use Airflow for mission-critical data pipelines. (11:08) Airflow 3 introduces major architectural updates. (13:58) Multi-cloud and edge deployments are supported in Airflow 3. (16:49) Event-driven scheduling makes Airflow more dynamic. (19:40) Tasks in Airflow 3 can run in any language. (22:30) Multilingual task support is crucial for enterprises. (25:21) Data assets and event-based integration enhance orchestration. (28:12) Community feedback plays a vital role in Airflow 3. Resources Mentioned: Vikram Koka - https://www.linkedin.com/in/vikramkoka/ Astronomer - https://www.linkedin.com/company/astronomer/ The Apache Software Foundation LinkedIn - https://www.linkedin.com/company/the-apache-software-foundation/ Apache Airflow LinkedIn - https://www.linkedin.com/company/apache-airflow/ Apache Airflow - https://airflow.apache.org/ Astronomer - https://www.astronomer.io/ The Apache Software Foundation - https://www.apache.org/ Join the Airflow slack and/or Dev list - https://airflow.apache.org/community/ Apache Airflow Survey - https://astronomer.typeform.com/airflowsurvey24 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Building a Data-Driven HR Platform at 15Five with Guy Dassa 20:25
20:25 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי20:25
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData and AI are revolutionizing HR, empowering leaders to measure performance and drive strategic decisions like never before. In this episode, we explore the transformation of HR technology with Guy Dassa , Chief Technology Officer at 15Five , as he shares insights into their evolving data platform. Guy discusses how 15Five equips HR leaders with tools to measure and take action on team performance, engagement and retention. He explains their data-driven approach, highlighting how Apache Airflow supports their data ingestion, transformation, and AI integration. Key Takeaways: (01:54) 15Five acts as a command center for HR leaders. (03:40) Tools like performance reviews, engagement surveys, and an insights dashboard guide actionable HR steps. (05:33) Data visualization, insights, and action recommendations enhance HR effectiveness to improve their people's outcomes. (07:08) Strict data confidentiality and sanitized AI model training. (09:21) Airflow is central to data transformation and enrichment. (11:15) Airflow enrichment DAGs integrate AI models. (13:33) Integration of Airflow and DBT enables efficient data transformation. (15:28) Synchronization challenges arise with reverse ETL processes. (17:10) Future plans include deeper Airflow integration with AI. (19:31) Emphasizing the need for DAG versioning and improved dependency visibility. Resources Mentioned: Guy Dassa - https://www.linkedin.com/in/guydassa/ 15Five - https://www.linkedin.com/company/15five/ Apache Airflow - https://airflow.apache.org/ MLflow - https://mlflow.org/ DBT - https://www.getdbt.com/ Kubernetes - https://kubernetes.io/ RedShift - https://aws.amazon.com/redshift/ 15Five - https://www.15five.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Intersection of AI and Data Management at Dosu with Devin Stein 20:18
20:18 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי20:18
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיUnlocking engineering productivity goes beyond coding — it’s about managing knowledge efficiently. In this episode, we explore the innovative ways in which Dosu leverages Airflow for data orchestration and supports the Airflow project. Devin Stein , Founder of Dosu , shares his insights on how engineering teams can focus on value-added work by automating knowledge management. Devin dives into Dosu’s purpose, the significance of AI in their product, and why they chose Airflow as the backbone for scheduling and data management. Key Takeaways: (01:33) Dosu's mission to democratize engineering knowledge. (05:00) AI is central to Dosu's product for structuring engineering knowledge. (06:23) The importance of maintaining up-to-date data for AI effectiveness. (07:55) How Airflow supports Dosu’s data ingestion and automation processes. (08:45) The reasoning behind choosing Airflow over other orchestrators. (11:00) Airflow enables Dosu to manage both traditional ETL and dynamic workflows. (13:04) Dosu assists the Airflow project by auto-labeling issues and discussions. (14:56) Thoughtful collaboration with the Airflow community to introduce AI tools. (16:37) The potential of Airflow to handle more dynamic, scheduled workflows in the future. (18:00) Challenges and custom solutions for implementing dynamic workflows in Airflow. Resources Mentioned: Apache Airflow - https://airflow.apache.org/ Dosu Website - https://dosu.dev/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 AI-Powered Vehicle Automation at Ford Motor Company with Serjesh Sharma 26:11
26:11 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי26:11
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיHarnessing data at scale is the key to driving innovation in autonomous vehicle technology. In this episode, we uncover how advanced orchestration tools are transforming machine learning operations in the automotive industry. Serjesh Sharma, Supervisor ADAS Machine Learning Operations (MLOps) at Ford Motor Company, joins us to discuss the challenges and innovations his team faces working to enhance vehicle safety and automation. Serjesh shares insights into the intricate data processes that support Ford’s Advanced Driver Assistance Systems (ADAS) and how his team leverages Apache Airflow to manage massive data loads efficiently. Key Takeaways: (01:44) ADAS involves advanced features like pre-collision assist and self-driving capabilities. (04:47) Ensuring sensor accuracy and vehicle safety requires extensive data processing. (05:08) The combination of on-prem and cloud infrastructure optimizes data handling. (09:27) Ford processes around one petabyte of data per week, using both CPUs and GPUs. (10:33) Implementing software engineering best practices to improve scalability and reliability. (15:18) GitHub Issues streamline onboarding and infrastructure provisioning. (17:00) Airflow's modular design allows Ford to manage complex data pipelines. (19:00) Kubernetes pod operators help optimize resource usage for CPU-intensive tasks. (20:35) Ford's scale challenges led to customized Airflow configurations for high concurrency. (21:02) Advanced orchestration tools are pivotal in managing vast data landscapes in automotive innovation. Resources Mentioned: Serjesh Sharma - www.linkedin.com/in/serjeshsharma/ Ford Motor Company - www.linkedin.com/company/ford-motor-company/ Apache Airflow - airflow.apache.org/ Kubernetes - kubernetes.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 From Task Failures to Operational Excellence at GumGum with Brendan Frick 24:06
24:06 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:06
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData failures are inevitable but how you manage them can define the success of your operations. In this episode, we dive deep into the challenges of data engineering and AI with Brendan Frick, Senior Engineering Manager, Data at GumGum. Brendan shares his unique approach to managing task failures and DAG issues in a high-stakes ad-tech environment. Brendan discusses how GumGum leverages Apache Airflow to streamline data processes, ensuring efficient data movement and orchestration while minimizing disruptions in their operations. Key Takeaways: (02:02) Brendan’s role at GumGum and its approach to ad tech. (04:27) How GumGum uses Airflow for daily data orchestration, moving data from S3 to warehouses. (07:02) Handling task failures in Airflow using Jira for actionable, developer-friendly responses. (09:13) Transitioning from email alerts to a more structured system with Jira and PagerDuty. (11:40) Monitoring task retry rates as a key metric to identify potential issues early. (14:15) Utilizing Looker dashboards to track and analyze task performance and retry rates. (16:39) Transitioning from Kubernetes operator to a more reliable system for data processing. (19:25) The importance of automating stakeholder communication with data lineage tools like Atlan. (20:48) Implementing data contracts to ensure SLAs are met across all data processes. (22:01) The role of scalable SLAs in Airflow to ensure data reliability and meet business needs. Resources Mentioned: Brendan Frick - https://www.linkedin.com/in/brendan-frick-399345107/ GumGum - https://www.linkedin.com/company/gumgum/ Apache Airflow - https://airflow.apache.org/ Jira - https://www.atlassian.com/software/jira Atlan - https://atlan.com/ Kubernetes - https://kubernetes.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 From Sensors to Datasets: Enhancing Airflow at Astronomer with Maggie Stark and Marion Azoulai 22:25
22:25 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי22:25
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיA 13% reduction in failure rates — this is how two data scientists at Astronomer revolutionized their data pipelines using Apache Airflow. In this episode, we enter the world of data orchestration and AI with Maggie Stark and Marion Azoulai, both Senior Data Scientists at Astronomer. Maggie and Marion discuss how their team re-architected their use of Airflow to improve scalability, reliability and efficiency in data processing. They share insights on overcoming challenges with sensors and how moving to datasets transformed their workflows. Key Takeaways: (02:23) The data team’s role as a centralized hub within Astronomer. (05:11) Airflow is the backbone of all data processes, running 60,000 tasks daily. (07:13) Custom task groups enable efficient code reuse and adherence to best practices. (11:33) Sensor-heavy architectures can lead to cascading failures and resource issues. (12:09) Switching to datasets has improved reliability and scalability. (14:19) Building a control DAG provides end-to-end visibility of pipelines. (16:42) Breaking down DAGs into smaller units minimizes failures and improves management. (19:02) Failure rates improved from 16% to 3% with the new architecture. Resources Mentioned: Maggie Stark - https://www.linkedin.com/in/margaretstark/ Marion Azoulai - https://www.linkedin.com/in/marionazoulai/ Astronomer | LinkedIn - https://www.linkedin.com/company/astronomer/ Apache Airflow - https://airflow.apache.org/ Astronomer | Website - https://www.astronomer.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Mastering Data Orchestration with Airflow at M Science with Ben Tallman 24:36
24:36 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:36
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיMastering the flow of data is essential for driving innovation and efficiency in today’s competitive landscape. In this episode, we explore the evolution of data orchestration and the pivotal role of Apache Airflow in modern data workflows. Ben Tallman, Chief Technology Officer at M Science, joins us and shares his extensive experience with Airflow, detailing its early adoption, evolution and the profound impact it has had on data engineering practices. His insights reveal how leveraging Airflow can streamline complex data processes, enhance observability and ultimately drive business success. Key Takeaways: (02:31) Benjamin’s journey with Airflow and its early adoption. (05:36) The transition from legacy schedulers to Airflow at Apigee and later Google. (08:52) The challenges and benefits of running production-grade Airflow instances. (10:46) How Airflow facilitates the management of large-scale data at M Science. (11:56) The importance of reducing time to value for customers using data products. (13:32) Airflow’s role in ensuring observability and reliability in data workflows. (17:00) Managing petabytes of data and billions of records efficiently. (19:08) Integration of various data sources and ensuring data product quality. (20:04) Leveraging Airflow for data observability and reducing time to value. (22:04) Benjamin’s vision for the future development of Airflow, including audit trails for variables. Resources Mentioned: Ben Tallman - https://www.linkedin.com/in/btallman/ M Science - https://www.linkedin.com/company/m-science-llc/ Apache Airflow - https://airflow.apache.org/ Astronomer - https://www.astronomer.io/ Databricks - https://databricks.com/ Snowflake - https://www.snowflake.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Welcome to The Data Flowcast: Mastering Airflow for Data Engineering & AI — the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Enhancing Business Metrics With Airflow at Artlist with Hannan Kravitz 23:51
23:51 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:51
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData orchestration is revolutionizing the way companies manage and process data. In this episode, we explore the critical role of data orchestration in modern data workflows and how Apache Airflow is used to enhance data processing and AI model deployment. Hannan Kravitz, Data Engineering Team Leader at Artlist, joins us to share his insights on leveraging Airflow for data engineering and its impact on their business operations. Key Takeaways: (01:00) Hannan introduces Artlist and its mission to empower content creators. (04:27) The importance of collecting and modeling data to support business insights. (06:40) Using Airflow to connect multiple data sources and create dashboards. (09:40) Implementing a monitoring DAG for proactive alerts within Airflow. (12:31) Customizing Airflow for business metric KPI monitoring and setting thresholds. (15:00) Addressing decreases in purchases due to technical issues with proactive alerts. (17:45) Customizing data quality checks with dynamic task mapping in Airflow. (20:00) Desired improvements in Airflow UI and logging capabilities. (21:00) Enabling business stakeholders to change thresholds using Streamlit. (22:26) Future improvements desired in the Airflow project. Resources Mentioned: Hannan Kravitz - https://www.linkedin.com/in/hannan-kravitz-60563112/ Artlist - https://www.linkedin.com/company/art-list/ Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Streamlit - https://streamlit.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Cutting-Edge Data Engineering at Teya with Alexandre Magno Lima Martins 23:46
23:46 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:46
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיData engineering is constantly evolving and staying ahead means mastering tools like Apache Airflow. In this episode, we explore the world of data engineering with Alexandre Magno Lima Martins, Senior Data Engineer at Teya. Alexandre talks about optimizing data workflows and the smart solutions they've created at Teya to make data processing easier and more efficient. Key Takeaways: (02:01) Alexandre explains his role at Teya and the responsibilities of a data platform engineer. (02:40) The primary use cases of Airflow at Teya, especially with dbt and machine learning projects. (04:14) How Teya creates self-service DAGs for dbt models. (05:58) Automating DAG creation with CI/CD pipelines. (09:04) Switching to a multi-file method for better Airflow performance. (12:48) Challenges faced with Kubernetes Executor vs. Celery Executor. (16:13) Using Celery Executor to handle fast tasks efficiently. (17:02) Implementing KEDA autoscaler for better scaling of Celery workers. (19:05) Reasons for not using Cosmos for DAG generation and cross-DAG dependencies. (21:16) Alexandre's wish list for future Airflow features, focusing on multi-tenancy. Resources Mentioned: Alexandre Magno Lima Martins - https://www.linkedin.com/in/alex-magno/ Teya - https://www.linkedin.com/company/teya-global/ Apache Airflow - https://airflow.apache.org/ dbt - https://www.getdbt.com/ Kubernetes - https://kubernetes.io/ KEDA - https://keda.sh/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Airflow Strategies for Business Efficiency at Campbell with Larry Komenda 26:10
26:10 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי26:10
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיManaging data workflows well can change the game for any company. In this episode, we talk about how Airflow makes this possible. Larry Komenda, Chief Technology Officer at Campbell, shares how Airflow supports their operations and improves efficiency. Larry discusses his role at Campbell, their switch to Airflow, and its impact. We look at their strategies for testing and maintaining reliable workflows and how these help their business. Key Takeaways: (02:26) Strong technology and data systems are crucial for Campbell’s investment process. (05:03) Airflow manages data pipelines efficiently in the market data team. (07:39) Airflow supports various departments, including trading and operations. (09:22) Machine learning models run on dedicated Airflow instances. (11:12) Reliable workflows are ensured through thorough testing and development. (13:45) Business tasks are organized separately from Airflow for easier testing. (15:30) Non-technical teams have access to Airflow for better efficiency. (17:20) Thorough testing before deploying to Airflow is essential. (19:10) Non-technical users can interact with Airflow DAGs to solve their issues. (21:55) Airflow improves efficiency and reliability in trading and operations. (24:40) Enhancing the Airflow UI for non-technical users is important for accessibility. Resources Mentioned: Larry Komenda - https://www.linkedin.com/in/larrykomenda/ Campbell - https://www.linkedin.com/company/campbell-and-company/ 30% off Airflow Summit Ticket - https://ti.to/airflowsummit/2024/discount/30DISC_ASTRONOMER Apache Airflow - https://airflow.apache.org/ NumPy - https://numpy.org/ Python - https://www.python.org/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Laurel Uses Airflow To Enhance Machine Learning Pipelines with Vincent La and Jim Howard 23:58
23:58 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:58
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe world of timekeeping for knowledge workers is transforming through the use of AI and machine learning. Understanding how to leverage these technologies is crucial for improving efficiency and productivity. In this episode, we’re joined by Vincent La, Principal Data Scientist at Laurel, and Jim Howard, Principal Machine Learning Engineer at Laurel, to explore the implementation of AI in automating timekeeping and its impact on legal and accounting firms. Key Takeaways: (01:54) Laurel's mission in time automation. (03:39) Solving clustering, prediction and summarization with AI. (06:30) Daily batch jobs for user time generation. (08:22) Knowledge workers touch 300 items daily. (09:01) Mapping 300 activities to seven billable items. (11:38) Retraining models for better performance. (14:00) Using Airflow for retraining and backfills. (17:06) RAG-based summarization for user-specific tone. (18:58) Testing Airflow DAGs for cost-effective summarization. (22:00) Enhancing Airflow for long-running DAGs. Resources Mentioned: Vincent La - https://www.linkedin.com/in/vincentla/ Jim Howard - https://www.linkedin.com/in/jameswhowardml/ Laurel - https://www.linkedin.com/company/laurel-ai/ Apache Airflow - https://airflow.apache.org/ Ernst & Young - https://www.ey.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing 23:51
23:51 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:51
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיDiscover the cutting-edge methods Vibrant Planet uses to revolutionize geospatial data processing and resource management. In this episode, we delve into the intricacies of scaling geospatial data processing and resource allocation with experts from Vibrant Planet. Joining us are Cyrus Dukart, Engineering Lead, and David Sacerdote, Staff Software Engineer, who share their innovative approaches to handling large datasets and optimizing resource use in Airflow. Key Takeaways: (00:00) Inefficiencies in resource allocation. (03:00) Scientific validity of sharded results. (05:53) Tech-based solutions for resource management. (06:11) Retry callback process for resource allocation. (08:00) Running database queries for resource needs. (10:05) Importance of remembering resource usage. (13:51) Generating resource predictions. (14:44) Custom task decorator for resource management. (20:28) Massive resource usage gap in sharded data. (21:14) Fail-fast model for long-running tasks. Resources Mentioned: Cyrus Dukart - https://www.linkedin.com/in/cyrus-dukart-6561482/ David Sacerdote - https://www.linkedin.com/in/davidsacerdote/ Vibrant Planet - https://www.linkedin.com/company/vibrant-planet/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io/ Vibrant Planet - https://vibrantplanet.net/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Future of AI in Data Engineering With Astronomer’s Julian LaNeve and David Xue 23:36
23:36 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:36
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe world of data orchestration and machine learning is rapidly evolving, and tools like Apache Airflow are at the forefront of these changes. Understanding how to effectively utilize these tools can significantly enhance data processing and AI model deployment. This episode features Julian LaNeve, CTO at Astronomer, and David Xue, Machine Learning Engineer at Astronomer. They delve into the intricacies of data orchestration, generative AI and the practical applications of these technologies in modern data workflows. Key Takeaways: (01:51) The pressure to engage in the generative AI space. (02:02) Generative AI can elevate data utilization to the next level. (02:43) The transparency issues with commercial AI models. (04:27) High-quality data in model performance is crucial. (06:40) Running new models on smaller devices, like phones. (12:19) Fine-tuning LLMs to handle millions of task failures. (16:54) Teaching AI to understand specific logs, not general passages, is a goal. (21:56) Using Airflow as a general-purpose orchestration tool. (22:00) Airflow is adaptable for various use cases, including ETL and ML systems. Resources Mentioned: Julian LaNeve - https://www.linkedin.com/in/julianlaneve/ Atronomer - https://www.linkedin.com/company/astronomer/ David Xue - https://www.linkedin.com/in/david-xue-uva/ Apache Airflow - https://airflow.apache.org/ Meta’s Open Source Llama 3 model: https://ai.meta.com/blog/meta-llama-3/https://ai.meta.com/blog/meta-llama-3/ Microsoft’s Phi-3 model: https://www.microsoft.com/en-us/research/publication/phi-3-technical-report-a-highly-capable-language-model-locally-on-your-phone/ GPT-4 - https://www.openai.com/research/gpt-4 Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
For the first episode of the Airflow Podcast, we met up with Maxime Beauchemin, creator of Airflow, to explore the motivations behind its creation and the problems it was designed to solve. We asked Maxime for his definition of Airflow, the design principles behind hook/operator use, and his vision for the project. Speaker list: Pete DeJoy - Product at Astronomer Viraj Parekh - Data Engineer at Astronomer Maxime Beauchemin - Software Engineer at Lyft, creator of Airflow Talk mentioned at the end of the podcast- Advanced Data Engineering Patterns with Apache Airflow: http://www.ustream.tv/recorded/109227704 Maxime's Blog: https://medium.com/@maximebeauchemin…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
A sneak peek at our upcoming podcast about Apache Airflow. Featured in this clip (in order of appearance): Pete DeJoy - Product Specialist at Astronomer Patrick Atwater - Water Data Projects Manager at ARGO Labs Maksime Pecherskiy - Chief Data Officer of the City of San Diego Bolke de Bruin - Head of Advanced Analytics at ING…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 The Power of Airflow in Modern Data Environments at Wynn Las Vegas with Siva Krishna Yetukuri 24:31
24:31 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי24:31
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיUnderstanding the critical role of data integration and management is essential for driving business success, particularly in a dynamic environment like a luxury casino resort. In this episode, we sit down with Siva Krishna Yetukuri, Cloud Data Architect at Wynn Las Vegas, to explore how Airflow and other tools are transforming data workflows and customer experiences at Wynn Las Vegas. Key Takeaways: (02:00) Siva designs and builds cutting-edge data pipelines and architectures. (02:54) Wynn is building a data platform to drive surveys and marketing strategies. (05:00) Airflow is the backbone of data ingestion, curation and integration. (07:00) Custom operators in Airflow enhance monitoring and reporting. (09:00) Excitement surrounds the use of Airflow 2.9 and its new features. (08:32) A metadata database drives Airflow workflows and captures metrics. (12:31) Understanding Airflow fundamentals in layman’s terms simplifies complexity. (16:33) Transitioning from Control-M to Airflow eases building complex workflows. (24:06) ML models for volume and freshness anomalies improve data quality. (20:15) DAGs are often auto-generated, simplifying the process for engineers. Resources Mentioned: Apache Airflow - https://airflow.apache.org/ Snowflake - https://www.snowflake.com/ Databricks - https://databricks.com/ Great Expectations - https://greatexpectations.io/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Powering the Texas Rangers World Series Win With AI on Airflow with Alexander Booth 23:38
23:38 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי23:38
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThe integration of data and AI in sports is transforming how teams strategize and perform. Understanding how to harness this technology is key to staying competitive in the rapidly evolving landscape of baseball. In this episode, we sit down with Alexander Booth, Assistant Director of Research and Development at Texas Rangers Baseball Club, to explore the intersection of big data, AI and baseball strategy. Key Takeaways: (03:00) Alexander Booth's role and responsibilities at the Texas Rangers. (03:33) The implementation of multiple cameras and pose tracking in stadiums. (06:16) The importance of Airflow in organizing data orchestrations. (06:22) The demand for faster data among modern baseball players. (11:01) The necessity of scalable solutions for handling large data sets. (15:00) How weather data influences game strategy. (15:46) The impact of advanced technology on decision-making in baseball. (18:00) The role of AI and machine learning in player and game analysis. (22:26) The use of dynamic tasks in Airflow for better data management. Resources Mentioned: Apache Airflow - https://airflow.apache.org/ Statcast - https://www.mlb.com/statcast Google BigQuery - https://cloud.google.com/bigquery/ Databricks - https://databricks.com/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #ai #automation #airflow #machinelearning…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Expanding the Data Engineering Toolkit at Reddit 45:48
45:48 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי45:48
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיWelcome back to the Airflow Podcast. This week, we met up with Ben Wisegarver, a staff data scientist at Reddit who runs their data warehousing and data engineering functions. Reddit users generate petabytes of data every day that needs to be processed, stored, and analyzed by a wide breadth of backend services. Our conversation with Ben touches on everything from Airflow as a tool for career mobility across the data stack to scaling out a self-service data architecture across many teams. For folks interested, our team at Astronomer is growing rapidly and we're on the hunt for new folks to join in a variety of different roles. If you're passionate about Airflow and interested in building the future of data engineering, please get in touch. You can check our current job postings at careers.astronomer.io, but we're constantly updating our listings to accommodate new hiring needs. Please feel free to email me directly at pete@astronomer.io if you're passionate about what we're doing and think you'd be a good addition to the team. Mentioned Resources: Careers: https://careers.astronomer.io Guest Profile: Ben Wisegarver: https://www.linkedin.com/in/ben-wisegarver-54566576…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 GDPR, Self-Service Data, and Infrastructure Automation with Typeform 31:26
31:26 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי31:26
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיWelcome back to the Airflow Podcast. This week, we met up with Albert Franzi and Carlos Escura from Typeform. Typeform is a tool that allows you to build beautiful interactive forms that you can use for a wide variety of use cases, including customer surveys, employee engagement, product feedback, and market research to name a few. In our conversation, we discussed Airflow as a tool for GDPR compliance, the concept of self-service data and how it allows your data operations team to function as a data platform team, and some of the more specialized infrastructure tooling that the Typeform team has built out to support their internal teams. For folks interested, our team at Astronomer is growing rapidly and we're on the hunt for new folks to join in a variety of different roles. If you're passionate about Airflow and interested in building the future of data engineering, please get in touch. You can check our current job postings at careers.astronomer.io, but we're constantly updating our listings to accommodate new hiring needs. Please feel free to email me directly at pete@astronomer.io if you're passionate about what we're doing and think you'd be a good addition to the team. Mentioned Resources: Dag Factory: https://github.com/ajbosco/dag-factory Astronomer Careers: https://careers.astronomer.io Guest Profiles: Albert Franzi: https://www.linkedin.com/in/albertfranzi/?originalSubdomain=es Carlos Escura: https://www.linkedin.com/in/carlosescura/en-us/…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
After a bit of a break, we're back with the third official episode bundle of The Airflow Podcast. In this batch, we'll get a little bit deeper with current Airflow users and maintainers on core fundamental concepts in data engineering, architectures for operating modern data platforms at scale, and the process of maintaining and operating Airflow, specifically as we go through the release process of Airflow 2.0. This week, we met up with Brian de la Motte and Florian Hines at Netlify. Netlify provides an extremely popular toolset for building and deploying JAMstack sites. They provide hosting services, CI, DNS, authentication, and managed backend tools that help users run and operate static sites at scale. The team over there recently adopted Airflow to help decouple orchestration logic from a complex collection Spark jobs and are currently in the process of expanding their Airflow footprint to accommodate a broader group of interesting use-cases. Disclaimer: we get a bit of a surprise about halfway through the episode when Brian tells us that they had recently signed up for Astronomer- we promise that it wasn't a planted ad :). Please contact pete@astronomer.io if you'd like to get in touch regarding future episodes. Hope you enjoy! Guest Profiles: Brian de la Motte: https://www.linkedin.com/in/brian-de-la-motte/ Florian Hines: https://www.linkedin.com/in/florianhines/…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
This week, we linked up with Airflow release manager, core committer, and Astronomer platform engineer Ash Berlin-Taylor to discuss the Airflow 2.0 roadmap [1]. There is some great stuff in the works around performance, autoscaling, and usability that we're excited about. In this episode, Ash lends his thoughts on the design, implementation, and value-add around all of the upcoming features, including: - The Knative Executor - A modern and real-time UI - A production-grade API - Improved scheduler and webserver performance - An official production Docker image for Airflow We hope you enjoy! Please email pete@astronomer.io if you have thoughts on topics you'd like to see covered in future episodes. Separately, some good folks from the Airflow community are running a user survey that will help collect some useful information around the Airflow UX. If you have five minutes to spare, filling out the following form will help the core Airflow committers to shape the project roadmap: https://forms.gle/XAzR1pQBZiftvPQM7 [1] https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
This week, we had the pleasure of meeting up with Jarek Potiuk, Principal Software Engineer at Polidea and Apache Airflow committer, to discuss his most recent contribution to the community, Airflow Breeze. Jarek deeply values developer productivity and realized while building a team of Airflow committers that, in order to open a PR on the project, passing unit tests and waiting for the CI build was a cumbersome process that could take up to a few hours. Breeze seeks to improve that experience for Airflow committers and lower the barrier-to-entry of contribution for folks that are new to the open-source community. You can read more about Airflow Breeze here: https://www.polidea.com/blog/its-a-breeze-to-develop-apache-airflow/#the-apache-airflow-projects-setup…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Open Source and Airflow at Google 39:03
39:03 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי39:03
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThis episode kicks off season 2 of The Airflow Podcast. In this next season, we'll focus on the future of Airflow and chat with leading members of the community to paint a picture of what's to come. We're pumped to be diving back into this project and look forward to the great conversations we have lined up. This week, we chatted with James Malone, Product Manager of Google's Cloud Composer. James had some interesting things to say about open source at Google and where his team plans on contributing most to the project going forward. As always, thanks for listening and please email pete@astronomer.io if you have any feedback or would like to be considered as a guest.…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
This week, we met up with Ash Berlin-Taylor to discuss the recent 1.10 release, what it's like to be a release manager for an open source project, Airflow's bid to graduate from incubating status, and the next phase of Airflow project development. As mentioned in our podcast intro, we at Astronomer are hiring Data Engineers who are passionate about contributing to open source and making Airflow great. Please shoot us an email at humans@astronomer.io if you're interested in hearing more about the fully-remote opportunity. Check us out at www.astronomer.io if you're interested in seeing a demo of our platform.…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
1 Role-Based Access Control (RBAC) 37:28
37:28 הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתי37:28
הפעל מאוחר יותר הפעל מאוחר יותר רשימות לייק אהבתיThis time, we met up with WePay's Joy Gao to talk through her work on the RBAC components in the recent Airflow 1.10 release. We dove deep into what inspired her work and took some time to discuss what it's like to be a woman contributing to a predominately male open-source community. Hope you enjoy! If you'd like to get started using Airflow in your org, check out our recently-launched Spacecamp program here: https://www.astronomer.io/spacecamp Feel free to email me at pete@astronomer.io with any feedback or if you'd like to be considered as a guest!…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
In this episode, we dove into the relationship between Airflow and Kuberenetes and interviewed Daniel Imberman, Senior Software Engineer at Bloomberg (1:30), and Greg Neiheisel, CTO here at Astronomer (37:31). Daniel has done most of the work on the Kubernetes executor for Airflow and Greg plans to take on a chunk of the development going forward, so it was really interesting to hear both of their perspectives on the project. Enjoy!…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
This week, we’ll examine conversations with both old guests and new to paint a comprehensive picture of Airflow’s pain points. While we still undoubtedly believe that Airflow is the future of ETL, it’s important to acknowledge that any incubating project will have issues, and bringing those issues to the forefront of the community’s attention will help shape the future of the project. We’ll talk with Thomas La Piana (1:36), Data Engineer at OrderMyGear, Frank Hsu (14:20), Data Engineer at mines.io, and Alan Cruickshank (27:41), business insights and data manager at tails.com. Check out our open-source library of Airflow plugins at github.com/airflow-plugins, and feel free to contribute anything that you've been working on! If you're interested in being on the podcast or have any feedback on how you think we could make it better, shoot me an email at pete@astronomer.io…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
On this episode, we linked up with Erik Bernhardsson (@erikbern), creator of Luigi and CTO of Better Mortgage. We chatted about everything from the motivations behind Luigi's creation and his current thoughts on Airflow- we hope you enjoy! Check out: - Erik's blog at erikbern.com - Our open-source library of Airflow plugins at github.com/airflow-plugins All podcast feedback is hugely appreciated- feel free to email me at pete@astronomer.io if you have any thoughts.…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
In this episode, we dive into Airflow Best Practices and include longer portions of interviews with Alan Cruickshank (1:30), Business Insights and Data Manager at Tails.com, Chris Riccomini (7:27), Principal Software Engineer at WePay, and Bolke de Bruin(31:45), Head of Advanced Analytics Technology at ING. Hope you enjoy! We're still working to get better at podcasting, so please send over any feedback to pete@astronomer.io. We really appreciate hearing what the community has to say, and your feedback is hugely helpful in making us better. If you're interested in Astronomer Spacecamp, a guided Airflow development course, you can find more info on that here: https://www.astronomer.io/blog/announcing-astronomer-spacecamp/ We also launched our Managed Airflow on Product Hunt last week- you can check that out here: https://www.producthunt.com/posts/apache-airflow-on-astronomer Thanks so much for listening!…
T
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Episode 2 of The Airflow Podcast is here to discuss six specific use cases that we’ve seen for Apache Airflow. Here’s the lineup: Patrick Atwater (@patwater), Water Data Projects Manager at ARGO Labs: 2:03-5:35 Maksime Pecherskiy (@mrmaksimize), CDO of San Diego: 5:35-23:06 Scott Halgrim (@shalgrim), Data Engineer at Zapier: 23:06-27:27 Bolke de Bruin (@bolke2028), Head of Advanced Analytics at ING: 27:27-39:46 Chris Riccomini (@criccomini), Principal Software Engineer at WePay: 39:46-54:20 Ben Gregory (@benbeingbin), Data Engineer (and noted craft soda enthusiast) at Astronomer: 54:20-1:14:38 Contribute to our open-source library of Airflow plugins at github.com/airflow-plugins Contact us at www.astronomer.io if you’re interested in Spacecamp: A guided development program to get your team up and running on Airflow.…
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.
Daily Deals
Daily Deals
דומה לThe Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
The award-winning WIRED UK Podcast with James Temperton and the rest of the team. Listen every week for the an informed and entertaining rundown of latest technology, science, business and culture news. New episodes every Friday.
…
continue reading
Thanks for visiting The Cell Phone Junkie! I will be taking the time each week to discuss my favorite topic, cell phones. Any feedback is appreciated and welcome. You can email me at: questions (AT) thecellphonejunkie (DOT) com or call: 206-203-3734 Thanks and welcome!
…
continue reading
T
The Ticket: Discover the Future of Customer Service, Support, and Experience, with Intercom
1
The Ticket: Discover the Future of Customer Service, Support, and Experience, with Intercom
Intercom
1k480
On The Ticket, you'll hear Intercom's Customer Support team in conversation with the customer service leaders, renowned customer experience thinkers, and influential authors who are shaping the field of customer support. Follow The Ticket to get the weekly episodes and sign up for our twice-monthly newsletter bursting with all the insights, trends, tips, and assets your team needs to embrace the future of customer service. https://www.intercom.com/blog/newsletter 🏠 www.intercom.com
…
continue reading
BBC Radio 5 live’s award winning gaming podcast, discussing the world of video games and games culture.
…
continue reading
T
The Talk Show With John Gruber
1
The Talk Show With John Gruber
Daring Fireball / John Gruber
2k353
The director’s commentary track for Daring Fireball. Long digressions on Apple, technology, design, movies, and more.
…
continue reading
Download This Show is your weekly guide to the world of media, culture, and technology. From social media to gadgets, streaming services to privacy issues. Each week Rae Johnston and guests take a fun, deep dive into how technology is reshaping our lives.
…
continue reading
Hosts Ben Brock Johnson and Amory Sivertson dig into the internet's vast and curious ecosystem of online communities to find untold histories, unsolved mysteries, and other jaw-dropping stories online and IRL.
…
continue reading
Media and tech aren’t just intersecting — they’re fully intertwined. And to understand how those worlds work, and what they mean for you, veteran journalist Peter Kafka talks to industry leaders, upstarts and observers - and gets them to spell it out in plain, BS-free English. Part of the Vox Media Podcast Network.
…
continue reading
The 21st Century began with the rise of the Internet and social media. The next decade will mark the rise of the Intelligent Machines. AI will inhabit all our devices from cars and appliances to smart phones and robots. The Intelligent Machines podcast explores the most exciting revolution humanity has ever seen, filled with promise and peril. More than ever we need to understand what these new devices will bring to our lives and how to make best use of them as the 21st century unfolds. On t ...
…
continue reading
WSJ’s Bold Names brings you conversations with the leaders of the bold-named companies featured in the pages of The Wall Street Journal. Hosts Tim Higgins and Christopher Mims speak to CEOs and business leaders in interviews that challenge conventional wisdom and take you inside the decisions being made in the C-suite and beyond.
…
continue reading
Player FM - אפליקציית פודקאסט
התחל במצב לא מקוון עם האפליקציה Player FM !
התחל במצב לא מקוון עם האפליקציה Player FM !
))
