How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Player FM - Internet Radio Done Right

הוסף לפני four שנים

תוכן מסופק על ידי The Data Flowcast. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The Data Flowcast או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.

The Innovators & Investors Podcast

1
Transforming Due Diligence with AI: Wokelo’s Automation of Private Market Research and Risk Analysis 44:11

לפני 19 ימים44:11

הפעל מאוחר יותר

רשימות

לייק

אהבתי

44:11

In this episode of The Innovators & Investors Podcast, host Kristian Marquez sits down with Sid Masson, co-founder and CEO of Wokelo AI, to explore how their platform is transforming due diligence and research workflows in private equity, venture capital, consulting, and banking. Sid shares the origins of Wokelo AI, born from firsthand experience with the tedious and error-prone aspects of manual data analysis, and how their agent-driven platform automates complex data synthesis—turning scattered public and proprietary datasets into tailored, actionable insights and well-crafted deliverables. The conversation delves into the nuanced approach Wokelo takes toward risk assessment by integrating unconventional signals from social media, user reviews, and expert interviews, aiming to provide a comprehensive and balanced view beyond traditional metrics. Sid also discusses the evolution of their product from a single-feature diligence tool to a versatile, no-code platform empowering users to build customized AI agents that fit unique workflows. The episode highlights key lessons learned in product design, emphasizing founder-led UX development early on, and the strategic decisions around hybrid remote work culture to attract and retain top talent globally. Sid offers candid reflections on fundraising challenges during a tough market, the importance of investor alignment beyond capital, and how they allocate resources prioritizing rapid product innovation ahead of scaling sales and marketing. Listeners gain valuable perspectives on navigating startup growth, maintaining culture, and leveraging AI to augment—not replace—human expertise across the investment lifecycle. Sid closes with personal insights on adaptability, discipline, and continuous learning that have shaped his entrepreneurial journey, making this episode a rich resource for founders, investors, and professionals interested in the intersection of AI and private market research. For more information, visit wokelo.ai or contact Sid directly at sid@wokelo.ai. Learn more about Wokelo AI work at: https://www.wokelo.ai/ Connect with Sid Masson on LinkedIn at: https://www.linkedin.com/in/siddhantmasson/ Think you'd be a great guest on the show? Apply at https://finstratmgmt.com/innovators-investors-podcast/ Want to learn more about Kristian Marquez's work? Check out his website at https://finstratmgmt.com…

לפני שנה 23:51

MP3•בית הפרקים

Discover the cutting-edge methods Vibrant Planet uses to revolutionize geospatial data processing and resource management. In this episode, we delve into the intricacies of scaling geospatial data processing and resource allocation with experts from Vibrant Planet. Joining us are Cyrus Dukart, Engineering Lead, and David Sacerdote, Staff Software Engineer, who share their innovative approaches to handling large datasets and optimizing resource use in Airflow. Key Takeaways: (00:00) Inefficiencies in resource allocation. (03:00) Scientific validity of sharded results. (05:53) Tech-based solutions for resource management. (06:11) Retry callback process for resource allocation. (08:00) Running database queries for resource needs. (10:05) Importance of remembering resource usage. (13:51) Generating resource predictions. (14:44) Custom task decorator for resource management. (20:28) Massive resource usage gap in sharded data. (21:14) Fail-fast model for long-running tasks. Resources Mentioned: Cyrus Dukart - https://www.linkedin.com/in/cyrus-dukart-6561482/ David Sacerdote - https://www.linkedin.com/in/davidsacerdote/ Vibrant Planet - https://www.linkedin.com/company/vibrant-planet/ Apache Airflow - https://airflow.apache.org/ Kubernetes - https://kubernetes.io/ Vibrant Planet - https://vibrantplanet.net/ Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning

73 פרקים

#Tech #Airflow

How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

published לפני שנה

שתפו

MP3•בית הפרקים

73 פרקים

#Tech #Airflow

כל הפרקים

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Scaling Airflow for Enterprise Data Platforms at PepsiCo with Kunal Bhattacharya 19:04

לפני 6 ימים19:04

19:04

PepsiCo’s data platform drives insights across finance, marketing and data science. Delivering stability, scalability and developer delight is central to its success, and engineering leadership plays a key role in making this possible. In this episode, Kunal Bhattacharya , Senior Manager of Data Platform Engineering at PepsiCo , shares how his team manages Airflow at scale while ensuring security, performance and cost efficiency. Key Takeaways: 00:00 Introduction. 02:31 Enabling developer delight by extending platform capabilities. 03:56 Role of Snowflake, dbt and Airflow in PepsiCo’s data stack. 06:10 Local developer environments built using official Airflow Helm charts. 07:13 Pre-staging and PR environments as testing playgrounds. 08:08 Automating labeling and resource allocation via DAG factories. 12:16 Cost optimization through pod labeling and Datadog insights. 14:01 Isolating dbt engines to improve performance across teams. 16:12 Wishlist for Airflow 3: Improved role-based grants and database modeling. Resources Mentioned: Kunal Bhattacharya https://www.linkedin.com/in/kunaljubce/ PepsiCo | LinkedIn https://www.linkedin.com/company/pepsico/ PepsiCo | Website https://www.pepsico.com Apache Airflow https://airflow.apache.org/ Snowflake https://www.snowflake.com dbt https://www.getdbt.com Kubernetes https://kubernetes.io Great Expectations https://greatexpectations.io Monte Carlo https://www.montecarlodata.com Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Building a Unified Data Platform at Pattern with William Graham 24:09

לפני 13 ימים24:09

24:09

The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines. In this episode, we are joined by William Graham , Senior Data Engineer at Pattern , who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management. Key Takeaways: 00:00 Introduction. 02:44 Structure of Pattern’s data teams across acquisition, engineering and platform. 04:27 How Airflow became the central scheduler for batch jobs. 08:57 Credential management challenges that led to decoupling scheduling and orchestration. 12:21 Heimdall simplifies multi-application access through a unified interface. 13:15 Standardized operators in Airflow using Heimdall integration. 17:13 Open-source contributions and early adoption of Heimdall within Pattern. 21:01 Community support for Airflow and satisfaction with scheduling flexibility. Resources Mentioned: William Graham https://www.linkedin.com/in/willgraham2/ Pattern | LinkedIn https://www.linkedin.com/company/pattern-hq/ Pattern | Website https://pattern.com Apache Airflow https://airflow.apache.org Heimdall on GitHub https://github.com/patterninc/heimdall Netflix Genie https://netflix.github.io/genie/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
How Astronomer Turns Proactive Monitoring Into Customer Success with Collin McNulty 25:34

לפני 20 ימים25:34

25:34

The evolution of Airflow continues to shape data orchestration and monitoring strategies. Leveraging it beyond traditional ETL use cases opens powerful new possibilities for proactive support and internal operations. In this episode, we are joined by Collin McNulty , Sr. Director of Global Support at Astronomer , who shares insights from his journey into data engineering and the lessons learned from leading Astronomer’s Customer Reliability Engineering (CRE) team. Key Takeaways: 00:00 Introduction. 03:07 Lessons learned in adapting to major platform transitions. 05:18 How proactive monitoring improves reliability and customer experience. 08:10 Using automation to enhance internal support processes. 12:09 Why keeping systems current helps avoid unnecessary issues. 15:14 Approaches that strengthen system reliability and efficiency. 18:46 Best practices for simplifying complex orchestration dependencies. 23:24 Anticipated innovations that expand orchestration capabilities. Resources Mentioned: Collin McNulty https://www.linkedin.com/in/collin-mcnulty/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ Prometheus https://prometheus.io/ Splunk https://www.splunk.com/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Overcoming Data Engineering Challenges at Daiichi Sankyo Europe GmbH with Evgenii Prusov 19:26

לפני 27 ימים19:26

19:26

The shift to a unified data platform is reshaping how pharmaceutical companies manage and orchestrate data. Establishing standards across regions and teams ensures scalability and efficiency in handling large-scale analytics. In this episode, Evgenii Prusov , Senior Data Platform Engineer of Daiichi Sankyo Europe GmbH , joins us to discuss building and scaling a centralized data platform with Airflow and Astronomer. Key Takeaways: 00:00 Introduction. 02:49 Building a centralized data platform for 15 European countries. 05:19 Adopting SaaS to manage Airflow from day one. 07:01 Leveraging Airflow for data orchestration across products. 08:16 Teaching non-Python users how to work with Airflow is challenging. 12:25 Creating a global data community across Europe, the US and Japan. 14:04 Monthly calls help share knowledge and align regional teams. 15:47 Contributing to the open-source Airflow project as a way to deepen expertise. 16:32 Desire for more guidelines, debugging tutorials and testing best practices in Airflow. Resources Mentioned: Evgenii Prusov https://www.linkedin.com/in/prusov/ Daiichi Sankyo Europe GmbH | LinkedIn https://www.linkedin.com/company/daiichi-sankyo-europe-gmbh/ Daiichi Sankyo Europe GmbH | Website https://www.daiichi-sankyo.eu Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Snowflake https://www.snowflake.com/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Building a Data-Driven Beauty and Wellness Marketplace at StyleSeat with Paschal Onuorah 23:05

לפני 5 weeks23:05

23:05

StyleSeat is revolutionizing how beauty and wellness professionals grow their businesses through data-driven tools. From streamlining scheduling to optimizing marketing, their platform empowers professionals to focus on their craft while expanding their client base. In this episode, Paschal Onuorah , Senior Data Engineer at StyleSeat , shares how the company leverages Airflow, dbt, and Cosmos to drive marketplace intelligence, improve client connections and deliver measurable growth for professionals. Key Takeaways: 00:00 Introduction. 05:44 The role of the data engineering team in driving business success. 08:52 Leveraging technology for real-time business intelligence. 10:52 Data-driven strategies for improving marketing outcomes. 13:05 How adopting the right tools can increase revenue growth. 14:25 Advantages of simplifying and integrating technical workflows. 18:45 Benefits of multi-environment configurations for development and production. 20:17 Foundational skills and best practices for learning Airflow effectively. 22:33 Opportunities for deeper tool integration and improved data visualization. Resources Mentioned: Paschal Onuorah https://www.linkedin.com/in/onuorah-paschal/ StyleSeat | LinkedIn https://www.linkedin.com/company/styleseat/ StyleSeat | Website https://www.styleseat.com Apache Airflow https://airflow.apache.org/ dbt https://www.getdbt.com/ Astronomer Cosmos https://www.astronomer.io/cosmos/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Building the Future of Airflow Execution at Astronomer with Ian Buss and Piotr Chomiak 22:25

לפני 6 weeks22:25

22:25

The evolution of orchestration in Airflow continues with innovations that address both scalability and security. From improving executor reliability to enabling remote execution, these advancements reshape how organizations manage data pipelines. In this episode, we’re joined by Ian Buss , Principal Software Engineer at Astronomer, and Piotr Chomiak , Principal Product Manager at Astronomer , who share insights into the Astro Executor and remote execution. Key Takeaways: 00:00 Introduction. 04:13 How product leadership drives scalability for enterprise needs. 08:23 Architectural changes that improve reliability and remove bottlenecks. 10:15 Metrics that enhance visibility into system performance. 12:54 The role of remote execution in addressing security requirements. 15:56 Differences between open-source solutions and managed offerings. 19:04 Broad industry adoption and applicability of remote execution. 20:39 Future advancements in language support and multi-tenancy. Resources Mentioned: Ian Buss https://www.linkedin.com/in/ian-buss/ Piotr Chomiak https://www.linkedin.com/in/piotr-chomiak-b1955624/ Astronomer | Website https://www.astronomer.io Apache Airflow https://airflow.apache.org/ Airflow Slack Community https://airflow.apache.org/community/ Beyond Analytics conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Scaling On-Prem Airflow With 2,000 DAGs at Numberly with Sébastien Crocquevieille 24:17

לפני 7 weeks24:17

24:17

Scaling 2,000+ data pipelines isn’t easy. But with the right tools and a self-hosted mindset, it becomes achievable. In this episode, Sébastien Crocquevieille , Data Engineer at Numberly , unpacks how the team scaled their on-prem Airflow setup using open-source tooling and Kubernetes. We explore orchestration strategies, UI-driven stakeholder access and Airflow’s evolving features. Key Takeaways: 00:00 Introduction. 02:13 Overview of the company’s operations and global presence. 04:00 The tech stack and structure of the data engineering team. 04:24 Running nearly 2,000 DAGs in production using Airflow. 05:42 How Airflow’s UI empowers stakeholders to self-serve and troubleshoot. 07:05 Details on the Kubernetes-based Airflow setup using Helm charts. 09:31 Transition from GitSync to NFS for DAG syncing due to performance issues. 14:11 Making every team member Airflow-literate through local installation. 17:56 Using custom libraries and plugins to extend Airflow functionality. Resources Mentioned: Sébastien Crocquevieille https://www.linkedin.com/in/scroc/ Numberly | LinkedIn https://www.linkedin.com/company/numberly/ Numberly | Website https://numberly.com/ Apache Airflow https://airflow.apache.org/ Grafana https://grafana.com/ Apache Kafka https://kafka.apache.org/ Helm Chart for Apache Airflow https://airflow.apache.org/docs/helm-chart/stable/index.html Kubernetes https://kubernetes.io/ GitLab https://about.gitlab.com/ KubernetesPodOperator – Airflow https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
How Moniepoint Group Uses Airflow for Exposure Monitoring with Adeolu Adegboye 21:32

לפני 8 weeks21:32

21:32

Managing financial data at scale requires precise orchestration and proactive monitoring to maintain operational efficiency. In this episode, we are joined by Adeolu Adegboye , Data Engineer at Moniepoint Group , who shares how his team uses data pipelines and workflow automation to manage high volumes of transactions, ensure timely alerts and support diverse stakeholders across the business. Key Takeaways: (00:00) Introduction. (02:48) The role of data engineering in supporting all business operations. (04:17) Leveraging workflow orchestration to manage daily processes. (05:20) Proactively monitoring for anomalies to prevent potential issues. (08:12) Simplifying complex insights for non-technical teams. (13:01) Improving efficiency through dynamic and parallel workflows. (14:19) Optimizing system performance to handle large-scale operations. (17:19) Exploring creative and innovative uses for workflow automation. Resources Mentioned: Adeolu Adegboye https://www.linkedin.com/in/adeolu-adegboye/ Moniepoint Group | LinkedIn https://www.linkedin.com/company/moniepoint-inc/ Moniepoint Group | Website https://www.moniepoint.com Apache Airflow https://airflow.apache.org/ ClickHouse https://clickhouse.com/ Grafana https://grafana.com/ Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Inside Bosch’s Airflow 3 Revolution: Remote Execution with Jens Scheffler 28:02

לפני 9 weeks28:02

28:02

The evolution of Airflow has reached a milestone with the introduction of remote execution in Airflow 3, enabling flexible orchestration across distributed environments. In this episode, Jens Scheffler , Test Execution Cluster Technical Architect at Bosch , shares insights on how his team’s need for large-scale, cross-environment testing influenced the development of the Edge Executor and shaped this major release. Key Takeaways: (02:39) The role of remote execution in supporting large-scale testing needs. (04:44) How community support contributed to the Edge Executor’s development. (08:41) Navigating network and infrastructure limitations within secure environments. (13:25) Transitioning from database-heavy processes to an API-driven model. (14:16) How the new task SDK in Airflow 3 improves distributed task execution. (16:54) What is required to set up and configure the Edge Executor. (19:36) Managing multiple queues to optimize tasks across different environments. (23:30) Examples of extreme distance use cases for edge execution. Resources Mentioned: Jens Scheffler https://www.linkedin.com/in/jens-scheffler/ Bosch | LinkedIn https://www.linkedin.com/company/bosch/ Bosch | Website https://www.bosch.com/ Apache Airflow https://airflow.apache.org/ Edge Executor (Edge3 Provider Package) https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/index.html Astronomer’s Astro Executor https://www.astronomer.io/docs/astro/astro-executor/ Beyond Analytics Conference https://astronomer.io/beyond/dataflowcast Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Inside Modern Data Infrastructure at Massdriver with Cory O’Daniel and Jake Ferriero 31:24

לפני 10 weeks31:24

31:24

Managing modern data platforms means navigating a web of complex infrastructure, competing team needs and evolving security standards. For data teams to truly thrive, infrastructure must become both accessible and compliant without sacrificing velocity or reliability. In this episode, we’re joined by Cory O’Daniel , CEO and Co-Founder at Massdriver , and Jacob Ferriero , Senior Software Engineer at Astronomer , to unpack what it takes to make data platform engineering scalable, sustainable and secure. They share lessons from years of experience working with DevOps, ML teams and platform engineers and discuss how Airflow fits into the orchestration layer of today’s data stacks. Key Takeaways: (03:27) Making infrastructure accessible without deep ops knowledge. (07:23) Distinct personas and responsibilities across data teams. (09:53) Infrastructure hurdles specific to ML workloads. (11:13) Compliance and governance shaping platform design. (13:27) Tooling mismatches between teams cause friction. (15:13) Airflow’s orchestration role within broader system architecture. (22:10) Creating reusable infrastructure patterns for consistency. (24:13) Enabling secure access without slowing down development. (26:55) Opportunities to improve Airflow with event-driven and reliability tooling. Resources Mentioned: Cory O’Daniel https://www.linkedin.com/in/coryodaniel/ Massdriver | LinkedIn https://www.linkedin.com/company/massdriver/ Massdriver | Website https://www.massdriver.cloud/ Jacob Ferriero https://www.linkedin.com/in/jacob-ferriero/ Astronomer https://www.linkedin.com/company/astronomer/ Apache Airflow https://airflow.apache.org/ Prequel https://www.prequel.co/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
The Future of Airflow Telemetry with Bolke de Bruin 21:55

לפני 12 weeks21:55

21:55

Telemetry has the potential to guide the future of Airflow, but only if it’s implemented transparently and with community trust. In this episode, we’re joined by Bolke de Bruin , Director at Metyis and a long-time Airflow PMC member. Bolke discusses how telemetry has been handled in the past, why it matters now and what it will take to get it right. Key Takeaways: (03:20) The role of foundations in establishing credibility and sustainability. (04:52) Why data collection is critical to open-source project direction. (07:24) Lessons learned from previous approaches to user data collection. (10:23) The current state of telemetry in the project. (10:53) Community trust as a prerequisite for technical implementation. (12:54) The importance of managing sensitive data within trusted ecosystems. (16:37) Ethical considerations in balancing participation and access. (18:45) Forward-looking ideas for improving workflow design and usability. Resources Mentioned: Bolke de Bruin https://www.linkedin.com/in/bolke/ Metyis | LinkedIn https://www.linkedin.com/company/metyis/ Metyis | Website http://www.metyis.com Apache Airflow https://airflow.apache.org/ Airflow Summit https://airflowsummit.org/ Airflow Dev List https://lists.apache.org/list.html?dev@airflow.apache.org https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28

לפני 13 weeks22:28

22:28

Contributing to open-source projects can be daunting, but it can also unlock unexpected innovation. This episode showcases how one engineer’s journey with Apache Airflow led to impactful UI enhancements and infrastructure solutions at scale. Shubham Raj , Software Engineer II at Cloudera , shares how his team built a drag-and-drop DAG editor for non-coders, contributions which helped shape the Airflow 3.0 Ul and introduced features like external XCom control and bulk APls. Key Takeaways: (02:30) Day-to-day responsibilities building platforms that simplify orchestration. (05:27) Factors that make onboarding into large open-source projects accessible. (07:35) The value of improved user interfaces for task state visibility and control. (09:49) Enabling faster debugging by exposing internal data through APIs. (13:00) Balancing frontend design goals with backend functionality. (14:19) Creating workflow editors that lower the barrier to entry. (16:54) Supporting a variety of task types within a visual DAG builder. (19:32) Common infrastructure challenges faced by orchestration users. (20:37) Addressing dependency management across distributed environments. Resources Mentioned: Shubham Raj https://www.linkedin.com/in/shubhamrajofficial/ Cloudera | LinkedIn https://www.linkedin.com/company/cloudera/ Cloudera | Website https://www.cloudera.com/ Apache Airflow https://airflow.apache.org/ 2023 Airflow Summit https://airflowsummit.org/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34

לפני 13 weeks19:34

19:34

Managing data pipelines at scale is not just a technical challenge. It is also an organizational one. At Lyft, success means empowering dozens of teams to build with autonomy while enforcing governance and best practices across thousands of workflows. In this episode, we speak with Yunhao Qing , Software Engineer at Lyft , about building a governed data-engineering platform powered by Airflow that balances flexibility, standardization and scale. Key Takeaways: (03:17) Supporting internal teams with a centralized orchestration platform. (04:54) Migrating to a managed service to reduce infrastructure overhead. (06:04) Embedding platform-level governance into custom components. (08:02) Consolidating and regulating the creation of custom code. (09:48) Identifying and correcting inefficient workflow patterns. (11:17) Replacing manual workarounds with native platform features. (14:32) Preparing teams for major version upgrades. (16:03) Leveraging asset-based scheduling for smarter triggers. (18:13) Envisioning GenAI and semantic search for future productivity. Resources Mentioned: Yunhao Qing https://www.linkedin.com/in/yunhao-qing Lyft | LinkedIn https://www.linkedin.com/company/lyft/ Lyft | Website https://www.lyft.com/ Apache Airflow https://airflow.apache.org/ Astronomer https://www.astronomer.io/ Kubernetes https://kubernetes.io/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

לפני 15 weeks22:19

22:19

Understanding the complexities of Apache Airflow can be daunting for newcomers and seasoned data engineers. But with the right guidance, mastering the tool becomes an achievable milestone. In this episode, Marc Lamberti , Head of Customer Education at Astronomer , joins us to share his journey from Udemy instructor to driving education at Astronomer, and how he's helping over 100,000 learners demystify Airflow. Key Takeaways: (02:36) Early exposure to Airflow while addressing inefficiencies in data workflows. (04:10) Common barriers to implementing open source tools in enterprise settings. (06:18) The shift from part-time teaching to a full-time focus on Airflow education. (07:53) A modular, guided approach to structuring educational content. (09:57) The value of highlighting underused Airflow features for broader adoption. (12:35) Certifications as a method to assess readiness and uncover knowledge gaps. (13:25) Coverage of essential Airflow concepts in the Fundamentals exam. (16:07) The DAG Authoring exam’s emphasis on practical, advanced features. (20:08) A call for more visible integration of Airflow with AI workflows. Resources Mentioned: Marc Lamberti https://www.linkedin.com/in/marclamberti/ Astronomer | LinkedIn https://www.linkedin.com/company/astronomer/ Astronomer Academy https://academy.astronomer.io/ Airflow Fundamentals Certification https://www.astronomer.io/certification/ DAG Authoring Certification https://academy.astronomer.io/plan/astronomer-certification-dag-authoring-for-apache-airflow-exam The Complete Hands-On Introduction to Airflow https://www.udemy.com/course/the-complete-hands-on-course-to-master-apache-airflow/?utm_source=adwords&utm_medium=udemyads&utm_campaign=Search_DSA_Beta_Prof_la.EN_cc.ROW-English&campaigntype=Search&portfolio=ROW-English&language=EN&product=Course&test=&audience=DSA&topic=&priority=Beta&utm_content=deal4584&utm_term=_._ag_162511579404_._ad_696197165418_._kw__._de_c_._dm__._pl__._ti_dsa-1677053911088_._li_9061346_._pd__._&matchtype=&gad_source=1&gad_campaignid=21168154305&gbraid=0AAAAADROdO3MpljfP-gssiYSmDEPdhZV9&gclid=Cj0KCQjw097CBhDIARIsAJ3-nxdjZA6G5-Y0-akk6Huksy2PLb04t92J4iNfUSIbMdrSAla_tb-o2N8aArOeEALw_wcB&couponCode=PMNVD3025 https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

1
Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

לפני 16 weeks30:09

30:09

The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams. In this episode, we speak with Alberto Crespi , Data Architect at lastminute.com , who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach. Key Takeaways: (02:17) Defining responsibilities within data architecture teams. (04:15) Consolidating multiple orchestrators into a single solution. (07:00) Scaling Airflow environments with shared infrastructure and DevOps practices. (10:59) Managing dependencies and readiness using SQL sensors. (14:23) Enhancing visibility and response through Slack-integrated monitoring. (19:28) Extending Airflow’s flexibility to run legacy systems. (22:28) Integrating transformation tools into orchestrated pipelines. (25:54) Enabling non-engineers to contribute to pipeline development. (27:33) Fostering adoption through collaboration and communication. Resources Mentioned: Alberto Crespi https://www.linkedin.com/in/crespialberto/ lastminute.com | Website https://lastminute.com Apache Airflow https://airflow.apache.org/ dbt Labs https://www.getdbt.com/ Astronomer Cosmos https://github.com/astronomer/astronomer-cosmos GitLab Slack https://slack.com/ Kubernetes https://kubernetes.io/ Confluence https://www.atlassian.com/software/confluence Slack https://slack.com/ https://www.astronomer.io/events/roadshow/london/ https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/ https://www.astronomer.io/events/roadshow/san-francisco/ https://www.astronomer.io/events/roadshow/chicago/ Thanks for listening to “ The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI .” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations. #AI #Automation #Airflow #MachineLearning…

ברוכים הבאים אל Player FM!

Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.

תקשיבו ל-500+ נושאים

דומה לThe Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Roblox Digital Gift Card - 1,000 Robux [Includes Exclusive Virtual Item] [Digital Code]

Roblox Digital Gift Card - 2,000 Robux [Includes Exclusive Virtual Item] [Digital Code]

Amazon Basics Dog and Puppy Pee Pads, 5-Layer Leak-Proof Super Absorbent, Quick-Dry Surface, Potty Training, Regular (22x22"), 100 Count, Blue & White

פודקאסטים ששווה להאזין

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI « » How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing

How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing

פודקאסטים ששווה להאזין

ברוכים הבאים אל Player FM!

TurboTax Deluxe 2024 Tax Software, Federal & State Tax Return [PC/MAC Download]

Amazon Fire HD 10 tablet, built for relaxation, 10.1" vibrant Full HD screen, octa-core processor, 3 GB RAM, 32 GB, Black

Tubi: Watch Free Movies & TV Shows

FogWorx Extreme High Density Fog Juice - Quart – 32 Fluid Ounces of Long Lasting, High Output, Odorless Water Based Machine Fluid for 400 to 1500 Watt Machines

דומה לThe Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

מדריך עזר מהיר

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI « »
How Vibrant Planet's Self-Healing Pipelines Revolutionize Data Processing