
התחל במצב לא מקוון עם האפליקציה Player FM !
Building a Unified Data Platform at Pattern with William Graham
Manage episode 509844747 series 2948506
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.
In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.
Key Takeaways:
00:00 Introduction.
02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.
04:27 How Airflow became the central scheduler for batch jobs.
08:57 Credential management challenges that led to decoupling scheduling and orchestration.
12:21 Heimdall simplifies multi-application access through a unified interface.
13:15 Standardized operators in Airflow using Heimdall integration.
17:13 Open-source contributions and early adoption of Heimdall within Pattern.
21:01 Community support for Airflow and satisfaction with scheduling flexibility.
Resources Mentioned:
https://www.linkedin.com/in/willgraham2/
Pattern | LinkedIn
https://www.linkedin.com/company/pattern-hq/
Pattern | Website
https://pattern.com
https://airflow.apache.org
https://github.com/patterninc/heimdall
https://netflix.github.io/genie/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
73 פרקים
Building a Unified Data Platform at Pattern with William Graham
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 509844747 series 2948506
The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.
In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.
Key Takeaways:
00:00 Introduction.
02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.
04:27 How Airflow became the central scheduler for batch jobs.
08:57 Credential management challenges that led to decoupling scheduling and orchestration.
12:21 Heimdall simplifies multi-application access through a unified interface.
13:15 Standardized operators in Airflow using Heimdall integration.
17:13 Open-source contributions and early adoption of Heimdall within Pattern.
21:01 Community support for Airflow and satisfaction with scheduling flexibility.
Resources Mentioned:
https://www.linkedin.com/in/willgraham2/
Pattern | LinkedIn
https://www.linkedin.com/company/pattern-hq/
Pattern | Website
https://pattern.com
https://airflow.apache.org
https://github.com/patterninc/heimdall
https://netflix.github.io/genie/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
73 פרקים
כל הפרקים
×ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.