התחל במצב לא מקוון עם האפליקציה Player FM !
פודקאסטים ששווה להאזין
בחסות


1 LIVE: Before the Chorus & Open Folk Present: In These Lines feat. Gaby Moreno, Lily Kershaw & James Spaite 33:58
Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot
Manage episode 468755901 series 2948506
Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets.
Key Takeaways:
(02:04) CSET provides data-driven analysis to inform government decision-makers.
(03:54) ETL pipelines merge multiple data sources for more comprehensive insights.
(04:20) Airflow is central to automating and streamlining large-scale data ingestion.
(05:11) Larger-scale databases create challenges that require scalable solutions.
(07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers.
(12:13) DAG Factory and dynamic task mapping can improve workflow efficiency.
(15:46) Tracking data lineage helps teams understand dependencies across DAGs.
(16:14) New Airflow features enhance visibility and debugging for complex pipelines.
Resources Mentioned:
https://www.linkedin.com/in/jennifer-melot-aa710144/
Center for Security and Emerging Technology (CSET) -
https://www.linkedin.com/company/georgetown-cset/
https://airflow.apache.org/
Zenodo -
https://zenodo.org/
https://openlineage.io/
https://cloud.google.com/dataplex
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 פרקים
Harnessing Airflow for Data-Driven Policy Research at CSET with Jennifer Melot
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 468755901 series 2948506
Turning complex datasets into meaningful analysis requires robust data infrastructure and seamless orchestration. In this episode, we’re joined by Jennifer Melot, Technical Lead at the Center for Security and Emerging Technology (CSET) at Georgetown University, to explore how Airflow powers data-driven insights in technology policy research. Jennifer shares how her team automates workflows to support analysts in navigating complex datasets.
Key Takeaways:
(02:04) CSET provides data-driven analysis to inform government decision-makers.
(03:54) ETL pipelines merge multiple data sources for more comprehensive insights.
(04:20) Airflow is central to automating and streamlining large-scale data ingestion.
(05:11) Larger-scale databases create challenges that require scalable solutions.
(07:20) Dynamic DAG generation simplifies Airflow adoption for non-engineers.
(12:13) DAG Factory and dynamic task mapping can improve workflow efficiency.
(15:46) Tracking data lineage helps teams understand dependencies across DAGs.
(16:14) New Airflow features enhance visibility and debugging for complex pipelines.
Resources Mentioned:
https://www.linkedin.com/in/jennifer-melot-aa710144/
Center for Security and Emerging Technology (CSET) -
https://www.linkedin.com/company/georgetown-cset/
https://airflow.apache.org/
Zenodo -
https://zenodo.org/
https://openlineage.io/
https://cloud.google.com/dataplex
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
63 פרקים
כל הפרקים
×
1 The Future of Airflow Telemetry at Metyis with Bolke de Bruin 21:55

1 Transforming the Airflow UI for Cloudera’s Users with Shubham Raj 22:28

1 Streamlining Thousands of Data Pipelines at Lyft with Yunhao Qing 19:34

1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.