Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Player FM - Internet Radio Done Right

הוסף לפני four שנים

תוכן מסופק על ידי The Data Flowcast. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The Data Flowcast או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.

Fixable

1
Unsolicited Advice: How to get your mojo back at work 35:43

לפני 13 weeks35:43

הפעל מאוחר יותר

רשימות

לייק

אהבתי

35:43

Are you feeling overwhelmed, uninspired, or burned out at work? In this special Unsolicited Advice episode, Anne and Frances are getting ready for summer by tackling the importance of rest and the power of intention. They offer unexpected tips and tricks for recovering from constant stress, owning what you need to feel alive and engaged, and creating an experience of work that unleashes your ambition. Follow Hosts: Anne Morriss ( @annemorriss | LinkedIn: @anne-morriss ), Frances Frei ( @francesxfrei | LinkedIn: @francesfrei ) Links https://anneandfrances.com/ Subscribe to TED Instagram: @ted YouTube: @TED TikTok: @tedtoks LinkedIn: @ted-conferences Website: ted.com Podcasts: ted.com/podcasts For the full text transcript, visit ted.com/podcasts/fixable-transcripts Hosted on Acast. See acast.com/privacy for more information.…

לפני שנה 30:09

MP3•בית הפרקים

The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.

In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.

Key Takeaways:

(02:17) Defining responsibilities within data architecture teams.

(04:15) Consolidating multiple orchestrators into a single solution.

(07:00) Scaling Airflow environments with shared infrastructure and DevOps practices.

(10:59) Managing dependencies and readiness using SQL sensors.

(14:23) Enhancing visibility and response through Slack-integrated monitoring.

(19:28) Extending Airflow’s flexibility to run legacy systems.

(22:28) Integrating transformation tools into orchestrated pipelines.

(25:54) Enabling non-engineers to contribute to pipeline development.

(27:33) Fostering adoption through collaboration and communication.

Resources Mentioned:

Alberto Crespi

https://www.linkedin.com/in/crespialberto/

lastminute.com | Website

https://lastminute.com

Apache Airflow

https://airflow.apache.org/

dbt Labs

https://www.getdbt.com/

Astronomer Cosmos

https://github.com/astronomer/astronomer-cosmos

GitLab Slack

https://slack.com/

Kubernetes

https://kubernetes.io/

Confluence

https://www.atlassian.com/software/confluence

Slack

https://slack.com/

https://www.astronomer.io/events/roadshow/london/

https://www.astronomer.io/events/roadshow/new-york/

https://www.astronomer.io/events/roadshow/sydney/

https://www.astronomer.io/events/roadshow/san-francisco/

https://www.astronomer.io/events/roadshow/chicago/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

71 פרקים

#Tech #Airflow

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

published לפני שנה

שתפו