Artwork

תוכן מסופק על ידי Data on Kubernetes Community. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי Data on Kubernetes Community או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.
Player FM - אפליקציית פודקאסט
התחל במצב לא מקוון עם האפליקציה Player FM !

DoK Talks #99- ETL/ELT on Kubernetes with Airbyte: K8s Development Insights // Abhi Vaidyanatha

1:00:45
 
שתפו
 

Manage episode 306403268 series 2865115
תוכן מסופק על ידי Data on Kubernetes Community. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי Data on Kubernetes Community או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.

https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
ETL/ELT on Kubernetes is currently an unsolved problem. There are a lot of different approaches vying for a spot as the de facto method, but none are clear winners. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions fall flat and don't mesh with the trajectory of the community.
Airbyte is an open-source ETL/ELT tool that harmonizes well with the cloud-native landscape and lives to enable your stateful workloads on Kubernetes. Previously, I have talked about a theoretical deployment on Kubernetes and the nuances behind deploying an ETL/ELT pipeline in such an environment. Now, I'm looking to follow that up with how we actually implemented that strategy as we launched our K8s beta. Additionally, I'll dive into some of the nitty gritty details that we needed to figure out in order to get this all working... stuff that isn't really found online!
Overall, this will be a really unique case of getting to do a retrospective on what we planned our architecture to look like and following up with some great development insights as we solidified the final implementation.
KEY TAKE-AWAYS FROM THE TALK
- Quick overview of Airbyte and open-source ETL/ELT [5 minutes]
- Why run your ETL/ELT in K8s? [3 minutes]
- A quick recap on the previous talk (what we thought the architecture would look like) [5 minutes]
- Display the actual architecture and implementation [10 minutes]
-> Talk about how to communicate with k8s pods on STDOUT and STDIN pipes
-> Describe parent-child process termination strategy
-> Describe persistence layer/strategy and config storage
- Quick demo of an Airbyte deployment on K8s [10 minutes]
BIO
Abhi is a confused economist who enjoys writing backend code for data management software. He now spends most of his time doing developer relations in the data integration space, where he looks to evangelize open source technologies. In his spare time he is a DJ, drummer, and competitive Super Smash Bros. player. He is a staunch advocate of proper semicolon usage, Oxford commas and overused grammar jokes.

  continue reading

243 פרקים

Artwork
iconשתפו
 
Manage episode 306403268 series 2865115
תוכן מסופק על ידי Data on Kubernetes Community. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי Data on Kubernetes Community או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.

https://go.dok.community/slack
https://dok.community/
ABSTRACT OF THE TALK
ETL/ELT on Kubernetes is currently an unsolved problem. There are a lot of different approaches vying for a spot as the de facto method, but none are clear winners. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions fall flat and don't mesh with the trajectory of the community.
Airbyte is an open-source ETL/ELT tool that harmonizes well with the cloud-native landscape and lives to enable your stateful workloads on Kubernetes. Previously, I have talked about a theoretical deployment on Kubernetes and the nuances behind deploying an ETL/ELT pipeline in such an environment. Now, I'm looking to follow that up with how we actually implemented that strategy as we launched our K8s beta. Additionally, I'll dive into some of the nitty gritty details that we needed to figure out in order to get this all working... stuff that isn't really found online!
Overall, this will be a really unique case of getting to do a retrospective on what we planned our architecture to look like and following up with some great development insights as we solidified the final implementation.
KEY TAKE-AWAYS FROM THE TALK
- Quick overview of Airbyte and open-source ETL/ELT [5 minutes]
- Why run your ETL/ELT in K8s? [3 minutes]
- A quick recap on the previous talk (what we thought the architecture would look like) [5 minutes]
- Display the actual architecture and implementation [10 minutes]
-> Talk about how to communicate with k8s pods on STDOUT and STDIN pipes
-> Describe parent-child process termination strategy
-> Describe persistence layer/strategy and config storage
- Quick demo of an Airbyte deployment on K8s [10 minutes]
BIO
Abhi is a confused economist who enjoys writing backend code for data management software. He now spends most of his time doing developer relations in the data integration space, where he looks to evangelize open source technologies. In his spare time he is a DJ, drummer, and competitive Super Smash Bros. player. He is a staunch advocate of proper semicolon usage, Oxford commas and overused grammar jokes.

  continue reading

243 פרקים

כל הפרקים

×
 
Loading …

ברוכים הבאים אל Player FM!

Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.

 

מדריך עזר מהיר

האזן לתוכנית הזו בזמן שאתה חוקר
הפעלה