158: The Orchestration Layer as the Data Platform Control Plane With Nick Schrock of Dagster Labs

The Data Stack Show

Highlights from this week’s conversation include:

• Nick’s background and journey in data (2:28)
• Founding Dagster Labs (7:50)
• The evolution of data engineering (12:32)
• Fragmentation in data infrastructure (15:04)
• The role of orchestration in data platforms (19:53)
• The importance of operational tools for data pipelines (25:01)
• Lessons learned from working with GraphQL (26:19)
• The role of the orchestrator in data engineering (34:51)
• The boundaries between data infrastructure and product engineering (37:33)
• Different orchestrators in the data infrastructure landscape(42:03)
• The role of MLOps in data engineering (46:04)
• Data Quality and Orchestration (51:04)
• Future of Data Teams and Orchestration (54:27)
• Final thoughts and takeaways from (58:01)

The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.

RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.