How Apache Iceberg and Flink Can Ease Developer Pain

The New Stack Podcast

Player FM - Internet Radio Done Right

521 subscribers

DevOps

הוסף לפני ten שנים

תוכן מסופק על ידי The New Stack Podcast and The New Stack. כל תוכן הפודקאסטים כולל פרקים, גרפיקה ותיאורי פודקאסטים מועלים ומסופקים ישירות על ידי The New Stack Podcast and The New Stack או שותף פלטפורמת הפודקאסט שלהם. אם אתה מאמין שמישהו משתמש ביצירה שלך המוגנת בזכויות יוצרים ללא רשותך, אתה יכול לעקוב אחר התהליך המתואר כאן https://he.player.fm/legal.

The Big Pitch with Jimmy Carr

1
Coming Soon...The Big Pitch with Jimmy Carr 1:08

לפני יומיים1:08

הפעל מאוחר יותר

רשימות

לייק

אהבתי

1:08

Join us each week as celebrity guests pitch an idea for a film based on one of the SUPER niche sub-genres on Netflix. From ‘Steamy Crime Movies from the 1970s’ to ‘Australian Dysfunctional Family Comedies Starring A Strong Female Lead’, our celebrity guests will pitch their wacky plot, their dream cast, the marketing stunts, and everything in between. By the end of every episode, Jimmy Carr, Comedian by night / “Netflix Executive” by day, will decide whether the pitch is greenlit or condemned to development hell! New episodes on Wednesdays starting May 28th! Listen on all podcast platforms and watch on the Netflix is a Joke YouTube Channel . The Big Pitch is a co-production by Netflix and BBC Studios Audio.…

לפני שנה 47:08

MP3•בית הפרקים

In the New Stack Makers episode, Adi Polak, Director, Advocacy and Developer Experience Engineering at Confluent discusses the operational and analytical estates in data infrastructure. The operational estate focuses on fast, low-latency event-driven applications, while the analytical estate handles long-running data crunching tasks. Challenges arise due to the "schema evolution" from upstream operational changes impacting downstream analytics, creating complexity for developers.

Apache Iceberg and Flink help mitigate these issues. Iceberg, a table format developed by Netflix, optimizes querying by managing file relationships within a data lake, reducing processing time and errors. It has been widely adopted by major companies like Airbnb and LinkedIn.

Apache Flink, a versatile data processing framework, is driving two key trends: shifting some batch processing tasks into stream processing and transitioning microservices into Flink streaming applications. This approach enhances system reliability, lowers latency, and meets customer demands for real-time data, like instant flight status updates. Together, Iceberg and Flink streamline data infrastructure, addressing developer pain points and improving efficiency.

Learn more from The New Stack about Apache Iceberg and Flink:

Unfreeze Apache Iceberg to Thaw Your Data Lakehouse

Apache Flink: 2023 Retrospective and Glimpse into the Future

4 Reasons Why Developers Should Use Apache Flink

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

899 פרקים

#DevOps #Software Development #Tech #News #Tech News #Technology #Thenewstack #New Stack #Development #Joab Jackson #Alex Williams #Software Engineering #Heather Joslyn #Cloud Native #Kubernetes #Open Source #Darry Taft

The New Stack Podcast

How Apache Iceberg and Flink Can Ease Developer Pain

The New Stack Podcast

521 subscribers

published לפני שנה

שתפו

MP3•בית הפרקים

Learn more from The New Stack about Apache Iceberg and Flink:

Unfreeze Apache Iceberg to Thaw Your Data Lakehouse

Apache Flink: 2023 Retrospective and Glimpse into the Future

4 Reasons Why Developers Should Use Apache Flink

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

899 פרקים

כל הפרקים

The New Stack Podcast

1
Google Cloud Next Wrap-Up 18:22

לפני 19 hours18:22

18:22

At the close of this year’s Google Cloud Next, The New Stack’s Alex Williams, AI editor Frederic Lardinois, and analyst Janakiram MSV discussed the event’s dominant theme: AI agents. The conversation focused heavily on agent frameworks, noting a shift from last year's third-party tools like Langchain, CrewAI, and Microsoft’s Autogen, to first-party offerings from model providers themselves. Google’s newly announced Agent Development Kit (ADK) highlights this trend, following closely on the heels of OpenAI’s agent SDK. MSV emphasized the significance of this shift, calling it a major milestone as Google joins the race alongside Microsoft and OpenAI. Despite the buzz, Lardinois pointed out that many companies are still exploring how AI agents can fit into real-world workflows. The panel also highlighted how Google now delivers a full-stack AI development experience — from models to deployment platforms like Vertex AI. New enterprise tools like Agent Space and Agent Garden further signal Google’s commitment to making agents a core part of modern software development. Learn more from The New Stack about the latest in AI agents: How AI Agents Will Change the Web for Users and Developers AI Agents: A Comprehensive Introduction for Developers AI Agents Are Coming for Your SaaS Stack Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Agentic AI and A2A in 2025: From Prompts to Processes 19:18

לפני 3 ימים19:18

19:18

Agentic AI represents the next phase beyond generative AI, promising systems that not only generate content but also take autonomous actions within business processes. In a conversation recorded at Google Cloud Next, Kevin Laughridge of Deloitte explains that businesses are moving from AI pilots to production-scale deployments. Agentic AI enables decision-making, reasoning, and action across complex enterprise environments, reducing the need for constant human input. A key enabler is Google’s newly announced open Agent2Agent (A2A) protocol, which allows AI agents from different vendors to communicate and collaborate securely across platforms. Over 50 companies, including PayPal, Salesforce, and Atlassian, are already adopting it. However, deploying agentic AI at scale requires more than individual tools—it demands an AI platform with runtime frameworks, UIs, and connectors. These platforms allow enterprises to integrate agents across clouds and systems, paving the way for AI that is collaborative, adaptive, and embedded in core operations. As AI becomes foundational, developers are transitioning from coding to architecting dynamic, learning systems. Learn more from The New Stack about the latest insights about Agent2Agent Protocol: Google’s Agent2Agent Protocol Helps AI Agents Talk to Each Other A2A, MCP, Kafka and Flink: The New Stack for AI Agents Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Your AI Coding Buddy Is Always Available at 2 a.m. 20:43

לפני 8 ימים20:43

20:43

Aja Hammerly, director of developer relations at Google, sees AI as the always-available coding partner developers have long wished for—especially in those late-night bursts of inspiration. In a conversation with Alex Williams at Google Cloud Next, she described AI-assisted coding as akin to having a virtual pair programmer who can fill in gaps and offer real-time support. Hammerly urges developers to start their AI journey with tools that assist in code writing and explanation before moving into more complex AI agents. She distinguishes two types of DevEx AI: using AI to build apps and using it to eliminate developer toil. For Hammerly, this includes letting AI handle frontend work while she focuses on backend logic. The newly launched Firebase Studio exemplifies this dual approach, offering an AI-enhanced IDE with flexible tools like prototyping, code completion, and automation. Her advice? Developers should explore how AI fits into their unique workflow—because development, at its core, is deeply personal and individual. Learn more from The New Stack about the latest AI insights with Google Cloud: Google AI Coding Tool Now Free, With 90x Copilot’s Output Gemini 2.5 Pro: Google’s Coding Genius Gets an Upgrade Q&A: How Google Itself Uses Its Gemini Large Language Model Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Google AI Infrastructure PM On New TPUs, Liquid Cooling and More 19:38

לפני 10 ימים19:38

19:38

At Google Cloud Next '25, the company introduced Ironwood, its most advanced custom Tensor Processing Unit (TPU) to date. With 9,216 chips per pod delivering 42.5 exaflops of compute power, Ironwood doubles the performance per watt compared to its predecessor. Senior product manager Chelsie Czop explained that designing TPUs involves balancing power, thermal constraints, and interconnectivity. Google's long-term investment in liquid cooling, now in its fourth generation, plays a key role in managing the heat generated by these powerful chips. Czop highlighted the incremental design improvements made visible through changes in the data center setup, such as liquid cooling pipe placements. Customers often ask whether to use TPUs or GPUs, but the answer depends on their specific workloads and infrastructure. Some, like Moloco, have seen a 10x performance boost by moving directly from CPUs to TPUs. However, many still use both TPUs and GPUs. As models evolve faster than hardware, Google relies on collaborations with teams like DeepMind to anticipate future needs. Learn more from The New Stack about the latest AI infrastructure insights from Google Cloud: Google Cloud Therapist on Bringing AI to Cloud Native Infrastructure A2A, MCP, Kafka and Flink: The New Stack for AI Agents Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Google Cloud Therapist on Bringing AI to Cloud Native Infrastructure 24:04

לפני 15 ימים24:04

24:04

At Google Cloud Next, Bobby Allen, Group Product Manager for Google Kubernetes Engine (GKE), emphasized GKE’s foundational role in supporting AI platforms. While AI dominates current tech conversations, Allen highlighted that cloud-native infrastructure like Kubernetes is what enables AI workloads to function efficiently. GKE powers key Google services like Vertex AI and is trusted by organizations including DeepMind, gaming companies, and healthcare providers for AI model training and inference. Allen explained that GKE offers scalability, elasticity, and support for AI-specific hardware like GPUs and TPUs, making it ideal for modern workloads. He noted that Kubernetes was built with capabilities—like high availability and secure orchestration—that are now essential for AI deployment. Looking forward, GKE aims to evolve into a model router, allowing developers to access the right AI model based on function, not vendor, streamlining the development experience. Allen described GKE as offering maximum control with minimal technical debt, future-proofed by Google’s continued investment in open source and scalable architecture. Learn more from The New Stack about the latest insights with Google Cloud: Google Kubernetes Engine Customized for Faster AI Work KubeCon Europe: How Google Will Evolve Kubernetes in the AI Era Apache Ray Finds a Home on the Google Kubernetes Engine Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
VMware's Kubernetes Evolution: Quashing Complexity 30:40

לפני 17 ימים30:40

30:40

Without this, developers waste time managing infrastructure instead of focusing on code. VMware addresses this with VCF, a pre-integrated Kubernetes solution that includes components like Harbor, Valero, and Istio, all managed by VMware. While some worry about added complexity from abstraction, Turner dismissed concerns about virtualization overhead, pointing to benchmarks showing 98.3% of bare metal performance for virtualized AI workloads. He emphasized that AI is driving nearly half of Kubernetes deployments, prompting VMware’s partnership with Nvidia to support GPU virtualization. Turner also highlighted VMware's open source leadership, contributing to major projects and ensuring Kubernetes remains cloud-independent and standards-based. VMware aims to simplify Kubernetes and AI workload management while staying committed to the open ecosystem. Learn more from The New Stack about the latest insights with VMware Has VMware Finally Caught Up With Kubernetes? VMware’s Golden Path Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Prequel: Software Errors Be Gone 5:13

לפני 18 ימים5:13

5:13

Prequel is launching a new developer-focused service aimed at democratizing software error detection—an area typically dominated by large cloud providers. Co-founded by Lyndon Brown and Tony Meehan, both former NSA engineers, Prequel introduces a community-driven observability approach centered on Common Reliability Enumerations (CREs). CREs categorize recurring production issues, helping engineers detect, understand, and communicate problems without reinventing solutions or working in isolation. Their open-source tools, cre and prereq , allow teams to build and share detectors that catch bugs and anti-patterns in real time—without exposing sensitive data, thanks to edge processing using WebAssembly. The urgency behind Prequel’s mission stems from the rapid pace of AI-driven development, increased third-party code usage, and rising infrastructure costs. Traditional observability tools may surface symptoms, but Prequel aims to provide precise problem definitions and actionable insights. While observability giants like Datadog and Splunk dominate the market, Brown and Meehan argue that engineers still feel overwhelmed by data and underpowered in diagnostics—something they believe CREs can finally change. Learn more from The New Stack about the latest Observability insights Why Consolidating Observability Tools Is a Smart Move Building an Observability Culture: Getting Everyone Onboard Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

The New Stack Podcast

1
Arm’s Open Source Leader on Meeting the AI Challenge 18:21

לפני 22 ימים18:21

18:21

At Arm, open source is the default approach, with proprietary software requiring justification, says Andrew Wafaa, fellow and senior director of software communities. Speaking at KubeCon + CloudNativeCon Europe, Wafaa emphasized Arm’s decade-long commitment to open source, highlighting its investment in key projects like the Linux kernel, GCC, and LLVM. This investment is strategic, ensuring strong support for Arm’s architecture through vital tools and system software. Wafaa also challenged the hype around GPUs in AI, asserting that CPUs—especially those enhanced with Arm’s Scalable Matrix Extension (SME2) and Scalable Vector Extension (SVE2)—are often more suitable for inference workloads. CPUs offer greater flexibility, and Arm’s innovations aim to reduce dependency on expensive GPU fleets. On the AI framework front, Wafaa pointed to PyTorch as the emerging hub, likening its ecosystem-building potential to Kubernetes. As a PyTorch Foundation board member, he sees PyTorch becoming the central open source platform in AI development, with broad community and industry backing. Learn more from The New Stack about the latest insights about Arm: Edge Wars Heat Up as Arm Aims to Outflank Intel, Qualcomm Arm: See a Demo About Migrating a x86-Based App to ARM64 Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

The New Stack Podcast

1
Why Kubernetes Cost Optimization Keeps Failing 17:22

לפני 24 ימים17:22

17:22

In today’s uncertain economy, businesses are tightening costs, including for Kubernetes (K8s) operations, which are notoriously difficult to optimize. Yodar Shafrir, co-founder and CEO of ScaleOps, explained at KubeCon + CloudNativeCon Europe that dynamic, cloud-native applications have constantly shifting loads, making resource allocation complex. Engineers must provision enough resources to handle spikes without overspending, but in large production clusters with thousands of applications, manual optimization often fails. This leads to 70–80% resource waste and performance issues. Developers typically prioritize application performance over operational cost, and AI workloads further strain resources. Existing optimization tools offer static recommendations that quickly become outdated due to the dynamic nature of workloads, risking downtime. Shafrir emphasized that real-time, fully automated solutions like ScaleOps' platform are crucial. By dynamically adjusting container-level resources based on real-time consumption and business metrics, ScaleOps improves application reliability and eliminates waste. Their approach shifts Kubernetes management from static to dynamic resource allocation. Listen to the full episode for more insights and ScaleOps' roadmap. Learn more from The New Stack about the latest in scaling Kubernetes and managing operational costs: ScaleOps Adds Predictive Horizontal Scaling, Smart Placement ScaleOps Dynamically Right-Sizes Containers at Runtime Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
How Heroku Is ‘Re-Platforming’ Its Platform 18:01

לפני 29 ימים18:01

18:01

Heroku has been undergoing a major transformation, re-platforming its entire Platform as a Service (PaaS) offering over the past year and a half. This ambitious effort, dubbed “Fir,” will soon reach general availability. According to Betty Junod, CMO and SVP at Heroku (owned by Salesforce), the overhaul includes a shift to Kubernetes and OCI standards, reinforcing Heroku’s commitment to open source. The platform now features Heroku Cloud Native Buildpacks, which let developers create container images without Dockerfiles. Originally built on Ruby on Rails and predating Docker and AWS, Heroku now supports eight programming languages. The company has also deepened its open source engagement by becoming a platinum member of the Cloud Native Computing Foundation (CNCF), contributing to projects like OpenTelemetry. Additionally, Heroku has open sourced its Twelve-Factor Apps methodology, inviting the community to help modernize it to address evolving needs such as secrets management and workload identity. This signals a broader effort to align Heroku’s future with the cloud native ecosystem. Learn more from The New Stack about Heroku's approach to Platform-as-a-Service: Return to PaaS: Building the Platform of Our Dreams Heroku Moved Twelve-Factor Apps to Open Source. What’s Next? How Heroku Is Positioned To Help Ops Engineers in the GenAI Era Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Container Security and AI: A Talk with Chainguard's Founder 20:51

לפני 4 weeks20:51

20:51

In this episode of The New Stack Makers , recorded at KubeCon + CloudNativeCon Europe, Alex Williams speaks with Ville Aikas, Chainguard founder and early Kubernetes contributor. They reflect on the evolution of container security, particularly how early assumptions—like trusting that users would validate container images—proved problematic. Aikas recalls the lack of secure defaults, such as allowing containers to run as root, stemming from the team’s internal Google perspective, which led to unrealistic expectations about external security practices. The Kubernetes community has since made strides with governance policies, secure defaults, and standard practices like avoiding long-lived credentials and supporting federated authentication. Aikas founded Chainguard to address the need for trusted, minimal, and verifiable container images—offering zero-CVE images, transparent toolchains, and full SBOMs. This security-first philosophy now extends to virtual machines and Java dependencies via Chainguard Libraries. The discussion also highlights the rising concerns around AI/ML security in Kubernetes, including complex model dependencies, GPU integrations, and potential attack vectors—prompting Chainguard’s move toward locked-down AI images. Learn more from The New Stack about Container Security and AI Chainguard Takes Aim At Vulnerable Java Libraries Clean Container Images: A Supply Chain Security Revolution Revolutionizing Offensive Security: A New Era With Agentic AI Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Kelsey Hightower, AWS's Eswar Bala on Open Source's Evolution 37:52

לפני 5 weeks37:52

37:52

In a candid episode of The New Stack Makers , Kubernetes pioneer Kelsey Hightower and AWS’s Eswar Bala explored the evolving relationship between enterprise cloud providers and open source software at KubeCon+CloudNativeCon London. Hightower highlighted open source's origins as a grassroots movement challenging big vendors, and shared how it gave people—especially those without traditional tech credentials—a way into the industry. Recalling his own journey, Hightower emphasized that open source empowered individuals through contribution over credentials. Bala traced the early development of Kubernetes and his own transition from building container orchestration systems to launching AWS’s Elastic Kubernetes Service (EKS), driven by growing customer demand. The discussion, recorded at KubeCon + CloudNativeCon Europe, touched on how open source is now central to enterprise cloud strategies, with AWS not only contributing but creating projects like Karpenter, Cedar, and Kro. Both speakers agreed that open source's collaborative model—where companies build in public and customers drive innovation—has reshaped the cloud ecosystem, turning former tensions into partnerships built on community-driven progress. Learn more from The New Stack about the relationship between enterprise cloud providers and open source software: The Metamorphosis of Open Source: An Industry in Transition The Complex Relationship Between Cloud Providers and Open Source How Open Source Has Turned the Tables on Enterprise Software Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
The Kro Project: Giving Kubernetes Users What They Want 21:51

לפני 5 weeks21:51

21:51

In a rare show of collaboration, Google, Amazon, and Microsoft have joined forces on Kro — the Kubernetes Resource Orchestrator — an open source, cloud-agnostic tool designed to simplify custom resource orchestration in Kubernetes. Announced during KubeCon + CloudNativeCon Europe, Kro was born from strong customer demand for a Kubernetes-native solution that works across cloud providers without vendor lock-in. Nic Slattery, Product Manager at Google and Jesse Butler, Principal Product Manager, AWS shared with The New Stack that unlike many enterprise products, Kro didn’t stem from top-down strategy but from consistent customer "pull" experienced by all three companies. It aims to reduce complexity by allowing platform teams to offer simplified interfaces to developers, enabling resource requests without needing deep service-specific knowledge. Kro also represents a unique cross-company collaboration, driven by a shared mission and open source values. Though still in its alpha stage, the project has already attracted 57 contributors in just seven months. The team is now focused on refining core features and preparing for a production-ready release — all while maintaining a narrowly scoped, community-first approach. Learn more from The New Stack about KRO: One Mighty kro; One Giant Leap for Kubernetes Resource Orchestration Kubernetes Gets a New Resource Orchestrator in the Form of Kro Orchestrate Cloud Native Workloads With Kro and Kubernetes Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

The New Stack Podcast

1
OpenSearch: What’s Next for the Search and Analytics Suite? 20:10

לפני 6 weeks20:10

20:10

OpenSearch has evolved significantly since its 2021 launch, recently reaching a major milestone with its move to the Linux Foundation. This shift from company-led to foundation-based governance has accelerated community contributions and enterprise adoption, as discussed by NetApp’s Amanda Katona in a New Stack Makers episode recorded at KubeCon + CloudNativeCon Europe. NetApp, an early adopter of OpenSearch following Elasticsearch’s licensing change, now offers managed services on the platform and contributes actively to its development. Katona emphasized how neutral governance under the Linux Foundation has lowered barriers to enterprise contribution, noting a 56% increase in downloads since the transition and growing interest from developers. OpenSearch 3.0, featuring a Lucene 10 upgrade, promises faster search capabilities—especially relevant as data volumes surge. NetApp’s ongoing investments include work on machine learning plugins and developer training resources. Katona sees the Linux Foundation’s involvement as key to OpenSearch’s long-term success, offering vendor-neutral governance and reassuring users seeking openness, performance, and scalability in data search and analytics. Learn more from The New Stack about OpenSearch: Report: OpenSearch Bests ElasticSearch at Vector Modeling AWS Transfers OpenSearch to the Linux Foundation OpenSearch: How the Project Went From Fork to Foundation Join our community of newsletter subscribers to stay on top of the news and at the top of your game.…

The New Stack Podcast

1
Kong’s AI Gateway Aims to Make Building with AI Easier 21:05

לפני 7 weeks21:05

21:05

AI applications are evolving beyond chatbots into more complex and transformative solutions, according to Marco Palladino, CTO and co-founder of Kong. In a recent episode of The New Stack Makers, he discussed the rise of AI agents, which act as "virtual employees" to enhance organizational efficiency. For instance, AI can now function as a product manager for APIs—analyzing documentation, detecting inaccuracies, and making corrections. However, reliance on AI agents brings security risks, such as data leakage and governance challenges. Organizations need observability and safeguards, but developers often resist implementing these requirements manually. As GenAI adoption matures, teams seek ways to accelerate development without rebuilding security measures repeatedly. To address these challenges, Kong introduced AI Gateway, an open-source plugin for its API Gateway. AI Gateway supports multiple AI models across providers like AWS, Microsoft, and Google, offering developers a universal API to integrate AI securely and efficiently. It also features automated retrieval-augmented generation (RAG) pipelines to minimize hallucinations. Palladino emphasized the need for consistent security in AI infrastructure, ensuring developers can focus on innovation while leveraging built-in protections. Learn more from The New Stack about Kong’s AI Gateway Kong: New ‘AI-Infused’ Features for API Management, Dev Tools From Zero to a Terraform Provider for Kong in 120 Hours Join our community of newsletter subscribers to stay on top of the news and at the top of your game .…

ברוכים הבאים אל Player FM!

Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.