התחל במצב לא מקוון עם האפליקציה Player FM !
Episode 51: Why We Built an MCP Server and What Broke First
Manage episode 491006621 series 3317544
What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?
In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.
We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world
If you’re working on LLMs in production, this one’s for you!
LINKS
- So We Shipped an AI Product: Did it Work? by Philip Carter
- Vanishing Gradients YouTube Channel
- Upcoming Events on Luma
- Hugo's recent newsletter about upcoming events and more!
🎓 Learn more:
- Hugo's course: Building LLM Applications for Data Scientists and Software Engineers — next cohort starts July 8: https://maven.com/s/course/d56067f338
📺 Watch the video version on YouTube: YouTube link
61 פרקים
Manage episode 491006621 series 3317544
What does it take to actually ship LLM-powered features, and what breaks when you connect them to real production data?
In this episode, we hear from Philip Carter — then a Principal PM at Honeycomb and now a Product Management Director at Salesforce. In early 2023, he helped build one of the first LLM-powered SaaS features to ship to real users. More recently, he and his team built a production-ready MCP server.
We cover:
• How to evaluate LLM systems using human-aligned judges
• The spreadsheet-driven process behind shipping Honeycomb’s first LLM feature
• The challenges of tool usage, prompt templates, and flaky model behavior
• Where MCP shows promise, and where it breaks in the real world
If you’re working on LLMs in production, this one’s for you!
LINKS
- So We Shipped an AI Product: Did it Work? by Philip Carter
- Vanishing Gradients YouTube Channel
- Upcoming Events on Luma
- Hugo's recent newsletter about upcoming events and more!
🎓 Learn more:
- Hugo's course: Building LLM Applications for Data Scientists and Software Engineers — next cohort starts July 8: https://maven.com/s/course/d56067f338
📺 Watch the video version on YouTube: YouTube link
61 פרקים
כל הפרקים
×ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.