19 subscribers
התחל במצב לא מקוון עם האפליקציה Player FM !
פודקאסטים ששווה להאזין
בחסות


1 The Southwest’s Wildest Outdoor Art: From Lightning Fields to Sun Tunnels 30:55
Experiencing Data w/ Brian T. O’Neill (UX for AI Data Products, SAAS Analytics, Data Product Management)
«
»
148 - UI/UX Design Considerations for LLMs in Enterprise Applications (Part 2)
Manage episode 430292849 series 2938687
Ready for more ideas about UX for AI and LLM applications in enterprise environments? In part 2 of my topic on UX considerations for LLMs, I explore how an LLM might be used for a fictitious use case at an insurance company—specifically, to help internal tools teams to get rapid access to primary qualitative user research. (Yes, it’s a little “meta”, and I’m also trying to nudge you with this hypothetical example—no secret!) ;-) My goal with these episodes is to share questions you might want to ask yourself such that any use of an LLM is actually contributing to a positive UX outcome Join me as I cover the implications for design, the importance of foundational data quality, the balance between creative inspiration and factual accuracy, and the never-ending discussion of how we might handle hallucinations and errors posing as “facts”—all with a UX angle. At the end, I also share a personal story where I used an LLM to help me do some shopping for my favorite product: TRIP INSURANCE! (NOT!)
Highlights/ Skip to:- (1:05) I introduce a hypothetical internal LLM tool and what the goal of the tool is for the team who would use it
- (5:31) Improving access to primary research findings for better UX
- (10:19) What “quality data” means in a UX context
- (12:18) When LLM accuracy maybe doesn’t matter as much
- (14:03) How AI and LLMs are opening the door for fresh visioning work
- (15:38) Brian’s overall take on LLMs inside enterprise software as of right now
- (18:56) Final thoughts on UX design for LLMs, particularly in the enterprise
- (20:25) My inspiration for these 2 episodes—and how I had to use ChatGPT to help me complete a purchase on a website that could have integrated this capability right into their website
Quotes from Today’s Episode
- “If we accept that the goal of most product and user experience research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings, is that all worth it? So, this is a design for AI problem.” - Brian T. O’Neill (8:09)
- “What’s in our data? Can the right people change it when the LLM is wrong? The data product managers and AI leaders reading this or listening know that the not-so-secret path to the best AI is in the foundational data that the models are trained on. But what does the word *quality* mean from a product standpoint and a risk reduction one, as seen from an end-users’ perspective? Somebody who’s trying to get work done? This is a different type of quality measurement.” - Brian T. O’Neill (10:40)
- “When we think about fact retrieval use cases in particular, how easily can product teams—internal or otherwise—and end-users understand the confidence of responses? When responses are wrong, how easily, if at all, can users and product teams update the model’s responses? Errors in large language models may be a significant design consideration when we design probabilistic solutions, and we no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path unknowingly, then AI is kind of like the team on the other side of the tug of war that we’re playing.” - Brian T. O’Neill (11:22)
- “As somebody who writes a lot for my consulting business, and composes music in another, one of the hardest parts for creators can be the zero-to-one problem of getting started—the blank page—and this is a place where I think LLMs have great potential. But it also means we need to do the proper research to understand our audience, and when or where they’re doing truly generative or creative work—such that we can take a generative UX to the next level that goes beyond delivering banal and obviously derivative content.” - Brian T. O’Neill (13:31)
- “One thing I actually like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that there is an opportunity for organizations here to do some fresh visioning work. And this is a place that designers and user experience professionals can help data teams as we bring design into the AI space.” - Brian T. O’Neill (14:04)
- “If there was ever a time to do some new visioning work, I think now is one of those times. However, we need highly skilled design leaders to help facilitate this in order for this to be effective. Part of that skill is knowing who to include in exercises like this, and my perspective, one of those people, for sure, should be somebody who understands the data science side as well, not just the engineering perspective. And as I posited in my seminar that I teach, the AI and analytical data product teams probably need a fourth member. It’s a quartet and not a trio. And that quartet includes a data expert, as well as that engineering lead.” - Brian T. O’Neill (14:38)
Links
105 פרקים
Manage episode 430292849 series 2938687
Ready for more ideas about UX for AI and LLM applications in enterprise environments? In part 2 of my topic on UX considerations for LLMs, I explore how an LLM might be used for a fictitious use case at an insurance company—specifically, to help internal tools teams to get rapid access to primary qualitative user research. (Yes, it’s a little “meta”, and I’m also trying to nudge you with this hypothetical example—no secret!) ;-) My goal with these episodes is to share questions you might want to ask yourself such that any use of an LLM is actually contributing to a positive UX outcome Join me as I cover the implications for design, the importance of foundational data quality, the balance between creative inspiration and factual accuracy, and the never-ending discussion of how we might handle hallucinations and errors posing as “facts”—all with a UX angle. At the end, I also share a personal story where I used an LLM to help me do some shopping for my favorite product: TRIP INSURANCE! (NOT!)
Highlights/ Skip to:- (1:05) I introduce a hypothetical internal LLM tool and what the goal of the tool is for the team who would use it
- (5:31) Improving access to primary research findings for better UX
- (10:19) What “quality data” means in a UX context
- (12:18) When LLM accuracy maybe doesn’t matter as much
- (14:03) How AI and LLMs are opening the door for fresh visioning work
- (15:38) Brian’s overall take on LLMs inside enterprise software as of right now
- (18:56) Final thoughts on UX design for LLMs, particularly in the enterprise
- (20:25) My inspiration for these 2 episodes—and how I had to use ChatGPT to help me complete a purchase on a website that could have integrated this capability right into their website
Quotes from Today’s Episode
- “If we accept that the goal of most product and user experience research is to accelerate the production of quality services, products, and experiences, the question is whether or not using an LLM for these types of questions is moving the needle in that direction at all. And secondly, are the potential downsides like hallucinations and occasional fabricated findings, is that all worth it? So, this is a design for AI problem.” - Brian T. O’Neill (8:09)
- “What’s in our data? Can the right people change it when the LLM is wrong? The data product managers and AI leaders reading this or listening know that the not-so-secret path to the best AI is in the foundational data that the models are trained on. But what does the word *quality* mean from a product standpoint and a risk reduction one, as seen from an end-users’ perspective? Somebody who’s trying to get work done? This is a different type of quality measurement.” - Brian T. O’Neill (10:40)
- “When we think about fact retrieval use cases in particular, how easily can product teams—internal or otherwise—and end-users understand the confidence of responses? When responses are wrong, how easily, if at all, can users and product teams update the model’s responses? Errors in large language models may be a significant design consideration when we design probabilistic solutions, and we no longer control what exactly our products and software are going to show to users. If bad UX can include leading people down the wrong path unknowingly, then AI is kind of like the team on the other side of the tug of war that we’re playing.” - Brian T. O’Neill (11:22)
- “As somebody who writes a lot for my consulting business, and composes music in another, one of the hardest parts for creators can be the zero-to-one problem of getting started—the blank page—and this is a place where I think LLMs have great potential. But it also means we need to do the proper research to understand our audience, and when or where they’re doing truly generative or creative work—such that we can take a generative UX to the next level that goes beyond delivering banal and obviously derivative content.” - Brian T. O’Neill (13:31)
- “One thing I actually like about the hype, investment, and excitement around GenAI and LLMs in the enterprise is that there is an opportunity for organizations here to do some fresh visioning work. And this is a place that designers and user experience professionals can help data teams as we bring design into the AI space.” - Brian T. O’Neill (14:04)
- “If there was ever a time to do some new visioning work, I think now is one of those times. However, we need highly skilled design leaders to help facilitate this in order for this to be effective. Part of that skill is knowing who to include in exercises like this, and my perspective, one of those people, for sure, should be somebody who understands the data science side as well, not just the engineering perspective. And as I posited in my seminar that I teach, the AI and analytical data product teams probably need a fourth member. It’s a quartet and not a trio. And that quartet includes a data expert, as well as that engineering lead.” - Brian T. O’Neill (14:38)
Links
105 פרקים
כל הפרקים
×
1 169 - AI Product Management and UX: What’s New (If Anything) About Making Valuable LLM-Powered Products with Stuart Winter-Tear 1:01:05

1 168 - 10 Challenges Internal Data Teams May Face Building Their First Revenue-Generating Data Product 38:24

1 167 - AI Product Management and Design: How Natalia Andreyeva and Team at Infor Nexus Create B2B Data Products that Customers Value 37:34

1 166 - Can UX Quality Metrics Increase Your Data Product's Business Value and Adoption? 26:12

1 165 - How to Accommodate Multiple User Types and Needs in B2B Analytics and AI Products When You Lack UX Resources 49:04

1 164 - The Hidden UX Taxes that AI and LLM Features Impose on B2B Customers Without Your Knowledge 45:25

1 163 - It’s Not a Math Problem: How to Quantify the Value of Your Enterprise Data Products or Your Data Product Management Function 41:41

1 162 - Beyond UI: Designing User Experiences for LLM and GenAI-Based Products 42:07

1 161 - Designing and Selling Enterprise AI Products [Worth Paying For] 34:00

1 160 - Leading Product Through a Merger/Acquisition: Lessons from The Predictive Index’s CPO Adam Berke 42:10

1 159 - Uncorking Customer Insights: How Data Products Revealed Hidden Gems in Liquor & Hospitality Retail 40:47

1 158 - From Resistance to Reliance: Designing Data Products for Non-Believers with Anna Jacobson of Operator Collective 43:41

1 157 - How this materials science SAAS company brings PM+UX+data science together to help materials scientists accelerate R&D 34:58

1 156-The Challenges of Bringing UX Design and Data Science Together to Make Successful Pharma Data Products with Jeremy Forman 41:37

1 155 - Understanding Human Engagement Risk When Designing AI and GenAI User Experiences 55:33

1 154 - 10 Things Founders of B2B SAAS Analytics and AI Startups Get Wrong About DIY Product and UI/UX Design 44:47

1 153 - What Impressed Me About How John Felushko Does Product and UX at the Analytics SAAS Company, LabStats 57:31

1 152 - 10 Reasons Not to Get Professional UX Design Help for Your Enterprise AI or SAAS Analytics Product 53:00

1 151 - Monetizing SAAS Analytics and The Challenges of Designing a Successful Embedded BI Product (Promoted Episode) 49:57

1 150 - How Specialized LLMs Can Help Enterprises Deliver Better GenAI User Experiences with Mark Ramsey 52:22

1 149 - What the Data Says About Why So Many Data Science and AI Initiatives Are Still Failing to Produce Value with Evan Shellshear 50:18

1 148 - UI/UX Design Considerations for LLMs in Enterprise Applications (Part 2) 26:36

1 147 - UI/UX Design Considerations for LLMs in Enterprise Applications (Part 1) 25:46

1 146 - (Rebroadcast) Beyond Data Science - Why Human-Centered AI Needs Design with Ben Shneiderman 42:07

1 145 - Data Product Success: Adopting a Customer-Centric Approach With Malcolm Hawker, Head of Data Management at Profisee 53:09

1 144 - The Data Product Debate: Essential Tech or Excessive Effort? with Shashank Garg, CEO of Infocepts (Promoted Episode) 52:38

1 143 - The (5) Top Reasons AI/ML and Analytics SAAS Product Leaders Come to Me For UI/UX Design Help 50:01

1 142 - Live Webinar Recording: My UI/UX Design Audit of a New Podcast Analytics Service w/ Chris Hill (CEO, Humblepod) 50:56

1 141 - How They’re Adopting a Producty Approach to Data Products at RBC with Duncan Milne 43:49

1 140 - Why Data Visualization Alone Doesn’t Fix UI/UX Design Problems in Analytical Data Products with T from Data Rocks NZ 42:44

1 139 - Monetizing SAAS Analytics and The Challenges of Designing a Successful Embedded BI Product (Promoted Episode) 51:02

1 138 - VC Spotlight: The Impact of AI on SAAS and Data/Developer Products in 2024 w/ Ellen Chisa of BoldStart Ventures 33:05

1 137 - Immature Data, Immature Clients: When Are Data Products the Right Approach? feat. Data Product Architect, Karen Meppen 44:50

1 136 - Navigating the Politics of UX Research and Data Product Design with Caroline Zimmerman 44:16

1 135 - “No Time for That:” Enabling Effective Data Product UX Research in Product-Immature Organizations 52:47

1 134 - What Sanjeev Mohan Learned Co-Authoring “Data Products for Dummies” 46:52


1 132 - Leveraging Behavioral Science to Increase Data Product Adoption with Klara Lindner 42:56

1 131 - 15 Ways to Increase User Adoption of Data Products (Without Handcuffs, Threats and Mandates) with Brian T. O’Neill 36:57

1 130 - Nick Zervoudis on Data Product Management, UX Design Training and Overcoming Imposter Syndrome 48:56

1 129 - Why We Stopped, Deleted 18 Months of ML Work, and Shifted to a Data Product Mindset at Coolblue 35:21

1 128 - Data Products for Dummies and The Importance of Data Product Management with Vishal Singh of Starburst 53:01

1 127 - On the Road to Adopting a “Producty” Approach to Data Products at the UK’s Care Quality Commission with Jonathan Cairns-Terry 36:55

1 126 - Designing a Product for Making Better Data Products with Anthony Deighton 47:38

1 125 - Human-Centered XAI: Moving from Algorithms to Explainable ML UX with Microsoft Researcher Vera Liao 44:42

1 124 - The PiCAA Framework: My Method to Generate ML/AI Use Cases from a UX Perspective 21:51

1 123 - Learnings From the CDOIQ Symposium and How Data Product Definitions are Evolving with Brian T. O’Neill 27:17

1 122 - Listener Questions Answered: Conducting Effective Discovery for Data Products with Brian T. O’Neill 33:46

1 121 - How Sainsbury’s Head of Data Products for Analytics and ML Designs for User Adoption with Peter Everill 39:40

1 120 - The Portfolio Mindset: Data Product Management and Design with Nadiem von Heydebrand (Part 2) 41:35

1 119 - Skills vs. Roles: Data Product Management and Design with Nadiem von Heydebrand (Part 1) 37:12

1 118 - Attracting Talent and Landing a Role in Data Product Management with Kyle Winterbottom 49:23

1 117 - Phil Harvey, Co-Author of “Data: A Guide to Humans,” on the Non-Technical Skills Needed to Produce Valuable AI Solutions 39:39

1 116 - 10 Reasons Your Customers Don’t Make Time for Your Data Product Initiatives + A Big Update on the Data Product Leadership Community (DPLC) 45:56

1 115 - Applying a Product and UX-Driven Approach to Building Stuart’s Data Platform with Osian Jones 45:19

1 114 - Designing Anti-Biasing and Explainability Tools for Data Scientists Creating ML Models with Josh Noble 42:05

1 113 - Turning the Weather into an Indispensable Data Product for Businesses with Cole Swain, VP Product at tomorrow.io 38:53

1 112 - Solving for Common Pitfalls When Developing a Data Strategy featuring Samir Sharma, CEO of datazuum 35:18

1 111 - Designing and Monetizing Data Products Like a Startup with Yuval Gonczarowski 33:15

1 110 - CDO Spotlight: The Value and Journey of Implementing a Data Product Mindset with Sebastian Klapdor of Vista 32:52

1 109 - The Role of Product Management and Design in Turning ML/AI into a Valuable Business with Bob Mason from Argon Ventures 32:43

1 108 - Google Cloud’s Bruno Aziza on What Makes a Good Customer-Obsessed Data Product Manager 50:43

1 107 - Tom Davenport on Data Product Management and the Impact of a Product Orientation on Enterprise Data Science and ML Initiatives 42:52

1 106 - Ideaflow: Applying the Practice of Design and Innovation to Internal Data Products w/ Jeremy Utley 44:14

1 105 - Defining “Data Product” the Producty Way and the Non-technical Skills ML/AI Product Managers Need 41:53

1 104 - Surfacing the Unarticulated Needs of Users and Stakeholders through Effective Listening 44:12

1 103 - Helping Pediatric Cardiac Surgeons Make Better Decisions with ML featuring Eugenio Zuccarelli of MIT Media Lab 42:33

1 102 - CDO Spotlight: The Non-Technical Roles Data Science and Analytics Teams Need to Drive Adoption of Data Products w/ Iván Herrero Bartolomé 35:05

1 101 - Insights on Framing IOT Solutions as Data Products and Lessons Learned from Katy Pusch 39:11

1 100 - Why Your Data, AI, Product & Business Strategies Must Work Together (and Digital Transformation is The Wrong Framing) with Vin Vashishta 45:08

1 099 - Don’t Boil the Ocean: How to Generate Business Value Early With Your Data Products with Jon Cooke, CTO of Dataception 48:28

1 098 - Why Emilie Schario Wants You to Run Your Data Team Like a Product Team 39:41

1 097 - Why Regions Bank’s CDAO, Manav Misra, Implemented a Product-Oriented Approach to Designing Data Products 35:22

1 096 - Why Chad Sanderson, Head of Product for Convoy’s Data Platform, is a Champion of Data UX 37:36

1 095 - Increasing Adoption of Data Products Through Design Training: My Interview from TDWI Munich 16:50

1 094 - The Multi-Million Dollar Impact of Data Product Management and UX with Vijay Yadav of Merck 46:02

1 093 - Why Agile Alone Won’t Increase Adoption of Your Enterprise Data Products 47:16

1 092 - How to measure data product value from a UX and business lens (and how not to do it) 34:46

1 091 - How Brazil’s Biggest Fiber Company, Oi, Leverages Design To Create Useful Data Products with Sr. Exec. Design Manager, João Critis 31:24

1 090 - Michelle Carney’s Mission With MLUX: Bringing UX and Machine Learning Together 31:43

1 089 - Reader Questions Answered about Dashboard UX Design 48:16

1 088 - Doing UX Research for Data Products and The Magic of Qualitative User Feedback with Mike Oren, Head of Design Research at Klaviyo 42:26

1 087 - How Data Product Management and UX Integrate with Data Scientists at Albertsons Companies to Improve the Grocery Shopping Experience 37:36

1 086 - CED: My UX Framework for Designing Analytics Tools That Drive Decision Making 27:57

1 085 - Dr. William D. Báez on the Journey and ROI of Integrating UX Design into Machine Learning and Analytics Solutions 44:42

1 084 - The Messy Truth of Designing and Building a Successful Analytics SAAS Product featuring Jonathan Kay (CEO, Apptopia) 39:56

1 083 -Why Bob Goodman Thinks Product Management and Design Must Dance Together to Create “Experience Layers” for Data Products 33:08

1 082 - What the 2021 $1M Squirrel AI Award Winner Wants You To Know About Designing Interpretable Machine Learning Solutions w/ Cynthia Rudin 37:55

1 081 - The Cultural and $ Benefits of Human-Centered AI in the Enterprise: Digging Into BCG/MIT Sloan’s AI Research w/ François Candelon 36:45

1 080 – How to Measure the Impact of Data Products…and Anything Else with Forecasting and Measurement Expert Doug Hubbard 46:00
ברוכים הבאים אל Player FM!
Player FM סורק את האינטרנט עבור פודקאסטים באיכות גבוהה בשבילכם כדי שתהנו מהם כרגע. זה יישום הפודקאסט הטוב ביותר והוא עובד על אנדרואיד, iPhone ואינטרנט. הירשמו לסנכרון מנויים במכשירים שונים.