In this episode, Collin and I continue building our list of questions to help increase the chances of your AI investments delivering a return. We focus in particular on the role non-technical C-level leaders should play in this effort.

We use Chip Huyen’s AI Engineering as a practical framework, exploring how it gives digital leaders and non-technical senior managers a structured way to engage more deeply with AI initiatives. In particular, it provides a way into the ecosystem and lifecycle of AI application development—helping leaders ask better questions around things like data, prompt design, fine-tuning, and evaluation—so they can have more meaningful discussions with technical teams about how these applications are built, where the risks sit, and what criteria should be used to define and measure success.

We walk through some of the key differences between traditional software and AI systems—particularly the shift from deterministic to probabilistic behaviour, and the central role of data in shaping outcomes.

From there, we build on the questions we believe leaders should be asking: What problem are we solving? How are we evaluating outputs? How are we managing risks around data quality, safety, and factual accuracy? What trade-offs are we making between quality, cost, and latency?

We spend some time looking at Chip’s section on evaluation criteria, using it as a springboard for non-technical senior leaders to delve deeper into the thinking behind—and expected outcomes of—AI applications. We also introduce the concept of “evals”—ongoing evaluation frameworks that extend beyond traditional testing—and why they require continuous iteration, collaboration, and oversight, even after deployment.

This episode continues our exploration of how leaders can better understand what they are funding, engage more effectively with product and delivery teams, and create the conditions for AI investments to deliver real value.

Links to Chip's book & interview on evals referred to in the episode:

Chip Huyen’s AI Engineering: https://www.oreilly.com/library/view/ai-engineering/9781098166298/

Lenny’s podcast - Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar: https://www.youtube.com/watch?v=BsWxPI9UM4c

A significant number of AI initiatives are consuming budget and executive attention, while failing to reach production or deliver measurable value. Episode 12 of Talking Product helps you pressure-test your organisation's AI spend today.

In this episode, Collin and I use a Financial Times article as a springboard for a conversation about the state of AI initiatives in large organisations. The article—"AI's awfully exciting until companies want to use it"—captures a familiar pattern in technology: high expectations, significant investment, but limited impact. The article is freely available with FT registration.

In this episode, we explore three themes:

• Why pilots aren't scaling (often it's your data, not the technology)
• Whether organisations are bringing experimental rigour to AI adoption, or just buying impressive demos
• The leadership knowledge gap—understanding not just what AI can do, but what it can't

What you'll get:

• Questions that will help reveal whether your organisation is truly learning from its AI experiments or just spending
• Insight into why data governance problems you've been kicking down the road are now becoming existential
• What Gartner found about hidden costs in AI initiatives—and questions to ask to bring them out into the open
• Practical guidance on what experimental rigour actually looks like in an AI context

#AI #DigitalTransformation #Leadership #ProductManagement

In this episode, Collin shares a case about a leadership team building a hardware and software product.

We dig into this case because it highlights familiar product dynamics—great, passionate people, working to tight deadlines, but stuck in semi-siloed patterns and not really operating as one team. Each group pushes its own goals but isn’t fully aligned on the end game. And when the product leaders aren’t working effectively as a team, it has a real impact on outcomes and the ability to deliver at pace.

We chose this topic to give product leaders footholds and practical ideas for starting similar conversations in your own organisation. We often call ourselves a team—but are we, as product leaders, really operating as a team? What do we really mean when we use the word “team”?

In this session, Collin describes how he asked the leadership group that he was working with this simple, but powerful question: “Are you working as a team?” To their credit, they took the time to work with Collin and reflect on this question, as a collective.

One area of the case that I found interesting was when Collin unpacks the complexity of “owning” requirements as new information emerges or thinking shifts about a particular problem. It is this inherent complexity of digital product development that makes siloed, and even semi-siloed, ways of working so dangerous.

Collin also highlights a deeper systemic issue: senior and executive leaders often don’t grasp the importance of ways of working in digital product development. Instead, they simply push the silos to deliver faster. I personally feel that this level of naivety at the top is a serious competitive problem.

I hope you find this case useful. We welcome recommendations and comments—and we’re looking forward to getting a few more episodes out, sooner rather than later.

At the end of the podcast, I refer to the work of Nonaka & Takeuchi and said I would include a link: 1) The New New Product Development Game and 2) The Knowledge Creating Company are probably their most well known work. But there is a wealth of other material that is also worth exploring.

In Episode 10, Collin and I dive into a question I was asked during an interview:

“We do an offsite with our leadership team, and it’s a great couple of days… but once we’re back in the day-to-day, it all sort of disappears. As a coach, how do you help with that?”

The question has lingered with me for quite some time, and I thought it would be an interesting topic to unpack with Collin.

In this episode, we explore:

• How coaches can help “hold” a team’s desire to change, providing structure and accountability without getting pulled into the daily fires
• Why working with digital leadership teams means going beyond agile frameworks into the messier world of team dynamics and organizational complexity
• The importance of examining your own unconscious beliefs about leadership
• Three common offsite mistakes we often see—and how to avoid them:
  • Senior leaders facilitating instead of participating
  • Agendas so full there’s no space for reflection
  • Avoiding the hard conversations that matter most

We share personal experiences to keep the conversation actionable and grounded in the day-to-day realities of organisational life.

I hope you find Episode 10 useful.

Thanks to everyone who’s been following the podcast and leaving kind reviews—it means a lot.

What if you could measure what really matters to your customer?

In our latest episode of Talking Product, Collin and I chat with Alex Hidalgo—author of the O’Reilly book Implementing Service Level Objectives. Alex shares how Service Level Objectives (SLOs) can help teams zero in on customer priorities and bring people across the organisation together to define what really matters.

SLOs are powerful, but they can also be tricky to implement. I’ve struggled with them myself, so it was great to have the chance to explore this topic in depth with Alex and learn from his expertise.

In this session, we cover:

• Practical steps to get started with Service Level Indicators (SLIs), the foundation of SLOs.
• How to start small and iterate as you refine your SLOs over time.
• The cultural challenges of implementing SLOs—and how to navigate them.

If you’re just starting out or looking to improve your approach, I hope you find this episode full of insights to help you on your SLO journey.

Alex’s book is an excellent resource for making SLOs meaningful and achievable in your organisation. I highly recommend it.

I hope you enjoy listening to this session as much as we enjoyed recording it.

If you liked the episode, please give us a rating—or go all out and write us a nice review. It makes a big difference!

A huge thank you to Alex for making the time to share his insights with us.

In this session Collin & I use Volkswagen’s struggle to build Electric Vehicle (EV) software as a way to explore two themes:

1. the importance of getting a digital transformation right, and
2. the opportunity presented to competitors when large players simply cannot deliver good quality digital products.

We use two articles from the Financial Times as a springboard for our conversation. The first article, dated Jun 28, 2024 “Oliver Blume, VW and Porsche boss steering an EV strategy shift” and the second one, published September 2, 2024 “VW considers closing factories in Germany and cutting jobs” (both behind a paywall - sorry)

The first article describes an investment of $5 billion in the US based EV startup Rivian. The purpose of the investment is to create a joint partnership that allows VW to use Rivian’s EV software. The article quotes VW’s CEO Oliver Blume as saying, “In terms of a big tech transformation, you can’t do it all on your own… It should be a win-win situation . . . The motivation from our side is to speed up software transformation at Volkswagen in all our brands. Rivian has best in class architecture . . . Volkswagen has the scale”.

Collin and I explore this thinking of an acquisition / strategic partnership as a means for fixing an internal digital transformation challenge. For those of you who have listened to Collin and I, you probably won’t be surprised to hear that we are not convinced that such an approach will easily remedy digital product development challenges.

As we ask in the show, what mental models will need to change for these two companies - a digital native and a non-digital native, to work effectively together? As Collin highlights, ultimately these transformations are culture changes, which require heavy-lifting.

Since recording this session, the FT has come out with more articles on the challenges Volkswagen is facing. I have also started listening to some of the VW YouTube influencers who have been discussing software issues for a number of years. I am finding this to be a fascinating case study, albeit quite sad for the workers of Wolfsburg if the leadership of VW can’t get its digital act together.

In this session, I also refer to a 2017 Medium post on how Revolut used the build-measure-learn cycle to develop digital products quickly and acquire customers.

Hope you find the session useful.

If you like the show, please give us a nice rating and recommend the show. It helps others find the podcast.

In this last episode of our three part series, Collin and I continue to role-play an imaginary data-heavy product development project. In our last episode, Collin was granted some authority to get more involved in the project in an effort to reduce the risk that Collin felt was inherent in the delivery team’s approach. The delivery teams is a consulting firm that has been brought in by another group in the company. In this episode, Collin elaborates on the types of risks he sees, as well as how he would address those risks through an alternative approach to product development.

Collin and I are using this fictitious scenario to explore how ways of working, power dynamics and group-think play out in digital product development. In my opinion, these three forces are always at play in this type of work. Our goal in exploring this is to help listeners become better equipped to work with these forces on their own projects.

In this episode, Collin and I discuss the fact that using an approach that pushes risk out into the open earlier in the product-development lifecycle brings reality into the room very quickly. It forces people to stop imagining “happy-path” outcomes through Power Point and Gantt charts. This can often be experienced as “raining on the parade.”

In this episode, I ask Collin to help a senior manager identify the risks themselves by role playing the person who procured the consulting firm. In this scenario, a senior manager wants to be able to see the types of risks that Collin is seeing. This manager wants to be able to ask the questions that Collin is asking. And, they want to mitigate some of those risks without having to rely on delivery managers like Collin - “What is it that allows Collin to see digital product development in the way he sees it?”

Through the role-play, we discover risks that were hidden in the consulting firm’s Power Point presentation and Gantt charts. Because of the approach the consulting firm was taking, made worse by shrouding it in Agile lingo, these risks would not have revealed themselves until very late in the consulting firm’s engagement. As Collin says, “If the consulting firm had followed that plan they would have delivered late and asking for an extension. That may be good for the bottom-line on the consulting side, but puts the company’s strategic goals at risk.”

Hope you enjoy it. As always we would look forward to hearing from people.

If you are liking these podcasts, please do throw us some love in the ratings.

Thanks

In this episode, Collin and I continue to explore a digital product development project. Our project is heavily dependent on data and is to be delivered by an outside consulting firm. Our consulting firm says that they are are using Agile practices and principles to deliver the project, but as we learned in part 1, this team is in fact using a much more sequential “black-box” approach to delivery.

For Collin and I, this is a realistic scenario. It is something that we have seen multiple times in our careers. It is also a high risk project, not simply because of the heavy dependence on data, but also because of the proposed ways of working combined with the organisational and power dynamics of the situation. We feel that such dynamics are present in all real-world digital product development situations, but often not taken into consideration when talking about ways of working.

Our goal in this exploration is to give listeners tangible actions that they can use when faced with a similar situation.

In this episode, we explore further Collin’s idea of latent assumptions, which he introduced in episode 1. These assumptions are baked into projects, unrecognised as assumptions. Collin and I feel that this is particularly true for projects with a heavy reliance on data.

Dependent upon the ways of working, teams may not learn that their assumptions were incorrect until very late in the project, often when the budget is nearly exhausted.

We delve into the mechanisms that Collin is using to assess the risks within such a project and how to bring these risks into the light early to prove or disprove them?

We spend a lot of time in this session exploring how do you get people with the right level of authority to recognise the risks of such a project, particularly when many of those people aren’t equipped to see such risks?

We hope you enjoy it. As always please let us know what you think. And, if you liked the session, we would be very appreciative for a positive rating.