The PM playbook is broken for AI. Here’s how to fix it.
In the last few years, we’ve seen a shift in both the types of products PMs are building and the tools they use to build those products. The AI-powered tools we build are probabilistic, context-dependent, and always evolving. Their value is defined less by feature presence and more by output quality, trust, and relevance.
At the same time, AI is expanding what PMs can do, enabling faster experimentation, functional prototyping, and earlier validation of ideas.
In other words, AI has accelerated product development. But it’s also exposed shortfalls in the traditional PM playbook. Many PM rituals, like prototyping, usability testing, A/B testing, and roadmapping, were designed for deterministic products with more predictable behavior. AI products don’t scale with that playbook, which forces PMs to rethink the product development process.
PMs need to adapt to this changing landscape. Here are a few ways to do that.
The traditional PM frameworks that don’t fully apply anymore
The processes that PMs relied on in the past no longer fit when building AI products (and, to a lesser extent, when using AI tools). PMs who want to continue to build useful products need to be aware of the ways AI has changed their work, including:
- Low-fi prototyping in tools like Figma doesn’t replicate personalization, relevance, or model reasoning. Instead, AI PMs are using tools like Replit to vibe code basic proof-of-concept versions.
- A/B testing may struggle because results vary by user context, not treatment alone. AI’s probabilistic nature means that even two identical inputs won’t necessarily lead to the same output.
- Traditional roadmaps need to move from linear cycles or feature checklists to ongoing model iteration and monitoring to account for model changes in accuracy and AI drift.
- User research has to shift from usability to corpus testing, prompt evaluation, and scenario simulation, using human insights but also agentic AI tools for evaluation.
- Success metrics shift from binary feature adoption to more intangible evals such as output quality.
The new questions AI PMs are asking (or should ask)
Alongside the challenges, PMs building AI products and using AI tools to help them build those products are also facing new opportunities. To seize those opportunities, they need to embrace new ways of thinking.
AI products and tools are non-deterministic and, by their nature, somewhat unpredictable. As Mixpanel’s Daniel Schmidt puts it,
“AI is like having a genius toddler in your codebase. The ceiling on what you can accomplish is extremely high, but the potential for things to go wrong is also much higher than traditional products. Engineers and product managers are learning to harness that potential while also limiting the risks.”
The first step to harnessing that potential is asking the right questions and being clear-eyed about how AI (and building AI products) changes processes. Here are a few questions that AI PMs can ask themselves at different stages of product development.
Ideation and roadmapping
One of the biggest changes that AI has brought to product development is vibe coding, or the ability to use AI to build apps and simple digital products without deep engineering knowledge.
For PMs, vibe coding offers valuable opportunities to test and flesh out ideas, without calling on engineering and design resources. “Inevitably, most product ideas are not going to work. That’s just the nature of product work,” Daniel says. “Vibe coding with tools like Replit allows me to realize the flaws in my ideas a lot faster, and I don’t need to devote design and engineering resources to a project until I’ve tested it out on my own first,” he explains.
“Any product team that’s operating solely in the world of Figma instead of vibe coding prototypes is going to be slower than the competition. And they will overinvest in product ideas that just aren’t going to work, instead of discarding them at the early stages.”
Important questions to consider during the ideation and roadmapping phase:
- How do we create tangible differentiation when core AI capabilities are increasingly commoditized and considered as table stakes?
- What does a roadmap look like when we need to iterate on model quality, rather than new features?
- How do we balance speed to market with long-term trust and quality?
Evaluation and QA
In many cases, AI products don’t have a static user interface, and interactions with AI products are entirely personalized and based on user inputs. That’s the nature of probabilistic products: We don’t know what AI is going to produce based on a specific input.
This presents new problems for product evaluation and quality control. Tools like tracing and AI evals help to solve these issues, but human oversight (at least in the early stages) has become more important than ever.
“There’s a temptation for PMs to immediately jump to a higher level of sophistication and immediately use tracing and evals, when really the first step needs to be reading transcripts of what happened and seeing what the user’s input was and what the AI response was…You can’t skip that step of having a human interpret what went right and what went wrong. Later down the line, you can graduate into more automations and AI evals. Jumping to automations prematurely is a mistake.”
Qualitative and quantitative insights from tools like session replay and heatmaps can also give a more complete picture of the user experience.
Important questions to consider for evaluation and QA:
- How do we test something whose quality is heavily dependent on input, user, business, or any other context?
- If model accuracy changes over time (and drifts), what does 'ship' even mean?
- What is the MVP for an AI feature, and what’s considered good enough?
- How do we think about launch blockers in the context of relevance and accuracy?
Prototyping and discovery
One of the new challenges that AI poses is the distance between proof of concept, early prototypes, and building an actual product that will work reliably for real users. “There’s a very big gap between having a good demo of an AI feature and having an AI feature that you can truly launch and bring to market,” explains Daniel. Product teams have to understand this gap when they’re prototyping and building, as it can affect budgets, viability, timelines, and decision-making.
Important questions to ask for prototyping and discovery:
- How do we validate the value of an AI feature if a low-fidelity prototype can’t replicate context or reflect any meaningful accuracy or relevance?
- Do we need to prototype with real customer data to get meaningful feedback?
- Do we still need to prioritize testing for usability or testing the UI?
User experience
Since every user journey is different with AI products, understanding those journeys and creating the best possible experiences becomes more difficult. It’s also more challenging to diagnose issues, since it’s harder to understand what they are: Is it a negative result, a prompting problem, or an engineering problem?
“Understanding what’s working, what’s not, and how your product is designed is trickier when every user could get a result that you can’t predict, and you don’t fully know or control what they’re going to be experiencing,” Daniel says.
Important questions to consider for UX:
- What’s the role of UX in expectation setting for the user?
- What’s the role of UX when it comes to accuracy or quality? (e.g., Are you supposed to expose uncertainty, or should you hide it?)
Join us as we explore how PMs PM in the AI era
At Mixpanel, our PMs are figuring out how to elevate their craft in the AI era. We may not have all the answers, but we want to share what we’re doing, where we feel like we’re onto something, where we’re getting it wrong, and bring you (our community) along for the journey. We’d love to hear about your AI experiments too. Join the Mixpanel Community!