Willy Wonka and the Product Leader

A few months ago, everything was (mostly) manageable. You built a great product for a small startup. You kept the vision intact, and you moved quickly across small and agile teams. Your founders got what they wanted: buzz, then traction, then explosive growth.

Suddenly, you’re not just a lean little startup that’s running and running freely. You’re a “real” company. There are executives and board members to report up to. You’re facing down regulatory scrutiny, contract negotiations, and adding more people to do more things and build more features and products. And growth requires new skill sets, too, like the ability to train and manage a cohesive unit, keeping everyone productive and creative.

You’re still the product leader, but you aren’t magic. Where do you go from here? How do you get a handle on the cross-functional chaos?

If these questions sound familiar, our very own Woody Schneider may have some answers for you. The erstwhile physicist was also a management consultant before coming to Mixpanel in early 2012. As our seventh hire, and current leader of Mixpanel’s strategic accounts program, he’s seen first-hand what’s worked (and hasn’t) at the many fast-growing companies that make up Mixpanel’s customer base.

And he’s seen the same struggle time and again as companies scale up: not enough reliance on data. Woody calls it the Willy Wonka Fallacy of Product Management.

“You go to Willy Wonka’s chocolate factory, and no one’s making candy,” he says. “They’re making magic wallpaper and floating bubbles and Everlasting Gobstoppers. The actual candy-making happens behind closed doors, and it’s all full of mysticism. That’s how some folks think of product development. I don’t believe that’s true at all.

“Product development is not about giant leaps forward – at least not most of the time. It’s about incremental progress, about using your head and your creativity to craft solutions that may or may not be good. And there’s so much more that data can do for you as you scale.”

Woody has learned a lot working directly with product people at dozens of high growth companies over the last four years – from Uber to Twitch. Here are five of his best pieces of advice for product leaders when their startup is taking off, but the Willy Wonka Fallacy is making it harder to grow.

When you’ve found product-market fit and it’s time to grow, you’re probably reaching a point where “moving fast and breaking things” isn’t an option. While this Facebook mantra seems to penetrate Silicon Valley culture, clinging to five magic words won’t get you to where you need to be. Such incantations won’t build awesome products either, especially as you scale. As Woody tells us, it’s time to scrap the magical thinking.

“I think product management at scale can be much more scientific than it is in the early days of a startup,” Woody says. “Of course, sometimes you want to make or take risks, you want to do big things, and you should. But even when you’re doing those big things, that ‘magic’ can be broken apart into pieces that can be measured and understood; that magic should have goals that tie into business strategy.

“It’s like that famous Henry Ford quote: ‘If I had asked people what they wanted, they would have said faster horses.’ But I don’t think that’s really how it works,” he continues. 

“Henry Ford made cars, and there was already a market for cars, there was already the assembly line, there were already interchangeable parts. What he did was take an existing product and move it downmarket, gradually. But the lore of the great inventor has us thinking about him like the Model T just came to him. It was less a blast of genius than it was a ton of small measurable steps.”

“Small, measurable steps” backed by data aren’t sexy, but they will help you and your product org make better, more effective decisions as you’re growing. Most importantly, they’ll keep you from reinventing the wheel.

TL;DR: Growth is the result of grinding on product day in and day out, not magic. Create and measure data around your product management process so you can make the best product decisions as you scale.

One of the biggest challenges that product leaders face at a fast-growing company is giving up portions of their hard-earned influence. Rapid growth can turn a PM from the go-to person for every aspect of a product into a bottleneck for new product management processes.

“When you’re small, power can and should be concentrated in the hands of very few people, and decision-making happens really quickly, top down” says Woody. “You’re able to maintain a very high bar for your decision making, arguing through each issue, and thinking about the big picture.“

After a product leader has reached the point where her limited bandwidth is blocking others like Augustus Gloop in the chocolate tube, it’s time to let data take over and empower everyone in your product org.

“It’s a very interesting transitional point for a business, when you have crossed over into the territory where one human can’t keep track of all the many threads in their head, or all the many pieces that go together to make that product magic,” says Woody.  

“Even if you found a superhuman, would you want such a concentrated bottleneck slowing development down? You need something better to keep track of all the moving parts, and you need to give everyone on your team access to the data you’re producing to do so.”

Take XO Group, the parent company of “lifestage” media sites like The Knot, The Nest, and The Bump. They released their bottleneck by sharing real-time analytics with everyone in the company – not just their data scientists. In doing so, they discovered that their users were sharing their content more frequently via email and iMessage. They made a product tweak that allowed users to more easily share their finds in the way they preferred.

This data democratization gives everyone on a product team the ability to measure and report on progress in a systematic, sophisticated way. As a bonus, this type of access and transparency leads to ownership and empowerment. And it releases the pressure behind the product leader, and fuels the engine for the next phase of product growth.

TL; DR: Don’t be the bottleneck for innovation in your org. Give your entire team access to the data so that everyone can make better, quicker, and smarter product decisions. It empowers the team and it ultimately empowers you.

Once everyone has access to the data, what does it mean to put together metrics you can use in the middle of rapid growth, when your goals keep changing? What do you do with all that data? Can you take it at face value?

“There is a bit of a fallacy out there right now about data, which is that it tells the truth,” says Woody. “If I say, ‘listen, this data says X; therefore, it’s unequivocal,’ that’s not true at all. You can make data say whatever you want, depending on how you put it together.”

If you make a decision based on a particular metric, it’s vitally important to know exactly how it might be leading or misleading your team. To keep your data honest, always ask: What are you measuring? Where are your metrics coming from? How are they calculated? Interrogation systems like paired metrics – tracking retention rates along with the number of installs, for example – can help measure the effects of your decisions and deter data myopia.  

Rand Fishkin of Moz fame expanded his SEO product empire through good data interrogation. He discovered that Google’s AdWords reporting was becoming less accurate and needlessly specific. He could have accepted the internet behemoth’s keyword research data as gospel. Instead, he dug deep into the data to figure out what was really going on. The result is a product that gives its users a more accurate range of searches and a much simpler workflow – something he would have never considered if he’d taken the data at face value.  

“A big lesson I think I’ve learned watching so many companies grow is that you must always remember to interrogate the data,” says Woody. “Just having a piece of data to point at that backs up a decision you made does not release you from your obligation to be a discerning human being.”

TL;DR: Just because something is measured doesn’t mean the results are the truth. A product leader’s job as a company grows is to keep data honest by asking good questions.

If you’ve been a product manager long enough, some of your most thankless tasks probably include managing expectations and mitigating failures. But at scale, these activities may actually stunt the growth of your company.

“I think that lot of businesses get gummed up in their product development because they’re not willing to put trust into people to make decisions because they might make mistakes,” says Woody. Avoiding mistakes in the face of rapid growth becomes an impossible task, even for the people in the org that you trust the most.

“Effective, error-free decision making is impossible at scale. Instead, you need to have tolerance for mistakes; in fact, you need to expect mistakes. That’s why you have data, after all, to let you know what worked and what didn’t.

“You need to be ready to be transparent and realistic when you do make mistakes. We’re all human, after all, and that’s how we learn. Otherwise, you’re going to stumble forward carrying the baggage of your good ideas and your bad ideas alongside you, and you won’t know which is which,” says Woody. That stifles decision making and growth.  

Instead, use data to define the parameters in which good ideas can take shape. Ironically enough, using data can make the team feel safer about failure. That safety, in turn, can lead them to take bigger swings with potentially bigger payoffs, since failure is an option.

When Harper Reed was CTO for Barack Obama’s 2012 reelection campaign, failure meant failing the leader of the free world. So he and his team designed a backend that integrated all of the data the campaign gathered across the organization and purposely crashed it, working through each and every way things could break. They examined all the data and learned how to make things right, just in time to secure the president’s second term.

Harper’s is an extreme example, but in most cases it’s not about the potential consequences of a product leader failing; it’s about the project showing data that didn’t match the hypothesis. By using data to take as much of the personal sting as possible out of a failure, you can motivate your team to move on quickly from mistakes and find the next thing to try.

TL;DR: Don’t get hung up on mistakes. Plan for them. With data transparency, you can trust people to make decisions on their own, knowing that a certain percentage of decisions won’t be the best, but will still move growth forward.

When a startup becomes a high-growth company, the distance between you and your customer can quickly go from inches to miles. It’s tempting to retreat into the details of building a product as your company gets bigger. If you stay there too long, however, you may discover that you’ve lost track of what your customers actually want.

Whether you’re taking incremental steps forward or you’re trying to stand up a giant structure, however, the product management fundamentals are still the same: take a level-headed look at what’s been built and what you’d like to build, and then implement some line of communication to the market to see what it thinks.

“You can think of a company as a black box where you put a dollar in one slot then two dollars come out the other slot, right?” Woody says. “What’s inside that box is your product, and its profitability is a function of what’s happening in the market for your product. The bad news is that what’s happening in the market is always changing. The good news is that what gets used, what sells, and what doesn’t are all pretty easily measurable. You just have to actually look at the data and get quality feedback on it from the market.”

Surveys are one of the more obvious choices when it comes to customer feedback at scale. Spotify was able to identify their optimal monthly price ($14.99) and how many accounts should be included (6) by querying their customers through simple surveys. By collecting this data from millions of users, they were able to implement these changes before they launched these two new service offerings.  

With rapid growth, data is absolutely critical to understanding the conversation. When you had a handful of customers, it was easy ask them directly for product feedback. But when accounting for thousands or millions of customer voices, you’re contending with a customer ecosystem. Lean on data to deliver the insights you need to drive customer engagement.

TL;DR: Keep an eye on the market as you scale. Think of your data as market signals. The signals are not just about what you’re building, but about what’s happening in the wider world.

Rapid growth looks a lot like a golden ticket, and product development can feel a lot like Willy Wonka’s factory. In reality, though, it’s not that simple.   

Balancing creativity with progress in that environment is what great product leaders do, but it’s seldom easy. The clarity that data transparency can bring to your growing product org will help propel your product forward, and keep you honest and on track as you shift from total product ownership to product leadership.

“So much time and so little to do. Wait a minute. Strike that. Reverse it.”
– Willy Wonka, product leader