How to create a culture of experimentation in product teams

Have you ever seen a product team make a major product change based on nothing but vibes? To be fair, we usually call vibes “assumptions” in a business setting—it makes them sound more legitimate. 

But really, the outcome is the same. Decisions are made based on what we think is true, or what we feel should be true, or what sounds like it’s probably true, instead of looking at the data. 

To remedy this, product teams need to foster a culture of experimentation, where decisions are made based on what the data shows and not how we feel. Experimentation is more than just A/B testing—it’s about creating a mindset of learning and iteration throughout all levels of your organization. 

This article will help product managers and team leaders implement a structured approach to experimentation. Let’s jump in. 

Creating a culture of experimentation across your company promotes data-driven decision-making and prevents teams from relying on assumptions. When your product managers, engineers, and growth teams are encouraged to experiment—and given the tools and resources to do so—you promote innovation through continuous iteration. 

By testing small changes instead of going all in on big decisions immediately, teams avoid expensive product failures. Those incremental changes reduce risk and help promote the culture of learning that product teams need to succeed. Experimentation helps teams quickly identify what works and what doesn’t and speeds up those learning cycles. This gives companies that embrace experimentation a competitive advantage: They can innovate faster and ultimately build a better product more quickly. 

Airbnb operates a two-sided online marketplace for booking travel accommodation, matching “hosts” with travelers looking for rooms or apartments. Since the beginning, Airbnb has embraced a culture of experimentation, using controlled testing to understand and validate product changes. Throughout the process, they’ve learned to run experiments for longer (to account for their complex ecosystem), tested different search layouts, and much more.  

Culture doesn’t create itself. If you want to create and promote a culture of experimentation at your company, you need to intentionally put the necessary building blocks in place. It’s important to actively promote the culture you want. 

Perhaps the most important element for creating a culture of experimentation is leadership buy-in: If your C-suite and executives are actively and enthusiastically supporting experimentation, the rest of your company will feel empowered to prioritize it too. 

Without leadership buy-in, changing organizational culture is tricky. Depending on the environment and tone at your company, people might even actively avoid experimentation because they are afraid of failure and the impact that failing (or even just being perceived as failing) can have on their careers. 

It’s important to foster a sense of psychological safety and encourage a culture where “failures” are viewed as learning opportunities. 

You want your team to feel safe to say, “We tested this hypothesis, the results weren’t what we expected, and now we have more data to work with,” and not be afraid this will be viewed as a failure. It’s important to foster the mindset that experiments that don’t yield the results you were hoping for still give you valuable information about your users and that you can incorporate this data into future experimentation. 

Once your leadership is promoting experimentation and your teams feel confident in exploring hypotheses and trying new things, you need to make sure that data is accessible. Teams need the right tools and data to measure experiment outcomes effectively. Without those, there is simply no way to create a culture of experimentation. Even the most enthusiastic organization will find it impossible to gain momentum. Teams need a structured experimentation process with clear guidelines and support on how to propose, run, and analyze experiments. 

Online streaming platform Netflix has been building its culture of learning for decades, since the days when it shipped DVDs to people’s homes. Learning and experimentation are ubiquitous across the company: As the Netflix technology blog explains, “There’s no better way to learn and make great decisions than to confirm or falsify ideas and hypotheses using the power of rigorous testing. Openly and candidly sharing test results allows everyone at Netflix to develop intuition about our members and ideas for how we can deliver an even better experience to them — and then the virtuous cycle starts again.”

Now that we’ve talked a little bit about the elements you need to create a culture of experimentation, let’s look at the step-by-step process you and your team can put in place to make the changes you need. 

Communicate the importance of experimentation to the team and align it with company goals. In addition to company-wide values (which, depending on your level of seniority and the size of your company, you might not be able to change), consider creating a set of team values that promote experimentation. These can include things like:

Different core values and priorities will make sense for different product teams, but building a culture of experimentation on your team starts with naming what that culture looks like. 

Valuing experimentation won’t get you anywhere if your team doesn’t know how to execute. A structured and documented experimentation process will help support your product team as they implement and run experiments. 

Your process should define how experiments are proposed, prioritized, and documented. Ensure each experiment has a hypothesis, success metrics, and a clear goal.

In order to run experiments, product teams need access to data and tools that enable them to analyze that data. Self-serve analytics tools that connect to your data warehouse make it possible to track and analyze real-time data without the help of data analysts. You’ll also want to create dashboards where teams can easily access results and share insights across the organization. 

Experimentation works best when you move quickly. Shorter iteration cycles mean that teams can adapt quickly to new insights without spending too much time on ideas that aren’t bringing value. 

Incorporate discussions of experiments into daily and weekly standups to brainstorm ideas and share updates. Encourage post-mortem discussions to document lessons learned and incorporate those learnings into future experiments. 

Positive feedback and reinforcement are some of the most effective ways to create lasting positive change. Recognize teams for running well-structured experiments, even if results don’t lead to success. Reinforce the idea that learnings from failed experiments are valuable. Make experimentation part of performance review conversations so that your team members know it’s an integral part of their job/duties. 

Music streaming platform Spotify is known for experimenting at scale—the company conducts more than 250 tests annually. A large contributing factor to this extensive testing and their success? The company culture. 

As Spotify’s engineering blog explains, “By fostering a culture of transparency and collaboration, we can closely scrutinize each experimental variant, minimizing errors and ensuring that every release contributes valuable, actionable insights for enhancing our user experience.”

There are a few common roadblocks that product teams encounter when they try to implement an experimentation culture. Here are a few we’ve seen and some tips on what to do when you feel stuck. 

Some teams may be skeptical about experiment-driven decision-making. There’s often resistance to changing how things are done, especially if it requires learning new methods and if there is a perception that the status quo was fine. 

Get your team on board by highlighting success stories from early experiments and sharing them widely. Celebrating wins will nudge more people to investigate the experimentation tools at their disposal and create a more positive culture. 

Every experiment should have a clear goal. If teams don’t define success clearly, they won’t get actionable insights.

Powerful, user-friendly analytics tools like Mixpanel offer your team a centralized platform to interrogate and analyze product, marketing, and revenue data. With all of your data easily accessible and queryable, you can easily measure experiment success and keep your approach consistent across all of your data. 

As we mentioned above, a major component of a successful culture of experimentation is creating an environment where your team feels safe to fail. Teams will avoid experiments if they fear negative outcomes. 

To counter this fear, you need to promote a learning-first mindset, both by rewarding it in your team and by making sure that leadership models that mindset too. Every experiment is a learning opportunity that contributes to knowledge. 

Mixpanel is built to make experimentation and analysis accessible to your entire organization, and it has the features and functionality to support your team as it scales its experiments. 

SaaS website builder GoDaddy wanted to deploy an analytics strategy that gave them the speed and flexibility they needed to answer questions about their product’s performance on the fly. The product team often course-corrects and experiments with newly shipped features essentially as they are launching. Building this habit of fast iteration and experimentation into their processes and culture was vital to their success.

As Senior Director of Product Management at GoDaddy Nissim Lehyani says, “With Mixpanel at everyone’s fingertips, GoDaddy has built a culture of experimentation where we put our customers first.”

Read the full case study. 

We’ve talked about why a culture of experimentation is important and some of the pitfalls you can encounter when you try to foster one at your organization. 

Before we go, here are a few best practices that will help your culture shift and grow in the direction you want: 

 A culture of experimentation leads to better decision-making, faster innovation, and continuous learning.

But remember that culture, like Rome, isn’t built in a day. Focus on rewarding experimentation and creating space for people to try things and share their results. Being consistent and patient makes it more likely that your cultural changes will stick around for the long term.