NerdWallet’s culture of experimentation

As VP of Product at NerdWallet, I guide product decisions that drive maximum value for our brand and consumers—helping them break down complicated financial topics (e.g. what the heck is a Roth IRA?), and make smart money moves (e.g. how can I get the highest interest rate on my savings?). This understanding of why we do what we do is core to our product strategy, and informs every decision we make.

No matter the industry, product experimentation should always be done with your customers top of mind. But in the consumer finance space, sensitivity towards personal financial data makes considerations around product experimentation all the more delicate.

After all, when credibility and trust is your brand, you’d better be solid as a rock.

Along with fostering a company culture that’s widely adopted and understood, smart product experimentation is about prioritizing drivers before solutions, taking a measured approach, and experimenting with purpose.

Sadly, I’ve come across too many companies where culture is no more than posters on a wall. But for culture to matter, it needs to be reinforced in both what you say, and what you do. For product leaders, this means being intentional about creating psychologically safe spaces where employees can fail and learn from experiments.

On the NerdWallet team, we go beyond a binary “pass / fail” grade on demo days with Engineering, Product and Design. Instead, we promote a practice of asking questions with the goal of gleaning knowledge and insight—not to dismantle someone’s point of view—and encourage thinking aloud to pressure-test logic, and validate hypotheses. We also host voluntary “turkey” demos where members of our team can share experiments that failed and what they learned from them.

These turkey demos have surfaced key learnings for our team. For example:

Recently, an important mortgage lead generation user flow was underperforming. Our analysis surfaced many issues: call-to-action was low on the page; content and copy was cluttered and hard to scan; information users wanted was hidden behind links; etc. So, we completely redesigned the user flow to correct all of these deficiencies, and launched an A/B test of what we felt would be a much more performant variant.

Boy, were we wrong. To our surprise, the variant underperformed not just by a little, but by a lot—but the team didn’t know why.

Thanks to this particular turkey demo, the team learned an important lesson about what not to do: change several variables at one time.

When it comes to building a culture of experimentation, adhering to established guardrails will also help your team make better, faster decisions about your product. Ours are:

In practice, ‘drivers before solutions’ is about getting to the root of the problem you want to solve. It’s all about doing the legwork upfront before presenting a hypothesis to your team. In meetings, “I think” statements become “I know, because I’ve done a lot of homework” statements.

Here’s an example of a key driver we uncovered for NerdWallet customers:

About half of consumers have real choice when it comes to their finances—they’re not living paycheck-to-paycheck—and most have cash sitting in the savings account that came with their checking account…the one paying 0.03% interest. There are plenty of credible, high-yield savings options out there, but because people don’t know, they do nothing. And doing nothing is a poor choice as well.

NerdWallet can help by figuring out why these folks are doing nothing, and then empower them to find the best financial products for their needs. In terms of our core product, this might mean adding capabilities that make it easy for users to transfer funds from one account to another.

For smarter experiments that put the customer first, encourage your team to dig into the “why” before your experimentation ship even sets sail.

With your culture in step, establish a framework to help you set the stage for confident experimentation that prioritizes brand consistency. 

Consider this framework the basis for your team’s ‘epic’—a term borrowed from literature that refers to a big chunk of work that has one common objective. This could be anything from a feature, to a customer request, or a business requirement. When used consistently in all your product communications to the company, epics will help you craft a compelling story and create a cycle of reinforcement around your process.

While Homer has The Odyssey, NerdWallet has HBML (hypothesize-build-measure-learn). Here’s what it looks like:

We believe that if we <create / offer / build feature / action / idea> for <primary user who needs / wants / has a problem with job / opportunity / pain point>, we will <observable change from user’s current state> and <observable change from NerdWallet’s current state>. We will know this to be true when <measurable KPI that will signal change>.

Second, follow a playbook for A/B testing that includes consistent standards (more on this below), and explicit requirements for statistical significance.

At NerdWallet, we conduct hundreds of A/B tests a year that touch our CRM, Marketplace, and Insights & Guidance teams. Staying focused on our “north star” metrics and leveraging lightweight prototyping makes for faster, cleaner feedback loops and clear-headed decision making.

Now that you’ve built the cultural foundation and framework for product iteration, it’s time to start experimenting. But heed my golden rule:

Don’t test what you wouldn’t ship.

We already know that experimentation is incredibly valuable. But here in the business of helping people make financial decisions with confidence, we’re extra mindful of not compromising our core brand tenets (namely, privacy, information security, and trust). We’ve established our sandbox for playing in, and we respect the outer limits.

Simply put, we can’t get it wrong.

Rather than blindly swinging for the fences and risking a brand misstep, I advise product teams to make a transparent decision on their “start value quality”—basic brand and quality standards that apply to even the smallest “painted door” feature test—and then leverage feedback loops from users to identify where to improve.

Lastly, make a meaningful investment in both your content and visual style guides—but don’t just post your design system standards somewhere and let them languish. Pre-build major components for future assembly and maintain a library for teams to use in experiments. An upfront investment here will pay off in a consistent brand voice in the long run, along with faster prototyping and experimentation.

By marrying agility with quality control standards, product leaders can drive experiments that are truly impactful, delivering value for their business and their customers.

My final words of advice: rules around product experimentation shouldn’t be so hard and fast as to stifle innovation. Listen, learn and ask questions about users. Rub shoulders with people from different facets of the business to truly understand your product end-to-end. 

If I’ve learned anything in my years as a product leader, it’s that good product management is a lot of science, and a little bit of art. 

Tracy Mahnken leads a diverse team of product managers and business development pros who are busy growing emerging businesses at NerdWallet. She has over 20 years of product management, marketing, research and sales operations experience in the real estate and CPG industries. Tracy has a BA in Economics from the University of Washington.

LinkedIn | Twitter