The pitfalls of being data-driven

In 2006, Facebook rolled out its News Feed feature–and boy, did people hate it.

The uproar among the website’s (then) small user base of 12M was big enough to draw headlines. Speaking about the rollout in 2010, Facebook executive Adam Mosseri said his team stuck to their guns because their experience and intuition told them that it was a good strategy – even in the absence of supporting data, and with a backlash from users.

And their conviction certainly paid off: News Feed became the primary driver of traffic and engagement on the site and helped Facebook grow to over 2B users globally.

This story illustrates the importance of being data-informed rather than data-driven.

Data-driven vs. data-informed: What’s the difference?

When you conduct any sort of research, you’ll typically draw on a combination of data and experience (yours and that of your peers). Leaning too heavily on personal experience and intuition leads to biased results, which is why academics in any field back their research with data and the established findings of their peers.

Of course, academic research takes many years, if not decades, and business decisions must often happen quickly. It’s easy to give data much more weight in the decision-making process, prioritizing it over experience and intuition. This is what’s called being data-driven.Being data-informed means finding a happy medium between experience and data, and making decisions by combining the two. Below are a few reasons why this approach leads to better decisions and, ultimately, better products.

Data is not the only variable

Think about how you typically book a flight. Do you book based on price alone? Of course not. “Sweet deals” that require an overnight stay, charge a fee for checking bags, or depart from a distant airport aren’t actually that sweet when you add up the cost of such variables. Like most travelers, you rely on price among many other variables to determine which itinerary to book.

In the same way, product design should be influenced by more than just data. Ask yourself and your team: What is the business environment like? Has there been positive or negative user feedback for your product? What are your competitors doing–and are you keeping up? While considering these things might seem obvious, it’s important to make sure they aren’t undervalued in favor of data.

Beware of lagging indicators

Putting absolute trust in data can sometimes run the risk of relying on the wrong metric. You may end up making decisions based on lagging indicators that are easier to measure but don’t actually tell you if you’ve made a sound decision until after the fact.

Let’s say that you want to launch a new product feature, but early user feedback suggests that it makes the product difficult to navigate. You run a quick A/B test over a week and see no change in sessions, so you push the update live. Over the first few weeks the churn rate remains stable, but after a few months it increases–and reviews suggest that the update is to blame.

What happened? The session data from the A/B test was a lagging indicator, and a week was not long enough to truly assess the effect of the update. In this case, the initial negative user feedback was a leading indicator and should have been given more consideration before launching the update.

Achieving a local maximum

Robert Moore, founder of data analytics company RJmetrics (now owned by Magento), wrote in 2014 about how they chose the headline for their first website:They let the data decide. SEO research showed that the term “eCommerce analytics” had high traffic and low competition, so they decided to go with it. However, they failed to consider how that choice would ultimately impact the identity of the business. Over the next year, their audience skewed heavily toward eCommerce, alienating major segments of their market. One data-driven decision led to an outcome that took years to correct.

This is what’s referred to as achieving a “local maximum” — an outcome that’s an improvement but fails to optimize for long-term business goals.

Taking a data-informed approach means making decisions based partly on data, but within a larger context of the market, user preferences, and competitor benchmarking in order to understand the full landscape of product possibilities and business impact. Becoming too narrowly focused on specific metrics can lead to missed opportunities or expensive course corrections.

Data-informed approach to product design

Data-driven teams that doggedly focus on optimization can get stuck on local maximums because they give up creativity in pursuit of data-driven decisions. The idea that data can inoculate you from the inherent risk that comes with creativity is tantalizing, but it’s a false promise.

There often comes a point in a product lifecycle where iteration isn’t enough, and only a disruptive change can take your product to the next level. Such disruption can’t happen without creativity-enabled innovation. And steamrolling great ideas because “the data said so” might feel safe and pragmatic, but only quashes your product’s true potential.

Dating app Hinge understands this well. By combining quantitative data from Mixpanel user analytics with qualitative data from focus groups and surveys, Hinge redesigned the app to improve its matchmaking. The data showed that endless swiping didn’t lead to many dates. That insight informed a new product focus on connecting people for meaningful relationships, not casual dates. As a result, the app moved away from the wildly popular profile-swiping feature and introduced expanded user profiles and “content liking” so users could better reveal their personality. This data-informed redesign helped with market differentiation for Hinge as the app focused on long-term relationships, not hookups.

Data-informed > Data-driven

Being data-informed is ultimately about striking a balance between experience and metrics that helps your team draw insights from data to spark their creativity and innovation. Being data-informed empowers your team to deliver valuable products that are backed not only by data, but also by the valuable and unique experiences that they bring to the table.