With product data integrity, the truth is relative
While giving a talk on how to build data integrity as a product manager, Rob Versaw, the Director of Mobile Apps at Overstock.com, showed a picture of green rolling hills in front of Utah’s snow-capped Mount Timpanogos. How would you get to the top, he asked. As luck would have it, in the foreground, there was a path. To get to the summit, just take the trail – right?
If you look at the picture more closely, however, you’ll see something glimmering in the middle distance: a lake, right in between you and the mountain. “If I don’t take time to frame the problem and just start making progress towards the summit, I’m going to end up very wet or needing to backtrack,” Rob noted.
Establishing data integrity has never been more important for product teams – and it’s also never been harder. In fact, you’re liable to take an unexpected bath and lose your way if you don’t have a detailed plan that anticipates as many hurdles as possible.
Of course, as a former airman in the Air Force and engineer at Lockheed Martin, Rob is no stranger to complex problems that require careful attention to detail. Whether figuring out how to make airplanes fly or patio furniture fly off the e-commerce shelves, wrangling data has often been an essential component for success.
And now, as Rob’s team works on apps across different devices, operating systems, geographies and more, it’s even more critical that they not only trust the data but also understand what it says about the consumers they’re trying to help.
At Mixpanel Office Hours, Rob shared six detailed steps — or, as he phrased it, “six conversations you must have” — for building data integrity, and you should watch the full video here.
But, if you’re short on time because you have your own data mountain to climb, here is one key insight you can bring to your team now and avoid an unexpected dip in the lake.
With data, the truth is relative.
“A data strategy must include an agreement on truth,” Rob said. Most people (including, probably, that executive who keeps pestering you about “being data-driven”) think that analytics is supposed to lead you to “the truth”, but those in the trenches like Rob will tell you it’s the other way around.
Data can be made to tell a lot of stories, depending on factors like which data source is looked at or which model is run. If everyone looks at different analyses and stories, it’s difficult to determine which story is the right one or whose story to believe and act on.
So, why doesn’t everyone just agree on the truth in the first place? Because that agreement always involves trade-offs. In Rob’s experience, there are three main trade-offs: consistency, accuracy, and reliability. You can definitely have one, probably two, but certainly not all three. To determine which areas are most important to you, here are some good questions to ask your team.
Consistency: Is it important to maintain consistency across all data sources? Should there be a “single source of truth” for data that everything else must match?
Accuracy: Do I need exact numbers, or will trends suffice? Do fractions or minute optimizations matter, or are we looking for orders of magnitude difference?
Reliability: Have we agreed on a place to benchmark against? Do you need your data every day, every hour, or every minute? Do you value speed or dependability?
At Overstock.com, the Product team’s metrics are based on year-over-year trends, not absolute numbers—that’s the truth they’ve all agreed to. They’ve traded off “accurate” data for very reliable and consistent data, instead. In other words, so long as they can compare year-over-year trends, even if the absolute numbers are different, then “we’re okay with that.” Rob stressed that every team and business would have different trade-offs; getting to an agreement on your truth and what you’re comfortable trading off is the important thing.
Returning to the big hike on Mount Timpanogos: if you want to avoid the lake in the middle, you probably need a map. Before starting, though, everyone on your team should agree on the map’s common unit of measurement: kilometers or miles. It doesn’t really matter which one you choose (sorry, geography fanatics); rather, it’s the initial agreement that’ll give your hiking party a greater chance of success of reaching the summit as a team.
Figuring out your team’s data truth isn’t easy, but it is necessary for success. If you’re ready to start your own journey or get back on course, coming to an agreement on what your organization values and what it can de-prioritize is vital. Of course, that’s only one out of the six conversations Rob recommends product teams have to build data integrity. To get the full scoop, watch Rob’s 35 minute presentation here.