What assumptions are you making?
If you aren’t measuring something, how can you improve it? As engineers and entrepreneurs, we often think that we know best. This confidence is vital to product development, but it’s also necessary to recognize the fact that we make a lot of assumptions – especially if you aren’t using analytics to make informed decisions. Every time you make a change to your product without metric data, you are assuming that you know best. How was the product doing before? Can’t be sure. How is it doing now? Still can’t be sure. Every business endeavor requires you to make some assumptions, but it is critical that you validate those assumptions with data. If you find the data doesn’t back you up, it’s a sign that you need to rethink your plans. Andrew Chen writes about startups and analytics, where he poses this question:
Question: Is it better to build 10 features where you don’t know what worked and what didn’t, or is it better to build LESS features but have a clear sense for what and why something worked? In my opinion, you want to learn as much as you can so you can “run up the score” on the features that work.
It is absolutely crucial to understand what aspects of your product are working so that you can focus your efforts in the most productive way. If you take a little time to add some metrics to what you are building, you will immediately see a few benefits – you can validate those assumptions, and you can start pinpointing bottlenecks and key problems. Once you decide that you need metrics, the problem of implementing them remains. In that same article, Andrew Chen also describes startups who build their own internal metrics:
As a rough estimate, I’ve found that it takes between 25-40% of your resources to do analytics REALLY well. So for every 3 engineers working on product features, you’d want to put 1 just on analytics. This may seem like a ton (and it is), but it throws off indispensable knowledge that you can’t get elsewhere, like:
- Validating your assumptions
- Pinpointing bottlenecks and key problems
- Creating the ability to predict/model your business to make future decisions
- It tells you which features actually are good and what features don’t matter
Metrics as a “product tax”
In fact, one way to view analytics is that they are a double-digit “tax” on your product development process because of a couple things:
- It takes engineers lots of time and development effort
- It produces numbers that people argue about
- It requires machines, serious infrastructure, its own software, etc
- Fundamentally, it slows down your feature development
So, it’s decision time: it’s clear that analytics give you crucial insight into your business, but they come at a dear cost – up to 40% of your resources to implement internally. What do you do? We founded Mixpanel hoping to answer this question.