Product development and experiments
Testing is an essential part of life. We touch the water with our toes before
we jump in. We taste a piece of new food before we eat the entire meal. We
test drive cars and skis before we buy them. In general, we try to collect
information when it is available to make future decisions about larger
investments. In some ways, measuring cause and effect is easy: When you can
hold “everything else constant,” its easy to determine that a new pair of
sneakers gave you blisters and a shampoo you used gave you dry scalp and not
the other way around. But in other cases it is more difficult: Was it the
coffee or the wine or the soda you drank yesterday (or a combination of all
three) that gave you a stomach ache last night?
When building software that is used by many customers the same problem
presents itself. Was it the new algorithm or the new module that generated an
increase in average engagement for users? Did the new user tutorial or the
coupon offer drive a higher conversion rate? One way to isolate the effects
from these potential sources is to introduce them one at a time and see what
measures change. But this is not realistic: other businesses who don’t test
this way will move much faster and a subset of these will beat their
competition. Also, pushing new features out one at a time does not account for
hourly, weekly or seasonal effects.
What do you do then when you need visibility on the impact of product features
and insight on where to take the next steps? The answer: experiments.
Experiments are actually ideally suited for rapid development in the web and
software industries that are filled with a large number of fluid users. Why?
- It is easy to segment users randomly based on a numerical identifier (by
taking the modulus of that identifier for example). - Software and particular features are easy to execute on those segments (if
id meets a condition then show new feature). - It is easy to measure results and compare segments if you are logging raw
data of user behavior.
Employing the scientific method ensures the credibility of the experiment.
This means forming a hypothesis, implementing a control group and collecting
the right data to make the measurements you need to draw conclusions. However,
in competitive markets, there is not much room for retesting and verifying
results as the scientific method encourages. In other words, a compromise
between experimental verification and intuition is a far more successful
strategy for business than intuition alone.