Feature flags and product analytics

Experimentation is a huge part of getting your product right, whether it’s trying out a new feature or potential tweaks to existing elements of your app. When it comes to running these kinds of product tests, using feature flags can be a really effective and minimally disruptive way to go.

We spoke to David Martin, Senior Solution Engineer at Split, about how feature flags open up dynamic, low-risk possibilities for product experimentation—and, thanks to Split and Mixpanel’s integration, without leaving data holes in your valuable product analytics.

Feature flags have been used by engineers since prehistoric times, even before Martin Fowler popularized them. Early flags turned snippets of code on and off, usually with configuration or properties files and sometimes in the database. Using a feature flag meant you could engage in trunk-based development.

When it comes to experimentation through feature flags, the big selling points are being able to easily turn a feature on or off for a specific cohort of users, sometimes as small as 1% to start, in order to minimize the disruption to your overall user base.

Manual approaches to feature flag implementation can require a lot of engineering work, and that’s even more true when it comes to building the kind of segmentation and targeting that goes into deploying experiments with them. Modern shops want to do these things to be competitive, but there’s a mounting technical debt associated with building it yourself. Testing with feature flags via a platform like Split reduces technical labor and implementation time.

I’m very fond of the story of Imperfect Foods. A product manager over there explained to me that she reached for feature flagging to be able to safely “shoot the moon.” What she did was make changes in their interface for a section of their user base to test out some really radical improvements—without fear that that radical change would have a rippling negative impact across her business.

Those really big audacious changes worked. From there, Imperfect Foods was able to leap forward in terms of their usability.

Now, this is a tech company that’s helping you pick tomatoes, cucumbers, and kale. But they’re no different than a lot of Split customers in that they’re facing intense competition from other companies in their markets. That means they can’t afford to make a mistake and push a radical change that isn’t received well by their customers. Meanwhile, spending time and resources conservatively iterating on something that might not impact your users’ experience in a positive way could wind up being a mistake in itself.

With modern product analytics, you know everything about your funnel, retention rates, conversions, and more. When you deploy feature flags in your funnel, you’re essentially skewing a percentage of your traffic. So while you’re on your way to uncovering bugs or user experience hiccups, now your analytics are confused, and it wouldn’t be incorrect to say you’ve created two funnels instead of one.

To achieve clarity, your feature flagging solution must tell the analytics about the customer experience. Split and MixPanel deftly fuse the feature flagging impression information with MixPanel’s premier analytics, meaning you can engage in not just advanced feature flagging, but advanced analytics.

The under-the-hood way it works: When a user gets a feature flag in Split, it’s called an impression. What we do with Mixpanel is we have a webhook that can, in batch, take impressions and transform them into “experiment started” events that get the payload of the impression. And that’s what gets passed over to Mixpanel. Then those events are actionable in Mixpanel, meaning you can build cohorts out of them and do the rest of your analysis.

There are often two kinds of roles in Split: the developers and the product owners or product managers. And those two roles have dramatically different paths through the product.

The developer tends to touch more of Split because they have to touch down at the ground level to see that the impressions are coming in. They’re typically configuring the feature flags. But what’s nice in Split is that the interface is amenable to product owners being able to make some configurations to feature flags, as well, and we have approval flows where people must check their changes before they’re accepted, preventing any mishaps.

But, since the product owners are the ones who are going to be consuming any testing results, we have accessible reports that can be taken off Split and passed around on Slack or in email. So not everybody has to be able to configure Split in order to engage in the results that come from it.

I’m usually asked how painful it is to get started. The good news is that it can be as modest an undertaking as you’d like. That’s because you can start by putting just one feature under flag.

My advice is to start with whatever you’re already making code changes to at the time. Put a flag in, test it, and get a feel for the process with that one flag. If it’s a good experience, you’ll work toward a policy of putting every new feature under flag.

David is a Solutions Engineer for Split Software. He enjoys programming in serverless Node.js these days, even though he’s an old-time Java programmer at heart. He is proud of his many Split product integrations, especially MixPanel. In his spare time, he likes writing fiction and photographing his family and pets.