How Popsa Experiments With Confidence Across Millions Of Users
Company
Popsa is an app-first photo memory curation platform that harnesses the power of AI to help everyday people make sense of overwhelming photo libraries – privately, automatically, and without a single photo ever leaving their phone. Despite being a small team based in London, Popsa ships to more than 50 countries in 12 languages, with over half of revenue coming from non-English-speaking markets. With the power of Mixpanel, Popsa’s team moves fast without gambling on any single release: naming discipline keeps the data trustworthy, and cohort testing keeps every rollout precise before it reaches everyone.
Challenge
Popsa now operates at a scale — millions of customers, 50+ countries, a fast-growing Android user base with a near-limitless spread of devices and OS versions — where a single bad rollout can’t be caught after the fact. Native apps make that harder: unlike web, every feature flag or experiment has to ship inside an app-store release, so there’s no instant rollback if something goes wrong. And as a consumer retailer with seasonal swings like Black Friday, the team can’t just look at last week’s numbers and assume they’ll hold.
We couldn’t imagine how any team of our size could reliably run accurate testing across all of those different device typesOliver McQuitty Product Director, Popsa
Solution
To operate safely at this scale, Popsa leans on Mixpanel’s cohorting to test precisely instead of broadly — targeting specific devices, operating systems, and processing tiers, which matters enormously given the sheer range of hardware that runs on Android.
Results
1. Rolling out impactful releases with confidence
Across millions of users, a sudden change to the product experience could easily have hurt average order value or conversion if it went wrong, so Popsa needed to validate it before any wide rollout. The team rewrote one of the on-device photo models behind its Memories suite of features and tested it against the old version: cohorted to the specific devices, operating systems, and processing tiers the new model needed to reach, so the test population matched the right customer base. Using Mixpanel’s segmentation feature to parse the results by device type, processing power, and photo library size, the team confirmed the lift held consistently across every segment — no single group was quietly underperforming behind an average. The result: a 17% lift in feature discoverability, with no drop in performance, AOV or conversion. So Popsa made it the default and retired the old model entirely.
We increased the discoverability over the course of the experiment by 17% . . . because the data was consistently positive, we decided to make that new version default to all new versions of that app, and eventually retire the old one entirely.Oliver McQuitty Product Director, Popsa
2. Treating flat and negative results are directional wins, not setbacks
At Popsa, there’s no such thing as a wasted test. A hypothesis that turns out wrong is still an answer: it tells the team not to spend engineering time rolling out a change that wouldn’t have worked, before that cost is ever incurred. When a test moves the wrong way, controlled exposure means the team can go back to the drawing board without real damage — the “negative” result is just as valuable as a lift, because it’s still telling them exactly where to spend their effort next.
Some of those tinkering experiments have just shown us this is having the opposite effect. . . we go back to the drawing board. That's what those experiments allow us to do without a really material negative impact.Oliver McQuitty Product Director, Popsa
3. Meticulous data discipline, powered by Mixpanel governance
Every event and feature flag at Popsa follows enforced naming conventions, meaning they can implicitly trust their cohorts and the results of any test or rollout.
Any event that doesn’t meet the required syntax or is missing a description or owner gets flagged for cleanup before it pollutes the lexicon. Events can also be verified, meaning anyone building a report or browsing the data can see at a glance that it’s been vetted rather than guessing whether to trust it. And with Event Approval switched on, no new event goes live to the wider team the moment it’s ingested — it sits pending until an admin reviews and approves it. That governance layer means the team can trust its data completely once a change is live in front of millions of users — there’s no chasing down which version of an event fired, or reconciling conflicting definitions across teams.
The bottom line
Mixpanel is what lets a lean team ship with confidence: cohort testing keeps every rollout precise before it reaches everyone, and governance keeps the underlying data trustworthy enough to act on without a second guess.
