Experimentation terms 101: Health checks that make your A/B test results trustworthy

Before trusting what an A/B test reveals, it’s crucial to make sure the experiment itself is set up correctly. A result may look statistically significant while bias, noise, and tracking bugs actually drive it. In reviewing experiments run in Mixpanel over the past year, we noticed that health checks frequently surfaced problems with setup (like broken randomization or tracking), yet—if the experiment moved forward anyway—the experiment still produced a significant result. 

It’s a reminder that without experiment health checks, even seemingly meaningful tests can lead to misleading conclusions. They act as guardrails that help you validate, stabilize, and debug your experiments, ensuring your product decisions are based on trustworthy results rather than statistical accidents.

Keep reading to learn about key experimentation terms, how they work, and why to use them, including: 

The five health checks below will help you validate your experiment so you can trust your results.

A sample ratio mismatch check ensures that your experiment is actually fair and each variant receives the right share of users.

SRM happens when the actual number of users in each variant differs from the originally planned split in a statistically significant way. For example, you planned a 50/50 test, but one group somehow ended up with 70% of users, which may indicate that the randomization process is broken, making the results unreliable. 

In a properly randomized experiment, user counts should closely match the planned split, aside from small random variation. SRM refers to differences that are larger than what randomness would reasonably explain.

SRM often happens because of mistakes in user bucketing, differences in the start times of each variant, or technical issues like broken redirects.

SRM can be a silent killer for experiments. If you haven't distributed users evenly:

Whenever you run an experiment that splits users into multiple variants, but especially when you’re:

Winsorization stabilizes metrics that can be skewed by extreme values, making results less sensitive to extreme users.

Winsorization is a way to handle extreme values (outliers) without removing them entirely by capping values at a certain threshold. You don’t remove extreme users; you simply replace their value with a maximum (or minimum) threshold.

Imagine you’re measuring the average number of messages sent per user in a social app. Most users send five to 15 messages per day, but a few power users send hundreds. Without winsorization, those extreme users would inflate the (mean) average, making it look like typical users are way more active than they really are. 

Most products have some users who behave differently from everyone else, which might skew metrics like revenue per user or session length.

If you’re running an experiment, these users can also skew the results by making it seem like your experiment is having a bigger or smaller effect than it really is. Winsorization reduces the impact of these outliers while preserving the sample size, which provides a more realistic view.

Note: While winsorization improves stability, it does change the definition of success. If extreme behavior is part of what you're testing, using it can actually undermine your results. For example, say you’re testing a pricing change: a few high-spend users are on the “extreme” end of purchases, but that isn’t noise—it’s an indication of opportunity. 

CUPED is a variance reduction technique that reduces noise in an experiment, making results more precise and easier to interpret. This helps you detect whether the experiment really had an impact, without needing huge sample sizes or long test periods.

CUPED works to “level the playing field” by accounting for how users were already behaving before the experiment began using historical, pre-experiment data. You adjust for prior user behavior and then measure how the actual result during the experiment differs from that baseline. By controlling for how users were already behaving, CUPED helps isolate the true impact of the experiment.

This reduces random variation and makes it easier to detect a real effect. Imagine you’re testing a new onboarding flow and measuring activation. Even before the experiment launches, your users all have very different baselines: some already complete those actions almost every time, while others might rarely get past step one.

If you just compare raw activation rates, those built-in differences can hide a small but real lift from the new flow. Importantly, CUPED doesn’t change how users are assigned to variants; it only adjusts how outcomes are measured.

Use retrospective A/A testing to validate your measurement and experiment setup. Run an A/A test when you want to be confident that differences in your experiments are valid and not the result of broken tracking or setup errors.

An A/A test compares two identical experiences, with the goal of confirming that any observed differences are consistent with normal random variation.

A/A tests help ensure your system isn’t generating unexpected or inflated differences. When two identical groups show unexpected differences, it can indicate tracking bugs, timing issues, or inconsistencies in how you define audiences. By catching these problems early, A/A testing gives you confidence that future A/B test results really reflect the impact of changes you made rather than errors in data collection or experiment setup.

The Bonferroni correction reduces the probability of a false positive when you’re looking at a large number of metrics, variants, or audience segments at once.

When you run many tests simultaneously with multiple metrics, segments, or variants, the chance of seeing a “significant” difference by pure luck increases.

The Bonferroni correction is a simple way to adjust the statistical threshold so the overall risk of false positives stays under control when making multiple comparisons. More tests mean more chances for randomness to look real; Bonferroni adjusts for that.

For example, if you’re testing 10 metrics at the standard 0.05 significance level, one might appear significant just as a fluke. The Bonferroni correction divides the threshold by the number of tests (0.05 ÷ 10 = 0.005), so only results that meet the stricter cutoff are considered meaningful.

Even seasoned teams can be tempted to skip health checks, especially when experiments feel urgent, timelines are tight, or executives are pressing for quick results. Health checks are a safeguard against shipping the wrong insights under pressure, and they’re especially important when:

These checks aren’t about second-guessing your experiment; they’re for verifying that the results are trustworthy. Think of this like a pre-flight checklist before making a product decision.

Before trusting your experiment’s results, ask:

If not, stop and investigate further (e.g., bucketing or form IDs).

If it’s unclear, try comparing effect sizes with and without outlier treatment, or re-run the analysis with winsorization or more robust metrics to understand how sensitive your results are to extreme values.

If you have past data available, apply CUPED (if it was defined in advance) and reassess your effect size and confidence intervals. Otherwise, document why and move on.

If not, consider running a retrospective A/A test if infrastructure issues are suspected. You don’t have to do this every time, but it’s good to confirm nothing infrastructural could be wrong.

If you’re testing multiple metrics, variants, or audience segments as separate hypotheses, apply the correction and re-evaluate the results.

Most experiment failures are preventable, especially when they come from hidden noise, bias, or measurement problems.

Before launching your next test, take some time to run through these health checks. They’ll help you catch problems early and separate real product insight from false signals, so you can trust what your data is actually telling you.

Learn more about experiments in Mixpanel today and turn every result—win or loss—into a deeper understanding of your product and your users.