AI can’t do all your A/B testing. That’s why we built Web Experiments.

When we started looking at a no-code editor for A/B tests, I wasn't sure we should build one.

There's a reasonable argument going around that AI killed no-code experimentation. If anyone can ask AI to make product changes for testing, why would you need a visual editor to make it? I took that seriously. So before we committed, we talked to the people who'd actually use it: growth PMs, marketers, and conversion teams, mostly at consumer companies.

None of them were shipping production changes with AI.

Last week, we launched Web Experiments, our new visual editor for no-code A/B testing, along with a handful of other updates to Mixpanel. But I want to share what we learned building Web Experiments, because it says a lot about where product work actually gets stuck right now. Then I'll cover four other new product updates that make it faster to run a test and easier to trust what it tells you.

AI made writing product changes easier. It didn't touch most of what stands between a team and a live test.

Some of the people we talked to had legal or compliance limits on which AI tools they could use at work. Others didn't know the codebase well enough to point an AI at the right file, let alone get a change through review and deployment. And almost everyone said engineering time was getting tighter, not looser.

That hits small tests hardest. A copy change or a new button style is a few lines of code, which makes it the first thing cut when a sprint fills up. But small tests can matter more than you'd expect. One team we talked to kept adjusting a single sign-in button, from the text to the border width, and kept seeing bigger swings in conversion than they ever predicted. So they kept running more.

This is what Web Experiments is for. You open your live site, make a change in the visual editor, and launch it as an A/B test without filing a ticket. It runs on the same feature flags, metrics, and results as every other experiment in Mixpanel.

We kept it intentionally simple. Visual editors are great for small, low-risk changes and get brittle fast when you push them further, so we built for the tests that tend to get stuck and left complex changes to code-based experiments. There are two ways to test, and you can reach for whichever one fits. (We'll share more about the trade-offs we made in a follow-up post.)

Here's what I kept coming back to while building this: Making it easier to launch a test doesn't help much if you can't trust the result. A fast test on a messy metric isn't progress. Neither is a winning variant nobody can explain.

Other teams at Mixpanel were working on that from different angles. None of these are exclusively experiment features. They work across Mixpanel. But each one makes a real difference when you're running tests, so here's how they show up there.

Every test is only as good as the metric you're measuring. If sign-ups are tracked twice under two different event names, or nobody's sure which "checkout" event is the real one, your result is shaky before the test even starts.

AI-powered data governance cleans that up for you. It finds and merges duplicate events, archives stale ones, and fills in missing descriptions, so the metric you pick is the one you meant. You can run it in Mixpanel through our MCP server.

Hand off the busywork to Mixpanel Agent

Mixpanel Agent now does more of the hands-on work. You can ask it to set up an editable web experiment for you. It can also edit your Boards, with every change tracked in version history so you can see what changed and undo it. And for bigger requests, it works in the background while you move on.

It keeps watch after launch, too. The Agent can set up KPI monitoring so a drop in a key metric comes to you, and it can kick off root-cause analysis when something moves unexpectedly, whether that's during a test or any other day.

Say a variant wins overall but underperforms on mobile. Your data can show you that. It usually can't tell you that another team shipped a mobile login change the same week.

Context Engine, the layer that gives Mixpanel AI an understanding of your business, now connects to Jira, GitHub, and Linear. So when you ask why a result looks the way it does, the answer can account for what your team actually shipped, not just what showed up in your events. That's the difference between "variant B lost on mobile" and knowing whether it was really the variant.

As more of this work runs through AI, whether that's Mixpanel Agent or tools like Claude and ChatGPT connected through our MCP server, teams need to see what those tools did. Changes made through MCP now appear in Mixpanel's audit log, labeled as MCP and tied to the person behind them. Admins can also pull audit logs through MCP to review activity or send it to their own security tools.

For experiments, that means if an AI tool touched a flag or a metric definition mid-test, there's a record. More broadly, it's often what gets security teams comfortable saying yes to AI in the first place.

What I took away from building Web Experiments is that AI didn't remove the bottleneck. It moved it. Writing the change got faster. Getting it live, measuring it on data you trust, and understanding the result didn't, at least not on their own.

That's what this set of launches goes after. And it brings me back to that sign-in button: a few lines of code, a result nobody predicted, and a team that kept going because they could. I want more teams to have that loop.

If you've got small tests stuck in an engineering backlog, sign up for early access to Web Experiments.

And if you want the full picture of how we’re building product intelligence that works for every team, explore more here.