Post-mortem: Data inconsistency on July 19, 2016
What happened and why
What might this look like in your Mixpanel projects? Here’s a breakdown of what you might see in your reports during the date range if your users were affected:
- Segmentation might show a spike in uniques from new distinct_ids.
- More users might have completed the first step in a funnel, with far fewer users having completed subsequent steps.
- There could be a much higher initial cohort in Retention with lower rates of retention.
- It’s also possible you might have duplicate people profiles, depending on your use of mixpanel.identify(). The best practice is to always pass through your unique identifier to the identify() calls to ensure that activity and people properties are set on the correct distinct_id. However, if your implementation has people.set() calls with an empty identify() call on the same page, the property would be set on the distinct_id that existed in the cookie at that time, thus creating a people profile. If the cookie reset during the user’s session, any subsequent people.set() and identify() calls would reference a different distinct_id.
You can check the impact on your project by adding a series of filters to see every event that was affected by the bug during the date range of July 19, 2016 – July 22, 2016. Go to the Segmentation report, view Top Events, and then add the following property filters: “Mixpanel Library” equals “web”, AND “Library Version” contains “2.9.” AND “Library Version” does not equal “2.9.4”.
We care deeply about data accuracy, and we apologize for the inconvenience. In the immediate term, we are strengthening our automated tests and casting a wider net by running multiple configurations of our library while dogfooding our changes. In the long term, we’ll focus on minimizing impact when making big changes by performing incremental rollouts, as well as considering creating a new release instead of automatically updating the library. If you have any questions regarding this incident, please don’t hesitate to reach out to Mixpanel Support at email@example.com.