Earlier this week, a vulnerability in the popular traffic security technology, OpenSSL, was exposed by security researchers. While Mixpanel has no evidence of a breach, we took immediate action to fix the issue and maintain the integrity and security of our customer’s data.
The “Transport Layer Security” or TLS provides an encrypted connection between devices and web servers, which one might recognize as the “HTTPS” in URLs or lock icon in a web browser.
The Mixpanel engineering team closed any exposures by working along the following timeline (PDT):
- April 08 10:00 – Identified affected hosts, developed a procedure for patching and restarting.
- April 08 10:30 – Patched and restarted status.mixpanel.com
- April 08 11:30 – All application servers (mixpanel.com) patched and restarted.
- April 08 13:00 – Started patching and restarting all api.mixpanel.com endpoints (Dallas, San Jose, and Amsterdam) and other affected servers.
- April 08 14:00 – New SSL certificate deployed to application servers (mixpanel.com)
- April 08 17:00 – Finished patching and restarting all api.mixpanel.com endpoints.
- April 08 17:00 – New SSL certificate deployed to API servers (api.mixpanel.com)
- April 08 21:00 – Other servers using OpenSSL (mail, data export API) were updated (these were not vulnerable but updated anyway)
We deployed new SSL certificates to all servers because these could have been compromised, which could allow 3rd parties to undermine secure connections. We are working with our SSL certificate provider to revoke any potentially compromised keys at this time, and once this process is complete, we will update this post the reflect this. Update: potentially compromised SSL Certificates were revoked on Apr 9 20:09:32 2014 GMT
Finally, all mixpanel.com user sessions were expired today (April 9 2014) at 1:20pm Pacific Time. After that time, you will be asked to log back into Mixpanel, and we encourage users to update to a new, strong password (in the Account > Password tab).
As always, feel free to email email@example.com with any questions or concerns.