One misdesigned analytics event can bring your tracking to its knees

I ran into a hot shard in Mixpanel. It happens when one event, or a property with very low cardinality, dominates traffic and overloads a single shard. Queries slow down, dashboards lag, and the project's event budget disappears.

Root cause

One event was being sent at massive scale with almost identical properties. A batch job looped over users and fired a new Mixpanel event for each one, 340k+ events a day, all landing on the same shard.

The job was doing what it was told. The event design was the problem.

The fix

That loop should have produced one bulk event that represents the whole batch, with the count and the batch id as properties. Same information, a fraction of the writes. After the change the shard dropped by about 89%, which works out to roughly 110 million redundant events a year that no longer exist.

Takeaway

Analytics is not "just logging". Events are writes into a system with real limits, and they need the same design care as your API payloads and database tables:

• Decide the grain of each event before anyone sends it.
• Prefer one event per batch over one event per item when the item does not matter on its own.
• Watch cardinality: a property that is always the same value is a shard key waiting to go wrong.

Design your events carefully, or your data platform will remind you later.