A metric drops on Monday and the question lands with you. Before anyone opens a dashboard, decide what kind of drop you are looking at. Most of the time it is one of four things, and each one has its own first check: the measurement changed, one segment moved, a release broke a step, or customers started saying something new.
This guide is the order to work in. You can run every step with the tools you already have. The last section shows where the same steps land in adduce.
1. Rule out a measurement change
The cheapest explanation is that nothing happened to users. Check whether the event still fires, whether a release renamed a property, and whether the date range compares like with like.
- Does the raw event count drop with the metric, or only the ratio?
- Did a release ship within a day of the drop? Read its diff for the event name and the property names it sends.
- Is a holiday, a weekend or a time-zone change inside the window?
- Did anyone change a filter, a cohort definition or a dashboard formula that week?
If the raw count is flat and only the ratio moved, you have a denominator story, not a user story. Write that down and move on.
A ratio can fall while the numerator stays flat. If sign-ups held steady and visits doubled, conversion halved and nothing is wrong with sign-up.
2. Find the segment that moved
Break the metric down by the four or five properties that split your users: platform, plan, country, acquisition channel, release version. A real change is rarely even. One segment carries most of it, and the rest barely move.
| Breakdown | Before | After | Change |
|---|---|---|---|
| Web · all | 64% | 52% | −12 |
| Web · returning | 71% | 70% | −1 |
| Web · new, mobile | 58% | 31% | −27 |
| App · all | 66% | 65% | −1 |
The table above is the whole job of this step: a drop of twelve points overall is a drop of twenty-seven points in new mobile visitors on the web, and almost nothing anywhere else. You now know where to look, and you know what not to touch.
Keep going until a breakdown stops splitting. If new mobile visitors on the web drop evenly across every country and channel, the segment is "new mobile web", and the next step is about what that segment saw.
3. Line up the releases and the errors
Put the drop on a timeline with every release and every new error group from the same window. A change that starts within an hour of a deploy, in the segment that got the deploy, is your first hypothesis. A change that starts a day before the deploy rules that deploy out, however tempting it looks.
Use the error tracker the same way. A new error group that first appears in the same hour, on the same platform, with a message that names the step where the metric is measured, is evidence. An old error group that happens to be loud this week is noise until the timeline says otherwise.
Write each candidate as a sentence you could be wrong about: "The address step started rejecting postcodes with a space after the 14:10 deploy." A hypothesis you can test is one you can also drop.
4. Read what customers said that week
Support conversations and call notes from the same days name the problem in the customer's words. Search them for the step where the segment leaves, not for the metric. Nobody writes in to say conversion fell; they write in to say the form will not accept their postcode.
A matching thread is a lead; threads from the same segment, in the same days, describing the same step, are evidence for a hypothesis. Keep the links to the conversations: the write-up will need them, and so will the engineer who fixes it.
| Hypothesis | Verdict | Evidence | Confidence |
|---|---|---|---|
| New mobile visitors hit the address step and leave | Supported | The segment table, the error group that first appears at 14:12, and the support threads about the postcode field from the same week | High |
| The price change on Tuesday put people off | Not supported | Returning visitors saw the same price and their rate did not move | Low |
5. Write it up with the evidence attached
State the hypothesis, the segment, the evidence for it, the evidence against it, and what you still do not know. Say which of your hypotheses the evidence supports, which it does not, and which you could not settle. A write-up that shows its sources is one your team can act on without re-checking every tool, and one you can come back to when the next drop looks the same.
Keep the write-up short. One paragraph per hypothesis, one line per piece of evidence, each with a link to where it lives. The reader should be able to open any claim and see the chart, the error or the conversation behind it.
The checklist
Run this in order. Stop at the first step that explains the whole drop; keep going when it explains only part of it.
- Confirm the raw event still fires and the window compares like with like.
- Break the metric down by platform, plan, country, channel and release until one segment carries the change.
- Put releases and new error groups from the same window on one timeline, in that segment.
- Search support and call notes from the same days for the step where the segment leaves.
- Write one sentence per hypothesis, mark each supported, not supported or open, and attach the evidence.
- Name what you still do not know and what would settle it.
In adduce
Where this lands in adduce.
An investigation tests each explanation against the records you connected, cites what it used, and says plainly when the cause is still unknown. You read the case, not the dashboards.
