Verification
Part of Advertising clean rooms
Evaluating clean room results without overstating causation
Separate matched and attributed outcomes from incremental effects by checking populations, comparison design and uncertainty.
A clean-room report can show that permitted exposure and outcome records matched under specified rules. It cannot show, on its own, that advertising caused the outcomes. An incremental-effect claim needs a credible estimate of what would have happened without the campaign, plus an account of uncertainty.
Name the measure
| Measure | What it describes | What it lacks for a causal claim |
|---|---|---|
| Matched outcomes | Outcomes found in the permitted joined population | A comparison with outcomes absent the campaign |
| Attributed outcomes | Outcomes assigned to an ad by a stated rule and window | A counterfactual for comparable people |
| Incremental outcomes | An estimated campaign effect from a suitable comparison design | Full certainty; the estimate still has uncertainty and design limits |
Google’s documentation includes a user-based Conversion Lift measure. It can help identify the measure in a report, but a clean-room query does not automatically create a valid lift study.
Audit the observed population
Record who could enter each source, which identifiers joined, and which rows the output controls withheld. An ad opportunity, a delivered impression and a viewable impression are distinct exposure definitions. An order count, a distinct purchaser count and revenue after returns are distinct outcomes.
A report may omit unmatched people, unsupported inventory, purchase channels not supplied, and groups filtered for privacy. Describe findings as applying to the measured population. Treat unavailable groups as unknown, not zero. Do not project a matched rate to the whole campaign without a justified method.
Key Metrics and Considerations in Clean Room Reporting
- Exposure definitions
- Ad opportunity, delivered impression, viewable impression
- Outcome types
- Order count, distinct purchaser count, revenue after returns
- Privacy filters applied
- Groups filtered for privacy; treated as unknown, not zero
Examine the comparison
For a planned holdout, identify the unit assigned to treatment or control, the assignment method, possible exposure of control units elsewhere, and whether outcomes were captured comparably. Set the principal outcome and observation window before seeing the result. Record other campaign or market changes that could affect the groups differently.
Random assignment can help create comparable groups, but delivery and measurement still matter. A simple before-and-after change or exposed-versus-unexposed comparison is descriptive unless its assumptions and alternative explanations are addressed. People selected for exposure may already differ from those not exposed.
Report the claim the design supports
Present the population, query and output-rule version, outcome window, available treatment and control counts, estimated difference and uncertainty. Include exclusions and suppressed cells. If the design supports only association, say “sales recorded after matched exposure” rather than “sales generated by the campaign”.
A review may reasonably conclude there is insufficient evidence of lift. State whether the obstacle is the comparison design, measurement coverage, precision or a result the product cannot release. That is more useful than relabelling an attribution total as incremental sales.
