ASO course · lesson 42 of 60
ASO experiment reporting and rollout decisions
Produce a result memo another practitioner can challenge, reproduce and act on.
By Gabriel Machuret · Editorial methodology
Before you start: A completed experiment or a clearly labeled hypothetical result set.
Module 7: Experiment design and interpretation
Plan about 55 minutes for reading, practice and review
You will produce: Module project: a preregistered brief, results memo and owned rollout decision.
Understand the decision
Reconstruct what actually happened
Start with the original question, control, treatment, population, metric and planned rule. Then record actual dates, sample context, interruptions and deviations. Do not rewrite the hypothesis to match the result. If setup differed from the plan, explain the consequence. The report should show whether the result answers the original decision or a narrower question.
Show evidence and competing explanations
Present counts, rates, effect and method-specific uncertainty together. Note relevant source mix, release events and operational issues. A randomized design addresses some confounding, but does not excuse broken implementation or unsupported generalization to another market. A nonrandom comparison needs stronger qualifications. Include neutral or negative findings rather than selecting only favorable charts.
Turn the result into controlled action
State adopt, retain control, investigate or inconclusive, with a reason tied to the original rule. Assign the rollout owner, affected surfaces, verification and follow-up. Preserve the losing concept and learning; a treatment may fail because its promise is wrong or because the execution was unclear. The next hypothesis should follow evidence, not an obligation to make every test sound successful.
Apply the method in practice
Reconstruct the experiment before interpreting it
Start with the original question, control, treatment, audience, dates and metric definitions. Confirm that the intended configuration actually ran. List interruptions and concurrent changes. If the treatment differed from the approved brief, explain the difference before showing the result. This protects the reader from interpreting an attractive chart as evidence about a design that was never tested. Keep the original hypothesis even when the outcome is disappointing.
Report the whole decision-relevant result
Show counts where available, rates, effect estimates and the native uncertainty measure. State the statistical method or link the platform guidance. Include relevant guardrails and their measurement limits. Segment analysis can be useful, but unexpected subgroup patterns should usually be treated as exploratory unless planned and supported. Avoid selecting only the segment where the preferred treatment looks strongest. A readout should make an informed reviewer able to disagree with the recommendation using the same evidence.
Turn the result into a bounded action
Choose adopt, reject, investigate or repeat with a specific reason. State where the conclusion applies and what remains uncertain. Before rollout, recheck product accuracy, locale suitability and operational readiness. Record the person responsible for applying the treatment and verifying the live result. Close the loop by adding the learning to the hypothesis ledger. A failed or inconclusive experiment can still be valuable when it prevents repeated work or clarifies the next question.
Platform references for this work: Apple — Product page optimization results · Google Play — Run store listing experiments
Your step-by-step procedure
- Compare the original brief with the actual configuration and timeline.
- Report the primary result with counts and the appropriate uncertainty interpretation.
- List limitations, guardrails and plausible explanations not resolved by the design.
- Make a rollout or follow-up decision with an owner and verification step.
Worked example and interpretation
Trail Notes and numerical research scenarios are fictional teaching examples. Platform limits, where shown, come from the linked official references.
Trail Notes’ journal-led concept produces a promising but inconclusive native result. The readout keeps the existing control, records that the concept passed comprehension review and proposes a stronger demonstration of the journal workflow for a future test. It does not use the positive point estimate as a public success statistic. The report still delivers value by narrowing the next question and preventing an unjustified rollout claim.
| Readout section | Content | Decision value |
|---|---|---|
| Setup | Actual treatment and audience | Establishes what was tested |
| Evidence | Counts, effect and uncertainty | Shows strength and magnitude |
| Context | Deviations and guardrails | Reveals limits and tradeoffs |
| Action | Owned rollout or next investigation | Makes the learning usable |
A result summary without setup or uncertainty is incomplete. A long report without an action is also incomplete. The structure above keeps the analysis connected to the original decision and the next operational step. If evidence is inconclusive, say why another test would be different before recommending repetition. Otherwise, the team may spend another cycle producing the same uncertainty.
Build a handover someone can use
Original brief
Attach the hypothesis, configuration and exact asset versions that actually ran.
Result evidence
Preserve native reports, counts, effect estimates and uncertainty with definitions.
Validity review
Explain deviations, audience changes and limits on subgroup interpretation.
Recommendation
State the bounded decision and why the evidence supports it.
Execution and learning
Assign rollout verification or further research and update the experiment history.
Your ASO assignment
Use your own app and evidence, or work through the teaching case. Keep your observations separate from assumptions and explain the reasoning behind your decisions.
- Write a one-page readout from a hypothetical or permissioned result.
- Include an explicit section on what the test does not establish.
- Prepare a rollout checklist or follow-up hypothesis based on the decision.
Scenario challenge
Write a five-sentence readout for an inconclusive test: question, setup, observed result, limitation and next action. Avoid using “failed” as a substitute for explaining what the team learned.
Assess your work
- The readout preserves the original question and decision rule.
- The conclusion matches the method and tested population.
- The next action is specific even when the result is inconclusive.
Common mistakes to catch
- Changing the story after seeing the result.
- Publishing a best-case point estimate without uncertainty or context.
Knowledge check
A valid test wins in one locale. Does the result justify replacing assets globally?
Read the answer and reasoning
It supports a decision for the tested context. Assess language, audience, product and source differences before extending it. A staged adaptation with verification or further testing is more defensible than treating one locale’s result as a universal law.
Sources and further reading
Platform references reviewed 29 September 2026. These sources support platform capabilities and constraints; the teaching frameworks, assignments and illustrative cases are original course material.
- Apple — Product page optimization results — Native result interpretation and Bayesian analysis.
- Google Play — Run store listing experiments — Experiment configuration and native interpretation guidance.
Your ASO lesson notes
Use this space for your assignment, evidence and handover. Select Save to keep a draft in this browser, or download a copy to take with you.
Notes stay on this device and are not submitted to ASO Agency.
Complete your assignment with the ASO experiment planner.
Loading your progress…
Progress is saved only in this browser. It does not sync across devices. Mark a lesson complete after doing its exercise.