About 19 minutes reading · 36 minutes practice
You will produce: Module project: a preregistered brief, results memo and owned rollout decision.
Reconstruct what actually happened
Start with the original question, control, treatment, population, metric and planned rule. Then record actual dates, sample context, interruptions and deviations. Do not rewrite the hypothesis to match the result. If setup differed from the plan, explain the consequence. The report should show whether the result answers the original decision or a narrower question.
Show evidence and competing explanations
Present counts, rates, effect and method-specific uncertainty together. Note relevant source mix, release events and operational issues. A randomized design addresses some confounding, but does not excuse broken implementation or unsupported generalization to another market. A nonrandom comparison needs stronger qualifications. Include neutral or negative findings rather than selecting only favorable charts.
Turn the result into controlled action
State adopt, retain control, investigate or inconclusive, with a reason tied to the original rule. Assign the rollout owner, affected surfaces, verification and follow-up. Preserve the losing concept and learning; a treatment may fail because its promise is wrong or because the execution was unclear. The next hypothesis should follow evidence, not an obligation to make every test sound successful.
Quick check · pick a response
Make the call before you scroll.
Go deeper: practical decisions and tradeoffs
Reconstruct the experiment before interpreting it
Start with the original question, control, treatment, audience, dates and metric definitions. Confirm that the intended configuration actually ran. List interruptions and concurrent changes. If the treatment differed from the approved brief, explain the difference before showing the result. This protects the reader from interpreting an attractive chart as evidence about a design that was never tested. Keep the original hypothesis even when the outcome is disappointing.
Report the whole decision-relevant result
Show counts where available, rates, effect estimates and the native uncertainty measure. State the statistical method or link the platform guidance. Include relevant guardrails and their measurement limits. Segment analysis can be useful, but unexpected subgroup patterns should usually be treated as exploratory unless planned and supported. Avoid selecting only the segment where the preferred treatment looks strongest. A readout should make an informed reviewer able to disagree with the recommendation using the same evidence.
Turn the result into a bounded action
Choose adopt, reject, investigate or repeat with a specific reason. State where the conclusion applies and what remains uncertain. Before rollout, recheck product accuracy, locale suitability and operational readiness. Record the person responsible for applying the treatment and verifying the live result. Close the loop by adding the learning to the hypothesis ledger. A failed or inconclusive experiment can still be valuable when it prevents repeated work or clarifies the next question.
Platform references for this work: Apple — Product page optimization results · Google Play — Run store listing experiments
Your turn · two-minute exercise
Try the scenario. Make your case.
Write a five-sentence readout for an inconclusive test: question, setup, observed result, limitation and next action. Avoid using “failed” as a substitute for explaining what the team learned.
Autosaved on this browser. Your work is private and does not sync between devices.
Your step-by-step procedure
Compare the original brief with the actual configuration and timeline.
Report the primary result with counts and the appropriate uncertainty interpretation.
List limitations, guardrails and plausible explanations not resolved by the design.
Make a rollout or follow-up decision with an owner and verification step.
Completed example · use the reasoning
A complete experiment decision note
Gabriel’s take: “Promising” is not a decision. It is what decks say when nobody wants to call it.
Constructed teaching material, not a client result. Adapt the evidence and product facts before using it for your app.
Observation
Control 20%, treatment 22% in a constructed comparison
Uncertainty
Illustrative interval spans loss and gain
Context
No verified product-value effect
Decision
Retain control for now and investigate the mechanism before committing rollout
Owner
ASO lead documents the next question and review date.
Your turn · repair the decision
Make the call. Then compare.
Turn this into a three-sentence executive readout without hiding the inconclusive result.
Autosaved on this browser. Your work is private and does not sync between devices.
Explore the supporting case and interpretation
Trail Notes’ journal-led concept produces a promising but inconclusive native result. The readout keeps the existing control, records that the concept passed comprehension review and proposes a stronger demonstration of the journal workflow for a future test. It does not use the positive point estimate as a public success statistic. The report still delivers value by narrowing the next question and preventing an unjustified rollout claim.
Readout section: Setup
Content: Actual treatment and audience
Decision value: Establishes what was tested
Readout section: Evidence
Content: Counts, effect and uncertainty
Decision value: Shows strength and magnitude
Readout section: Context
Content: Deviations and guardrails
Decision value: Reveals limits and tradeoffs
Readout section: Action
Content: Owned rollout or next investigation
Decision value: Makes the learning usable
A result summary without setup or uncertainty is incomplete. A long report without an action is also incomplete. The structure above keeps the analysis connected to the original decision and the next operational step. If evidence is inconclusive, say why another test would be different before recommending repetition. Otherwise, the team may spend another cycle producing the same uncertainty.
Build a handover someone can use
Original brief
Attach the hypothesis, configuration and exact asset versions that actually ran.
Result evidence
Preserve native reports, counts, effect estimates and uncertainty with definitions.
Validity review
Explain deviations, audience changes and limits on subgroup interpretation.
Recommendation
State the bounded decision and why the evidence supports it.
Execution and learning
Assign rollout verification or further research and update the experiment history.
Your ASO assignment
Use your own app and evidence, or work through the teaching case. Keep your observations separate from assumptions and explain the reasoning behind your decisions.
Build something · track your work
Your assignment checklist
Tick each item after you have produced the work. Reading it does not count. The strategy deck will survive.
0 of 3 tasks checked.
Autosaved on this browser. Your work is private and does not sync between devices. Course completion is recorded separately.
Assess your work
- The readout preserves the original question and decision rule.
- The conclusion matches the method and tested population.
- The next action is specific even when the result is inconclusive.
Common mistakes to catch
- Changing the story after seeing the result.
- Publishing a best-case point estimate without uncertainty or context.
Sources and further reading
Platform references reviewed 29 September 2026. These sources support platform capabilities and constraints; the teaching frameworks, assignments and illustrative cases are original course material.
- Apple — Product page optimization results — Native result interpretation and Bayesian analysis.
- Google Play — Run store listing experiments — Experiment configuration and native interpretation guidance.
Your ASO lesson notes
Use this space for your assignment, evidence and handover. Your notes save automatically in this browser. Download a copy to take with you.
Notes stay on this device and are not submitted to ASO Agency.
Complete your assignment with the ASO experiment planner.
Loading your progress…
Progress is saved only in this browser. It does not sync across devices. Mark a lesson complete after doing its exercise.