ASO course · lesson 36 of 60
ASO creative testing: write an experiment brief
Write a complete hypothesis and production brief that can lead to an interpretable test.
By Gabriel Machuret · Editorial methodology
Before you start: Reviewed creative concepts, a metric dictionary and available store traffic context.
Module 6: Creative strategy and production
Plan about 55 minutes for reading, practice and review
You will produce: Module project: a test-ready creative pack with an approved experiment brief.
Understand the decision
State the causal idea
Write “Changing X for audience Y should affect metric Z because…” and make the reason specific. “New screenshots will perform better” is not enough. Explain the current misunderstanding and how the treatment addresses it. A concept may change several coordinated elements, but then the conclusion concerns the concept as a whole rather than an isolated color or headline.
Define the comparison before production
Describe the control, treatment, eligible population, primary metric and guardrails. State what stays fixed and list concurrent work that could interfere with interpretation. Confirm that the chosen platform supports the design. A targeted page with a different audience is not automatically a randomized creative test, and a before-and-after comparison carries different limitations.
Plan all possible outcomes
Write what you will do if the treatment looks better, worse or inconclusive. Include operational stop conditions for incorrect assets or product problems. Do not prescribe one universal duration or sample size without the design assumptions. The next module develops evaluation in detail; this lesson ensures the design team produces an answerable question rather than simply another asset set.
Apply the method in practice
Write a mechanism, not just a variant name
A useful hypothesis connects an observation, a proposed change and an expected behavior. “Version B will win” explains nothing. “Showing a completed journal first may reduce navigation confusion and improve acquisition among suitable visitors” identifies a mechanism. Specify the control and treatment exactly. If multiple assets change together, label the treatment as a package and limit the interpretation accordingly. Accuracy corrections should be completed and logged before testing uncertain persuasion choices.
Specify the audience and measurement boundaries
Name store, locale, eligible traffic, primary metric and its denominator. Include a downstream guardrail where measurement supports it. Record concurrent campaigns and app changes that could complicate interpretation. Verify the platform's experiment eligibility and current method rather than imposing a generic calculator on native results. Plan what the team will do for improvement, decline and inconclusive evidence. The brief should make those decisions possible before anyone sees a favorable-looking chart.
Prepare a reproducible production handoff
Give the publisher exact asset IDs, ordering, localizations and approval status. Keep the previous control and the proposed treatment side by side. Record launch verification, experiment configuration and the person responsible for reading results. A design folder alone is not an experiment brief. If a late revision changes the treatment after review, update the hypothesis and manifest before launch. Otherwise, the readout may describe an experiment that never actually ran.
Platform references for this work: Apple — Product page optimization overview · Google Play — Run store listing experiments
Your step-by-step procedure
- Write the hypothesis with audience, change, metric and mechanism.
- Specify the control and treatment assets, including what remains unchanged.
- Verify platform support, traffic context and measurement definitions.
- Agree decision rules, operational checks and owners before production sign-off.
Worked example and interpretation
Trail Notes and numerical research scenarios are fictional teaching examples. Platform limits, where shown, come from the linked official references.
Trail Notes tests a journal-led opening sequence against its existing scenery-led sequence. The hypothesis is improved recognition among eligible new visitors, measured through the chosen native listing outcome with product-quality context where available. Text and artwork change together, so any result concerns the whole opening concept. The brief explicitly rejects the narrower claim that the background color caused the outcome.
| Brief field | Teaching example | Reason |
|---|---|---|
| Observation | Navigation confusion in research | Defines the problem |
| Treatment | Completed journal shown first | Targets the suspected mechanism |
| Primary measure | Native acquisition measure, exactly defined | Prevents denominator ambiguity |
| Decision branches | Adopt, reject or investigate | Prevents forced winner selection |
The hypothesis is plausible, but the qualitative observation does not prove that the treatment will improve acquisition. The experiment is designed to learn about that specific audience and implementation. If the result is inconclusive, the brief should allow a useful next step, such as revising the concept or choosing another research method. Keeping an honest inconclusive branch protects the team from rolling out an unsupported winner.
Build a handover someone can use
Hypothesis
Connect observation, change and expected user behavior with explicit uncertainty.
Asset manifest
Provide exact control and treatment files, order, locale and approvals.
Measurement
Define primary outcome, denominator, guardrails and platform method.
Operating plan
Assign launch QA, monitoring and responsibility for concurrent-change records.
Decision rules
Write actions for improvement, decline and inconclusive evidence before launch.
Your ASO assignment
Use your own app and evidence, or work through the teaching case. Keep your observations separate from assumptions and explain the reasoning behind your decisions.
- Complete the experiment planner for one creative hypothesis.
- Attach the exact control and treatment asset versions.
- Write positive, negative and inconclusive decision paths.
Scenario challenge
A designer changes the opening image, icon and all captions together. Rewrite the brief as a package test and explain which individual causal claims the eventual result will not support.
Assess your work
- The proposed conclusion matches what the design can identify.
- Assets and eligible audience are specified before launch.
- An inconclusive outcome has a useful next action.
Common mistakes to catch
- Changing several elements and attributing the result to one of them.
- Choosing the primary metric only after seeing which one improved.
Knowledge check
A test changes headline, artwork and order. What can a positive result establish?
Read the answer and reasoning
With a valid design, it can support the combined treatment for the tested population and metric. It cannot isolate which individual element caused the difference. If that distinction matters, plan a later focused test rather than inventing an explanation from the winning design.
Sources and further reading
Platform references reviewed 29 September 2026. These sources support platform capabilities and constraints; the teaching frameworks, assignments and illustrative cases are original course material.
- Apple — Product page optimization overview — Supported experiment assets, eligibility and setup.
- Google Play — Run store listing experiments — Experiment configuration and native interpretation guidance.
Your ASO lesson notes
Use this space for your assignment, evidence and handover. Select Save to keep a draft in this browser, or download a copy to take with you.
Notes stay on this device and are not submitted to ASO Agency.
Complete your assignment with the ASO experiment planner.
Loading your progress…
Progress is saved only in this browser. It does not sync across devices. Mark a lesson complete after doing its exercise.