# Lesson 36: ASO creative testing: write an experiment brief

Write a complete hypothesis and production brief that can lead to an interpretable test.

Prerequisite: Reviewed creative concepts, a metric dictionary and available store traffic context.

Teaching examples are fictional unless explicitly identified as platform definitions.

## Foundations

### State the causal idea

Write “Changing X for audience Y should affect metric Z because…” and make the reason specific. “New screenshots will perform better” is not enough. Explain the current misunderstanding and how the treatment addresses it. A concept may change several coordinated elements, but then the conclusion concerns the concept as a whole rather than an isolated color or headline.

### Define the comparison before production

Describe the control, treatment, eligible population, primary metric and guardrails. State what stays fixed and list concurrent work that could interfere with interpretation. Confirm that the chosen platform supports the design. A targeted page with a different audience is not automatically a randomized creative test, and a before-and-after comparison carries different limitations.

### Plan all possible outcomes

Write what you will do if the treatment looks better, worse or inconclusive. Include operational stop conditions for incorrect assets or product problems. Do not prescribe one universal duration or sample size without the design assumptions. The next module develops evaluation in detail; this lesson ensures the design team produces an answerable question rather than simply another asset set.

## Apply the method

### Write a mechanism, not just a variant name

A useful hypothesis connects an observation, a proposed change and an expected behavior. “Version B will win” explains nothing. “Showing a completed journal first may reduce navigation confusion and improve acquisition among suitable visitors” identifies a mechanism. Specify the control and treatment exactly. If multiple assets change together, label the treatment as a package and limit the interpretation accordingly. Accuracy corrections should be completed and logged before testing uncertain persuasion choices.

### Specify the audience and measurement boundaries

Name store, locale, eligible traffic, primary metric and its denominator. Include a downstream guardrail where measurement supports it. Record concurrent campaigns and app changes that could complicate interpretation. Verify the platform's experiment eligibility and current method rather than imposing a generic calculator on native results. Plan what the team will do for improvement, decline and inconclusive evidence. The brief should make those decisions possible before anyone sees a favorable-looking chart.

### Prepare a reproducible production handoff

Give the publisher exact asset IDs, ordering, localizations and approval status. Keep the previous control and the proposed treatment side by side. Record launch verification, experiment configuration and the person responsible for reading results. A design folder alone is not an experiment brief. If a late revision changes the treatment after review, update the hypothesis and manifest before launch. Otherwise, the readout may describe an experiment that never actually ran.

## Procedure

1. Write the hypothesis with audience, change, metric and mechanism.
2. Specify the control and treatment assets, including what remains unchanged.
3. Verify platform support, traffic context and measurement definitions.
4. Agree decision rules, operational checks and owners before production sign-off.

## Worked example

Trail Notes tests a journal-led opening sequence against its existing scenery-led sequence. The hypothesis is improved recognition among eligible new visitors, measured through the chosen native listing outcome with product-quality context where available. Text and artwork change together, so any result concerns the whole opening concept. The brief explicitly rejects the narrower claim that the background color caused the outcome.

| Brief field | Teaching example | Reason |
| --- | --- | --- |
| Observation | Navigation confusion in research | Defines the problem |
| Treatment | Completed journal shown first | Targets the suspected mechanism |
| Primary measure | Native acquisition measure, exactly defined | Prevents denominator ambiguity |
| Decision branches | Adopt, reject or investigate | Prevents forced winner selection |

The hypothesis is plausible, but the qualitative observation does not prove that the treatment will improve acquisition. The experiment is designed to learn about that specific audience and implementation. If the result is inconclusive, the brief should allow a useful next step, such as revising the concept or choosing another research method. Keeping an honest inconclusive branch protects the team from rolling out an unsupported winner.

## Your ASO assignment

1. Complete the experiment planner for one creative hypothesis.
2. Attach the exact control and treatment asset versions.
3. Write positive, negative and inconclusive decision paths.

A designer changes the opening image, icon and all captions together. Rewrite the brief as a package test and explain which individual causal claims the eventual result will not support.

## Handover

### Hypothesis

Connect observation, change and expected user behavior with explicit uncertainty.

### Asset manifest

Provide exact control and treatment files, order, locale and approvals.

### Measurement

Define primary outcome, denominator, guardrails and platform method.

### Operating plan

Assign launch QA, monitoring and responsibility for concurrent-change records.

### Decision rules

Write actions for improvement, decline and inconclusive evidence before launch.

## Check your work

- The proposed conclusion matches what the design can identify.
- Assets and eligible audience are specified before launch.
- An inconclusive outcome has a useful next action.

## Common mistakes

- Changing several elements and attributing the result to one of them.
- Choosing the primary metric only after seeing which one improved.

## Knowledge check

A test changes headline, artwork and order. What can a positive result establish?

With a valid design, it can support the combined treatment for the tested population and metric. It cannot isolate which individual element caused the difference. If that distinction matters, plan a later focused test rather than inventing an explanation from the winning design.

## Sources

- [Apple — Product page optimization overview](https://developer.apple.com/help/app-store-connect/create-product-page-optimization-tests/overview-of-product-page-optimization/) — Supported experiment assets, eligibility and setup.
- [Google Play — Run store listing experiments](https://support.google.com/googleplay/android-developer/answer/12053285?hl=en) — Experiment configuration and native interpretation guidance.

Platform references reviewed 29 September 2026.
www.asoagency.com