Why onboarding and paywall should be one experiment
Split tests on onboarding and paywalls separately hide interaction effects. Run one experiment on the full journey to measure real lift on trial starts and revenue.
Growth teams often run an "onboarding test" in one tool and a "paywall test" in another. Onboarding wins on completion. Paywall wins on ARPU. Leadership asks which to ship. Nobody knows.
The problem is interaction effects. Users experience one journey. Split experiments assume independence that does not exist. This post explains why combined experiments matter, when separation is valid, and how to structure tests in Rheo so trial starts and revenue move together.
The split-experiment trap
Typical setup:
- Team A shortens onboarding in a flow builder. Completion rises 8%.
- Team B changes paywall layout in RevenueCat. Trial conversion rises 5%.
- Both ship to production in the same release.
Observed outcome: Trial starts flat. Completion dropped on the new paywall path because users arrive with less context. Or paywall wins evaporate because onboarding now attracts lower-intent users.
| Split tests assume | Reality |
|---|---|
| Onboarding effect is independent of paywall | Message and intent carry forward |
| Paywall effect is independent of arrival quality | Same UI converts differently by segment |
| Metrics compose linearly | They multiply and sometimes cancel |
You optimized two local maxima and missed the global one.
What "one experiment" means
One experiment here means:
- One hypothesis about the end-to-end journey
- Multiple variants, each a full published flow version (onboarding sequence + paywall placement + post-purchase branches)
- One traffic split on a channel so each user stays in one arm
- Primary metrics on trial starts, purchase, or completion for the whole path
It does not mean you cannot change paywall templates in RevenueCat. It means assignment is unified: user 123 always sees variant B's screens and variant B's timing into the RC paywall.
Interaction effects you should expect
1. Message match
Short onboarding that skips a feature demo may reduce paywall conversion even if more users arrive. Long onboarding may increase conversion but reduce arrival. Only a joint metric (trials per install or per flow start) reveals the winner.
2. Permission and attribution order
ATT before paywall vs after changes both opt-in and revenue. Testing permissions in isolation on a channel that later changes paywall timing invalidates results.
3. Dismiss and recovery paths
Paywall-only tests ignore Fallback branches defined in onboarding. A dismiss-friendly onboarding variant may pair with a second-chance screen that only exists in that flow graph.
4. Segment heterogeneity
Paid social users may need social proof screens; organic users may not. A single combined experiment with segment analysis (custom property or MMP attribute) beats separate tools with no shared bucketing.
When separate tests are OK
Separation is fine when:
- Changes are orthogonal by design (e.g., backend entitlement bug fix, not user-visible)
- You are sequential, not parallel: onboarding test concludes, winner pinned, then paywall test on the winner only
- Traffic is too low for multi-cell tests and you accept exploratory results
Even then, document the sequence. Do not run overlapping splits on the same population without a factorial plan.
How to structure a combined experiment in Rheo
1. Duplicate the baseline flow
Create variant flows in the builder. Each variant is a complete graph:
| Arm | Description |
|---|---|
| Control | Current production path |
| Treatment A | e.g., shorter path + same paywall position |
| Treatment B | e.g., same length + paywall after preview step |
2. Publish each arm as its own version
Version labels should be human-readable. Experiments reference published pins only.
3. Attach experiment to one channel
Configure weights (e.g., 34/33/33) and a required end date UTC. Start when QA passes on a test channel clone.
4. Pick primary and guardrail metrics
| Type | Examples |
|---|---|
| Primary | Trial starts per flow start, purchase completed rate |
| Guardrail | Completion, ATT opt-in, paywall dismiss rate |
Decide before launch. Avoid switching primary metric when results look disappointing.
5. Read pending decision holistically
At end date, Rheo enters pending decision. Traffic keeps bucketed arms; you promote one version to the channel pin.
Compare pairwise vs control at ~5% significance. Multi-arm tests inflate false positives; treat secondary arms as exploratory unless you pre-register them.
Example: one hypothesis, two treatments
Hypothesis: Users who see a personalized result before the paywall start more trials than users who see a shorter generic path.
- Control: 4 generic screens → paywall
- Variant A: 2 generic + 1 personalized → paywall (same offering)
- Variant B: 2 generic → paywall → personalization after trial
Primary metric: iap_purchase with trial period type per flow_started.
Analysis: If B wins on arrival but loses on conditional purchase, A may still win overall. Split experiments on only screens 1-4 or only paywall would miss that.
RevenueCat's role in a unified test
RevenueCat still hosts offerings and paywall UI. Rheo controls:
- Which screens precede the paywall node
- Which offering id the node passes
- Branches on
purchase_completed,dismissed, etc.
For a true combined test, keep the same offering across arms unless the hypothesis is explicitly about plan mix. Change one journey variable at a time when learning.
Organizational habits that help
- One DRI per experiment across growth, product, and design
- Shared doc with hypothesis, metrics, start/end, and variant screenshots
- No mid-flight weight changes without recording reason
- Weekly funnel review on the same experiment id
Anti-patterns to avoid
| Anti-pattern | Why it fails |
|---|---|
| Ship onboarding winner, start paywall test same day on same channel | Overlapping assignments |
| Compare RC experiment to Rheo experiment numerically | Different buckets |
| Optimize completion only | Ignores revenue |
| Optimize paywall only | Ignores arrival quality |
Factorial vs one-factor experiments
Advanced teams sometimes run 2x2 factorial tests (e.g., short vs long onboarding crossed with early vs late paywall). That requires four arms and roughly 4x traffic. Default to one-factor until you have Scale-level volume.
If you must run factorial:
| Arm | Onboarding | Paywall timing |
|---|---|---|
| A | Control | Control |
| B | Short | Control |
| C | Control | Late |
| D | Short | Late |
Interpret main effects and interaction in your stats tooling. Rheo supplies variant-level event streams; export to your warehouse for factorial analysis.
Communicating results to leadership
Translate variant performance into revenue language:
- "Variant B delivered +12% trial starts per install with no ATT regression."
- "We should promote B and run a follow-up paywall framing test on B only."
Avoid showing twelve metrics without a decision. Pending decision exists to force promote or extend.
Relationship to paywall-only tools
RevenueCat Experiments, Superwall tests, and similar tools optimize billing UI assignment. They do not reorder your Rheo onboarding graph. If onboarding assignment is independent of paywall assignment, you reintroduce the interaction problem.
Preferred stack:
| Layer | Tool |
|---|---|
| Onboarding graph + channel experiment | Rheo |
| Products, offerings, paywall templates | RevenueCat |
| Attribution | MMP |
One channel experiment id should be the source of truth for "which journey did this user see?"
After you promote a winner
- Pin the winning version on the channel.
- Stop the experiment (channel serves direct pin).
- Archive losing variants in docs, do not delete published versions (analytics history).
- Queue the next one-factor test from the backlog.
Within two quarters, compounding single-factor wins on a unified journey often beat one heroic redesign.
Holding a learning backlog
Maintain a ranked list of journey hypotheses, not screen tickets. Example entries: "preview before paywall," "ATT after trial," "second chance on dismiss." When an experiment ends, pull the next item that targets the current bottleneck metric. One backlog prevents parallel conflicting tests and keeps onboarding plus paywall decisions in one prioritized queue.
Why this matters for revenue compounding
A 5% lift on trial starts and a 5% lift on paywall conversion multiply to roughly 10% more paying users from the same install base. Split experiments that optimize each factor on different cohorts never capture that multiplicative effect. Unified experiments align the whole machine toward one revenue outcome.
The compound case
Small lifts multiply. If onboarding sends 20% more users to the paywall and the paywall converts 10% more trials, combined lift is ~32% on trials, not 20% or 10%. Interaction can also subtract. Measure the product.
Start one combined test
Take your current production flow. Name the single metric leadership cares about (usually trial starts or paid conversion per install). Duplicate the graph twice with coherent alternative journeys. Run one experiment for two weeks. Promote the global winner.
Stop arguing which team "won" separate tests. Win the journey. Document the promoted version and schedule the next holistic hypothesis before momentum fades.