What is paywall A/B testing?
Paywall A/B testing means splitting your users into two or more groups, showing each group a different paywall variant, and measuring which variant produces more revenue per user. Done well, it's the single highest-ROI growth lever in a subscription app. Done badly, it produces a string of false positives that don't reproduce in cohort retention.
You need three things to do it right:
A way to render multiple paywall variants quickly, ideally without shipping an app update — while absolutely adhering to each App Store's respective rules and guidelines for paywalls.
A way to assign users to variants deterministically and track them through to revenue.
The discipline to wait long enough for the data to mean something.
How to set up a paywall A/B test
The setup pattern that works regardless of tooling:
Pick one variable to change. "New paywall" with five things different is not a test. Tha's more in line with a complete redesign.
Define one primary metric. Pre-register it before the test starts. Usually trial start rate as a fast signal and Day 30 retained ARPU as the decision metric.
Define sample size and stop date. Decide both before you launch. Stopping when results "look significant" inflates false-positive rate dramatically.
Hash-assign users. Use a stable hash of user ID → variant. Don't randomize per session; users will see different paywalls on different opens and your data will be junk.
Track at the cohort level. Users assigned on day 1 of the test get followed all the way through D30+, regardless of when they convert.
In Superwall this looks like a Campaign with two paywalls with a 50/50 split, triggered on a specific placement.
In a hand-rolled system you implement the same logic yourself — the structure doesn't change. If you hand-roll assignment, make it deterministic. This isn't perfect, but something similar to this:
ts
import { createHash } from "crypto"
function assignVariant(userId: string) {
const hash = createHash("sha256").update(userId).digest("hex")
const bucket = Number.parseInt(hash.slice(0, 8), 16) % 100
return bucket < 50 ? "control" : "treatment"
}What sample size do I need?
Most paywall teams test on far too little data. A rough calculator using normal-approximation:
Baseline trial-start rate | Minimum detectable lift | Sample per variant |
5% | +20% relative (5% → 6%) | ~9,000 views |
5% | +50% relative (5% → 7.5%) | ~1,500 views |
15% | +20% relative (15% → 18%) | ~2,500 views |
15% | +50% relative (15% → 22.5%) | ~400 views |
30% | +20% relative (30% → 36%) | ~1,000 views |
Rule of thumb: if your paywall sees fewer than 1,000 views per week and the baseline conversion is below 10%, you'll need to test bigger changes (offer matrix, full-screen redesigns) rather than micro-copy. Don't test button-color changes if you have onboarding-volume data, because you'll likely spend three months chasing noise.
If you want to test smaller changes, run them on the highest-volume placement you have (almost always onboarding) and let the test run for 2–4 full weeks, including a full weekly cycle.
What should I test first?
In rough order of expected lift:
1. Offer matrix (biggest lever)
Annual only vs. monthly + annual.
Trial length (3-day vs. 7-day vs. 14-day).
Free trial vs. introductory price ($1 first week).
Price points themselves (this requires App Store / Play Store product changes, it's slower but the highest-impact test you're likely to run).
Successful lift: Any where in the1 10–40% range, sometimes much more.
2. Headline + value prop
Benefit-first ("Lose weight without counting calories") vs. feature-first ("Unlock 200+ workouts").
Personalized headline using onboarding answers.
Length (1 line vs. 3 lines of bullets).
Successful lift: 5–20%.
3. Social proof
Star rating + review count near the CTA.
Quoted user review carousel.
Press logos, or high-value clients/users.
"Used by X million people." Don't make up numbers here, either. People eventually sense that, and it erodes brand trust.
Successful lift: 5–15%, larger for unknown brands than for well-known ones.
4. CTA copy and offer framing
"Start free trial" vs. "Continue" vs. "Try $product free for 7 days".
"Cancel anytime" microcopy under the button.
Showing the price-after-trial vs. hiding it.
Successful lift: 2–10%, but compounds with other tests.
5. Layout, animation, length
Long-scroll vs. above-the-fold single screen.
Multi-step paywall (quiz → recommendation → offer) vs. single-screen.
Lottie animation vs. static hero.
Successful lift: This one is variable. Multi-step paywalls often beat single-screen for cold traffic but can hurt warm traffic.
What not to test first: button colors, corner radii, gradient angles. You can run those tests, but only on big-volume placements where the data is essentially free. They don't move the needle on small apps.
How long should I run a paywall test?
Three rules:
At least 7 days, always. Weekday vs. weekend traffic mixes differently, and your test needs to cover a full weekly cycle.
At least 14 days if your decision metric is trial-to-paid conversion or D30 ARPU. A 7-day trial cohort that converted on day 8 hasn't actually retained yet.
Stop only at the pre-registered end date. "Peeking" at significance daily and stopping when p < 0.05 inflates your false-positive rate to ~25–30%. If you must peek, use sequential testing methods (mSPRT, Bayesian) that account for repeated looks.
How do I decide the winner?
Pick the decision metric before the test, not after. The four common choices, in increasing order of trustworthiness:
Trial start rate — This one is fast, noisy, easy to game with offers users will never pay for.
Trial-to-paid conversion — Generally, a bit better; reveals whether the paywall over-promised.
D30 retained revenue per install — Here, it's slow, but the actual money number.
D90 retained revenue per install — Finally, the gold standard for serious subscription teams; pairs with LTV modeling.
For most teams the right answer is: optimize on trial start rate for fast iteration, then verify the winner on D30 retained ARPU before declaring it the new default. If the trial-start winner loses on D30 ARPU, that variant is over-promising — a worse product experience masked as a paywall win.
How do I avoid the most common A/B test mistakes?
Mistake 1: Running multiple tests on the same users. Two simultaneous tests on the same placement contaminate each other. Run them sequentially or on different placements.
Mistake 2: Stopping early. See above. Always wait to the pre-registered end date.
Mistake 3: Ignoring segment differences. A paywall that wins overall might lose badly in your highest-LTV segment (e.g., paid-acquisition users from Meta). Always look at the winner by acquisition channel and by country.
Mistake 4: Re-running a test to confirm a result. If you re-test a winner and it's slightly worse, you have no idea whether the original test was a false positive or the re-test is noise. Trust the original outcome unless you have a real reason to suspect it.
Mistake 5: Skipping the holdout. Keep 5–10% of users on the previous baseline forever. Periodically check that your "winners" are still winning in aggregate against the long-term holdout. They surprisingly often aren't.
Tools you'll need
A paywall platform with remote-config and built-in A/B logic — Superwall, RevenueCat, or hand-rolled. Required because you can't easily iterate weekly through App Stores.
An analytics layer that captures paywall events through to revenue — Amplitude, Mixpanel, or your own data warehouse. The platform layer (Superwall) usually pipes events automatically. For Superwall users, user superwall.ai.
A statistical significance calculator — There are free online ones, or just use a t-test in BigQuery / Snowflake. Basically, don't try to eyeball it, the tool is not as important.
(Optional) An offline analysis notebook — Most serious teams pull raw event data into Python or R for proper cohort analysis. The platform dashboards are good for triage, not for decision-making.
A reference test plan
Here's the structure I'd ship for a team running their first real paywall test:
Hypothesis: Changing the onboarding paywall from monthly-default-selected to annual-default-selected will increase D30 retained ARPU by ≥10%, because annual subscribers have higher LTV.
Variants: Control (monthly default) vs. Treatment (annual default). All other elements identical.
Placement: Onboarding paywall only.
Sample size: 10,000 paywall views per variant (baseline trial-start ≈ 12%, MDE 15% relative).
Primary decision metric: D30 retained revenue per install.
Secondary metrics: Trial start rate, trial-to-paid conversion, refund rate (watch for over-promising).
Duration: Minimum 14 days, maximum 30 days, or until sample size hits.
Stop conditions: Pre-registered end date, OR refund rate diverges by >2 percentage points (safety stop).
Decision rule: Treatment ships if D30 retained ARPU lift ≥ +5% with p < 0.05.
That's it. Boring, structured, and far more likely to find real wins than ten chaotic tests in parallel.
FAQ
How long should a paywall A/B test run? Minimum 7 days to cover a full weekly cycle. Minimum 14 days if your decision metric requires trial-to-paid data. Never stop early based on peeking.
Do I need statistical significance to call a winner? You need enough confidence — usually p < 0.05 or a Bayesian probability of being better above 95%. If your sample is small and the effect is large, you might call a winner with less; if the sample is huge and the effect is small, you need more.
How many paywall views do I need per variant? Roughly 2,000–10,000 per variant to detect a 15% relative lift in trial start rate at 95% confidence, depending on your baseline rate. Big changes need fewer views; tiny changes need many more.
Can I test more than two variants at once? Yes, but each additional variant divides your traffic. Stick to 2 variants if your weekly paywall volume is under 5,000 views.
What's the difference between A/B testing and personalization? A/B testing decides which paywall is best for everyone. Personalization picks the best paywall for this user based on their attributes (acquisition channel, country, onboarding answers). Personalization is the natural next step once you have a few proven winners, and Superwall handles both.
Next steps
Set up your first test in the Superwall dashboard.
Decide on your first test.