Holdout Groups

A holdout is a control group inside an audience. Users in it match the audience but see no paywall. Use one when you want to know whether a paywall helps at all, not just which paywall wins: put a share of the audience in a holdout, show paywalls to the rest, and compare the two groups in the audience's results.

Requirements

• A campaign with a placement and an audience. See Audiences.
• At least one paywall in the audience, so the holdout has something to be compared with. An audience with only a holdout holds out everyone it matches.

Add a holdout

These steps are for the new dashboard at app.superwall.com.

1. Open the campaign and select the audience.
2. Click the Paywalls tab. Its label counts what's in it, like 1 paywall or 2 paywalls + holdout.
3. If there's no Holdout tile, click Add holdout, then Add holdout again in the dialog. The holdout starts at 0%. New audiences already include a holdout at 0%, so you may only need step 4.
4. Click Edit split, or the percentage on any tile. Set each paywall's share. The holdout gets the rest, so one paywall at 80% leaves 20% in the holdout. Shares are whole numbers, and the paywalls can't add up to more than 100%.
5. If the audience is live, choose what happens to users already in it:
   • Keep their paywall: users already assigned stay where they are. Only new users follow the new split.
   • Re-assign everyone: every assignment is reset, so all users follow the new split on their next paywall. The audience's results start over too.
6. Click Save changes. A message confirms the holdout's new share.

What happens on the device

When a user lands in the holdout, the SDK shows no paywall. Your app sees the same things it sees whenever a paywall is skipped:

• The feature block runs. The code you pass to register runs, just as it does when no paywall is configured. If the paywall would have been gated, a held-out user gets the feature without paying.
• The presentation handler's onSkip fires with the holdout reason: .holdout(experiment) on iOS, PaywallSkippedReason.Holdout on Android. The experiment tells you which experiment and variant the user is in.
• The trigger_fire event fires with result set to holdout, along with experiment_id and variant_id. Forward it to your analytics tool to cohort held-out users there.
• getPresentationResult returns a holdout result for the placement: .holdout(experiment) on iOS, PresentationResult.Holdout on Android. Use it to check the outcome without registering the placement.

let handler = PaywallPresentationHandler()
handler.onSkip { reason in
  if case .holdout(let experiment) = reason {
    print("Held out of experiment \(experiment.id), variant \(experiment.variant.id)")
  }
}

Superwall.shared.register(placement: "caffeineLogged", handler: handler) {
  // Runs for held-out users too.
  logCaffeine()
}

val handler = PaywallPresentationHandler()
handler.onSkip { reason ->
  if (reason is PaywallSkippedReason.Holdout) {
    println("Held out of experiment ${reason.experiment.id}, variant ${reason.experiment.variant.id}")
  }
}

Superwall.instance.register(placement = "caffeineLogged", handler = handler) {
  // Runs for held-out users too.
  logCaffeine()
}

For every SDK's handler and result types, see Using the Presentation Handler.

How users are assigned

Superwall assigns each user a variant at random, weighted by the shares you set. A 20% holdout gets about one in five new users.

Assignments are sticky. A user stays in the holdout, or on their paywall, until you reset assignments or remove that variant. Changing the split with Keep their paywall only affects users who haven't been assigned yet.

To start over, hover the Holdout tile and click Reset all assignments. It resets every variant in the audience, not just the holdout, and users are assigned again on their next evaluation. The assigned counts and the audience's results start over.

Read the results

Open the audience's Results tab. The holdout has its own row, named Holdout, with no paywall preview. Compare each paywall's row with it:

• Users is how many users are in each group. Check that the holdout is big enough to compare before you read anything into it.
• Proceeds per user divides proceeds by users, so it compares groups of different sizes. If paywall rows earn more per user than the holdout, showing a paywall at that placement is worth it.
• Views count paywall opens, so the holdout has none.

The Users tab lists recent matches, and held-out users show Holdout as their variant. It's a quick way to check that the holdout is getting users.

For what each column means, see Understanding Experiment Results.

Things to know

• One holdout per audience. Add holdout only appears when the audience doesn't have one.
• The holdout tile's assigned count stays at 0. That count is paywall viewers, and held-out users never see a paywall. Use Users in the Results tab for the holdout's size.
• Held-out users don't move on to the next audience. Audiences are checked top to bottom, and the first match wins, holdouts included.
• A holdout counts toward the audience's limit. If the audience has a limit, a held-out match uses it up the same way a paywall does.
• Autopilot sets the holdout to 0%. Where Autopilot is available, turning it on resets assignments and splits users evenly across the paywalls, and you can't edit the split while it runs.

Related

• Starting an Experiment
• Understanding Experiment Results
• Audiences
• Using the Presentation Handler
• Cohorting in 3rd Party Tools