Starting an Experiment

You run experiments in Superwall by adding multiple paywalls to an audience. To start an experiment:

1. Select an audience.
2. Click the Paywalls tab.
3. Add two or more paywalls.

Here’s a .gif, beginning to end, setting up an experiment:

It’s as simple as that to start a paywall experiment.

​

Setting presentation percentages

You must set a presentation percentage between your paywalls within the experiment. This determines how often they’ll show based off of the percentage set for each one.

To set a percentage, click the pencil icon above any of the paywalls attached to the audience:

Then, assign percentages from 0%-100% for each of them. In total, your percentages should equal 100 (i.e. paywall A shows 10%, paywall B shows 30%, and paywall c shows 60% of the time) unless you’re purposely creating a holdout. When you’re done, click the Checkmark icon below any paywall:

​

Resetting assignments

If you change your experiment, or simply want to change the presentation percentages between your paywalls, you might want to reset your assignments. Remember that when an audience matches a user, it’s sticky — and the same is true of when someone is matched to a paywall within an experiment.

So, if you want to make sure everyone is matched again to a paywall based off new percentages, click the refresh button below a paywall when editing percentages (next to where it says “X assigned”):

​

Creating holdouts

A holdout occurs when you purposely edit an audience to not present a paywall in some cases. Setting a holdout is useful when you want to test the effectiveness of showing a paywall.

To create a holdout, set your paywall presentation percentages to be less than 100% across all of the paywall you’re using.

Here’s an example of one paywall set to show 50% of the time, meaning the other 50% of users who match this audience will be in a holdout:

It’s common to pair holdouts to certain placements to see whether a holdout increases or decreases transactions. The holdout group will act as a control which you can compare against.

​

Removing variants

During an experiment, you may find that one or more paywalls are performing significantly worse than the others. In that case, you would probably consider removing it. You can simply remove the paywall, or set its presentation percent to 0%, and your experiment will continue. No metrics will be affected or reset. Resetting assignments will reset metrics, removing paywalls will not.