Quick A/B test

What is a quick A/B test?

The quick A/B test enables you to test different ad monetization strategies without any additional setup.

Traffic is automatically allocated 50/50 between the A and B groups. Performance KPIs can be broken down by A vs. B in the platform Performance, User Activity, and Bidding report pages to compare results.

What can you test with the quick A/B test?

Waterfall A/B tests

Test different waterfall settings including:

• Country groups
• New networks
• Instance pricing
• Cross promotion

Ad strategy 

Create any type of ad strategy test for all of the supported features on the ad unit and placement level: 

• Ad unit capping and pacing
• Placement capping and pacing
• Rewarding
• Banner refresh rate

How the quick A/B test works

Once activated, the quick A/B test divides traffic equally between groups A and B.  At this stage, the A and B groups are identical and no changes will be applied to either group.

• Each group will include all of your active instances and mediation groups
• The groups are identical in settings and waterfalls
• No pre-test setup is required

How to start a quick A/B test

Step 1: Navigate to the A/B Testing page 

Click the Create Test button next to the Quick A/B Test option

Step 2: Name and activate the test

Give the test a name and description. Then choose the type of test you want to run: a waterfall setup test or an ad strategy test that features changes to ad unit capping and pacing. When you’re ready, click the Start Test button to start the test.

Step 3: Modify your test group’s settings

To test your ad strategy: 

Go to the ad unit & placement page and modify the test group’s settings

You’ll see a toggle on the ad unit & placement page page that allows you to switch between groups A and B. To test your capping and pacing settings by ad unit and placement, adjust the settings for the test group according to the variables that interest you. 

Each change you make will apply ONLY to the group you’ve selected with the following exceptions that will update in both A and B groups: 

• ironSource instances 
• Network test mode status
• Activation or deactivation of ironSource as a bidder
• Callbacks

To test your waterfall setup:

Go to your Mediation Management page and modify each group’s waterfall settings

You’ll see a toggle on the mediation management page that allows you to switch between groups A and B. Adjust the waterfall settings for the test group according to the variables that interest you. Each change you make will apply ONLY to the group you’ve selected.

Supported waterfall components include:

• Capping 
• Addition (or subtraction) of instances from the mediation group
• New group 
• Delete group
• Duplicate group 
• Export CSV 
• Rate 

 Step 4: Terminate the test

Once you’ve analyzed your results and reached a conclusion about which group performs better, you can choose the winning group and terminate the test by clicking on the Continue with A or Continue with B buttons.

Terminating a test means the winning group’s settings will apply to 100% of the traffic on your app. 

Monitoring and reporting

To monitor an active bidding test, use the platform reporting pages—Performance, User Activity, and Bidding reports.

Breakdown by A/B is available on the Performance, User Activity, and Bidding reports, with all metrics supported. To view reports broken by A/B, select one app that completed or is currently running an A/B test.

Click Show Reports to view the Performance report with the app pre-filtered, test start/end dates selected, and the data broken down by A/B.

Here’s an example of a Performance Report that’s broken down into A and B groups: