A/B test

The A/B test tool enables you to test different ad monetization strategies on the ad unit level. Once you activate a test for an ad unit, traffic is automatically allocated 50/50 between groups A (control) and B (test). You have the flexibility to add ad units to the test at any time and remove an individual ad unit without closing the entire test.

What you can test

Waterfall configuration

From the Mediation Management page, you can test different waterfall settings, including:

Waterfall setup
Mediation groups
New networks
Instance pricing

Ad strategy

From the Ad Units and Placement page, you can create any type of ad strategy test for all of the supported features on the ad unit and placement level:

Ad unit capping and pacing
Placement capping and pacing
Rewarding
Banner refresh rate

How to start an A/B test

There are 2 places on the platform where you can start an A/B test:

Mediation Management page – To test your waterfall configuration
Ad Units and Placements page – To test your ad strategy

From either page, the setup is almost identical. In this article, we’ll show the setup from the Mediation Management page.

Step 1: Select the test you want to create

Under Management, click on Mediation. Select the app, then the ad unit you want to run an A/B test on. Then click Start Test.

In the pop-up screen, choose A/B Test. Give your test a name and description, and click Start Test. Your traffic for the ad unit you selected will now automatically split 50/50 between groups A and B.

Step 2: Modify your waterfall setup

On the Mediation Management page, use the toggle to switch between group A (control) and B (test). Adjust the waterfall configuration for group B (test group) according to the variables you want to test. Each change you make will apply only to group B.

Supported waterfall components include:

Addition (or subtraction) of instances from the mediation group
Group configurations
Delete / add groups
Instance rates
Instance level capping

Step 3: Include additional ad units

You can add additional ad units to the test by clicking an ad unit name (rewarded video, interstitial, or banner) at the top of the page. Then click Add to test.

Click Confirm and the ad unit will immediately be added to the test. Adjust the waterfall configuration for your test group (B) according to the variables you want to test.

Step 4: Test your ad strategy (optional)

At the same time you’re testing your waterfall configuration, you can test your ad strategy. From the left side nav bar, go to the Ad Unit and Placement page. Select the app, then the ad unit you want to run an A/B test on. Then, click Add to test at the top of the page.

Use the toggle at the top of the page to switch between group A (control) and group B (test). In Group B, under Capping & Pacing, click the edit icon to adjust the number of ads you want users to see, and the frequency at which the ads are shown.

Under Reward, click the edit icon to adjust the type of reward or the number of rewards a user gets for watching an ad.

Each change you make will apply only to the test group with the following exceptions that will update in both A and B groups:

ironSource instances
Network test mode status
Activation or deactivation of ironSource as a bidder
Callbacks

Test monitoring and reports

From the left side nav bar, select A/B testing. There you’ll see a list of both the apps that currently are running an A/B test and apps that have ended an A/B test. To see the performance of the test, select an app, click the action button, then, click View report.

This will take you to the Performance page. Break the data by A/B to see an overview of the difference in performance between the 2 groups. Click on the different ad units to see data from that particular ad unit.

To see additional data click on User Activity or Cohorts in the left side nav bar.

End a test

Individual ad units

You can remove an ad unit from your test at any time without it ending the entire test. From the Mediation Management page, click on the ad unit you want to remove from the test. Click on the dropdown menu at the top of the page and select to continue with either group A or B.

Click Approve and the test for that specific ad unit will end. Now the winning group’s settings will immediately apply to 100% of the traffic for the selected ad unit.

End all ad unit tests

To continue with group A or B across all ad units, go to the A/B Management page, and select the app with the active test. Then, click on the action button and select to continue with either group A or B.

Click Approve and the test will end. Now the winning group’s settings will immediately apply to 100% of the traffic on your app.