最佳实践:广告变现A/B测试

A/B测试可以确保您的广告变现方法获取最大化收益和业务增长。使用ironSource的A/B测试工具,您能够测试不同的广告变现策略,了解用户与广告的互动习惯并最终选择获胜策略。

A/B测试较为复杂,如果未正确进行测试,有可能得出错误结论。在本指南中,您将找到A/B测试的最佳实践,以便根据准确的结果做出最佳决策。

  1. 每次测试一个变量:优化广告策略时,您可能会发现有多个功能需要进行测试。评估数据有效性的唯一方法是隔离一个变量并测量其在测试中收到影响。这意味着您无法在测试一个新的广告网络的同时改变一个视频广告位的奖励数量。您可以测试多个变量,只要确保每个变量都进行了自己的独立测试即可。
  2. 不要变更对照组:确定要测试的变量后,请确保对照组(A组)保持不变。这意味着A组设置作为当前广告变现策略部署,不应做任何更改。您只需对B组进行更改,以挑战当前策略并比较结果。
  3. 确定目标:测试前,花一些时间考虑您希望改进哪些指标(KPI)以及这些改进的重要性。如果多个KPI可能受到影响,请确定哪个KPI最重要,以免在它们受到不同影响时对您的决策造成干扰。这将确保您可有效检查结果,以便对最佳策略做出明智的决策。
  4. .逐步扩大测试:即使是一项简单的更改也可能产生重大影响,因此,重要的是首先对一小部分用户(即10-20%)进行测试,以确保任何潜在的负面结果不会对您的业务造成大的影响。然而,如果一小部分用户的测试结果是有正面的,并不意味着应用至所有用户时会有同样的效果。您可以将B组上的配置部署至更多的用户(即80-90%),以模拟完整部署时的情形并确认测试结果是正确的。
  5. 给予足够的测试时间:每个测试都需要足够的时间来生成有效数据。否则就很难知道结果是否可靠。此外,B组使用的instance ID将是全新的,各家平台都需要一些时间来收集数据。每项测试都需要2-3天的时间来使同一层在A/B两组的效果趋同。我们建议测试至少进行一周,理想情况下应进行2周或更长时间。
  6. 给予足够的流量:如果您的测试组没有获得足够的流量,也会导致结果不可靠。若流量较少,测试结果会有很大的波动。根据我们的经验,B组每天应该获得至少1万展示,或者每层每天获得1千展示。如果不满足这个标准,请考虑增加测试组获得的流量比例。
  7. 不断进行测试:每改善一小点都可以为您带来额外的收入。任何变现策略都有优化空间,所以不要只停留在一项测试上,不断进行测试!

수익화 A/B 테스트

소개

수익화 A/B 테스트는 무엇인가?

A/B 테스트는 사용자를 여러 그룹으로 나누고 각 그룹을 다른 구성에 노출시키는 테스트입니다. 이는 다양한 광고 수익화 전략을 테스트하고 데이터 분석을 통해 결과를 비교할 수 있으므로 어떤 전략이 더 효과적인지 이해하는 데 도움이 됩니다.

ironSource의 수익 창출 A/B 테스트는 어떻게 실행되나요?

ironSource A/B 테스트 툴을 사용하면 별도 개발 노력 없이 A와 B 그룹 간의 트래픽 할당을 정의하고 테스트하려는 변수에 따라 앱 구성을 설정할 수 있습니다. 테스트가 실행되면 각 사용자는 두 그룹 중 하나에 할당되며 각각의 그룹에 대해 정의한 구성에 노출됩니다. 사용자가 한번 특정 그룹에 속하면 해당 사용자는 테스트가 종료되거나 그룹 간의 트래픽 할당이 업데이트 될 때까지 동일한 그룹에 남아 있습니다. 테스트 결과에 따라 어떤 그룹의 퍼포먼스가 더 좋았는지 결론을 내면 선택한 그룹의 구성을 모든 사용자에게 적용하여 테스트를 종료 할 수 있습니다.

A/B 테스트 할 수 있는 것은 무엇입니까?

아이언소스의 툴을 이용하여 수 많은 테스트를 해볼 수 있습니다. 새로운 광고 유닛 추가, 광고 배치당 보상 (Placement level reward) 또는 캡핑/간격 (Capping/Pacing) 설정, 다양한 워터폴 최적화 전략 등에 대한 조정 등이 포함됩니다. 어떤 전략이 적합할 지에 대한 자세한 상담이 필요한 경우 여러분의 담당자 또는 아이언소스에 문의해주세요.

테스트 설정

1단계. A/B 테스트 페이지로 이동하여 ‘Create New Test’를 클릭하세요.

1

2단계. 테스트 이름과 간단한 설명을 설정하고 그룹 간 트래픽 분할을 선택하세요.

2

3단계. 그룹 B의 네트워크 구성을 설정하세요.

이 단계에서 앱의 구성된 인스턴스 각각에 대해 새로운 인스턴스 ID를 입력해야합니다. 인스턴스 ID를 통해 그룹 간 퍼포먼스를 비교하기 위한 수익이 측정 가능합니다.

3

참고: 특정 인스턴스를 복사하고 싶지 않으시다면, 해당 인스턴스를 비활성으로 표시함으로써 그룹 B 구성에서 제외 시킬 수 있습니다.

참고: ironSource의 인스턴스 ID는 그룹 B에 자동으로 생성됩니다.

4 단계. ‘Start A/B Test’를 클릭하고 그룹 B에서 원하는 앱 구성을 적용합니다.

테스트는 다음 날 00:00 UTC에 시작됩니다.4

테스트 시작이 예약되는 즉시 ironSource는 모든 광고배치 설정, 워터폴 우선순위 지정 등 A 그룹 (제어 그룹)에서 B 그룹 (테스트 그룹)까지의 모든 기존 앱 구성을 복제합니다.

참고: ironSource 플랫폼에서 멀티플 인스턴스를 지원하지 않는 네트워크 및 광고 유닛은 테스트 중  A버전(제어 그룹)에만 포함될 수 있습니다.

그룹 B가 만들어지면 테스트 할 변수에 따라 플랫폼 설정을 조정할 수 있습니다. 각 관련 플랫폼 페이지에 토글이 추가되어 그룹 A와 그룹 B 사이 선택할 수 있습니다. 각 변경 사항은 그 특정 플랫폼 페이지의 그룹에만 적용됩니다.

5

관련 플랫폼 페이지는 다음과 같습니다.

  • 광고 유닛과 배치
  • 서버 측 네트워크
  • SDK 네트워크
  • 미디에이션 관리
  • 세그먼트

5단계. 테스트를 종료하세요.

결과를 분석하여 어느 그룹이 더 좋은 성과를 보였는지 결론을 내린 후에는 “Continue with A” 또는 “Continue with B” 버튼을 클릭하여 더 나은 그룹을 선택하고 테스트를 종료할 수 있습니다. 테스트를 종료하면 애플리케이션 트래픽의 100%가 성과가 우수했던 설정이 적용됩니다.

6

참고: 테스트 종료는 취소할 수 없습니다.

참고: 테스트는 00:00 UTC에 따라 다음 날 종료됩니다.

테스트가 종료되면 이름, 시작 날짜 및 종료 날짜를 포함하여 히스토리 탭에 추가됩니다.

모니터링과 보고서

활성화 되어있는 A/B 테스트를 모니터링하거나 종료된 A/B 테스트의 결과를 검토하려면 A/B 테스트 개요 모듈 또는 아이언소스 플랫폼의 성과 및 유저 활동 레포트(Performance and User Activity reports)를 사용하세요.

A/B 테스트 개요:

A/B 테스트 개요 페이지에서 활성 A/B 테스트에 대해 다음 작업을 수행할 수 있습니다:

2

  • 테스트 편집: 테스트 도중 이름, 설명 및 트래픽 변동을 수정할 수 있습니다. 트래픽 변동은 00:00 UTC에 따라 다음 날에 적용됩니다.
  • 보고서 보기: 사전 필터링 된 앱, 테스트 시작일/종료일 선택 및 A/B별로 구분된 데이터에 따라 퍼포먼스 보고 페이지로 연결됩니다.
  • 테스트 종료: 테스트를 종료하면 우승한 그룹의 구성이 애플리케이션 트래픽의 100 %에 적용됩니다.

개요 페이지에는 마지막 트래픽 변동 변경부터 현재 날짜까지 그룹 A와 B 사이의 측정 항목 비교 차트도 표시됩니다.

3

앱 레벨 측정 항목:

  • 수익
  • DAU
  • ARPDAU
  • 1일차 잔존율
  • 7일차 잔존율

광고 유닛 레벨 측정 항목:

  • 수익
  • 노출
  • eCPM
  • 필 레이트
  • 노출/ DAU
  • 노출/DEU

보고 페이지에서 A/B 테스트

퍼포먼스 및 유저 활동 보고서에서 A/B 테스트를 분석할 수 있으며 모든 측정 항목이 지원됩니다. A/B로 구분된 보고서를 보려면 A/B 테스트를 실행했거나 현재 실행중인 애플리케이션을 하나 선택해야 합니다.

5

모든 ‘분류’ 옵션은 A/B 테스트 중에도 지원됩니다. 그룹 B의 중복된 인스턴스, 태그, 게재 위치 및 세그먼트 (테스트 생성 전에 이미 존재했던 인스턴스)는 연결되어 있습니다. 예를 들어 인스턴스별로 분류 할 때 중복된 각 인스턴스 이름에 대해 A그룹과 B그룹의 통합된 데이터가 표시된다는 점 참고하시길 바랍니다.

이제 모범 사례 가이드에서 효과적인 A/B 테스트를 실행하는 방법에 대한 주요 팁을 확인해보세요.

모범 사례: 광고 수익화 A/B 테스트

모든 비즈니스 전략에서 A/B 테스트는 수익 극대화 및 사업적 성장을 위한 중요 구성 요소입니다.  ironSource의 A/B 테스트 툴을 이용하면 다양한 광고 구현을 테스트 해볼 수 있으며, 사용자가 광고를 어떻게 사용하는지 파악하고 최고의 전략을 선택할 수 있습니다.

A/B 테스트는 복잡할 수 있으며 제대로 수행되지 않으면 잘못된 결론을 내릴 위험이 있습니다. 최적의 A/B 테스트에 대한 모범 사례를 참고하여 정확한 결과 분석을 통한 최선의 판단을 내리세요.

  1. 한번에 하나의 변수를 테스트 하기: 광고 전략을 최적화하면 테스트해보고 싶은 기능이 여러 개 있을 수 있습니다. 어떤 변경 요소의 유효성을 효과적으로 확인하기 위해서는 하나의 변수를 분리하여 그 영향을 측정하는 것입니다. 예를 들어 비디오 배치의 보상금을 변경하면서, 동시에 새로운 네트워크를 라이브하면 안됩니다. 두개 이상의 변수를 모두 테스트 해 볼 수는 있으나, 한 테스트당 하나의 변수만 둘 것을 명심하십시요.
  2. 제어 그룹을 변경하지 마세요: 테스트할 변수를 식별했으면 현재 설정을 기준으로 비교하고 결과를 측정하기 위해 제어그룹(A그룹)은 그대로 두고 그룹 B에서만 변경하세요. 즉, 그룹 A의 설정은 현재 광고 적용 하에 이미 되어 있는 설정 그대로 두어야 합니다.
  3. 목표를 정확하게 설정하세요: 테스트를 시작하기 전에, 어떤 KPI를 개선하고 싶은지, 그리고 얼마나 개선해야 할지에 대해 생각해 보세요. 여러 KPI가 영향을 받을 수 있다면, 어떤 KPI에 가중치를 둘 것인지 결정하십시오. 이렇게 하면 결과를 효과적으로 검사하여 최상의 전략에 대해 객관적 정보에 입각한 결정을 내릴 수 있습니다.
  4. 점진적으로 테스트하세요: 간단한 변경 사항도 큰 영향을 미칠 수 있으므로 소수의 사용자(예: 10-20%)를 대상으로 테스트를 시작하여 잠재적인 부정적 결과가 비즈니스에 큰 영향을 미치지 않도록 하는 것이 중요합니다. 소수의 사용자 그룹에서 좋은 결과가 있었다고 해도 모든 사용자에게 배포되었을때 동일한 결과를 얻지 못할 수도 있습니다. 때문에, 그룹 B의 트래픽 할당을 훨씬 더 크게 (예: 80-90 %) 업데이트하여 전체 출시와 유사한 환경을 설정하고 테스트 결과가 유효함을 확인하십시오.
  5. 충분한 시간을 가지세요: 각 테스트는 유용한 데이터를 생성하기에 충분한 시간이 필요합니다. 그렇지 않으면 결과가 믿을 만한지 알기 어렵습니다. 또한 그룹 B는 모든 신규 인스턴스 ID와 함께 테스트를 진행하게 되는데 이 신규 인스턴스 ID의 경우 데이터를 수집할 시간이 필요합니다. 각 테스트는 그룹 B의 인스턴스가 그룹 A와 공정하게 다뤄질 수 있기 까지 2~3일 간의 시간이 걸립니다. 테스트는 최소 1주일 이상 실시되어야 하며 2주 이상이 이상적입니다.
  6. 충분한 트래픽이 필요합니다: 테스트 그룹에 트래픽이 충분하지 않으면 데이터의 정확도와 신뢰도가 떨어집니다. 소량의 트래픽에만 노출 될 경우 퍼포먼스가 크게 변동 할 수 있습니다. 어림법칙으로 그룹 B는 총 일일 노출 수 1만 또는 현재 사용중인 각 인스턴스 당 일일 노출 수 1천번을 받아야합니다. 만일 충족되지 않으면 테스트 그룹 할당량을 늘리는 것이 좋습니다.
  7. 항상 테스트해야 합니다: 작은 변동사항들이 합쳐져 추가 매출을 올릴 수 있습니다. 최적화가 가능한 부분은 항상 있으므로 한 번의 테스트만으로 그치지 마십시오. 꾸준히 테스트를 하시길 추천드립니다!

Reporting API

Use this API to receive all the reporting data from the monetization ad units you serve with ironSource, including multiple metrics such as revenue, impressions and active users, etc. across several breakdowns and optional filters.

Before you start make sure you go over our API introduction article.

Note: Our Reporting API is limited to 40 requests every 10 minutes.

Authentication type

Bearer API authentication

Method

GET  platform.ironsrc.com/partners/publisher/mediation/applications/v6/stats?

Required Parameters

Name Type Description Default
startDate String YYYY-MM-DD (UTC Timezone)
endDate String YYYY-MM-DD (UTC Timezone)

Optional Parameters

With the ironSource reporting API, reports can be filtered by any one of the optional parameters. Doing so searches for a direct match to any of the filters.

Name Type Description Default
appKey String Application Key (as seen on our platform). All App keys
country String 2 letter country code, as per ISO 3166-1 Alpha-2. All Countries
adUnits String Options are ‘rewardedVideo’, ‘offerWall’, ‘interstitial’ or ‘banner’. All Ad Units
adSource String Mediated ad networks, Including ironSource All Ad Sources
metrics String For available metrics – See below All Metrics
breakdowns String For available breakdowns – See below adUnits, date


Supported Breakdowns

  1. date
  2. app
  3. platform
  4. adSource
  5. adUnits
  6. instance
  7. country
  8. segment
  9. placement
  10. OS version
  11. Connection type
  12. SDK version
  13. App version
  14. ATT
  15. IDFA
  16. A/B

Check out all of our supported breakdowns by metric 

Supported Metrics

Check out all of our available metrics 

Note: All our parameters support multiple comma separated values.

Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/applications/v6/stats?startDate=2018-08-01&endDate=2018-08-01&appKey=6fa4475&adSource=ironSource&metrics=revenue,impressions,eCPM&breakdown=adUnits

JSON Example Response

[
  {
    "adUnits": "Rewarded Video",
    "date": "2018-08-01",
    "data": [
      {
        "revenue": 71188.8,
        "impressions": 13321624,
        "eCPM": 5.34
      }
    ]
  },
  {
    "adUnits": "Interstitial",
    "date": "2018-08-01",
    "data": [
      {
        "revenue": 39437.81,
        "impressions": 6077264,
        "eCPM": 6.49
      }
    ]
  },
  {
    "adUnits": "Offerwall",
    "date": "2018-08-01",
    "data": [
      {
        "revenue": 43.78,
        "impressions": 14873,
        "eCPM": 2.94
      }
    ]
  }
]

API Versions

Ad Revenue Measurement API

Description

Use this API to get device level ad revenue data for ironSource and your mediated ad networks. The revenue per device is calculated based on the eCPM of each impression, according to the app, day, network, country, and instance.

Before you start, make sure you review our API introduction article.

Notes:

  • Pulling Ad Revenue Measurement data is done in 2 steps:
    • Make an API request which returns a URL link to download the report
    • Download the report by clicking on the provided link
  • Each API call must be made for a single app and single date.
  • The data for each date is stored and available for only 14 days.
    • Example: Data for January 1st will be available only until January 14th
  • Each day, the data for yesterday is available starting 02:00 PM UTC. Make sure to only call the API after 02:00 PM UTC each day.
    • Example: The data for January 1st is only available on January 2nd, after 02:00 PM UTC
  • Each day, we update the data for yesterday and the day before yesterday. The data will not change after 2 days.
    • Example: The data for January 1st will update on January 2nd and January 3rd. After January 3rd, the data will not change

API request for report URL

Note: A valid API hit returns a link to the requested file. The file will be available to download for one hour only

Authentication Type

Bearer API authentication

Method

GET  platform.ironsrc.com/partners/userAdRevenue/v3?

Required Parameters 

Name Type Description
date String YYYY-MM-DD (UTC Timezone)
appKey String Application Key (as seen on our platform)
reportType Integer Supporting different report types. At the moment, should always set to 1


Request Example URL

https://platform.ironsrc.com/partners/userAdRevenue/v3?appKey=12345678&date=2018-12-15&reportType=1

Response Example URL

https://user-ad-revenue.s3.amazonaws.com/data/bi_databases/uar/tables/network/report_version%3Dv1/publisher_id%3D138101/application_id%3D100411/day%3D2018-07-13/000000.gz?AWSAccessKeyId=AKIAILCCGIXP5G2XNXBA&Expires=1532524582&Signature=1U%2FbrLmb%2F7TwXXdoHtb%2F%2FQjrmz0%3D

Download the report

Simply click the URL to download the report

Understand the report

The report includes the following columns:

  • Ad Unit
  • Advertising ID
  • Advertising ID Type
  • User ID
  • Segment (if applicable)
  • Placement
  • Ad Network
  • A/B Testing (if applicable)
  • Impressions
  • Revenue

Note: Country breakdown is not included in the report in order to minimize tracking discrepancies with the attribution partners. 

Fields Definition

Field Description Example
Ad Unit The ad unit from which the revenue was generate from rewarded_video
Advertising ID The unique ID of the user’s device 21d0e51a-772d-4b23-9fc4-296126b4f123
Advertising ID Type The unique ID type of the user’s device
IDFA/IDFV/GAID
User ID The user ID being set by the developer 135920
Segment The segment the user is related to as defined on the platform Tier 1
Placement The placement the user is related to as defined on the platform Home_Screen
Ad Network The ad network name 
Admob
A/B Testing  A/B testing group the user was allocated to (can be ‘A’ or ‘B’) ‘A’
Impressions The number of ads displayed to the user 1000
Revenue The Revenue generated by the user (USD) 0.5

* Date & App Key aren’t included as they are unique per report (file)

Mediation Management API

Use this API to manage all group and waterfall configurations you have set on ironSource’s dashboard.

Before you start make sure you go over our API introduction article.

Note: Mediation Management API does not support apps during a live A/B test

Get Mediation Groups

Description
Use this API to retrieve a list of each of your application’s groups and their waterfall configurations. The response will include the following: group name, group ID, group segment, group countries, group position, group waterfall, and more.

Authentication Type

Bearer API authentication

Method

GET https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String application Key (as seen on our platform)

Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/management/v1?appKey=813df9s5

Response Parameters

Name Description
appName the application name
appStatus app status can be active or inactive
platform iOS or Android
bundleId the native app store bundle_id
adUnits includes the active ad sources per ad unit
groupName name as defined on the platform
groupId the mediation group’s ID
groupPosition the group’s hierarchy vs. other groups
groupSegments targeted segments (if defined)
groupCountries targeted countries (country codes)
adSourcePriority waterfall setup
providerName ad source provider name (e.g Facebook, ironSource)
instanceName instance name as defined on the platform 
instanceId either auto-generated by ironSource network or input manually for other networks

JSON Example Response

{
  "appKey": "a123y419",
  "appName": "ironsource App",
  "appStatus": "active",
  "platform": "Android",
  "bundleId": "com.ironsource.app",
  "adUnits": {
    "rewardedVideo": [
      {
        "groupName": "US_High",
        "groupId": 275212,
        "groupPosition": 1,
        "groupSegments": [
          {
            "id": "6112",
            "name": "age_Over_16"
          }
        ],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "US"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1636704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1646701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "All Countries",
        "groupId": 275310,
        "groupPosition": 2,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1636704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1646701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ],
    "interstitial": [
      {
        "groupName": "All Countries",
        "groupId": 275111,
        "groupPosition": 1,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1699706
            }
          ],
          "auto": [
            {
              "providerName": "ironSource",
              "instanceName": "is_IS_Bidder",
              "instanceId": 163302
            },
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ]
  }
}

Create Mediation Groups

Description
Use this API to create mediation groups and define each groups waterfall setup. 

General notes:

  1. Supports creation of only one mediation group per call.
  2. If ‘adSourcePriority’ is missing in the call, all networks that are active in the app will be placed within the waterfall’s “auto optimization” section of the newly created group.
  3. If ‘adSourcePriority’ is included in the call, and “manualHigh” and “manualLow” parameters are set, the instances will be placed based on their order in the call itself.
  4. If ‘groupPosition’ is missing in the call, the newly created group will be placed on top of the existing groups.

Authentication Type

Bearer API authentication

Method

POST https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String application Key (as seen on our platform)
adUnit String “rewardedVideo”, “interstitial”, “banner”

groupName

String the newly created group’s name 
groupCountries
String array the newly created group’s targeted countries, countries are defined by a 2 letter country code, as per ISO 3166-1 Alpha-2. To include all countries simple add asterisk (*) 

Optional Parameters

Name Type Description
groupPosition Integer
the newly created group’s hierarchy vs. existing groups 
groupSegments String array the newly created group’s targeted segments (ID)

adSourcePriority

String array the newly created group’s waterfall setup. Can include 3 sections: ‘manualHigh’, ‘auto’, ‘manualLow’, ‘bidders’


Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/management/v1?appKey=8a5513f5

Request Body Example 

{
  "appKey": "8a5513f5",
  "adUnit": "rewardedVideo",
  "groupCountries": [
    "JP"
  ],
  "groupName": "Japan_High",
  "adSourcePriority": {
    "bidders": [
      {
        "providerName": "AdColony",
        "instanceId": 1626704
      },
      {
        "providerName": "ironSource",
        "instanceId": 1626701
      }
    ],
    "manualHigh": [
      {
        "providerName": "ironSource",
        "instanceName": "Default",
        "instanceId": 0
      }
    ],
    "auto": [
      {
        "providerName": "AdColony",
        "instanceName": "Default",
        "instanceId": 0
      }
    ]
  }
}

JSON Example Response

{
  "appKey": "8a5513f5",
  "adUnits": {
    "rewardedVideo": [
      {
        "groupName": "Japan_High",
        "groupId": 275903,
        "groupPosition": 1,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "JP"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "manualHigh": [
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "US_High",
        "groupId": 275112,
        "groupPosition": 2,
        "groupSegments": [
          {
            "id": "6112",
            "name": "age_Over_16"
          }
        ],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "US"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "All Countries",
        "groupId": 275110,
        "groupPosition": 3,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ],
    "interstitial": [
      {
        "groupName": "All Countries",
        "groupId": 275111,
        "groupPosition": 1,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626706
            }
          ],
          "auto": [
            {
              "providerName": "ironSource",
              "instanceName": "is_IS_Bidder",
              "instanceId": 1626702
            },
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ]
  }
}

Update Mediation Groups

Description
Use this API to update the mediation group’s name, position, targeted countries, targeted segments and waterfall setup.

General notes:

  1. Supports updating of only one mediation group per call.
  2. If ‘adSourcePriority’ is included in the call, and “manualHigh” and “manualLow” parameters are set, the instances will be placed based on their order in the call itself.
  3. Any optional parameter that is not included in the call (‘groupName’, ‘groupPosition’, ‘groupSegments’, ‘groupCountries’ or ‘adSourcePriority’) will not impact the group’s existing setup.
  4. Any optional parameter that is included in the call (‘groupName’, ‘groupPosition’, ‘groupSegments’, ‘groupCountries’ or ‘adSourcePriority’) will replace the group’s existing setup.

Authentication Type

Bearer API authentication

Method

PUT https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String application Key (as seen on our platform)
groupId Integer unique group ID

Optional Parameters

Name Type Description
groupName String the desired group’s new name
groupPosition Integer the desired group’s new position

groupSegments

String array the desired group’s newly targeted segments
groupCountries String array the desired group’s newly targeted countries. countries are defined by a 2 letter country code, as per ISO 3166-1 Alpha-2. To include all countries simple add asterisk (*)
adSourcePriority String array the desired group’s newly waterfall setup. Can include 3 sections: ‘manualHigh’, ‘auto’, ‘manualLow’, ‘bidders’


Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/management/v1?appKey=8a2cbac5

Request Body Example 

{
  "appKey": "8a2cbac5",
  "groupId": "275903",
  "groupName": "Japan_High",
  "groupPosition": "1",
  "countries": [
    "JP"
  ],
  "adSourcePriority": {
    "bidders": [
      {
        "providerName": "AdColony",
        "instanceId": 1626704
      },
      {
        "providerName": "ironSource",
        "instanceId": 1626701
      }
    ],
    "manualHigh": [
      {
        "providerName": "ironSource",
        "instanceName": "Default",
        "instanceId": 0
      }
    ],
    "auto": [
      {
        "providerName": "AdColony",
        "instanceName": "Default",
        "instanceId": 0
      }
    ]
  }
}

JSON Example Response

{
  "appKey": "8a2cbac5",
  "adUnits": {
    "rewardedVideo": [
      {
        "groupName": "Japan_High",
        "groupId": 275903,
        "groupPosition": 1,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "JP"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "manualHigh": [
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "US_High",
        "groupId": 275112,
        "groupPosition": 2,
        "groupSegments": [
          {
            "id": "6112",
            "name": "age_Over_16"
          }
        ],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "US"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "All Countries",
        "groupId": 275110,
        "groupPosition": 3,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626704
            },
            {
              "providerName": "ironSource",
              "instanceId": 1626701
            }
          ],
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ],
    "interstitial": [
      {
        "groupName": "All Countries",
        "groupId": 275111,
        "groupPosition": 1,
        "groupSegments": [],
        "groupAbVersion": "N/A",
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "bidders": [
            {
              "providerName": "AdColony",
              "instanceId": 1626706
            }
          ],
          "auto": [
            {
              "providerName": "ironSource",
              "instanceName": "is_IS_Bidder",
              "instanceId": 1626702
            },
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ]
  }
}

Delete Mediation Group

Description
Use this API to delete a mediation group

General notes:

  1. Can delete one mediation group per call.
  2. ‘All Countries’ default group cannot be deleted.
  3. Deleted mediation groups cannot be restored.
  4. Response includes all of the app’s mediation groups excluding the deleted ones.

Authentication Type

Bearer API authentication

Method

DELETE https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String application Key (as seen on our platform)
groupId Integer unique group ID


Request Example URL

https://platform.ironsrc.supersonicads.com/partners/publisher/mediation/management/v1?groupId=182641&appKey=8a2edd51

JSON Example Response

{
  "appKey": "8a2edd51",
  "adUnits": {
    "rewardedVideo": [
      {
        "groupName": "Tier 1",
        "groupId": 182951,
        "groupPosition": 1,
        "groupSegments": [],
        "groupCountries": [
          "AU",
          "CN",
          "DE",
          "GB",
          "US"
        ],
        "adSourcePriority": {
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        }
      },
      {
        "groupName": "All Countries",
        "groupId": 182637,
        "groupPosition": 2,
        "groupSegments": [],
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "auto": [
            {
              "providerName": "AdColony",
              "instanceName": "Default",
              "instanceId": 0
            },
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ],
    "interstitial": [
      {
        "groupName": "All Countries",
        "groupId": 182638,
        "groupPosition": 1,
        "groupSegments": [],
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "auto": [
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ],
    "banner": [
      {
        "groupName": "All Countries",
        "groupId": 182639,
        "groupPosition": 1,
        "groupSegments": [],
        "groupCountries": [
          "*"
        ],
        "adSourcePriority": {
          "auto": [
            {
              "providerName": "ironSource",
              "instanceName": "Default",
              "instanceId": 0
            }
          ]
        },
        "isAllCountries": true
      }
    ]
  }
}

Mediation

Use this API to get all the groups and waterfalls you have set on ironSource’s dashboard.

Before you start make sure you go over our API introduction article.

Get Mediation Groups

Description
Use this API to retrieve a list of all of your application’s groups and its matching waterfalls setup. The response will include the following: group name, group ID, group segment, group countries, group position and group waterfall.

Authentication Type

Bearer API authentication

Method
GET https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String Application Key (as seen on our platform)

Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/management/v1?appKey=813df9s5

Response Parameters

Name Description
appKey
adUnit
groupName
countries
segments
groupPosition
provider
instanceName

JSON Example Response

Mediation Management

Use this API to get all the groups and waterfalls you have set on ironSource’s dashboard.

Before you start make sure you go over our API introduction article.

Get Mediation Groups

Description
Use this API to retrieve a list of all of your application’s groups and its matching waterfalls setup. The response will include the following: group name, group ID, group segment, group countries, group position and group waterfall.

Authentication Type

Bearer API authentication

Method
GET https://platform.ironsrc.com/partners/publisher/mediation/management/v1?

Required Parameters

Name Type Description
appKey String Application Key (as seen on our platform)

Request Example URL

https://platform.ironsrc.com/partners/publisher/mediation/management/v1?appKey=813df9s5

Response Parameters

Name Description
appKey
adUnit
groupName
countries
segments
groupPosition
provider
instanceName

JSON Example Response


	

Best Practices: Monetization A/B Testing

A/B testing is a crucial component of every business strategy and involves testing your assumptions against a control to ensure your new approach maximizes revenue and grows your business. With Unity LevelPlay’s A/B testing tool, you can experiment with different monetization variables to understand how your users engage with ads, and choose a winning strategy.

A/B testing requires smart planning and defined goals to get the most conclusive results. The following best practices will help you conduct clean tests using the Unity LevelPlay A/B testing tool, so you can make the best decisions based on accurate data.

Best practices for setting up an A/B test

  1. Identify your goal: Take some time to think about the KPIs you want to improve, and how significant you want the improvement to be. If multiple KPIs can be affected, decide which carries the most weight in case they’re impacted differently. This will ensure you examine your results effectively so you make an informed decision on the best strategy.
    Make sure you have the answers to these questions before you start a test:
    • What is the purpose of the test?
    • What are the KPIs I want to test?
    • What would be considered a success?
  2. Test 1 variable at a time: As you optimize your ad strategy, you’ll probably find that there are multiple variables you want to test. The only way to effectively evaluate the significance of any change is to isolate one variable and measure its impact. This means, for example, that you can’t set a new network live while also changing the reward amount of a video placement.
  3. Changes should only be made in group B: Once you’ve identified the variable you want to test, leave your control group (group A) unaltered. The settings in group A already exist as your current ad implementation. Changes should only be made in group B, to challenge the current implementation and compare the results.
  4. Use instance rates in group B for waterfall tests: When setting up the test, use instance rates for the first 3 days. This will allow the instances in group B to “learn” and stabilize their eCPMs. In combination with the “Sort by CPM” function, this will automatically sort the waterfall by descending eCPM from the start of the test. IMPORTANT – remove the rates at the end of the 3rd day.

Best practices while an A/B test is running

  1. Monitor the test data: Keep track of your data to make sure that nothing too drastic occurs as a result of the change you made. However, it’s important not to terminate the test for the first 3 days, even if group B performance is lower than group A, because the data might be inaccurate at the beginning of the test.
    • Ignore data from the test initiation date, since it includes activity reported prior to the test
    • Ignore data from the first 2 days for waterfall tests, since group B includes new instance IDs that need time to learn
  2. Leave group B alone: Don’t make changes in group B during the test. Every test should address only one assumption and variable during the entire test. If you want to test a different assumption or even make small adjustments to the current test, terminate the test and start another one. This is crucial and will help to keep the data organized and clean for analysis.
  3. Give it enough time: Each test needs sufficient runtime to produce useful data. A good rule of thumb is to analyze the test results after one week. When testing apps with a low amount of users or impressions, wait for more than a week to gather enough data.

Best practices for analyzing the results

  1. Look at the right data: Exclude the first day of the test from your analysis. When testing the waterfall setup, exclude the first 2 days.
  2. Ignore daily data: When testing an assumption, you are aiming for long-term improvement rather than daily changes. Since daily performance can be volatile, analyze the data for the entire test as a whole without breaking it into days.
  3. Close the loop: Compare the data and the performance of the two groups in accordance with the KPIs you set before initiating the test. Make sure to answer the following questions:
    • Did the test reach the goals you set?
    • What is your winning group?
    • What is the next assumption that should be tested?
  4. Dig into the data: Try to understand the root cause of why Group B performed better or worse than Group A. The insights will guide you in setting up the next test.

Make an A/B testing routine to keep improving your app performance. Small and incremental changes can quickly add up to drive significant revenue growth, and there’s always room for more optimization. Use the insights from the analysis of the previous tests to understand how you can further improve in your next tests.

You can read this article in:

Monetization A/B testing

What is monetization A/B Testing?

An A/B test is a controlled test that splits users into different groups, and exposes each group to different app configurations. This allows you to test different monetization strategies and compare results through data analysis, thus helping you understand which strategy performs better.

How does ironSource’s monetization A/B Testing work?

With ironSource A/B testing module, you can define traffic allocation between A and B groups, and setup the app configurations according to the variables you want to test – all with no development effort required. Once the test is live, each user is assigned to one of two groups, and will be exposed to the configurations that you have defined for each respective group. Once a user belongs to a group, that user will remain in the same group until either the test is terminated or the traffic allocation between groups is updated. Based on test results, once you’ve reached a conclusion on which group performs better, you can terminate the test by applying the configurations from the chosen group to all users.

What can you A/B test?

There’s an endless amount of tests you can run with our tool. Some of these tests include rolling out new ad units, adjustments to placement level reward or capping/pacing settings, bidding vs traditional waterfall, different waterfall optimization strategies, etc. Reach out to your account manager or contact us to discuss which strategy is right for you.

Test setup

Step 1. Navigate to the A/B testing page and click on Create New Test.

1
Step 2. 
Setup your test name, description and define the traffic split between groups.
2

Step 3. Setup network configurations on group B.
At this stage you are required to input new and unique instance IDs for each one of your app’s configured instances. This allows us to track unique revenue to compare performance between groups.

3

Keep in mind: 

  • If you do not want to duplicate a specific instance, you can mark it as inactive and this instance will not be included within group B’s configurations.
  • ironSource’s instance IDs are automatically generated for group B.

Step 4. Click on ‘Start A/B Test’ and apply the desired app configurations on group B.
The test will start the next calendar day according to 00:00 UTC.

4

Once the test is scheduled to start, ironSource duplicates all existing app configurations from group A (the control group) to group B (the test group), including all placement settings, waterfall positioning, etc.

Note: Networks and ad units that do not support multiple instances on ironSource platform can only be included in A version of the test. 

Important!
When setting up the test and for the first few days of the test, use rates for the instances. This will allow the instances in group B to “learn” and stabilize their eCPMs.  Using Rates (combined with “Sort by CPM” function) at the beginning of the test will allow the waterfall to be sorted according to descending eCPM from the start of the test. This is especially important when using In-App Bidding, so the Bidders will be placed properly in the waterfall. Once Rates are set, the Bidder can be activated immediately

Once group B is created, you can adjust the platform settings according to the desired variables you would like to test. A toggle will be added to each relevant platform page which allows you to switch between group A and B. Each change applies ONLY to the group you are on.

5

Relevant platform pages include:

  • Ad Units & Placements
  • Server Side Networks
  • SDK Networks
  • Mediation Management
  • Segments

Step 5. Terminate Test
Once you’ve analyzed results and reached a conclusion about which group performs better, you can choose the winning group and terminate the test by clicking on the “Continue with A” or Continue with B” buttons. Terminating a test means the winning group’s settings will apply to 100% of the traffic on your application.

6

Note: Once you’ve terminated a test, this action can’t be undone.  The test will be terminated the next calendar day according to 00:00 UTC.

Once a test is terminated, it will be added to the History tab including the name, start date, and end date. 

Monitoring and Reporting

To monitor an active A/B test or review the results of a terminated A/B test, use either the A/B test Overview module or the platform reporting pages (Performance and User Activity reports).

A/B Testing Overview:

On the A/B testing Overview page you are able to perform the following actions on an active A/B test:

6

  • Edit Test: Name, description and traffic variation are editable during a test. Note that a traffic variation change will take place the next calendar day according to 00:00 UTC.
  • Show Reports: Redirect to Performance reports page, with the app pre-filtered, test start/end dates selected, and the data broken down by A/B.
  • Terminate Test: Terminating a test will apply the winning group’s configurations to 100% of the traffic on your application.

The Overview page also displays metrics comparison charts between group A to B, from the last traffic variation change to the current day’s date.

3

App Level metrics:

  • Revenue
  • DAU
  • ARPDAU
  • Day 1 Retention
  • Day 7 Retention


Ad Unit Level Metrics:

  • Revenue
  • Impressions
  • eCPM
  • Fill Rate
  • Impressions / DAU
  • Impressions / DEU

A/B Testing on reporting pages

A breakdown by A/B is available on the Performance and User Activity reports, with all metrics supported. To view reports broken by A/B, you must select one application which have run or is currently running an A/B test.

5

All ‘break by’ options are still supported during an A/B test. Note that duplicated instances, tags, placements and segments on group B (those that already existed prior to test creation) are connected by name. For example, when breaking down by instance, you will see aggregated data from both A and B groups for each duplicated instance name.

Best Practices

A/B testing is a crucial component of any business strategy, to ensure your monetization approach is maximizing revenue and growing your business. With ironSource’s A/B testing tool, you’ll be able to experiment with different ad implementations to conclusively understand how your users engage with ads and choose a winning strategy.

  1. Use rates for the instances in group B: When setting up the test and for the first few days of the test, set rates for the instances. This will allow the instances in group B to “learn” and stabilize their eCPMs. Using rates (combined with the “Sort by CPM” function) at the beginning of the test will allow the waterfall to be sorted according to descending eCPM from the start of the test. This is especially important when using in-app bidding, and will ensure that the bidders are placed properly in the waterfall. Once rates are set, the bidder can be activated immediately.
  2. Test one variable at a time: As you optimize your ad strategy, you’ll probably find that there are multiple features you want to test. The only way to effectively evaluate the significance of any change is to isolate one variable and measure its impact.  This means that you can’t set a new network live while also changing the reward amount of a video placement. You can test more than one variable, just make sure each variable gets its own unique test.
  3. Leave your control group alone: Once you’ve identified the variable you want to test, make sure you leave your control group (group A) unaltered. Meaning that the settings on group A exists already as your current ad implementation, and no changes should be made. Changes should only be made on group B, to challenge the current implementation and compare the results.
  4. Identify your goal: Before starting a test, take some time to think about the KPIs you want to improve, and how significant you want the improvement to be.  If multiple KPIs can be affected, decide which carry the most weight in case they’re impacted differently. This will ensure you examine your results effectively so you make an informed decision on the best strategy.
  5. Test in increments: Even a simple change can have major impact, so it’s important to start your test on a small percentage (10-20%) of users to ensure that any potential negative results don’t significantly impact your business. Once you see positive results on a small group of users, it’s not a certainty that those results will hold when rolling out to all users. Update the traffic allocation on group B to be much larger (80-90%) to mimic a full rollout, and confirm the test results hold true before expanding to 100%.
  6. Give it enough time: Each test needs sufficient time run to produce useful data. Otherwise it’ll be hard to know if your results are reliable. Also, group B is going live with all new instance IDs, which need time to learn and gather experience. Each test should be given a 2-3 day runway for the instances on group B to equalize with group A. We recommend that a test should run for an absolute  minimum of one week, with 2 weeks or more being ideal.
  7. Give it enough traffic: If your test group doesn’t have enough traffic, that can also make your data unreliable. Performance can fluctuate a lot when only exposed to a small amount of traffic. A good rule of thumb is that group B should receive 10K daily impressions, or 1K daily impressions for each active instance. Consider increasing the test group allocation if this threshold is not met.
  8. Always be testing: Incremental changes can quickly add up to drive additional revenue. There’s always room for more optimization, so don’t just stop at one test. Keep testing!

Now check out our best practices guide for top tips on how to run an effective A/B test.

You can read this article in