Creative A/B test settings

  • Updated

In this section, you'll learn how to set up a creative A/B test. A/B tests are set up through ad groups within each campaign and can be used to compare the performance of different creative groups.

For example, the creative A/B test feature allows you to allocate a part of the existing budget to see if the new concept/theme of Creative Group B is more efficient than the existing Creative Group A.

Note: We don't recommend using A/B testing to compare the performance of different creative formats (Image vs. Video or Video vs. Native). This is because the ad placements for each format are different, Thus, the difference in performance can be caused due to the different ad placements of the creative formats themselves as opposed to the performance of the actual creatives.


Set up a creative A/B test 

A/B tests need to be set up within each campaign under the ad group section.

  1. Select the ad account from the drop-down menu on the top left.

  2. Select the app on the left sidebar.

  3. Click the campaign title from the campaign lists.

  4. Scroll down to see the ad group lists under that campaign. Under the list of ad groups, you'll see a section that is named A/B Test Groups.


  5. Click Get Started to set up the A/B Test.

  6. Fill out the Name and Description (option) of the A/B test.

  7. Select Creative Group for the variable

  8. Use the drop-down menu to select an ad group for Group A. Group A is a control group to compare the test group's results. ___________2022-06-10______2.38.40.png

    Important: Make sure you create the ad groups that you want to test in advance before setting up the test. Learn more about ad group settings
  9. Once you select an ad group for Group A, you will see the list of creative groups that are within that ad group that will be used for the creative A/B test.
    Note: Control Group (Group A)'s creative groups cannot be modified once created.
  10. Group B is a test group using the same ad group as Group A. The target will be randomly divided to avoid overlap between the two groups. Use the drop-down menu to select all of the creative groups you want to add to Group B.
    Note: Once the test is activated, you cannot edit the creative groups being tested.
  11. Once creative groups for both Group A (control) and Group B (test) have been set, you will be able to set the budget, key metric, and schedule for the test.
  12. Enter the percentage of budget from the original campaign you wish to apply to the test. 
    • The budget ratio between the control group and the test group is fixed 1:1 within the set budget share. For example, if the total daily budget is $1,000 with 30% of the budget share, Group A and B's daily budget will be $150, respectively.___________2021-09-16______8.48.44.png
    • You can increase the percentage of the budget set for the A/B test up to 100%.
      Note: If the daily budget is set differently for each day of the week, the daily A/B test budget will be changed according to the daily budget allocated for each day of the week.
    • The budget cannot be modified once the test has been activated.
  13. Select the Key Metric. The key metric refers to the main metric you wish to compare for the test. 
    • CPA is available only for in-app event optimization campaigns and the event used to measure the performance of the A/B test is the same as the campaign's optimization target events.
  14. Set the schedule of the test. It is required to set a start and end date.Screen_Shot_2021-07-20_at_5.16.49_PM.png 
    • The timezone is dependent on the time zone of the ad account.
    • Even if you set a specific end date, you will be able to change the end date if needed. 
      Note: The test must run for a minimum of 24 hours and less than a total of 30 days.
  15. Once all of the fields are properly filled out, click Create.
    Important: Once the test has been completed, you cannot re-activate the test. You must set up a new A/B test if you want to test the creatives again. You may extend the end date of the test if there is insufficient data but each test cannot exceed 30 days.


NOTE FOR iOS Campaigns:
Probabilistic Attribution must be enabled for iOS campaigns using the creative AB testing feature. This feature is unavailable for campaigns running only on Unattributable or SKAdNetwork attributable only traffic.


Creative A/B Test Performance Overview

Once the A/B test has been created, you will be able to see the status and also check the detailed performance of active creative tests.

  1. Select the ad account from the drop-down menu on the top left.

  2. Select the app on the left sidebar.

  3. Click the campaign title from the campaign lists.

  4. Scroll down to see the ad group lists under that campaign. Under the list of ad groups, you'll see a list of created A/B tests.

  5. Click the Screen_Shot_2021-07-20_at_1.57.42_PM.png on the left side of the A/B test for a summarized overview of the test.___________2022-06-10______2.55.13.png
  6. To see a detailed performance overview, click on the title of the A/B test you wish to see the details for.Screen_Shot_2021-07-20_at_2.17.29_PM.png
    You will be able to compare the following performance metrics by using the dropdown menu on the top left corner of the graph: 
    • Spending
    • Impressions
    • Clicks
    • Installs
    • Actions
    • CTR
    • IPM
    • CPC
    • CPI
    • CPA
  7. There is also a section under Test Result that will notify you of the current status of the test. 
  • The results will vary depending on the amount of data gathered during the test. Once there is enough data to determine a winner between Group A and Group B, the results will be shown here.
  • The confidence level must exceed 75% to determine a winner. If the confidence level is less than 75%, the final result of the test will result in "There is no significant difference."
  • Test results can differ by selected key metric. (e.g. Group A is better in terms of CPC, but Group B is better in terms of CPA) We recommend analyzing the creative A/B test results with multiple key metrics together.


NOTE FOR iOS Campaigns:
You can only check the results of the creative A/B test based on MMP attribution for iOS campaigns. (Results based on SKAN attribution are not available).



Was this article helpful?

1 out of 1 found this helpful

Have more questions? Submit a request