Creative A/B test settings

  • Updated

In this section, you'll learn how to set up a creative A/B test. A/B tests are set up through ad groups within each campaign and can be used to compare the performance of different creative groups.

For example, the Creative A/B test feature allows you to allocate a part of the existing budget to see if the new concept/theme of Creative Group B is more efficient than the existing Creative Group A.

Note: We don't recommend using A/B testing to compare the performance of different creative formats (Image vs. Video or Video vs. Native). This is because the ad placements for each format are different, Thus, the difference in performance can be caused due to the different ad placements of the creative formats themselves as opposed to the performance of the actual creatives.

 

Set Up Creative A/B Test 

A/B Tests need to be set up within each campaign under the ad group section.

  1. Go to Home page.

  2. Select the Ad Account you wish to set up an A/B test for.

  3. Select the App & Campaign on the left sidebar menu. 

  4. Go to the list of Ad Groups under that campaign.

    • Under the list of ad groups, you see a section that is named A/B Test Groups

      MOLOCO_Cloud_2021-07-21_11-55-03.png

  5. Click Get Started to set up the A/B Test.

  6. Once you click Get Started, fill out the Name and Description (option) of the A/B test.

  7. Select Creative Group for the variable

  8. Use the drop-down menu to select the Ad Group of the first group (group A) you want to test.Screen_Shot_2021-07-20_at_1.01.02_PM.png

    Important: Make sure you create the Ad Groups that you want to test in advance before setting up the test. Learn more about Ad Group Settings.

     

  9. Once you select the Ad Group for Group A, you will see the list of creative groups that are within that ad group that will be used for the creative A/B test
    Note: Control Group (Group A)'s Creative Groups cannot be modified once created.AB_TEST_SCREENSHOTS_-_Google_Slides_2021-07-21_12-10-51.png
  10. The Ad Group for Group B (test group) will be automatically populated. Use the drop-down menu under Creative Groups to select all of the creative groups you would like to add to Group B.Screen_Shot_2021-07-20_at_1.20.23_PM.pngScreen_Shot_2021-07-20_at_1.23.35_PM.png
  11. Once the test is activated, you may not edit the creative groups being tested.
  12. Once both Group A (control) and Group B (test) creative groups have been set, you will be able to set the budget, key metric, and schedule for the test.Screen_Shot_2021-07-20_at_1.25.30_PM.png
  13. Enter the percentage of budget from the original campaign you wish to apply to the test. This is based on the daily budget.
    Screen_Shot_2021-07-20_at_1.29.48_PM.png
    • The budget ratio between the Control Group and the Test Group is fixed 1:1 within the set budget share. For example, if the total daily budget is $1,000 with 30% of the budget share, Group A and B's daily budget will be $150, respectively.___________2021-09-16______8.48.44.png
    • You can increase the percentage of the budget set for the A/B test up to 100%.
      Note: If the daily budget is set differently for each day of the week, the daily A/B test budget will be changed according to the daily budget allocated for each day of the week.
    • The budget cannot be modified once the test has been activated.
  14. Select the Key Metric. The key metric refers to the main metric you wish to compare for the test.
    Screen_Shot_2021-07-20_at_1.33.22_PM.png
    • In the case you select CPA as your key metric in an in app optimization goal campaign, the event that will be used to measure performance for the A/B test will be the same event set in the campaign.
      Screen_Shot_2021-07-20_at_1.33.38_PM.pngNote: For install or click optimization goal campaigns, you will only be able to select CPI or CPC for the key metrics.
  15. Set the schedule of the test. It is required to set a start and end date.Screen_Shot_2021-07-20_at_5.16.49_PM.png 
    • The timezone is dependent on the time zone of the ad account.
    • Even if you set a specific end date, you will be able to change the end date if needed. Note: The test must run for a minimum of 24 hours and less than a total of 30 days.
  16. Once all of the fields are properly filled out, click Create.
    Important: Once the test has been completed, you cannot re-activate the test. You must set up a new A/B test if you want to test the creatives again. You may extend the end date of the test if there is insufficient data but each test cannot exceed 30 days.

 

NOTE FOR iOS Campaigns:
  • Probabilistic Attribution must be enabled for iOS campaigns using the creative AB testing feature. This feature is unavailable for campaigns running only on Unattributable or SKAdNetwork attributable only traffic.

 

Creative A/B Test Performance Overview

Once the A/B test has been created, you will be able to see the status and also check the detailed performance of active creative tests.

  1. Go to Home page.

  2. Select the Ad Account you've set up an A/B test for.

  3. Select the App & Campaign on the left hand side menu. 

  4. Go to the list of Ad Groups under that campaign.

    • Under the list of ad groups, you see a section that is named A/B Test Groups. You will see a list of created A/B testsScreen_Shot_2021-07-20_at_1.54.35_PM.png

  5. Click the Screen_Shot_2021-07-20_at_1.57.42_PM.png on the left side of the A/B test for a summarized overview of the test.Screen_Shot_2021-07-20_at_2.03.57_PM.png
  6. To see a detailed performance overview, click on the title of the A/B test you wish to see the details for.Screen_Shot_2021-07-20_at_2.17.29_PM.png
    You will be able to compare the following performance metrics by using the dropdown menu on the top left corner of the graph
    • Spending
    • Impressions
    • Clicks
    • Installs
    • Actions
    • CTR
    • IPM
    • CPC
    • CPI
    • CPA
  7. There is also a section under Test Result that will notify you on the current status of the test. 
    Screen_Shot_2021-07-21_at_5.56.19_PM.png
  • The results will vary depending on the amount of data gathered during the test. Once there is enough data to determine a winner between Group A and Group B, the results will be shown here.
  • The confidence level must exceed 75% to determine a winner. If the confidence level is less than 75%, the final result of the test will result in "There is no significant difference."
  • Test Results can differ by selected Key Metric. (e.g. A group is better in terms of CPC, but B group is better in terms of CPA) We recommend you analyze the Creative A/B Test results with multiple key metrics together.

 

NOTE FOR iOS Campaigns:
  • You can only check the results of Creative A/B Test based on MMP Attribution events for iOS campaigns. (Results based on SKAN attribution events are not provided).

 

 

Was this article helpful?

1 out of 1 found this helpful

Have more questions? Submit a request

Comments

0 comments

Article is closed for comments.