Doing an A/B Test
A/B Testing is used in order to test different versions of your e-mail in real-time and to easily and quickly find out which version is the best to send to your profiles. In this way, you can test content variability within the same mail, ranging from simply having two different subjects to having two completely different versions of the message. This will allow you to draw conclusions on what worked best and for which reason.
After creating your A/B campaign, ACTITO will send each version of the e-mail to a sample of profiles in your database. You can have up to 4 different versions.
After a test period of the duration of your choice, ACTITO will calculate which e-mail worked best according to either the opening or the click rate. The best performing e-mail will then be sent to the rest of the target (people who have not been reached yet).
Creating your A/B Campaign
A/B testing is a campaign mode in itself, like Marketing or Scenarized e-mails. In order to create an A/B campaign, is therefore necessary to make your selection right at the creation of your message.
In "Manage e-mail campaigns" (Catalog > E-mail campaigns > Manage e-mail campaigns), click on "Create a new campaign".
Next, select the "A/B TESTING" mode.
In practice, after the test period, an A/B campaign will
behave like a Marketing campaign. It is therefore used to contact
a large number of profiles.
Good to know
If you want to make A/B tests on Scenarized e-mails, it is possible to use the "A/B split" block in the Scenario editor.
Step 1: General Data
As for every campaign, the first step will consist in giving a name to your campaign, selecting the target Database, the entity and the language of the campaign.
Tip
In A/B campaigns, you only have the possibility to select one definition language. Indeed, A/B testing is used to determine which version of your e-mail works best, but the behavior of your profiles might vary greatly according to their culture and their language. A more informal message might work best in a language but not in another, and idioms that cannot be translated might prevent you from having exactly the same message in two languages, in which case the selection criteria will be distorted.
If you want to contact your whole database, it is possible to select a default language at step 2 in order to contact all your profiles in the same language.
AB test parameters
Next, you will be able to select:
-
The number of versions of your e-mail: you can test up to 4 four different versions.
-
The parts of the e-mail on which you want to test variations. You can combine a different version of:
-
The sender
-
The subject
-
The pre-header
-
The content
-
Message headers
If you decided to test variation on the sender, the subject
and/or the pre-header, you will be invited to provide each
version in the 'Message headers' section.
Step 2: Target Group
As for any campaign, you will have to select the Subscription and the target population of your e-mail. You should target all the profiles that you want to contact: the test population will be selected randomly and the repartition criteria will be set at a later stage.
If you want to contact profiles no matter what their mother language is, you can un-check the 'Language' box to contact everyone in the language chosen at step 1.
Step 3: Message
You can create your message from a template, by importing an HTML or by coding your own HTML.
If you import HTML files to do an AB test on the content, the ZIP file must include as many HTML files as versions, with the version being specified in the suffix of the file name, under the format "_a", "_b", etc.
If you chose to do A/B testing on the content at step 1, you will have to define a specific content for each version.
You will be able to jump from a version to another by using the drop-down list in the top right corner of the message creation screen.
In the template editor, the version that you are editing
will be displayed in the top left corner.
Tip
What if you only have minimal variation between your versions? In the template editor, you can copy the content of another version by clicking in the 'Options' in the top left corner and choosing 'Import content from another version'. After that, you are free to apply the variations on which you want to do A/B testing.
Step 4: Check and Test
After creating your message, you will be able to test it by selecting a sample of profiles from your testing group. Then you will have to provide one or more e-mail addresses to which the test e-mails will be sent.
The "A/B" column enables you to select which version
you want to generate for each test profile. If you leave the
default value "A/B", each version of the e-mail will
be generated for this test profile (including version C and D if
they exist).
Alternatively, you can choose to only select
one version from the drop-down list for each profile.
Step 5: Sending
At step 5, you will get a summary of the general data, the message headers and the target group of your campaign.
In addition, you will be able to set the AB testing parameters.
AB testing parameters
-
Sample size for versions: a sliding bar will allow you to choose the repartition between the test version and the winner version. If you have more than 2 versions (A/B/C/D testing), each version will automatically have the same sample size.
-
Optimization criteria: you can choose the criteria used to determine the winner version, either the opening rate or the clicks rate.
-
Period: determine the test period at the end of which the winner version will be decided. It can be set either in hours or in days.
After selecting the AB testing parameters, you can choose to
send the campaign right now or to schedule it for later.
Tip
It is possible to set up an AB test with 100% of the target as sample, for example if you want to identify the best version in order to optimize your next campaigns.
To do so, you will still need to choose an optimization criteria and a test period. They will have no impact on sending, as the e-mail will be sent immediately sent to the whole target, but they will be used to identify the winner version in the campaign report.
Analyzing your A/B campaign
Once you're a/B campaign has been sent, it will be available in the "In progress" tab during the duration of the test period that you defined at step 5 of the campaign creation.
During the test period, it is
not possible to "Correct"
this campaign. However, it is possible to "Finish" it if
you do not want to send the winner version of the e-mail.
If you want to check the repartition of the sample
population or view each version of your e-mail, you can click on
the "View definition" button.
View reporting during the test period
You can view the campaign report at any time in order to check
the evolution of the opening or click rate of each version in
real time.
Do to so, click on the "View
reporting/results" button.
The "Summary" tab, which is found in every
campaign report, will provide the global stats of your campaign.
To view the repartition of you're A/B campaign, go to the
"A/B testing" tab, which is only present for this
specific type of campaigns.
This lets you see the number of e-mails sent for each
version. The target population sample will always be the same
for each version, but slight variations may appear, when
addresses are filtered for instance.
Next, you can compare the delivery, opening and click rate
of the different versions of your e-mail. It can be done at any
time during the test period.
The "Optimized" column, which remains empty
during the test period, will display the stats of the winner
version once it has been determined.
Viewing the optimized reporting
When the test period is finished, the campaign will appear in the "Sent" tab and you will be able to view the complete campaign report.
In the "A/B testing" tab, the header of the "Optimized" column will now display the winner version.
The campaign stats found in the "Optimized" column only relate to the optimized version, which means the e-mails sent after the test period, without including the related test sample.
At the bottom of the page, the "Winner" tab
specifies the winner version and reminds the criteria used to
choose it.
Once the winner version is sent, your A/B campaign will
appear in your last campaigns on your Dashboard, among the other
mass campaigns.