How CoSchedule improved open rates by systematically A/B testing email subject lines

Are you A/B testing your email subject lines?

If so, you are taking a step in the right direction towards better results from your email marketing.

However, if you are just creating two different subject lines at random for each campaign then you may not be getting the best results.

So when CoSchedule, a startup that uses Campaign Monitor to send their popular email newsletter The Content Marketing Update, recently wrote about how they systematically A/B test email subject lines to find out what works for their audience – we knew it was something we had to share with you.

In this post, we outline the step by step process CoSchedule use to test subject lines with the hope that you can steal their process and improve your email marketing results as well.

A step by step process for testing subject lines

Step 1: Decide what you want to learn

The first step towards systematically testing subject lines is deciding what you want to learn about your audience.

To do this, try to think of all the questions or uncertainties you’ve ever had when writing subject lines for email campaigns:

• Do long or short subject lines work best for my audience?
• Does including a number in the subject line get my audience to open emails?
• Does subject line personalisation work for my audience?
• Should I phrase the subject line as a question rather than a statement?

Once you’ve worked out all the things you want to learn, you can then turn that into a spreadsheet for measuring the results.

For instance, if you wanted to know whether long or short subject lines work best for your audience then you’ll need to add a column for ‘Number of characters’ to record the effect this has on key metrics like open rates.

Let’s look at the spreadsheet CoSchedule used to get a better idea of how this works.

Although what you are interested in may vary, CoSchedule wanted to be able to answer the following questions about their audience:

• Do higher open rates lead to more social shares of their blog posts?
• Does having a number in the subject line increase opens?
• Do shorter subject lines work better?
• Do subject lines with higher Emotional Marketing Value scores work better?
• Does asking a question in the subject line increase opens?

Because that is what they wanted to learn, that’s what they decided to measure. You can see they added columns to record number of characters, EMV score and social shares to name just a few.

Step 2: Lay out the testing ground rules

Conducting a single A/B test on two subject lines is easy, both tests go out at the same time, to the same amount of people and the winner is automatically selected for you.

However, when you start trying to compare multiple A/B subject line tests against each other, you start to introduce a lot of variables that can affect your results.

Things like number of recipients, length of test, day sent and time sent can all impact the results of a specific test and therefore give you misleading information if you are trying to compare two different tests together to get some insight.

So before CoSchedule started running A/B tests on their subject lines, they laid out some ground rules to help reduce these variables, including:

1. Deciding what percentage of their list would get sent the test emails (10%)
2. Deciding how long the A/B tests will run for (1 hour)
3. Deciding what time to the A/B tests will go out (10am CST)

By deciding to send the newsletter out at the same time each week – and run the subject line test for the same length and with the same amount of people – they were able to minimise the variables between each test and get more reliable insights.

Step 3: Run the A/B tests

Each week Co-Schedule would send out their newsletter and A/B test two different subject lines against each other using Campaign Monitor’s built-in A/B testing tool.

Given that each newsletter contained different content, every email had different subject lines that were tested against each other, but there were some subtle similarities.

Sometimes they would test using numbers in the subject line, other times they would test using a question as opposed to a direct statement.

Each time they ran an A/B test they would document the results, and over the period of 2 months they built a good set of data on how each type of subject line was performing which allowed them to then analyse the results and pull some insight.

Step 4: Analyse the results and use the insights to improve your email marketing

Now that they had accumulated the results of a number of different subject line A/B tests, they were able to do some digging into the data to learn a bit more about what works for their audience.

For instance, one of the original insights CoSchedule was looking for was ‘Does asking a question in the subject line increase opens?’

By looking at the open rates for all the subject lines that contained questions, they were able to work out that asking a question in the subject line does not work for their audience and in fact decreases open rates.

They were then able to apply this learning to future subject lines, removing questions and improving their email marketing results.

What CoSchedule learnt about their audience

By recording their results over the period of a few months, CoSchedule was able to ask a number of questions of their data and find out what works best for their audience.

Does having a number in the subject line help?

In their testing, they pitted subject lines with numbers in them against subject lines without numbers a total of 5 times. Of those five, three won.

Their conclusion was that numbered headlines did perform slightly better, but not significant enough to say that they are a sure thing.

Do longer or shorter subject lines do better?

In their testing, they found that shorter subject lines were less successful for them than longer ones, and that around between 40 and 50 characters was their sweet spot.

Does asking a question in the subject line get more opens?

Despite what they previously thought, when they tested subject lines with questions against those without they found that the question lost every time.

In conclusion

Just to be clear, the purpose of sharing this with you was not for you to takeaway that the right subject line length is between 40 and 50 characters, or that asking questions reduces open rates.

The purpose of sharing this with you was to show that sometimes popular wisdom and research from different authorities doesn’t always apply to your unique audience, and the only way for you to know what is going to work for you is to systematically test and learn from your results.

So go ahead and follow the steps outlined in this article, as the things you’ll learn about your audience will help you get better results from your email marketing moving forward.