10 minute read time

How CoSchedule Improved Open Rates by Systematically A/B Testing Email Subject Lines

Campaign Monitor - May 17, 2019

Article first published October 2014, updated May 2019

Are you A/B testing your email subject lines?

If so, you’re taking a step in the right direction toward better results from your email marketing.

However, if you’re just creating two different subject lines at random for each campaign, then you may not be getting the best results.

CoSchedule—a startup that uses Campaign Monitor to send their popular email newsletter, The Content Marketing Update—recently wrote about how they systematically A/B test email subject lines to find out what works for their audience.

Read on to learn more about the step-by-step process CoSchedule uses to test subject lines to help give you some ideas on how you can improve your email marketing results.

A step by step process for testing subject lines

Step 1: Decide what you want to learn

The first step toward systematically testing subject lines is deciding what you want to learn about your audience.

To do this, try to think of all the questions or uncertainties you’ve ever had when writing subject lines for email campaigns:

• Do long or short subject lines work best for my audience?
• Does including a number in the subject line get my audience to open emails?
• Does subject line personalization work for my audience?
• Should I phrase the subject line as a question rather than a statement?

Once you’ve worked out all the things you want to learn, you can then turn that into a spreadsheet for measuring the results.

For instance, if you wanted to know whether long or short subject lines work best for your audience, then you’ll need to add a column to your spreadsheet for “Number of characters” to record the effect this has on key metrics like open rates.

Examine the spreadsheet CoSchedule used to get a better idea of how this works:

In this example, CoSchedule wanted to be able to answer the following questions about their audience:

• Do higher open rates lead to more social shares of their blog posts?
• Does having a number in the subject line increase opens?
• Do shorter subject lines work better?
• Do subject lines with higher Emotional Marketing Value scores work better?
• Does asking a question in the subject line increase opens?

Because that’s what they wanted to learn, that’s what they decided to measure. You can see they added columns to record the number of characters, EMV score, and social shares, to name just a few.

Step 2: Lay out the testing ground rules

Conducting a single A/B test on two subject lines is easy; both tests go out at the same time, to the same amount of people, and the winner is automatically selected for you.

However, when you start trying to compare multiple A/B subject line tests against each other, you start to introduce a lot of variables that can affect your results.

Things like number of recipients, length of test, day sent, and time sent can all impact the results of a specific test and, therefore, give you misleading information if you’re trying to compare two different tests together to get some insight.

So, before CoSchedule started running A/B tests on their subject lines, they laid out some ground rules to help reduce these variables, including:

1. Deciding what percentage of their list would get sent the test emails (10%)
2. Deciding how long the A/B tests will run for (1 hour)
3. Deciding what time to the A/B tests will go out (10am CST)

By deciding to send the newsletter out at the same time each week—and run the subject line test for the same length and with the same amount of people—they were able to minimize the variables between each test and get more reliable insights.

Step 3: Run the A/B tests

Each week, CoSchedule would send out their newsletter and A/B test two different subject lines against each other using Campaign Monitor’s built-in A/B testing tool.

Given that each newsletter contained different content, every email had different subject lines that were tested against each other, but there were some subtle similarities.

Sometimes they would test using numbers in the subject line. Other times, they would test using a question as opposed to a direct statement.

Each time they ran an A/B test, they would document the results, and, over the period of two months, they built a good set of data on how each type of subject line was performing, which allowed them to then analyze the results and pull some insight.

Step 4: Analyze the results and use the insights to improve your email marketing

After that they had accumulated the results of a number of different subject line A/B tests, they were able to do some digging into the data to learn a bit more about what works for their audience.

For instance, one of the original insights CoSchedule was looking for was: Does asking a question in the subject line increase opens?

By looking at the open rates for all the subject lines that contained questions, they were able to work out that asking a question in the subject line does not work for their audience, and, in fact, decreases open rates.

They were then able to apply this learning to future subject lines, removing questions and improving their email marketing results.

What CoSchedule learned about their audience

By recording their results over the period of a few months, CoSchedule was able to ask a number of questions of their data and find out what works best for their audience.

Does having a number in the subject line help?

In their testing, they pitted subject lines with numbers in them against subject lines without numbers a total of five times. Of those five, three won.

Their conclusion was that numbered headlines did perform slightly better, but not significantly enough to say that they’re a sure thing.

Do longer or shorter subject lines do better?

In their testing, they found that shorter subject lines were less successful for them than longer ones, and that between 40 and 50 characters was best.

Does asking a question in the subject line get more opens?

Despite what they previously thought, when they tested subject lines with questions against those without, they found that the question lost every time.

How do you write a good subject line?

All this important data serves one main purpose: to help you write good subject lines. So how do you go about that? Using data that you get from your subject line will help you improve not only your subject line, but your conversions as well.

Understand your audience

The first step in writing a subject line that converts is to understand your audience. The importance of this is that you are better able to personalize the subject line by using the language your readers are familiar with.

Use numbers

As much as possible, try and incorporate numbers into your subject line. Numbers are great for making your headlines precise and set expectations for your readers.

Make your subject lines mobile friendly

According to CoSchedule’s email subject line tester, your subject line should be between 40-50 characters because it keeps your subject line within the right character limit for viewing on a mobile device. Research shows that mobile devices are the preferred device for opening and reading emails.

This means your subject line has to be mobile friendly, so be sure to use concise language. If it’s not mobile friendly, it’ll likely be cut off and won’t have as great an impact.

Include a sense of urgency

The purpose of subject lines is simple: to get readers to open your email. One great way of achieving this is by adding a sense of urgency to your subject line, which could be by way of a limited-time offer, for example.

By making your subject lines urgent, you elicit a quick response from your readers, resulting in higher response rates (and a more successful campaign).

Other A/B test to improve your emails

Subject lines aren’t the only part of your email that you need to test. Your whole email needs to be optimized for opens and conversions. Here are some other A/B tests you can carry out (that go beyond A/B testing email subject lines) to improve your emails:

Email copy

The email copy itself is a great part of your emails. Therefore, you should always split test your copy, particularly in regard to length and language.

Images vs. no images

Images are a great way of conveying a message more clearly. However, as great an idea as it is to include images in your emails is, you should always A/B test the effect of images on your email marketing campaigns. In some cases, images may actually draw attention away from the most important aspect of your email, thereby reducing your conversions.

The call to action

One element of your email you should always work on improving is your call to action—both the button and the text. As small an element as it may be, your call to action is the most important part of your email. A simple tweak in your call to action can result in a massive difference in conversions. Test to see if a button or text works better for you. As for the button copy, you can also test which words drive more clicks.

Templates

Email templates have a big impact on the effectiveness of your email. That’s why it’s very important that you test to see which templates result in better engagement.

Source: Campaign Monitor

The above are two different template styles that work well for the respective businesses. If, however, they were to switch, there’s a strong possibility their conversion rates would change drastically. Testing your email template style is a great way to ensure that you’re giving your readers the type of emails they enjoy.

Wrap up

Sometimes popular wisdom and research from different authorities doesn’t always apply to your unique audience, and the only way for you to know what is going to work for you is to systematically test and learn from your results.

So go ahead and follow the steps outlined in this article, as the things you’ll learn about your audience will help you get better results from your email marketing moving forward.

If you need more information on A/B testing your emails, check out our guide on A/B testing your email campaigns.