Why I love A/B split testing

I love split tests as a marketing tool.  Love 'em.

First, a quick definition:  Split testing is when you take a large number of potential respondents to an offer and split them up into different groups - or cells we call them - to receive offers or communications that are somehow different from each other.  If the respondents are homogeneous across the groups, then  you can reliably test how the differences in the offer or communication affect the respondents.  A simple example.  I could offer the same product in direct mail to the same list with the same creative, but in one case I could charge $99 while in the other I charge $79.  Now I have tested to see which of these two price points makes me more money.

The above example would be an A/B split test because there are two cells, A and B.  If there were three cells it would be an A/B/C split test.  Etc.  In general, however, the number of cells isn't that important to the concept of the test, and therefore I notice that in the real world A/B split testing and split testing are used almost interchangeably.

In theory you can conduct split tests under all sorts of circumstances.  For example, large retail chains conduct them by sending different in-store promotions out to half their stores.  Once upon a time the bulk of split testing took place in conjunction with direct mail.  Today there's still a lot of split testing in direct mail and direct e-mail, but the most interesting work is happening on web pages.

That's because web pages are the dream environment for this kind of test.  Homogenizing the traffic is a piece of cake.  You can get results in real-time and change the tests as often as you like.  There are many robust tools out there to help with your split testing, and they're super easy to implement and use.

And of course because the rewards are so high.  Split testing is the single best tool for taking you out of speculation and into facts.  The example above is incredibly important in terms of that product's revenue and profit.  You can test all kinds of things - message, purchase flow, offer, product name, headline, you name it.  The depth and variety of market knowledge you can gain is limited only by your tools, your traffic, and your imagination.