To understand A/B testing, we need to first look at the fundamental method of problem solving known as Trial and Error. It is characterized by repeated, varied attempts which are continued until success, or until the practitioner stops trying.
According to W.H. Thorpe, the term was devised by C. Lloyd Morgan (1852–1936) after trying out similar phrases "trial and failure" and "trial and practice".
Testing different variations is one of the quickest ways to learn, and Digital Advertising is no exception. When you begin talking about A/B testing, many of us get nervous and all sorts of possible catastrophes begin swimming around in our heads.
But the truth is, advertising is a quickly evolving tactic, and in order to gain momentum, testing has become a necessity. Whether your advertising is social, display, native or search, the key to better results and return on investment is testing.
An A/B split needs to be shown to the exact same audience, during the same times of day, same days of the week and in the same areas in order for you to have a firm grasp on which variation is truly performing better.
It’s important to have a firm grasp on your data set or audience so that you can split the information and create an equitable ad delivery.
In order to measure success, you need to make sure you’re giving your tests enough time to run. We typically recommend running tests at a minimum of 1 week with most tests running for 1 month.
Volume
How much data is required for you to have statistically significant results? If your program sees low volumes (clicks, impressions, conversions), you need to run longer tests. If you have a high volume account you can call tests much faster.
Seasonality
If you have drastic peaks and lows due to seasonality, I suggest finding the middle ground for testing. I never recommend testing during a seasonal low unless you dealing with a high volume account.
Day of Week/Month Activity
Make sure that both groups have equal coverage during your peak days and weeks. Again, try to avoid running tests during your low periods, especially if you see significant swings in CTR and conv. rates.
It’s also essential to make sure that your test is going to have impact. Sometimes slight copy variations and image changes aren’t going to tell you much. You want to make sure the variant has a very clear goal and hypothesis. For example, by changing a button from “Download” to “Get My Guide”, I’m hoping to see a 10% lift in conversions.
What Types of A/B Tests Does 3.14 test?
- Imagery Testing
- Message Testing
- Ad Placement Testing