A/B Testing

Why Every Ecommerce Team Should Be A/B Testing

Most people would never commit to a winter coat or a pair of slacks without trying them on first. Why wouldn’t website managers and ecommerce teams apply the same rigor to website changes?

To Lindsay Pugh, Digital Marketing Analyst here at Something Digital, it’s a question of good website stewardship. A/B testing is the best way to ensure the changes you make will achieve your goals. Foregoing the exercise can needlessly cost you money over the long haul.

Still, for many people A/B testing is intimidating, which is why we sat down with Lindsay for a frank talk on A/B testing.

Q: Why is A/B testing important?

A: A lot of people act on hunches. They’ll say: I think we’ll get more conversions if we change this button from black to pink. They may ask around and their colleagues will tell them it’s a great idea. But it’s all just speculation. What if your site visitors don’t respond to that change? Or ignore that button because it’s pink?

A/B testing validates these things with statistically significant numbers. It stops you from essentially implementing your opinion by letting you know quantifiably whether the change is good, or if it will cost you more money in the long run.

Q: Let’s say you want to deploy A/B testing this quarter…where should you begin?

A: Start by assessing your site and identifying problems. For instance, are there sections of your site where engagement is disappointing? Areas or functions you suspect are problematic?

I worked with a client who was disappointed that visitors just weren’t engaging with their social media icons. So we tested adding the icons to the header as well as the footer and got important insight. These are really smart applications for A/B testing.

Q: Why doesn’t everybody A/B test?

A: It’s a combination of not enough bandwidth, money, or a good handle on how to use the technologies. Someone may install a free copy of Optimizely and then feel overwhelmed at the prospect of learning a new platform. So they decide to not bother with A/B testing.

Q: That brings up a good question: Is A/B testing difficult, time-consuming and expensive?

A: Not at all! First, I really want to emphasize that A/B testing is very easy to do, especially after you’ve set up your system and learned how the platform works. It’s not the daunting, time-consuming task people assume it will be.

And as for expense, there are definitely some costs involved, especially if your site traffic volumes exceed the free versions of A/B testing solutions. But what you really need to consider is the ROI. No site owner wants to make changes that confuse or dissuade potential customers. You want to put your best foot forward, and A/B testing helps you do just that.

Q: How do you know when a change you’re testing is statistically significant?

A: That’s the benefit of using Optimizely or any of the A/B testing tools out there. These tools will only stop a test when they deem you’ve reached a statistically significant result. They use algorithms to automate the process, and can determine whether the change under consideration accomplished your business goal.

Q: That’s the second time you  mentioned Optimizely, which begs the question: Are A/B testing tools a requirement?

A: They certainly make the process efficient and reliable, which is why we recommend them.  There are a lot of options on the market. At Something Digital we tend to use Optimizely, which at this point is synonymous with A/B testing. But Google Optimize, a newer product, is also good, and we’re happy to use that if the client prefers it.

The reason why these tools are helpful is that they’ll split your traffic randomly, allowing you to test a change in scenarios where all variables are equal. Let’s say you want to test a button-color change yourself, with no help from a software solution. You’ll probably change the color for one day, collect the data, then change it back the next day and collect the data again. Next you’ll compare the two datasets.

The problem is that a lot can change from one day to the next. You might have different promotions running, you might have a different composition of visitors to your site due to an affiliate partner’s promotion. It’s difficult to sync up all of the different variables so that you actually test the data in the same environment.

A software solution like Optimizely will ensure your A and B test elements are running concurrently, using all of the same variables. These programs ensure traffic is sent randomly to test the change. This leads to truer, more reliable results.

Q: Let’s say you want to make that button-color change…do you need to change that button in your site?

A: No. With Optimizely you receive a pixel which you implement on your site, and all changes are made via that pixel. Within the platform you have access to screens that let you preview the changes to your site. You can create multiple versions of what you want to test. There’s no need to hardcode the text elements into your site.

Q: Is Optimizely user-friendly?

A: It’s not too bad, especially if you have an understanding of how HTML and CSS works (or you’re good at Googling!). For complex tests you may need the help of a developer.

Q: Are there things that shouldn’t be A/B tested?

A: There are some complex process changes that are difficult to A/B test. If you plan to re-architect your site navigation A/B testing will be difficult, because there are too many elements changing all at once. We recommend A/B or multivariate testing when you have a few, isolated elements to test.

Q: Google has been making changes to its browsers, and plans to block website elements it believes are bothersome to users. Can A/B testing help site owners prepare for that?

A: Companies that rely on a pop-up to solicit email sign-ups should definitely plan to A/B test some alternatives. They can start now by testing an email drop down.

It’s not clear at this point if Google will completely ban pop-ups. They may just ban ones that take over the entire page, or don’t allow the user to close it. A/B testing allows companies to prepare for that eventuality and have a fully-tested alternative in place.


Looking for help with A/B Testing on your site? Or have questions for us? Contact us now.