Taguchi testing

Types of Testing and Software

Many people in the marketing world are familiar with split testing. It’s the stuff Claude Hopkins wrote about in his book “Scientific Advertising” way back in 1923. Not as many are familiar with multivariate testing.

Split testing is testing version A versus version B. That’s about all you could do before the age of computers and the internet. You ran two different ads in similar periodicals to see which one did better. Of course back then, there were way to many variables to give you perfect results if it got close.

Now, one of the most perfect split testing operations is Google AdWords. Not only can you rotate A vs B, but you can test A vs B vs C… as far out as you care to write copy for.

So what’s multivariate testing? If you’re only writing a headline and two lines of copy, you can probably do without it. On the other hand, if you have much more than that, you need to know which headlines work best with which leads with which calls to action. If you introduce multiple variations for every variable you want to test, the possible combinations are practically impossible to sort out.

Even so, sorting out and pre-planning is what Taguchi testing is all about. Think of Taguchi testing as advanced split testing with many different combinations pre-planned. It was originally designed to test factory production lines.

What Taguchi can’t tell you is what relationship all of the variables have to each other outside those preplanned combinations. So essentially, it’s split testing with more versions.

For a tangible example, consider the following:

You have an email you’d like to test. You want to test 3 different headlines, greetings, opening paragraphs, body copy and closing. That’s 3 to the 5th power or 243 variations. If you had just two versions of 15 different variables, you’re talking over 32,000 total variations. Are you really going to try to split test that? No, that’s not possible. Taguchi testing would try to pick several of the best ones and split test them. Multivariate testing would test them all until it found the best combinations.

If you’d like more detail about why multivariate testing kicks butt over split or Taguchi testing, check out the white paper below. It was written by a Google Authorized Consultant who uses Google’s solution, Google Website Optimizer.


If you realize how much better multivariate testing is, how can you take advantage of it? Here are some of the different solutions so far:

  • Google Website Optimizer: a free solution. It’s great if you can write code or pay someone to.
  • MuVar or now TestiVar: a former product of James Brausch. It’s written in php so you also need to know code. It’s a one time fee for the software and you can install it on as many websites as you have. The reason it’s better than Google’s solution is that it “watches” the results and over time favors the winning combinations. If you give it some time, you’ll end up with a fully optimized website. Otherwise, you have to have someone constantly setting up tests and analyzing the results.
  • ConversionMultiplier.com: These guys SAY they multivariate test email and web pages but they don’t have enough info on their website (which doesn’t look like they optimized it either) to know. There’s no 3rd party credibility so I can’t say whether they could deliver or how they work.
  • Vertster.com: These guys look reputable from their website. Unfortunately, they only do Taguchi testing. The good thing though is that it looks like it only requires a small amount of coding (good if you know how) and then it’s an easy user interface.
  • Optimost.com: They also look like a reputable company. They do full multivariate testing. They require someone to do coding and ongoing technical support.
  • Sitespect.com: This looks like the easiest solution out there so far. They don’t require you to make any changes to your website but instead act as a filter for your site. When people go to your site, they intercept the traffic and send out the variations you wanted to test. Of course you have ongoing fees to support it but unless something easier comes along, it’s probably well worth it depending on your business.

All of these solutions required some amount of coding at one time or another. Personally, I’ve been challenged with MuVar because I’m not a programmer and realize that to get the most out of it (and be able to test the nuances of NLP language patterns in different formats), I probably need to go learn CSS.

What is NO ONE doing? Easy, multivariate software for marketing people… your typical business owner can barely manage a website IF there’s no coding involved. They aren’t going to figure out Google Website Optimizer or any of these other solutions aside from maybe Sitespect if they can afford it.

What would be the ideal solution? Drag and drop interface. You click, highlight and insert different variables to test. You can test depending on different sources of traffic. Your test results automatically favor the winners and let you know when those winners have been decided with a certain degree of certainty.

There needs to be one click installation or a technical support rep available to walk you through it (needs to be even easier than installing WordPress). And it ought to have the option of being a web app so you can log in from anywhere and add more variables.

There aren’t many companies out there positioned to take advantage of this. Partially that’s because the market has been driven by IT department purchases. That’s changing now as web tools become more accessible to marketers. As marketers start driving software infrastructure purchases, they’re going to start asking for essential modern marketing tools like multivariate testing.

And what about email? I don’t know of anyone multivariate testing email. Conversion Multiplier claims to but their website has no proof so maybe they are, maybe they aren’t.

Here’s how an email multivariate testing program could work:

Instead of the typical split testing approach (testing 2 or more emails on a small percentage of your list), a multivariate tester could return results in real time.

Typically, you’re wanting to measure a number of different actions: bounce rate (or if the servers think it’s spam), open rate, click through rate, unsubscribe rate, and purchases once they click through. Almost no one measures customer longevity but that would be a good metric too.

What if you multivariate test by trickling out emails to a percentage of your list? Your email program would send out versions generated from your different variables and rank them according to however you weighted the different action metrics. The whole setup would need to be an easy interface, drag and drop, and point and click too.

If you had optimizing capabilities (the program favored winners), you could send it once and be done. By the time you were halfway through the list, the program would be sending out only a few different variations that were all close winners.

How do you think that would affect your conversion rates or future copywriting? It would change everything.

And if the barrier between programmers and marketers were removed… everyone would be doing this and we’d all be seeing much better marketing messages.

Any company out there want to make this happen? We’re all waiting.

The Holy Grail of Copywriting

There’s lately been some discussion on the value of NLP in copywriting.

I haven’t seen too much hype about NLP Copywriting being the holy grail of copywriting. Jon McCulloch posted today about how it’s just another tool for copywriters. I don’t know that anyone is arguing that point.

Copywriters worth their salt will tell you these are the most important elements of a promotion:

  1. The list
  2. The offer
  3. The copywriting

These aren’t percentage contributions either as I saw one marketer teaching. This is a situation where if you miss on #1, it doesn’t matter how good the other two are. You can nail #1 but if you miss #2 you still don’t have a very successful promotion.

Conversely, you can do a terrible job of #3 but still have a moderate success if you got the first two down. Obviously you’d want to do well on all three to optimize your chances for success.

That said, NLP Copywriting is about optimizing the effectiveness of #3. It can’t rescue a promotion that missed on the first two elements.

The other point I wanted to add to the discussion is on testing.

James Brausch does as much testing as anyone else out there. Through using his multivariate testing software, he turned the visitor value of one of his sites from $5 to now $31.35 . And that’s without deliberately testing any NLP. [FYI: Visitor Value means for every unique visitor to that site, the average payoff is over $30]

James also posted a good and brief explanation of the different types of testing (including Taguchi). Split testing is better than nothing but why would you do it when you can get results so much faster these days?

As an aside, I’ve participated as a subject in several double blind studies and they’re not usually as blind as people would like to think. There are ALWAYS incentives at work that people tend not to take into account.

It would be ridiculous for an entrepreneur to spend time trying to set up a double blind study. In a medical setting, double blind means neither the doctor nor the subject knows which medication is being dosed and tested. But believe me, sometimes you KNOW which one is the placebo.

Why would an entrepreneur want to set up a situation where the winning variables were in a black box (double blind) that they had to sort through after the fact? James’ software, MuVar, automatically sorts it out for you and selects the best ones to display.

The point of all of this is the overall approach I’ve intended to take with this blog:

NLP Copywriting is simply another variable to multivariate test while you’re improving your copywriting.

There’s no perfect sales letter. There’s no magic pill persuasion technique. There’s only a point of diminishing returns where it becomes more profitable to either focus more energy on increasing traffic or create a new offer.

If there is a holy grail of copywriting, it’s multivariate testing.