Types of Testing and Software

Many people in the marketing world are familiar with split testing. It’s the stuff Claude Hopkins wrote about in his book “Scientific Advertising” way back in 1923. Not as many are familiar with multivariate testing.

Split testing is testing version A versus version B. That’s about all you could do before the age of computers and the internet. You ran two different ads in similar periodicals to see which one did better. Of course back then, there were way to many variables to give you perfect results if it got close.

Now, one of the most perfect split testing operations is Google AdWords. Not only can you rotate A vs B, but you can test A vs B vs C… as far out as you care to write copy for.

So what’s multivariate testing? If you’re only writing a headline and two lines of copy, you can probably do without it. On the other hand, if you have much more than that, you need to know which headlines work best with which leads with which calls to action. If you introduce multiple variations for every variable you want to test, the possible combinations are practically impossible to sort out.

Even so, sorting out and pre-planning is what Taguchi testing is all about. Think of Taguchi testing as advanced split testing with many different combinations pre-planned. It was originally designed to test factory production lines.

What Taguchi can’t tell you is what relationship all of the variables have to each other outside those preplanned combinations. So essentially, it’s split testing with more versions.

For a tangible example, consider the following:

You have an email you’d like to test. You want to test 3 different headlines, greetings, opening paragraphs, body copy and closing. That’s 3 to the 5th power or 243 variations. If you had just two versions of 15 different variables, you’re talking over 32,000 total variations. Are you really going to try to split test that? No, that’s not possible. Taguchi testing would try to pick several of the best ones and split test them. Multivariate testing would test them all until it found the best combinations.

If you’d like more detail about why multivariate testing kicks butt over split or Taguchi testing, check out the white paper below. It was written by a Google Authorized Consultant who uses Google’s solution, Google Website Optimizer.


If you realize how much better multivariate testing is, how can you take advantage of it? Here are some of the different solutions so far:

  • Google Website Optimizer: a free solution. It’s great if you can write code or pay someone to.
  • MuVar or now TestiVar: a former product of James Brausch. It’s written in php so you also need to know code. It’s a one time fee for the software and you can install it on as many websites as you have. The reason it’s better than Google’s solution is that it “watches” the results and over time favors the winning combinations. If you give it some time, you’ll end up with a fully optimized website. Otherwise, you have to have someone constantly setting up tests and analyzing the results.
  • ConversionMultiplier.com: These guys SAY they multivariate test email and web pages but they don’t have enough info on their website (which doesn’t look like they optimized it either) to know. There’s no 3rd party credibility so I can’t say whether they could deliver or how they work.
  • Vertster.com: These guys look reputable from their website. Unfortunately, they only do Taguchi testing. The good thing though is that it looks like it only requires a small amount of coding (good if you know how) and then it’s an easy user interface.
  • Optimost.com: They also look like a reputable company. They do full multivariate testing. They require someone to do coding and ongoing technical support.
  • Sitespect.com: This looks like the easiest solution out there so far. They don’t require you to make any changes to your website but instead act as a filter for your site. When people go to your site, they intercept the traffic and send out the variations you wanted to test. Of course you have ongoing fees to support it but unless something easier comes along, it’s probably well worth it depending on your business.

All of these solutions required some amount of coding at one time or another. Personally, I’ve been challenged with MuVar because I’m not a programmer and realize that to get the most out of it (and be able to test the nuances of NLP language patterns in different formats), I probably need to go learn CSS.

What is NO ONE doing? Easy, multivariate software for marketing people… your typical business owner can barely manage a website IF there’s no coding involved. They aren’t going to figure out Google Website Optimizer or any of these other solutions aside from maybe Sitespect if they can afford it.

What would be the ideal solution? Drag and drop interface. You click, highlight and insert different variables to test. You can test depending on different sources of traffic. Your test results automatically favor the winners and let you know when those winners have been decided with a certain degree of certainty.

There needs to be one click installation or a technical support rep available to walk you through it (needs to be even easier than installing WordPress). And it ought to have the option of being a web app so you can log in from anywhere and add more variables.

There aren’t many companies out there positioned to take advantage of this. Partially that’s because the market has been driven by IT department purchases. That’s changing now as web tools become more accessible to marketers. As marketers start driving software infrastructure purchases, they’re going to start asking for essential modern marketing tools like multivariate testing.

And what about email? I don’t know of anyone multivariate testing email. Conversion Multiplier claims to but their website has no proof so maybe they are, maybe they aren’t.

Here’s how an email multivariate testing program could work:

Instead of the typical split testing approach (testing 2 or more emails on a small percentage of your list), a multivariate tester could return results in real time.

Typically, you’re wanting to measure a number of different actions: bounce rate (or if the servers think it’s spam), open rate, click through rate, unsubscribe rate, and purchases once they click through. Almost no one measures customer longevity but that would be a good metric too.

What if you multivariate test by trickling out emails to a percentage of your list? Your email program would send out versions generated from your different variables and rank them according to however you weighted the different action metrics. The whole setup would need to be an easy interface, drag and drop, and point and click too.

If you had optimizing capabilities (the program favored winners), you could send it once and be done. By the time you were halfway through the list, the program would be sending out only a few different variations that were all close winners.

How do you think that would affect your conversion rates or future copywriting? It would change everything.

And if the barrier between programmers and marketers were removed… everyone would be doing this and we’d all be seeing much better marketing messages.

Any company out there want to make this happen? We’re all waiting.

8 Responses

  1. ohh, i got to learn much more from here, untill now i was knowing of only two test, Black Box testing and White Box testing, i haven’t heard of multivariate testing and Taguchi testing, wow, but are those general testing, or related to PHP,JAVA and some other languages ?

  2. @Alex – According to wikipedia, it appears black box and white box refer to testing software itself to make sure it works properly. The testing I’m talking about here is about testing and optimizing the effectiveness of your marketing.

Leave a Reply