Posts Tagged ‘analytics’

Blog Analytics: How do you measure the company blog’s performance?

December 16th, 2013 4 comments

Measuring the performance of your company’s blog is not easy. In truth, I would say the endeavor is an exercise in the illusive. By design, blogs don’t give up customer behavior secrets as easily or as comfortably as a landing page.

There are different topics covered in blog posts, the appeal of those topics to your audience, add in the frequency of posts and multiple authors, and any true measurement becomes difficult, if not impossible altogether.

Consequently, there’s no real surprise when marketers tasked with assessing a blog’s performance struggle measuring it. A common question we hear from marketers is, “What are critical metrics to measure a company blog’s performance would you recommend, and why?”

So, I asked a few of my colleagues at MECLABS this question and here’s what they had to say.


Pay attention to social media metrics

I would include a social measurement of some sort based upon the Google Hummingbird algorithm update, such as +1s or tweets. The reason is because increasingly social syndication is highly correlated with performance in organic search rankings.

My other metric would be something to measure the total impact of the blog upon a core organizational goal, such as revenue based upon an advanced attribution model like the time-decay methodology. The reason for that is to identify how closely blog performance mirrors organizational performance.

 – Jonathan Greene, Business Intelligence Manager, MECLABS


Track where visitors go and how long they stay

My initial thoughts here are time on site and pages per visit. A blog is not a typical site where you build a specific funnel to accomplish a goal. These two metrics would be very useful to measure performance especially if your blog is meant to be informative and build brand awareness.

– Matthew Hertzman, Senior Research Manager, MECLABS


And don’t forget about relevance

For starters, I would recommend looking at metrics that help you assess the relevancy of your content to your audience. One good example of two metrics to help you do this would be number of visits divided by the number of page views.

Sure, there are some other factors to consider depending on your goals, but the overall idea here it to get you started on the path of understanding where people go on your blog and how much content they are consuming on average.

– Ben Filip, Data Sciences Manager, MECLABS

Read more…

Interpreting Results: Absolute difference versus relative difference

November 21st, 2013 No comments

A common question that often arises among our Research Partners and audience, even after running many A/B split tests, is: “What’s the difference between relative and absolute difference?”

This is understandable given marketers are not notorious for being data-savvy mathematicians, but let’s face it, interpreting analytics effectively to gain useful insights about your customers is no easy task.

Customer behavior can easily become lost in the math behind campaign results, or worse, overlooked and simply taken at face value.


This is troubling given the significant number of marketers who use analytics to improve their messaging (67%) and make decisions (47%) according to the MarketingSherpa 2013 Marketing Analytics Benchmark Report (free excerpt at that link).

In today’s MarketingExperiments Blog post, we’ll take a deeper look at how having a clear understanding of absolute difference versus relative difference can help you interpret your test results.


Absolute difference gives you the real number difference between treatments

For the sake of example, we’ll use a simple A/B split test (control and a single treatment) for a hypothetical landing page test and some dummy data to interpret.


We ran our test and here are our results. The treatment outperformed the control by an absolute difference of 0.4% and a relative difference of 16.1%.

But what does that mean?

Well, the absolute difference is the difference of two real numbers.

Think of it literally as X – Y.

In this case, the control had a 2.5% conversion rate and the treatment had a 2.9% conversion rate.

2.9% (treatment conversion rate) minus 2.5% (control conversion rate) equals 0.4%

Therefore, our absolute (real number) difference was 0.4%.

We usually look at the absolute difference in conversion rates between a control and treatments to determine the overall difference in performance.

In regard to interpretation, this tells me that the treatment’s conversion rate was literally 0.4% higher than that of the control.


Relative difference gives you the scale of difference between treatments 


In keeping with our example, we know the treatment outperformed the control by an absolute difference of 0.4% and the literal implications of that.

But what is the real magnitude of the observed relationship difference between our control and treatment that we captured in our data?

This is where relative difference comes in.

Relative difference allows us to understand the comparative ratio of two numbers – often expressed as a percentage – that gives us a direct insight into the true scale of difference between our control and treatment.

In short, for every behavior recorded in the control’s conversion rate, there is 16.1% more of that behavior taking place in the treatment.


Here’s an illustration of how relative difference is calculated to help make it a little easier to understand. If we use the data from our test, here’s how it would plug into the formula:

[(2.9 – 2.5)/2.5] * 100 = 16.1%

By understanding relative difference, we can now say the treatment increased conversion by 16.1%.

  Read more…


October 17th, 2013 No comments

Imagine a mad scientist and his lab assistant developing a powerful explosive using two newly discovered chemicals – Alpharium and Betamium.

In their pursuit, they come up with a research question to test:

“Which combination of Alpharium and Betamium will produce the most powerful explosion?”

So, they put together some tests combining Alpharium and Betamium at different ratios while carefully measuring the force generated when each compound explodes.

After the first test, they compare results and discover the mad scientist’s results are different from his assistant’s results.

Was this due to human error in the setup of the experiment? Was it due to environmental conditions? Was it because their gauges were calibrated differently?

They pondered over these questions and decided to try the experiment a few more times.

At the end of their of the testing, they once again compared notes and discovered no two trials had resulted in exactly the same measurements, but the difference between in their average findings for each treatment was minimal.

In a blog post for Mad Science Magazine, they shared their findings:

“Treatment A, a 3:1 combination of Alpharium and Betamium, produced a 36% higher average level of explosive force than Treatment B, an equal combination, N=43, p=.024, α=.05.”

A week later, the mad scientist and his assistant were surprised to find themselves answering more questions about the symbols used in reporting than the actual results themselves.


Results are only useful to the extent they are understood

“So, what exactly do those little symbols mean?”

This is a common question our data sciences team receives frequently from readers of our research publications.

When you’re immersed in data on a daily basis, it becomes all too easy to forget that everyone is not fluent in statistical terminology.

In today’s MarketingExperiments Blog post, I wanted to share three statistical concepts every marketer should understand.

My goal here is not to give you a Ph.D. in statistics, but rather, I want to try and demystify a few common symbols used in statistical reporting you can use to aid your team’s next discussion of test results.


N= how many samples were taken

N means “number” and it tells you how many samples were taken.

Why is this important?

Because the lower number of samples you take, the greater likelihood your results could be skewed by random variation (randomness, in a mathematical sense).

This means our mad scientist and his assistant had higher odds their initial results were erroneous after collecting two samples for each treatment because they were working with a very small sample size.

However, once they collected 43 samples, their sample size was large enough that the statistics told them there was significant difference between the two compounds with an acceptable level of certainty.


p= an estimate of how likely something is to happen

And, what about that p?

p (for probability) refers to the statistical concept of significance.

Any time someone calculates a statistic from a set of samples, they are making an assumption about the entire population or about all possible outcomes from the experiment.

The p-value is a calculation which indicates the probability of observing results as far apart as those recorded in the experiment. It is essentially the probability of a false positive result.

The smaller the N (number of samples taken), the less confident the statistics are about any possible patterns in the data and the larger the p-value becomes.

For example, a p-value of 0.3 would indicate that there is about a 30% chance that the observed differences between the results are actually due to random variation.

Very few scientists, however mad they might be, are willing to risk those odds.

However in this case, our mad scientist and his assistant calculated that p=.024. This means that there is only a 2.4% chance that the observed differences are really just an illusion.

One thing also worth mentioning here is that your statistical level of confidence is also the inverse of your p value. (Hint: This really comes in handy when talking about statistical significance.)

This means the 2.4% chance of illusion is also a 97.6% chance you observed a real difference between the two treatments.


α= the line in the sand that tells you if something is statistically significant

At some point in the history, a few scientists decided to use .05 as the cutoff point for their p-values.

This means that if p was .05 or less (which is also called 95% level of confidence), then your findings were deemed to be “statistically significant” within the context of an industry standard.

This is where the alpha value, or α, also likely became a standard in reporting stats.

There was no special reason that I know of for using p=.05 or less as the line in the sand, but most research today continues to use that cutoff when determining if results are significant. There are a few fields for which a different value is used, such as nuclear physics, where 5% risk of error is unacceptable. In practical business, a level of confidence of 93% (p=7%) might represent an acceptable level of risk for some businesses. For any particular situation, α is determined by agreement of those involved.

So, what about the results for our mad scientist and his assistant?

They reported that their p-value (p=.024, or 2.4%) was less than the α value (α=.05, or 5%) they used for the experiment.

Remember how I explained that the 2.4% chance of false positive also means there is a 97.6% chance that you observed real difference between your treatments?

Well, what p=.024, α=.05 really means is the results were significant and can be used toward future research because their 97.6% level of confidence exceeds the industry standard of a 95% level of confidence or higher.

Therefore, the conclusion that the 36% difference they observed was in fact statistically significant.

Read more…

Marketing Analytics: 4 tips for productive conversations with your data analyst

September 19th, 2013 No comments


I think there’s a reason for this – poor communication.

































Read more…

Marketing Analytics: 4 tips to boost confidence in your analytics reporting

August 15th, 2013 No comments






























Read more…

A/B Testing: Example of a good hypothesis

July 11th, 2013 2 comments

Want to know the secret to always running successful tests?

The answer is to formulate a hypothesis.





So, what is a hypothesis?


Let’s break that down:

It is a proposed statement.




It can be proved or disproved.


It is used as a starting point for further investigation.



How do I write a hypothesis?




So now the question is …


How do I write a GOOD hypothesis?






  1. Presumed problem
  2. Proposed solution
  3. Anticipated result

Read more…