Oct 9, 2014

How to present Usability findings to your clients- “With confidence.. Intervals”

Note: This article will teach you how to use readily available calculators to calculate the confidence interval and use MS Excel to prepare the graphs for task completion rates with the confidence interval. For in-depth knowledge of how to calculate confidence intervals, read Jeff Sauro’s blog

You have convinced your clients about Usability testing. You have done a UT on some design option and your results look very interesting. You have an 80 to 90 % Success rates in Task completion for all tasks. So you happily go to your clients and say that since the task completion rates for the top critical tasks are in the range of 80-90 %, we can safely assume that 80 to 90 percent of our customer will complete the tasks and everyone will live happily ever after!
This is not exactly true and when you are dealing with the data driven type decision makers, they will correctly view your results with the skepticism it deserves.

As User Experience professionals, you are dealing with highly knowledgeable professionals who will definitely question you about small sample sizes and the validity of such results. Well, presenting the results with a confidence interval, will give them some confidence that there is a scientific basis for these calculations and these results may be statistically valid.

How do we get the Confidence Interval ?

When you are doing Usability testing, you typically have a small sample size of participants. If you have a task success rate of a certain percentage, it does not mean that every time you run a test, you will get the same results. In reality the average value of your task completion rates will fluctuate when you repeat your Usability tests.
For.e.g. if you tested a task with 15 users and 13 successfully completed them, if you apply the confidence intervals, what it actually means is that if you run a test 95 times, your Task completion rate would fluctuate between 60 % to 97 % for these tests.

Jeff Sauro has an online calculator on his site http://www.measuringusability.com/wald.htm
  1. Enter the number of passed participants and the total number of tested participants in the input table section of the calculator
  2. Give a confidence level of 95%
  3. Hit calculate: See the Adjust Wald row results
  4. You have a lower range of 0.6086 and 0.9752 which translates to a lower value of 60.86 percent and an upper value of 97.52 percent. That is the fluctuation in your results that you will get when you will repeat the tests with a confidence level of 95% ( repeating the tests 95 times)
  5. For knowing more how it was calculated and what Adjust Wald means, read the article at Jeff Sauros blog

How to present the Task completion rates with the confidence intervals in a Graph

  1. Create a table like the one below and note the success rates as 1 and failure rates as 0.
  2. Notice that I have added 2 columns on the right side of the table; Number of Successful task completion and the task completion percentage
  3. The task completion percentage is calculated by multiplying the number of completions by 100 and dividing it by number of participant
  4. Next is generating the task completion graph.
  5. Insert a 2D Chart  from the insert menu
  6. Click on the design Tab and click on the Select Data menu.
  7. First select the task’s column and click the icon next to chart data and then select the task completion percentage column, following which you will get a chart as shown below

  8. Now we have to show the confidence interval that is range that we got from the Adjust Wald calculator for each Task completion rate.
  9. Now create a table as per the following diagram which contains the following columns: Task completion Rate, Lower Bound , Upper Bound, Lower difference and upper difference
  10. You already know the data from the first three columns. For the lower difference Column, subtract the Task completion rate value from the lower bound value and for the higher Difference value subtract the Upper bound Value from the Task Completion Range.
  11. Select your task Completion Rates graph
  12. From the Layout tab, click on the Error Rates button and select the “More Error Bars” option
  13. From the Layout tab, click on the Error Rates button and select the “More Error Bar" option"
  14. Select the Custom option and click on specify value
  15. On clicking the Specify Value button, you will get a custom Error Bars modal
  16. Click on the button next to Positive error value and select the Higher
    column as shown in the diagram below the Custom Error bars diagram
  17.  Click the Custom Error Bars button again and repeat the same process for theNegative Error Value and now select the lower difference column and click the Custom Error Bars again

No comments: