Take the 2-minute tour ×

I have a list of sets of integers (run times in seconds, so all are greater than zero), with a varying amount in each set:

test suite A: 12, 15, 16
test suite B: 120, 130, 125, 90, 110
test suite C: 3

I will be running test suites A, B and C together, and I want to predict how long it will take. Summing the mean values of suites A, B, C gives me an expected run-time, but doesn't say anything about how certain I can be of that number. Ideally, I'd like a variance (and therefore standard deviation) as well.

Given that I want to give each suite equal "weighting" in any such calculation, what's the most reasonable way to go about this? I've seen Adding/Combining Standard Deviations , which is similar, but different (they aren't summing the values in the sets, as I am).

share|improve this question
This is in a real sense purely a statistics question. The fact that your data come from running a test suite is immaterial. You might look on Stats.SE. –  dmckee Nov 26 '11 at 3:21
add comment

1 Answer

up vote 0 down vote accepted

If you're willing to do assume independence between the runtimes of the different test suites, then you can calculate the variance of the time it takes to run A, B, and C together as the sum of the variances for the three. If you can't assume independence, you will need some measure of the way in which they are dependent. In particular, you will need the three pair-wise covariances.

The full calculation is

Var(A + B + C) = Var(A) + Var(B) + Var(C) + 2Cov(A,B) + 2Cov(B,C) + 2Cov(A,C)

When you assume that the random variables are independent, you get

Cov(A,B) = Cov(B,C) = Cov(A,C) = 0.
share|improve this answer
That works for 3 suites, but my actual data includes dozens of suites, which would seem to imply (N choose 2) "covariant terms". If my run-times are related to other factors (speed of the test computer) which would change the runtime of all suites -- in a non-uniform fashion -- can I consider them independent? Sorry, still a little new to statistics. –  Jordan A. Nov 25 '11 at 21:16
You're right that n suites will mean (n choose 2) covariance terms. And, unfortunately, you're also right that if the differences in times from different trials is affected by something that would affect all the runtimes (like the speed of the test computer), then the independence assumption will probably not work. One option would be to assume independence and see how accurate your numbers are. Another would be to actually calculate the covariances. The wikipedia page on covariance gives the equation to calculate the covariance of two variables. –  David Nov 25 '11 at 21:23
Alright, so I'll have to calculate the covariance of each pair of suites. Based on what I've read on Wikipedia, Cov(A,B) is defined for A, B both of size N; as previously stated, my data doesn't look like that at all; is there some way to calculate the covariance in that case? If not, is there a reasonable way to approximate that? Alternatively, is there some other measure of "coherence" that I could apply to my data to give an indication of how long the complete test will take? –  Jordan A. Nov 25 '11 at 21:34
It's not just that it requires that A and B have to be of size N. It also requires that the N samples come in pairs. So if you don't know which times from set A go with which times from set B, you can't compute Cov(A,B). If you do know which times go with which, you can just calculate the covariance over the pairs that go together and just ignore the times for B that don't have a an associated time for A (and vice-versa). –  David Nov 25 '11 at 21:56
Alright, thanks. I'll have to take a closer look at my data...it should be possible based on other characteristics. Thanks for your help; this stats stuff is hard. –  Jordan A. Nov 25 '11 at 22:38
add comment

Your Answer


