Correlation analysis in automated testing
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Wcisło, Łukasz
Correlation Analysis is a statistical method that is used to discover if there is a relationship between two variables, and how strong that relationship might be. A correlation coefficient is a numerical measure of such correlation. According to the Cauchy–Schwarz inequality it has a value between +1 and −1, where 1 is total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation. One of the axioms of automated testing is that tests are independent and in spite of that correlation coefficient should be equal to 0. But often it isn't. In this work, we are going to present a method of evaluation of tests suites quality based on correlation coefficient and finding their weak points. Using PC Engines open-source firmware regression test results, which are based on over 140 automated tests run with 2 flavors of software on 4 different platforms, we will show how its quality can be described numerically, and how that results can be used to optimize test criteria. As far as automated testing is considered all the tests can have only two expected output values - pass or fail. Originally Pearson's correlation coefficient is the covariance of the two variables divided by the product of their standard deviations - the first question was how to do it for Boolean variables. We assumed that the only value that matters can be a failure of a test. During the lecture, we will present how mathematical analysis can reveal potential flaws in test criteria by targeting cases that have a large chance to fail simultaneously.
 Thank you very much for coming. My name is Lukasz Wcisło. I'm here to talk about collation analysis in automated testing, as you can read. At the beginning, I want to introduce myself, tell something about who I am and how the history of my education moved me to such a spec of this presentation. Formerly, I thought about something slightly different, but it evolved into this. Then I will tell what was the problem and what was the purpose of this research. I will tell you about some math that stands behind all this thing. And I will go to the use cases and some conclusions, maybe some anecdote. So first of all, what was my main purpose? I was a former computational biologist, bioinformatic. I was a person who used very strong supercomputers to simulate biological processes like protein folding, in example. The most problematic issue with such research is that very pronounced to any performance issues. Because there are billions of particles that can be connected to each other in many ways and can have influence one or two in each other. So this is the performance was crucial. As I moved to informatics to IT, I started to be a tester in 3MD app in Poland. I started to work with big test suits, not very big, but big enough to consider that the speed of the test, the time that it would be done and resources consumed are important. So I thought about making it more reasonable to do some optimization of this test. And I think that nowadays it's a big team, machine learning. So why don't have all the tests learn one from each other? You know, from the history, one test will know that other tests was run and almost the exact time if some other tests failed, this test also failed. Or if some other test was run several iterations and it couldn't reach its expected outcome, the other test would be either shortened or not done. Then I realized it was better from the beginning because if tests are correlated in such way, the test suit is badly created. It has some wrong assumptions and the test conditions are badly formulated. So this is my assumption. I don't want to read it again. I want to make test suits more elegant and save some time and resources. Because for the perfect test suit, there should be no correlation between the tests. If there are some correlations, tests should be, I don't know, passed because we don't need to be made. Then that may be some value measured that can even benchmark test suits, the correlation coefficient. Let's see for an example. There are tests in rows and software versions in columns. Everything is passed and red is failed. It seems obvious. I considered tests at Boolean functions, which are true or false. Normally, when you compute correlation coefficient, you need to do regression first because it's Boolean function even from 0 to 1. That wouldn't make any sense. What I took was the probability, historical probability of passing the test. That was the value that I was measuring if the test is below that probability or above that probability. Next, what I've done, it's obvious that test that will have 100% pass, it doesn't really matter for us. Just the same as software version that fails all the tests. If we have power issue at the moment of testing, it doesn't really matter. It's not a matter of tests. I left only the meaningful test cases, meaningful tests that have sometimes failed. I created a covariance matrix from them. This may look suspicious, but it's very easy math. It can be done in Google spreadsheets. From this, I've created a covariance matrix, which as I said before, where on the diagonal is just a variance of each test that is meaningful and the other fields are covariances between the tests. Next, I computed a piece in correlation coefficient. It may look scary, but it isn't. Trust me. And create a correlation matrix. What we can see above, it's just a shape of the test outcome that was from the beginning that was our input. And below, it's scattered by me. It's only half of the matrix because to be clear for clarification, it should have on a diagonal one because in each field on the diagonal because correlation between a test and the same test is one. This coefficient is always between minus one and one. Minus one is negative correlation. If something passes, the other thing fails. And one is positive correlation. If something fails or passes the other test, the value is always the same. Zero means no correlation. And in a big test suite, or many test suits are aggregated in regression, the real values are near zero. Plus or minus per new zero. If something is like in this example, I don't consider one because it would be obviously error, but 0.5 or something like this, it is significant. We can do that just to validate the test suite, but it wouldn't have much value for us. What can have value? We can, with each regression in continuous integration, continuous delivery, we can test each version and check the dynamic of changes of this coefficient. If we can see that some values that shouldn't be connected seems to be correlated, correlated from some version in the past. Correlation is getting bigger with each iteration. We should see yellow light and check whether the conditions of the tests are prepared correctly. This is a real test outcome from, we are in the free and the app maintenance of Swiss firewalls firmware of PC engines. I pull maybe some of you used this routers. And this is from the test results of the firmware of one of the lines of the one of the routers. We are maintaining them for above two and a half year, but these tests are not a good example because our test suits are evolving too. Some tests were created and I don't know, started to be used half a year ago. So we have not completed a large enough base of tests for long enough history to make this really useful, but large circumventors or hardware producers, I suppose it could be useful just because the very often cases that one person or one team is responsible for one suit in regression. All in all, they might not be aware of correlations between the regression suits. So in the conclusion, the dynamics can be useful in a large sets with a large history. And this is a proof of concept. Do not consider this to be a white paper something like this. Idea is quite new and we started to work on this more seriously on our firmware with I think this is an anecdote. As a squad, it may be some of the some of you know it have heard of it because these four sets of data have the same as a certain below mean of X and Y variance of six and Y correlation and so on. Although they are obviously different. So we have to be very, we have to look. I don't consider the statistics to be a red light to be fail if something rapidly changes and something is rapidly grow because it may be some from some other reason but yellow light should be considered visible if something occurred like this. This is bibliography and thank you for your attention. For us can we have Warrio number here. That. the tests which are most effectively killing all the possible computations? I don't do any computations, all the computations I've done in Google's page sheets. Thank you. Yes, I heard about it, but this is a new concept. We are just looking for ways that could improve our test suits, and I think it should be considered. I can hear you. People who are watching the stream have not heard that, so you should probably mention that there is a thing called mutation testing. Yes. Have I heard about the mutation testing? Because it can be used. It can simplify all the computations that we used here with statistics. Yes, we've heard about it, but it's in progress. We are looking for new ways to do it, and I think it would be considered. Thank you. Any other questions? Yes. Thank you very much. Thank you very much for your attention.
 Thank you very much for coming. My name is Lukasz Wcisło. I'm here to talk about collation  analysis in automated testing, as you can read. At the beginning, I want to introduce  myself, tell something about who I am and how the history of my education moved me to  such a spec of this presentation. Formerly, I thought about something slightly different,  but it evolved into this. Then I will tell what was the problem and what was the purpose  of this research. I will tell you about some math that stands behind all this thing. And  I will go to the use cases and some conclusions, maybe some anecdote. So first of all, what  was my main purpose? I was a former computational biologist, bioinformatic. I was a person who  used very strong supercomputers to simulate biological processes like protein folding,  in example. The most problematic issue with such research is that very pronounced to any  performance issues. Because there are billions of particles that can be connected to each  other in many ways and can have influence one or two in each other. So this is the performance  was crucial. As I moved to informatics to IT, I started to be a tester in 3MD app in  Poland. I started to work with big test suits, not very big, but big enough to consider that  the speed of the test, the time that it would be done and resources consumed are important.  So I thought about making it more reasonable to do some  optimization of this test. And I think that nowadays it's a big team, machine learning.  So why don't have all the tests learn one from each other? You know, from the history,  one test will know that other tests was run and almost the exact time if some other tests  failed, this test also failed. Or if some other test was run several iterations and it couldn't  reach its expected outcome, the other test would be either shortened or not done. Then  I realized it was better from the beginning because if tests are correlated in such way,  the test suit is badly created. It has some wrong assumptions and the test conditions  are badly formulated. So this is my assumption. I don't want to read it again. I want to  make test suits more elegant and save some time and resources. Because for the perfect  test suit, there should be no correlation between the tests. If there are some correlations,  tests should be, I don't know, passed because we don't need to be made. Then that may be  some value measured that can even benchmark test suits, the correlation coefficient.  Let's see for an example. There are tests in rows and software versions in columns.  Everything is passed and red is failed. It seems obvious. I considered tests at Boolean  functions, which are true or false. Normally, when you compute correlation coefficient,  you need to do regression first because it's Boolean function even from 0 to 1. That wouldn't  make any sense. What I took was the probability, historical probability of passing the test.  That was the value that I was measuring if the test is below that probability or above  that probability. Next, what I've done, it's obvious that  test that will have 100% pass, it doesn't really matter for us. Just the same as software  version that fails all the tests. If we have power issue at the moment of testing, it doesn't  really matter. It's not a matter of tests. I left only the meaningful test cases, meaningful  tests that have sometimes failed. I created a covariance matrix from them. This may look  suspicious, but it's very easy math. It can be done in Google spreadsheets. From this,  I've created a covariance matrix, which as I said before, where on the diagonal is just  a variance of each test that is meaningful and the other fields are covariances between  the tests. Next, I computed a piece in correlation coefficient. It may look scary, but it isn't.  Trust me. And create a correlation matrix. What we can see above, it's just a shape of  the test outcome that was from the beginning that was our input. And below, it's scattered  by me. It's only half of the matrix because to be clear for clarification, it should have  on a diagonal one because in each field on the diagonal because correlation between a  test and the same test is one. This coefficient is always between minus one and one. Minus  one is negative correlation. If something passes, the other thing fails. And one is  positive correlation. If something fails or passes the other test, the value is always  the same. Zero means no correlation. And in a big test suite, or many test suits are  aggregated in regression, the real values are near zero. Plus or minus per new zero.  If something is like in this example, I don't consider one because it would be obviously  error, but 0.5 or something like this, it is significant. We can do that just to validate  the test suite, but it wouldn't have much value for us. What can have value? We can, with  each regression in continuous integration, continuous delivery, we can test each version  and check the dynamic of changes of this coefficient. If we can see that some values that shouldn't  be connected seems to be correlated, correlated from some version in the past. Correlation  is getting bigger with each iteration. We should see yellow light and check whether the  conditions of the tests are prepared correctly. This is a real test outcome from, we are in  the free and the app maintenance of Swiss firewalls firmware of PC engines. I pull maybe  some of you used this routers. And this is from the test results of the firmware of one  of the lines of the one of the routers. We are maintaining them for above two and a half  year, but these tests are not a good example because our test suits are evolving too. Some  tests were created and I don't know, started to be used half a year ago. So we have not  completed a large enough base of tests for long enough history to make this really useful,  but large circumventors or hardware producers, I suppose it could be useful just because  the very often cases that one person or one team is responsible for one suit in regression.  All in all, they might not be aware of correlations between the regression suits. So in the conclusion,  the dynamics can be useful in a large sets with a large history. And this is a proof  of concept. Do not consider this to be a white paper something like this. Idea is quite new  and we started to work on this more seriously on our firmware with I think this is an anecdote.  As a squad, it may be some of the some of you know it have heard of it because these  four sets of data have the same as a certain below mean of X and Y variance of six and  Y correlation and so on. Although they are obviously different. So we have to be very,  we have to look. I don't consider the statistics to be a red light to be fail if something  rapidly changes and something is rapidly grow because it may be some from some other reason  but yellow light should be considered visible if something occurred like this. This is bibliography  and thank you for your attention.  For us can we have Warrio number here.  That.  the tests which are most effectively killing all the possible computations?  I don't do any computations, all the computations I've done in Google's page sheets.  Thank you.  Yes, I heard about it, but this is a new concept.  We are just looking for ways that could improve our test suits, and I think it should be  considered.  I can hear you.  People who are watching the stream have not heard that, so you should probably mention  that there is a thing called mutation testing.  Yes.  Have I heard about the mutation testing?  Because it can be used.  It can simplify all the computations that we used here with statistics.  Yes, we've heard about it, but it's in progress.  We are looking for new ways to do it, and I think it would be considered.  Thank you.  Any other questions?  Yes.  Thank you very much.  Thank you very much for your attention.
testing_correlation_analysis_automated_testing_0_167.png testing_correlation_analysis_automated_testing_168_371.png testing_correlation_analysis_automated_testing_372_1487.png testing_correlation_analysis_automated_testing_1488_3131.png testing_correlation_analysis_automated_testing_3132_5663.png testing_correlation_analysis_automated_testing_5664_7355.png testing_correlation_analysis_automated_testing_7356_8879.png testing_correlation_analysis_automated_testing_8880_9803.png testing_correlation_analysis_automated_testing_9804_10235.png testing_correlation_analysis_automated_testing_10236_10703.png testing_correlation_analysis_automated_testing_10704_10955.png testing_correlation_analysis_automated_testing_10956_14183.png testing_correlation_analysis_automated_testing_14184_16199.png testing_correlation_analysis_automated_testing_16200_16763.png testing_correlation_analysis_automated_testing_16764_18023.png testing_correlation_analysis_automated_testing_18024_18107.png testing_correlation_analysis_automated_testing_18108_18215.png testing_correlation_analysis_automated_testing_18216_18347.png testing_correlation_analysis_automated_testing_18348_21875.png testing_correlation_analysis_automated_testing_21876_21990.png