How I auto-refactored GDAL's test suite
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Stigter, Craig de
Craig's talk was the fourth and final talk in the "Cloud / Development" session at FOSS4G SotM Oceania 2019, organised by OSGeo Oceania and held at The National Library in Wellington, New Zealand from November 12-15 2019. FOSS4G SotM Oceania is the coming together of Oceania's geospatial open source and open data community - with four days of workshops, presentations, a community sprint and social events.
 This is a talk which I'm going to bang through really quickly because it was originally a 30 minute talk about the depths of automated refactoring in Python. I've been at coordinates for 11 years now, that's how long ages is, and if this is the first you've heard of coordinates, you've unfortunately missed all our other talks, but come see us afterwards if finding or publishing data is a pain for you. I spend a lot of time messing around with Python, and this talk is going to be a little bit Python heavy, so sorry about that, but tough. So I went to Pycon 2018, and this kind of started there, where I came across this talk by this guy called John Reese, works at Facebook, and you should watch that talk, it's online if you want a more gentle introduction to automated refactoring, because as I say I'm going to rush through it a little bit. John released a tool called Bola, which is used for large scale refactoring, when you've got a change that you want to make to your code, and it's a lot of changes, and it's a pain to do manually, this can help you do it. And this image shows him, he's making a change in the changes, he's finding these to-do comments, to-do internationalisation, followed by a print statement, and he's adding a translation function to the print statement and removing the to-do comment. Just a little change there, and he can run that over how many files he's got in his code base. And I thought that was pretty cool, and I thought I can do some things with this, so I went off and tinkered with it, downloaded Bola, installed it, played with it, made a couple of small changes on our code base, thought it was pretty awesome, mentioned it to the CTO, and went to bed, and that could have been the end of my talk. Except I woke up the next morning and Rob had just casually volunteered me to refact to the whole GDEL test suite. Thanks, Rob. So I assume you've used GDEL, probably most of you have. If you don't think you have, you probably still have. If you've used QGIS or you've used PostGIS, you've probably used GDEL. And it's a big C++ thing that underlies a lot of the data transformation tools we have. It has a huge Python test suite, about 270,000 lines of code, so you can understand why I was a bit mortified by this endeavor. But I thought I'd dive in and have a look and see, well, at least I'll have a look and see what's wrong with GDEL's tests. Why do they need refactoring? I'll just work through this quickly. Basically it's got a really big old test suite, and the runner and the format of all the tests is really, well, it's all homegrown, there's no frameworks or anything. The tests are very verbose, they're boilerplate-y, the output is ugly, the tests ran in a specific order. You can't just run one test, you have to run the whole file in the order that they are written. It's just generally very inflexible. So I'll just demonstrate with an example. Here's a test. It's actually a made-up test, they made it up. But it's very idiomatic for the style that the GDEL tests were. Now, there's a few weird things going on if you're used to testing in Python that you might immediately notice. Obviously there's an if statement with some condition that it's checking, the test is checking if CS is not equal to 4783, then fail the test, basically. And that's unusual because normally you'd have some kind of assertion and you'd throw an error in a test, but this doesn't do it, it just returns the string fail. So basically the test contract in GDEL tests was to return a value and if that value was fail, the test was failed, if it was success, the test succeeded. If the test threw an exception, the whole test suite stopped, or at least that file did, and it all just exploded. So if you're running thousands of tests and it just explodes halfway through, you've got to start from scratch once you've fixed it. So that was annoying. It also does this thing, GDEL test.post reason fail. You might wonder what that's about. I still wonder what that was about. I think back in the day maybe it sent something to a log or something. Anyway, you could put whatever string you wanted in there with a reason why the test failed. So that was there with the string fail. So the reason was not terribly useful, kind of pointless, it'd be nice to just remove that. And you can also see just at the bottom there it doesn't unlink. It's created a file and it has to clean up after itself, and if it doesn't, no one else will. So there's a temp file sitting in your GDEL code directory, and if your test explodes halfway through, that's just going to stay there. So those are all kind of bad. It'd be nice if tests could clean up after themselves automatically. So I thought maybe this test should look like this at the bottom here. This is the pie test, or an easy, if it was written in pie test style, it might look like this. Firstly, the if statement is gone, it's replaced with an asset. This will throw an exception if the condition is false, rather than return a value. So there's no return success at the bottom. If the test doesn't throw an error, it's succeeded. The unlink thing at the bottom has gone. It's been replaced by, you might notice there's an argument to this test function, tempter, that's called a pie test fixture. And it sets up some stuff before the test runs and automatically cleans it up at the end of the test. So you can kind of assume or reply that this is going to create a temporary directory called temp and whatever folder we're in, I guess, and it will delete it and all its contents at the end of the test. The... Yeah. I'll just point out as well, no matter how good your regex is, if you wanted to do this with find and replace in your editor, you'd probably have a hard time. So we need something a little bit more structured. And what we need is called syntax-aware refactoring. Now, this encode here has actually simplified a lot. The original code is kind of awful. Sorry about that, but you can see the original code if you want. There's a link and stuff. And, yeah, so regex-replace isn't going to cut it. We need a syntax-aware find and replace. And it turns out that Python actually defines a simple pattern grammar. I found it in the Python source code. For these find patterns, and this is a... it's kind of like a regex, but it finds stuff in Python source files, such as the syntax tree. So let's walk through this quickly. We're doing a Bola query, and we are searching some file names. There will be Python files. And we're looking for a pattern. I'll come back to the pattern. And we're going to find... so the pattern finds some stuff in a Python file. And then we're going to modify that stuff by calling a callback, which is at the bottom there. The syntax of the patterns looks a little bit complex, but it is kind of not too difficult to learn. I'll run through it. We're searching for a simple statement, and it contains a function call. That's actually what power means. I don't know why it's called power. A simple statement that contains a function call to gdeltest.postreason with one string argument. And we're going to assign that string argument to the name reason. That's what that pattern means. We'll go through that once more. We're looking for a simple statement that contains a function call to gdeltest.postreason with one string argument. And we'll capture that argument in the name reason. So that capture thing is a little bit like a regex group capture. And then we're going to call the callback, which is at the bottom. And the callback is basically going to check if the reason was the string fail, then remove the whole statement. Cool. Okay. Hopefully you followed that. And that's what it does. So I thought that was handy. It seems like a lot of work to just do that. You probably possibly could do that one with a fine in replace. You haven't seen it run on a big project, of course. When you do, it looks like this. So that one little change saved over 9000, or almost 9000 lines of pointless code. So I went a little bit nuts over that and I got really excited. And I'll just quickly whip through some of the other changes that I made. The test functions, so to run tests and py tests, they have to be named test underscore something. So I just did that. Found things that looked like tests and called them test something. I got rid of those return value things and I made it just throw assertion errors or exceptions. And that was as easy as just getting rid of the stuff on the top left. I mean, there were like helper functions called throughout the code and the helper functions also returned success or fail sometimes. So I had to change the helper function manually and then I just went through and automated these changes to the things that called the helper functions. In the bottom right, I changed the if statements to assertions. And you might notice that print statement is gone as well. And I decided that was okay. I didn't need to print the variables that I was testing because py test actually automatically prints them. If your assertion fails, it will look at the variables you were checking and it will print out the values of all the things you were checking against. So I let my code happily get rid of assertion print statements as long as they were only printing things that I was asserting on. Whew! This took a while. It was really fun. I enjoyed it. But I got a little bit borderline obsessive over it. It was not very borderline actually. It was just really quite getting unhealthy. And my wife wasn't very impressed because every night I would spend like an hour and a half on this and it was so cool and it was so much fun. Ha! And I just kept going. And meanwhile, GDAL didn't stop. It was actually a very active project. It's not the Linux kernel but you could expect 10, 15, maybe 20 commits a day on average. And sometimes more. And almost all the commits as you would expect would touch the test suite which I had mostly munged. And I couldn't really keep up with it because there's a lot of conflicts. Luckily, it's not too hard when you're doing this to avoid conflicts on a lot of these things. So I quickly decided that if I was going to make any manual changes, they had to be separate commits. And if I was going to make automated changes, I had to rebase them as well as rebasing all my manual things. And I had to recreate the automated commits from scratch. I wasn't going to commit them to my get thing and then have to resolve conflicts against them because what a waste of time. So instead I would rebase my manual changes on what GDAL was doing and then I would totally recreate all my automated commits because computers are better at doing that than I am at resolving conflicts. And then obviously I had to do as little as possible. This was more difficult than you might imagine because I was having a lot of fun. So I had to resist the temptation to fix everything. All that rebasing got tedious so every day I'd sit down and I'd spend 15 minutes rebasing before I started doing anything useful. So I wrote a script and that script looked a little bit like this when it was running, only about 10 times slower. I sped it up because it'd be too boring otherwise. And any second now, boom, boom, boom. You can see at the end of that script after it had recreated all the automated changes it just cherry-picked a whole bunch of manual stuff on top that I'd had on another branch. Yeah, it took about three minutes to run and that was well sped up. But it was so much fun to watch it every time I saw that go I was like, yes, awesome. And it felt like this. I'm just going to finish up with some notes on why not to do this. Most of the time at least. It actually takes quite a while to get up to speed with all this and most of the time you might be better off with a regex replace if you can. But for very large projects, I think this was awesome. Absolutely worth doing. It only really works if you have well-tested code because no one's going to review your 100,000 line pool request. So the only thing that's reviewing it are the tests. Luckily, this was a test suite, so I assume it's well-tested. And yeah, don't do it if you can do a regex replace. I ended up with this, which I felt good about, but possibly I shouldn't have. No one wants to review that. A couple of people did. I don't think they actually went through all of it. They only went through the manual commits. So no one can review that. I had to come up with some tips for myself to get this thing regeged, otherwise it was around forever. Obviously I had to be polite and humble. I had to get feedback from the mailing lists. I had to leave out anything that was kind of controversial because I didn't want it jeopardizing the rest of the pool request. And I had to be responsive and patient. And possibly quite importantly, I had to give people shiny stuff because otherwise no one wanted to look at this thing. It was a monster. So what I did was I did things like adding green Unicode check marks when your tests pass and stuff like that. And voila, it got regeged. I doubt even Roel is listening, but thanks to him for reviewing this and reaging it, as well as the other reviewers. He probably did read every single line. He possibly did read it all. It's very thorough. Thanks a lot. Awesome work, Greg. A monumental task. Kind of amazing it got merged. Yeah, it's great. Questions? Come on. What is the most obscure thing that you automatically removed? That's a good question. I might go back a few slides to jog my memory. Is it going to play? Question? There are actually only eight changes in the end. But they needed a lot of testing and rewriting and re-basing to get there. Has anybody written non-conforming rubbish tests since you're Pia? It's still a work in progress. That Pia has merged, but there's a lot of things I didn't get to. For instance, you still can't run an individual test by itself. You have to run the whole file. That's because there's no way that I could figure out which tests required things from earlier tests in the same file. I hope that people are writing things properly now with Pytest fixtures instead of having tests that set things up and then other tests that depend on that. But I haven't been watching super closely, to be honest. Sorry, I would be at the end of the night. I get nervous when I submit a pull request to a new project, even if it's like one function. How long did it take from the point where you approached this with them and said, hey, I'd like to do this? How long was that process from that initial bit to it actually being merged? About three days. So there was a couple of back and forths on the mailing list. But I quickly got the impression that all of the developers involved would welcome this change because they couldn't stand the old test suite. It was a major pain point for them. So that was pretty good. All right, good work. That's it. On behalf of everybody that uses GDAL, thank you very much for your contribution to GDAL. Really appreciate it. Thanks, people.
 This is a talk which I'm going to bang through really quickly because it was originally a  30 minute talk about the depths of automated refactoring in Python.  I've been at coordinates for 11 years now, that's how long ages is, and if this is the  first you've heard of coordinates, you've unfortunately missed all our other talks,  but come see us afterwards if finding or publishing data is a pain for you.  I spend a lot of time messing around with Python, and this talk is going to be a little bit  Python heavy, so sorry about that, but tough.  So I went to Pycon 2018, and this kind of started there, where I came across this talk by this guy  called John Reese, works at Facebook, and you should watch that talk, it's online if you want  a more gentle introduction to automated refactoring, because as I say I'm going to rush through it a little bit.  John released a tool called Bola, which is used for large scale refactoring, when you've got a change  that you want to make to your code, and it's a lot of changes, and it's a pain to do manually,  this can help you do it.  And this image shows him, he's making a change in the changes, he's finding these to-do comments,  to-do internationalisation, followed by a print statement, and he's adding a translation function  to the print statement and removing the to-do comment.  Just a little change there, and he can run that over how many files he's got in his code base.  And I thought that was pretty cool, and I thought I can do some things with this, so I went off and tinkered with it,  downloaded Bola, installed it, played with it, made a couple of small changes on our code base,  thought it was pretty awesome, mentioned it to the CTO, and went to bed, and that could have been the end of my talk.  Except I woke up the next morning and Rob had just casually volunteered me to refact to the whole GDEL test suite.  Thanks, Rob. So I assume you've used GDEL, probably most of you have.  If you don't think you have, you probably still have.  If you've used QGIS or you've used PostGIS, you've probably used GDEL.  And it's a big C++ thing that underlies a lot of the data transformation tools we have.  It has a huge Python test suite, about 270,000 lines of code, so you can understand why I was a bit mortified by this endeavor.  But I thought I'd dive in and have a look and see, well, at least I'll have a look and see what's wrong with GDEL's tests.  Why do they need refactoring?  I'll just work through this quickly.  Basically it's got a really big old test suite, and the runner and the format of all the tests is really, well, it's all homegrown, there's no frameworks or anything.  The tests are very verbose, they're boilerplate-y, the output is ugly, the tests ran in a specific order.  You can't just run one test, you have to run the whole file in the order that they are written.  It's just generally very inflexible.  So I'll just demonstrate with an example.  Here's a test.  It's actually a made-up test, they made it up.  But it's very idiomatic for the style that the GDEL tests were.  Now, there's a few weird things going on if you're used to testing in Python that you might immediately notice.  Obviously there's an if statement with some condition that it's checking, the test is checking if CS is not equal to 4783, then fail the test, basically.  And that's unusual because normally you'd have some kind of assertion and you'd throw an error in a test, but this doesn't do it, it just returns the string fail.  So basically the test contract in GDEL tests was to return a value and if that value was fail, the test was failed, if it was success, the test succeeded.  If the test threw an exception, the whole test suite stopped, or at least that file did, and it all just exploded.  So if you're running thousands of tests and it just explodes halfway through, you've got to start from scratch once you've fixed it.  So that was annoying.  It also does this thing, GDEL test.post reason fail.  You might wonder what that's about. I still wonder what that was about.  I think back in the day maybe it sent something to a log or something.  Anyway, you could put whatever string you wanted in there with a reason why the test failed.  So that was there with the string fail. So the reason was not terribly useful, kind of pointless, it'd be nice to just remove that.  And you can also see just at the bottom there it doesn't unlink.  It's created a file and it has to clean up after itself, and if it doesn't, no one else will.  So there's a temp file sitting in your GDEL code directory, and if your test explodes halfway through, that's just going to stay there.  So those are all kind of bad. It'd be nice if tests could clean up after themselves automatically.  So I thought maybe this test should look like this at the bottom here.  This is the pie test, or an easy, if it was written in pie test style, it might look like this.  Firstly, the if statement is gone, it's replaced with an asset.  This will throw an exception if the condition is false, rather than return a value.  So there's no return success at the bottom. If the test doesn't throw an error, it's succeeded.  The unlink thing at the bottom has gone. It's been replaced by, you might notice there's an argument to this test function, tempter, that's called a pie test fixture.  And it sets up some stuff before the test runs and automatically cleans it up at the end of the test.  So you can kind of assume or reply that this is going to create a temporary directory called temp and whatever folder we're in, I guess,  and it will delete it and all its contents at the end of the test.  The...  Yeah. I'll just point out as well, no matter how good your regex is, if you wanted to do this with find and replace in your editor,  you'd probably have a hard time.  So we need something a little bit more structured.  And what we need is called syntax-aware refactoring.  Now, this encode here has actually simplified a lot.  The original code is kind of awful. Sorry about that, but you can see the original code if you want. There's a link and stuff.  And, yeah, so regex-replace isn't going to cut it. We need a syntax-aware find and replace.  And it turns out that Python actually defines a simple pattern grammar. I found it in the Python source code.  For these find patterns, and this is a... it's kind of like a regex, but it finds stuff in Python source files, such as the syntax tree.  So let's walk through this quickly.  We're doing a Bola query, and we are searching some file names. There will be Python files.  And we're looking for a pattern. I'll come back to the pattern.  And we're going to find... so the pattern finds some stuff in a Python file.  And then we're going to modify that stuff by calling a callback, which is at the bottom there.  The syntax of the patterns looks a little bit complex, but it is kind of not too difficult to learn.  I'll run through it. We're searching for a simple statement, and it contains a function call.  That's actually what power means. I don't know why it's called power.  A simple statement that contains a function call to gdeltest.postreason with one string argument.  And we're going to assign that string argument to the name reason.  That's what that pattern means. We'll go through that once more.  We're looking for a simple statement that contains a function call to gdeltest.postreason with one string argument.  And we'll capture that argument in the name reason.  So that capture thing is a little bit like a regex group capture.  And then we're going to call the callback, which is at the bottom.  And the callback is basically going to check if the reason was the string fail, then remove the whole statement.  Cool. Okay. Hopefully you followed that.  And that's what it does. So I thought that was handy.  It seems like a lot of work to just do that.  You probably possibly could do that one with a fine in replace.  You haven't seen it run on a big project, of course.  When you do, it looks like this.  So that one little change saved over 9000, or almost 9000 lines of pointless code.  So I went a little bit nuts over that and I got really excited.  And I'll just quickly whip through some of the other changes that I made.  The test functions, so to run tests and py tests, they have to be named test underscore something.  So I just did that.  Found things that looked like tests and called them test something.  I got rid of those return value things and I made it just throw assertion errors or exceptions.  And that was as easy as just getting rid of the stuff on the top left.  I mean, there were like helper functions called throughout the code  and the helper functions also returned success or fail sometimes.  So I had to change the helper function manually and then I just went through and automated these changes  to the things that called the helper functions.  In the bottom right, I changed the if statements to assertions.  And you might notice that print statement is gone as well.  And I decided that was okay.  I didn't need to print the variables that I was testing because py test actually automatically prints them.  If your assertion fails, it will look at the variables you were checking  and it will print out the values of all the things you were checking against.  So I let my code happily get rid of assertion print statements  as long as they were only printing things that I was asserting on.  Whew!  This took a while.  It was really fun. I enjoyed it.  But I got a little bit borderline obsessive over it.  It was not very borderline actually. It was just really quite getting unhealthy.  And my wife wasn't very impressed  because every night I would spend like an hour and a half on this  and it was so cool and it was so much fun.  Ha!  And I just kept going.  And meanwhile, GDAL didn't stop. It was actually a very active project.  It's not the Linux kernel but you could expect 10, 15, maybe 20 commits a day on average.  And sometimes more.  And almost all the commits as you would expect would touch the test suite  which I had mostly munged.  And I couldn't really keep up with it because there's a lot of conflicts.  Luckily, it's not too hard when you're doing this to avoid conflicts on a lot of these things.  So I quickly decided that if I was going to make any manual changes, they had to be separate commits.  And if I was going to make automated changes, I had to rebase them as well as rebasing all my manual things.  And I had to recreate the automated commits from scratch.  I wasn't going to commit them to my get thing and then have to resolve conflicts against them  because what a waste of time.  So instead I would rebase my manual changes on what GDAL was doing  and then I would totally recreate all my automated commits  because computers are better at doing that than I am at resolving conflicts.  And then obviously I had to do as little as possible.  This was more difficult than you might imagine because I was having a lot of fun.  So I had to resist the temptation to fix everything.  All that rebasing got tedious so every day I'd sit down and I'd spend 15 minutes rebasing  before I started doing anything useful.  So I wrote a script and that script looked a little bit like this when it was running,  only about 10 times slower.  I sped it up because it'd be too boring otherwise.  And any second now, boom, boom, boom.  You can see at the end of that script after it had recreated all the automated changes  it just cherry-picked a whole bunch of manual stuff on top that I'd had on another branch.  Yeah, it took about three minutes to run and that was well sped up.  But it was so much fun to watch it every time I saw that go I was like, yes, awesome.  And it felt like this.  I'm just going to finish up with some notes on why not to do this.  Most of the time at least.  It actually takes quite a while to get up to speed with all this and most of the time  you might be better off with a regex replace if you can.  But for very large projects, I think this was awesome.  Absolutely worth doing.  It only really works if you have well-tested code because no one's going to review your 100,000 line pool request.  So the only thing that's reviewing it are the tests.  Luckily, this was a test suite, so I assume it's well-tested.  And yeah, don't do it if you can do a regex replace.  I ended up with this, which I felt good about, but possibly I shouldn't have.  No one wants to review that.  A couple of people did.  I don't think they actually went through all of it.  They only went through the manual commits.  So no one can review that.  I had to come up with some tips for myself to get this thing regeged, otherwise it was around forever.  Obviously I had to be polite and humble.  I had to get feedback from the mailing lists.  I had to leave out anything that was kind of controversial because I didn't want it jeopardizing the rest of the pool request.  And I had to be responsive and patient.  And possibly quite importantly, I had to give people shiny stuff because otherwise no one wanted to look at this thing.  It was a monster.  So what I did was I did things like adding green Unicode check marks when your tests pass and stuff like that.  And voila, it got regeged.  I doubt even Roel is listening, but thanks to him for reviewing this and reaging it, as well as the other reviewers.  He probably did read every single line.  He possibly did read it all.  It's very thorough.  Thanks a lot.  Awesome work, Greg.  A monumental task.  Kind of amazing it got merged.  Yeah, it's great.  Questions?  Come on.  What is the most obscure thing that you automatically removed?  That's a good question.  I might go back a few slides to jog my memory.  Is it going to play?  Question?  There are actually only eight changes in the end.  But they needed a lot of testing and rewriting and re-basing to get there.  Has anybody written non-conforming rubbish tests since you're Pia?  It's still a work in progress.  That Pia has merged, but there's a lot of things I didn't get to.  For instance, you still can't run an individual test by itself.  You have to run the whole file.  That's because there's no way that I could figure out which tests required things from earlier tests in the same file.  I hope that people are writing things properly now with Pytest fixtures  instead of having tests that set things up and then other tests that depend on that.  But I haven't been watching super closely, to be honest.  Sorry, I would be at the end of the night.  I get nervous when I submit a pull request to a new project, even if it's like one function.  How long did it take from the point where you approached this with them and said,  hey, I'd like to do this?  How long was that process from that initial bit to it actually being merged?  About three days.  So there was a couple of back and forths on the mailing list.  But I quickly got the impression that all of the developers involved would welcome this change  because they couldn't stand the old test suite.  It was a major pain point for them.  So that was pretty good.  All right, good work. That's it.  On behalf of everybody that uses GDAL, thank you very much for your contribution to GDAL.  Really appreciate it.  Thanks, people.
Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_0_1439.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_1440_2963.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_2964_3059.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_3060_3131.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_3132_3191.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_3192_3371.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_3372_3419.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_3420_4235.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_4236_16631.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_16632_16763.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_16764_16811.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_16812_16895.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_16896_17255.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_17256_17303.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_17304_17351.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_17352_17399.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_17400_20783.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_20784_20915.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_20916_22091.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_22092_22271.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_22272_22319.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_22320_22391.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_22392_22475.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_22476_23783.png Craig_de_Stigter_-_How_I_auto-refactored_GDAL_s_test_suite_Y_opOoDrYyM_23784_28094.png