Test-driven code search and reuse coming to Python with pytest-nodev
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Amici, Alessandro
Alessandro Amici - Test-driven code search and reuse coming to Python with pytest-nodev We will present the test-driven reuse (TDR) development strategy, a natural extension of test-driven development (TDD), and how to execute it with [pytest-nodev] an Open Source test- driven search engine for Python code. When developing new functionalities developers spend significant efforts searching for code to reuse, mainly via keyword-based searches, e.g. on StackOverflow and Google. Keyword-based search is effective in finding code that is explicitly designed and documented to be reused, e.g. libraries and frameworks, but typically fails to identify reusable functions and classes in the large corpus of auxiliary code of software projects. TDR aims to address the limits of keyword-based search with test- driven code search that focuses instead on code behaviour and semantics. Developing a new feature in TDR starts with the developer writing the tests that will validate candidate implementations of the desired functionality. Before writing any functional code the tests are run against all functions and classes of available projects. Any code passing the tests is presented to the developer as a candidate implementation for the target feature. [Pytest-nodev] and other nodev tools that help implement TDR for Python are newer than the JAVA counterparts, in spite of that we will present several applications of the technique to more and more complex examples.
 Okay, welcome to this talk. The speaker is Alessandro Amici, a good developer, I think. He will explain some stuff about pie tests. So welcome. So this talk is about test-driven code search. And this is a rather new technique, not so new because someone already tried it a few years ago with Java. And the... But it's the first time that I see it applied to Python. The idea is pretty simple. What we want to... What we produced, what we did was a very basic search engine. It's a pie test nodev, it's a pie test plugin that enables you searching for code inside your machine on packages that you have installed on your local machine. The special thing about this, the test-driven search, is that you use a test as part of the search query. So you may use some... Also metadata and try to refine your search, but the core... At the core, what you are looking for is what you describe within a test. We call it a specification test. There is something that tries to specify a behavior or a feature without going too much into the details on how it is implemented. Once you run your search engine, you will get some search results. So this is a list of functions or classes or whatever object, actually, that pass the specification test. The documentation that the main... The core of the tool is the pie test nodev plugin, and there you have the main documentation, but there are a couple of other tools that I will show during the talk. Now, since this is something new, at the beginning I organized this talk to be somehow theoretical, but then I completely rewrote it yesterday because I think really good examples make people understand much faster. How it works. Do people here know pie test and pie test features? Who does... Okay. Now basically the base implementation detail is that the plugin provides a special fixture that's called candidate, and you need to use this feature when you write a test that you want to use to search for code. What will happen is that the fixture will effectively parameterize your test by passing it all the objects that it will manage to find in your environment. So if you install 10 packages in the virtual environment together with pie test, it will collect all the objects, all the live objects in your standard library and all the packages that you installed. Then obviously since this will be a parameterized test, the test will be run every thousandth of time most probably, and once for every object and the object will be passed, a reference to the object will be passed into the candidate variable. So you basically will use this candidate as if it was the function that you are looking for, and then the search engine will just tell you which functions, classes, or objects in general actually appear to behave exactly as you intended. So let's do our first search. You want to search for some kind of a function that has a feature, for example, let's search for a function that even the name of it executable returns the path to it. This is not just a nice example, this is actually the first real case that we have. We had exactly this need, and we started searching for it on the web, we didn't like the results, and we said, okay, no, this is the perfect test case because it's something easy, it's easy to write a test for it, and maybe there is something somewhere in my environment already that does it. So you could just write something like a sub process call to which and then pass the result, et cetera, that would be hacky and it would not work on Windows, so it's not the best. So what is the specification test? I write a standard test function for PyTest. I use the candidate fixture, then I just as to have the test more readable, I basically rename the candidate to which, which is more or less the idea that I'm looking something that works like the which command in the standard library. So then I assert the behavior that I expect. If I ask SH, I want to the function I'm looking for should return bin SH, and if I pass it the string m, it should return usrbinm. These two are two very common UNIX commands, and they are the one among the most tables because a lot of commands can't be in usrbin or in bin or spin or, but these two are the more common. So once I have written this test, I write it to a file and then I just run it as usual with PyTest as usual, just I need to add a candidate from all. This means that the candidate function, the candidate fixture will be parameterized by everything I find in my environment. So this starts a standard test session, and I get usually something like 5,000, 6,000 objects. This depends very much on how many packages you have installed. This is not many, it's easy to go into the 30,000 or 50,000. And then it just runs for a while, we will see in a minute. And since the test is expected to fail, the PyTest will print a small x when the test fails. You are throwing random functions to the test, so you expect it to fail most of the time. Then you have capital X, which means that the test passed in, it was not expected, but it passed. At the end of the run, you have many, many x. What you expect to do is to have a result, and in this case we found three functions, three objects that passed the test. And this is the report. So for my test which file, we found a case in which the test which function passed, and this is the executable. Now I have the test function as well, and let's see how it works, how much time it takes. So right now, I'm not using PyTest, pure PyTest, I'm using kind of a boxed run on PyTest inside the Docker container, because when you throw random arguments to random functions, anything can happen. So if you try to do it on your machine, you will find backup files with crazy names or probably connections to run costs or whatever. So you prefer to do it in Docker, and at the end of the run, you throw away your Docker environment. So what happens is that right now it's collecting all the objects. Now I have a little bit less objects than when I did the test, because I blacklist objects all the time, because they might crush your environment or, I don't know, open up a browser, et cetera. And this is what happens. Now all the tests, that test is running with all the functions. We see small x, and it means that we didn't find n much, but here we have one x. So this is one of the, we find at least one function that actually worked. This takes approximately 60 seconds, and everything goes okay. And now we should also have some garbage on the screen, because since you are using functions and classes in unexpected ways, you're always throwing random stuff to it. Exactly. You end up discovering a lot of bugs in the package that you have, because most of the printouts is exception in the Dell method that are ignored but printed to the standard error. Well, I finished. So this is the, now what happens once you get the result? You say, okay, I have tested past that very easy, very basic test. And what do I do? Well, since I have a manageable number of results, I can just have a look at them and decide if this is really what I want. This is the, sorry, the distutils.pwn.findexecutable. The name looks like what we are interested in. And this is inside the standard library. So it's very useful. Maybe I don't need to write any code for my findexecutable, for my which function, because I may just use this one. You see, that's more or less what I thought. It gets path, then it splits somehow in OS independent way, then it makes, it does some win32 checks that I even didn't think I needed because I don't use windows usually, but yes, might be useful. And then it just tries to see if the file exists inside the path. It's not really the best. I mean, this file, it doesn't check if the file is unexecutable. So not really perfect, but at least I have a template if I want to improve on that. Then I have pxpect.udeal.switch. I don't care too much about that because I already have a function in the standard library, so I don't need to add a dependency to my project if I want to use that. But then they have shd.witch. So, this is even more standard in the standard library, and this is the code, and if you have to look at the code, it's much, much more complex than it has any real access check. That means it checks that you can read it, that you can read the file, and you can execute it. And it has several details that I would not have thought I would not, it would have taken me one year of production to get right. So very nice. Unfortunately, if you go into documentation, you'll learn that this is only a Python 3, actually Python 3.3. So if your code, if your use case needed to work in Python 2.3, you might get a very nice find executable, it's still in the standard library, it's not as nice, but okay. Or maybe you can just take it as template and get it better. Or if you are Python 3 only, you have the luxury to use which. Which is great. Well, how many of you already know which function or how to solve this problem? Okay, a few. Right, I mean it's in the standard library, but I mean I didn't know it, and it was faster this way than to look for it. Okay, let's go back. Now this is a very simple example, but it also shows how things work. Now one of the point is that in this case, input and output of the function were really easy. When you have something that were the reasonable implementation, it's really easy. It's easy to write a test, but as soon as you look for more complex stuff, writing a test that is somehow implementation agnostic, that doesn't make too many assumptions of the implementation, it's complicated. It's more complicated. But actually Python is really great to write stuff that it's not too tight to the implementation, to the details of the implementation. Because it has dynamic measure, for example using that typing, you are not forced to guess the right data type. The in operator is extremely powerful, and a lot of classes even work nicely with the in operator. That is, instead of looking if the result of your function is a list, and the first element of a list is what you were looking for, you just use the in operator to see if somewhere inside your function, inside your result, there is what you expected it to be. And then there are, you may write specific helper, in particular, we wrote the not-aspects that helps you that leverages the inspect module to go even deeper into the search of where if your result actually contains what you expected it to contain, even if in crazy ways. So let's see how you would write a specification test in a way that is, that tries to be more independent on from the implementation. Here I want to parse in fc3986. This is also a real test, a real case. So I use the candidate fixture, I just rename it for, so I read it nicer, I use the test URI, and then I get all the functions that I will get will be passed this URI and I expect it to return some kind of tokens. And then here I will check if the schema and the path that I put in my URI are correctly parsed. Now, since most, there are a lot of false positives that are just strings, I mean just functions that return the same string as the inputs, I check that the return of my functions is not string. I don't care, I really want the string to be divided into tokens, so I don't want one string. I want some kind of list of strings. So let's see how it goes. This is the naive implementation in the sense that I didn't use any special trick except Python standard in operator, that is overloading, etc. Now this is going to run, come on, and usually I have different common lines that can be passed and those mostly need to restrict the search space. If you already know that some packages are not useful, you want to restrict the search space so you get faster. But this one candidates from all, it's the more powerful, it's just search for everything and anything in your environment. So this is where... Obviously, I tested just before the... I don't know. Let's see, I have a second run. Now let's see, since it takes a little bit of time to run, I also tried the second example, that is the same parsing function that is instead tested, the test is written using some advanced functionalities. The splact container in other specs generic, it gets an object and it makes it's a proxy object that when you use the infunction, it tries really hard to see if the item that you're looking for is somewhere in the object. So for example, it looks into the attributes, into the properties, even if it's an iterable, it looks inside every item inside the iterable. So it's extremely thorough. Let's see if we manage to not kill the queue. So apparently they're both running. So on this screen, I have the naive test, the one that crashed before. It was some kind of race condition because it's going okay now. And now let's see what the results are. Okay. I got several results. Now the first three results in collections doesn't look very good because keys view, chain map, user string really look like false positives. They're not trying to do anything with RFC or URL parsing but it's just their packaging somehow, the string that you are giving them. But then you have this RFC3986 API, you reference that looks very nice. But also the URL parts, you are parsed. That means you hit function that are able to do this both in a package and also in the standard library. Now what is interesting is that in both cases, both URL, the URL parts inside the RFC3988 package and also the one in the standard library, they don't return lists. They return classes. So how exactly this worked with the class? The point is that a lot of people are quite smart and they give you some way to get to access stuff or to test stuff in an implementation independent way. That is the two implementation that they used actually provide a underscore, underscore, or contains underscore, underscore method that tests exactly if they managed to find pieces that test exactly like a string, like a, sorry, a table or list. So it's not a simple type but it's a class that behaves like a type. Very nice. So you can happily use this one for most of your need but if you need more features, you may explore the code and you see this special package as more features. For example, it's able to recognize the username which the standard library function doesn't. Now there's something even more interesting. The other test that one that uses a dedicated proxy object to do the containment test has found one more object that matches. And this is a class in the PIP product that actually does the right thing but doesn't provide the nice containment helper functionality. So we managed to get it as well because the helper function tried very hard to find if the postgres and the path, the schema and the path were inside the class. So this is a way to get to test results in an implementation-independent way but then I want also to pass arguments in the independent way, in the implementation-independent way. Still in this case, what helps me is the parameterized marker of Pyvest. For example, in this case I'm looking for a function that just removes comments from a stream. And the main point is how do I represent the stream because this is my text and this is the readout of my configuration file, for example, and I want to strip these comments here. So how do I do? I use a parameterized argument so that I can say, okay, you have different functions that will make this text, this comment, into different shapes. I can pass it as it is. I can pass it as a list of individual lines. Or I can pass it as a list of individual lines with numbers. This is how my application actually was doing this part. Or I can pass it as a file. Now in this case, since I have a lot of parameters, I will run not just 5,000 times, but this will run 20,000 times. So I prefer to restrict my search by just including any function whose name matches this regular expression. So I want something that has to do with comments. This makes everything much, much faster. And here it is. So I find an ignore comments function in PIP, very good, because PIP is something that I might assume it's a light dependency. And this tells me that the text to stream that passes is the third one. So I go back here. It's 0, 1, 2. So this is the way, which was exactly the way I preferred. I could have worked with all the other trees, but this means that I don't even need to change my application to use that function. So by the way, extremely fast. This is the ignore comments. It's very simple. And it has also the feature that keeps the line. If it's empty. It also returns the line number because it doesn't return all the lines that you pass. And loop at the other function just below. This function takes options, which is a special class. And this class must have the skip requirements regx, otherwise it crashes. Oh, God. Even if I needed this, I would never, ever manage to pass the correct parameters to it because these parameters is extremely tied to the implementation. So I do tests that are quite loosely coupled with the implementation. I try to be as implementable as possible, but I only find functions, callables, or classes that are good code. They are not, they don't mix implementation details uselessly. It could just have you skip. This skip requirements regx could just have been a keyword argument with the same default. And the function would be as useful. And I would have been able to search for it or to use it in general. So when you search, you may get only relevant results, which means your query is just perfect. Or you have to refine your query. If you don't get any results at all, which happens quite often, it means that your test is too strict. And you probably need to remove test cases, edge cases, or probably just use a lower number of normal cases. If you find a lot of results, but they are not relevant, it means that your test is not, it's too weak. It's not strict enough, so you need to add more cases, more describe your feature better, and probably add more corner cases. If you appear to go from no result at all to no relevant result and back, it means that you don't find anything you most probably are looking for a function that is not in your environment. Now this is the base of code test driven reuse, which is something that has been studied a little bit in the Java community. And the idea is to use test driven reuse, it's just that you start like test driven development, you start your test, maybe you try to write it in a more independent way than you would do if you already know what is the implementation that you are doing. And then you try to search if you find a function that already works. If any code, if any function pass your test, then you have three options. So if you don't find any function, it's test driven development, you have to develop it, so fine. Otherwise, you may just import it, that means you get the dependency, and all you may fork it, that is, you get exactly the same code, test, you accept the license, and copy it to your project, or you may just have a look at it to see how many details you didn't think already. Another trick is that you may just use the test driven code search, which is a tool by itself, unit test validation. If you wrote a test, you think it's a good test, then you make a search with it, and you find a couple of totally unrelated functions, it means that your test is too weak. It finds false hit. So limitation of future work. The main point right now is performance. Then you may do a lot of things like extending the search space and making more tools, but then you get even more work to do, and so performance, performance, performance, and validation, et cetera. It would be very nice if this was not done on your machine, but on the web. So what we are trying to do is to make kind of a search engine on the web. If you want to know when things are starting to roll, write an email to the email here, and we are looking for people who are willing to test. Conclusions. If you start using it, you will recognize much better what are good tests and what are good codes, and you will tend, at least this is what we notice, we tend to write your code so that all the implementation details are filed as way as possible, as simple or as intuitive as possible. Thank you for your attention. Do you have any questions? Do you filter somehow already on, for example, the number of arguments that can be passed and similar things, because if a function doesn't take any argument and you need something that takes one, then there's not a valid candidate, for example. I didn't understand. When you look for candidates of things that solve your problem, do you filter already on the things that you've done? Right now, no. This is one of the reasons a web search, I mean, a curated index of objects would be nice, but it's very difficult to do it on your machine. I mean, with Python, you may tell how many arguments you are in a function, but not much more because you do type, you might not actually want to be too strict. So the idea behind having a web search engine is that you have a curated index of what kind of function may fit a particular test or not. Is there anything for timing out functions that could take a long time? This is already taken into account. Every test has a timeout of one second. So a lot of the stuff that are tried are timeouts. I use prompt, row input, et cetera, doesn't give any problem. The real problem is when you call C extensions and they just crashed in Teppadur. I have a long list of, I have a long blacklist for this kind of things. Another question? Yeah? No? Ah, see. So how do you deal with multi-argument functions where I don't know what the order of the arguments is going to be and what's the time complexity of that? Okay. So this is what he, in the first row, tried to do at the beginning. There is an automatic permutation of arguments. I refuse that because right now to get correct, it's more important than to a large search space. But as I was writing this talk, I noticed that you can easily use the parameter. You can easily parameterize your function just switching arguments. So it can be done right now with, by just passing the parameterized with switching. And the complexity, it's very hard if you have two arguments, it's two, but then it's n factorial. So with four arguments, you are already very, very heavy. It's very easy to go into the hundreds of thousands of tests. Now it was really a small environment and for educational purposes, etc. Thank you very much for everything. See you next time.
 Okay, welcome to this talk.  The speaker is Alessandro Amici, a good developer, I think.  He will explain some stuff about pie tests.  So welcome.  So this talk is about test-driven code search.  And this is a rather new technique, not so new because someone already tried it a few  years ago with Java.  And the...  But it's the first time that I see it applied to Python.  The idea is pretty simple.  What we want to...  What we produced, what we did was a very basic search engine.  It's a pie test nodev, it's a pie test plugin that enables you searching for code inside  your machine on packages that you have installed on your local machine.  The special thing about this, the test-driven search, is that you use a test as part of  the search query.  So you may use some...  Also metadata and try to refine your search, but the core...  At the core, what you are looking for is what you describe within a test.  We call it a specification test.  There is something that tries to specify a behavior or a feature without going too much  into the details on how it is implemented.  Once you run your search engine, you will get some search results.  So this is a list of functions or classes or whatever object, actually, that pass the  specification test.  The documentation that the main...  The core of the tool is the pie test nodev plugin, and there you have the main documentation,  but there are a couple of other tools that I will show during the talk.  Now, since this is something new, at the beginning I organized this talk to be somehow theoretical,  but then I completely rewrote it yesterday because I think really good examples make  people understand much faster.  How it works.  Do people here know pie test and pie test features?  Who does...  Okay.  Now basically the base implementation detail is that the plugin provides a special fixture  that's called candidate, and you need to use this feature when you write a test that you  want to use to search for code.  What will happen is that the fixture will effectively parameterize your test by passing  it all the objects that it will manage to find in your environment.  So if you install 10 packages in the virtual environment together with pie test, it will  collect all the objects, all the live objects in your standard library and all the packages  that you installed.  Then obviously since this will be a parameterized test, the test will be run every thousandth  of time most probably, and once for every object and the object will be passed, a reference  to the object will be passed into the candidate variable.  So you basically will use this candidate as if it was the function that you are looking  for, and then the search engine will just tell you which functions, classes, or objects  in general actually appear to behave exactly as you intended.  So let's do our first search.  You want to search for some kind of a function that has a feature, for example, let's search  for a function that even the name of it executable returns the path to it.  This is not just a nice example, this is actually the first real case that we have.  We had exactly this need, and we started searching for it on the web, we didn't like the results,  and we said, okay, no, this is the perfect test case because it's something easy, it's  easy to write a test for it, and maybe there is something somewhere in my environment already  that does it.  So you could just write something like a sub process call to which and then pass the result,  et cetera, that would be hacky and it would not work on Windows, so it's not the best.  So what is the specification test?  I write a standard test function for PyTest.  I use the candidate fixture, then I just as to have the test more readable, I basically  rename the candidate to which, which is more or less the idea that I'm looking something  that works like the which command in the standard library.  So then I assert the behavior that I expect.  If I ask SH, I want to the function I'm looking for should return bin SH, and if I pass it  the string m, it should return usrbinm.  These two are two very common UNIX commands, and they are the one among the most tables  because a lot of commands can't be in usrbin or in bin or spin or, but these two are the  more common.  So once I have written this test, I write it to a file and then I just run it as usual  with PyTest as usual, just I need to add a candidate from all.  This means that the candidate function, the candidate fixture will be parameterized by  everything I find in my environment.  So this starts a standard test session, and I get usually something like 5,000, 6,000  objects.  This depends very much on how many packages you have installed.  This is not many, it's easy to go into the 30,000 or 50,000.  And then it just runs for a while, we will see in a minute.  And since the test is expected to fail, the PyTest will print a small x when the test fails.  You are throwing random functions to the test, so you expect it to fail most of the time.  Then you have capital X, which means that the test passed in, it was not expected, but  it passed.  At the end of the run, you have many, many x.  What you expect to do is to have a result, and in this case we found three functions,  three objects that passed the test.  And this is the report.  So for my test which file, we found a case in which the test which function passed, and  this is the executable.  Now I have the test function as well, and let's see how it works, how much time it takes.  So right now, I'm not using PyTest, pure PyTest, I'm using kind of a boxed run on PyTest inside  the Docker container, because when you throw random arguments to random functions, anything  can happen.  So if you try to do it on your machine, you will find backup files with crazy names or  probably connections to run costs or whatever.  So you prefer to do it in Docker, and at the end of the run, you throw away your Docker  environment.  So what happens is that right now it's collecting all the objects.  Now I have a little bit less objects than when I did the test, because I blacklist objects  all the time, because they might crush your environment or, I don't know, open up a browser,  et cetera.  And this is what happens.  Now all the tests, that test is running with all the functions.  We see small x, and it means that we didn't find n much, but here we have one x.  So this is one of the, we find at least one function that actually worked.  This takes approximately 60 seconds, and everything goes okay.  And now we should also have some garbage on the screen, because since you are using functions  and classes in unexpected ways, you're always throwing random stuff to it.  Exactly.  You end up discovering a lot of bugs in the package that you have, because most of the  printouts is exception in the Dell method that are ignored but printed to the standard  error.  Well, I finished.  So this is the, now what happens once you get the result?  You say, okay, I have tested past that very easy, very basic test.  And what do I do?  Well, since I have a manageable number of results, I can just have a look at them and  decide if this is really what I want.  This is the, sorry, the distutils.pwn.findexecutable.  The name looks like what we are interested in.  And this is inside the standard library.  So it's very useful.  Maybe I don't need to write any code for my findexecutable, for my which function, because  I may just use this one.  You see, that's more or less what I thought.  It gets path, then it splits somehow in OS independent way, then it makes, it does some  win32 checks that I even didn't think I needed because I don't use windows usually, but yes,  might be useful.  And then it just tries to see if the file exists inside the path.  It's not really the best.  I mean, this file, it doesn't check if the file is unexecutable.  So not really perfect, but at least I have a template if I want to improve on that.  Then I have pxpect.udeal.switch.  I don't care too much about that because I already have a function in the standard library,  so I don't need to add a dependency to my project if I want to use that.  But then they have shd.witch.  So, this is even more standard in the standard library, and this is the code, and if you  have to look at the code, it's much, much more complex than it has any real access check.  That means it checks that you can read it, that you can read the file, and you can execute  it.  And it has several details that I would not have thought I would not, it would have taken  me one year of production to get right.  So very nice.  Unfortunately, if you go into documentation, you'll learn that this is only a Python 3,  actually Python 3.3.  So if your code, if your use case needed to work in Python 2.3, you might get a very  nice find executable, it's still in the standard library, it's not as nice, but okay.  Or maybe you can just take it as template and get it better.  Or if you are Python 3 only, you have the luxury to use which.  Which is great.  Well, how many of you already know which function or how to solve this problem?  Okay, a few.  Right, I mean it's in the standard library, but I mean I didn't know it, and it was faster  this way than to look for it.  Okay, let's go back.  Now this is a very simple example, but it also shows how things work.  Now one of the point is that in this case, input and output of the function were really  easy.  When you have something that were the reasonable implementation, it's really easy.  It's easy to write a test, but as soon as you look for more complex stuff, writing a  test that is somehow implementation agnostic, that doesn't make too many assumptions of  the implementation, it's complicated.  It's more complicated.  But actually Python is really great to write stuff that it's not too tight to the implementation,  to the details of the implementation.  Because it has dynamic measure, for example using that typing, you are not forced to guess  the right data type.  The in operator is extremely powerful, and a lot of classes even work nicely with the  in operator.  That is, instead of looking if the result of your function is a list, and the first element  of a list is what you were looking for, you just use the in operator to see if somewhere  inside your function, inside your result, there is what you expected it to be.  And then there are, you may write specific helper, in particular, we wrote the not-aspects  that helps you that leverages the inspect module to go even deeper into the search of  where if your result actually contains what you expected it to contain, even if in crazy  ways.  So let's see how you would write a specification test in a way that is, that tries to be more  independent on from the implementation.  Here I want to parse in fc3986.  This is also a real test, a real case.  So I use the candidate fixture, I just rename it for, so I read it nicer, I use the test  URI, and then I get all the functions that I will get will be passed this URI and I expect  it to return some kind of tokens.  And then here I will check if the schema and the path that I put in my URI are correctly  parsed.  Now, since most, there are a lot of false positives that are just strings, I mean just  functions that return the same string as the inputs, I check that the return of my functions  is not string.  I don't care, I really want the string to be divided into tokens, so I don't want one  string.  I want some kind of list of strings.  So let's see how it goes.  This is the naive implementation in the sense that I didn't use any special trick except  Python standard in operator, that is overloading, etc.  Now this is going to run, come on, and usually I have different common lines that can be  passed and those mostly need to restrict the search space.  If you already know that some packages are not useful, you want to restrict the search  space so you get faster.  But this one candidates from all, it's the more powerful, it's just search for everything  and anything in your environment.  So this is where...  Obviously, I tested just before the...  I don't know.  Let's see, I have a second run.  Now let's see, since it takes a little bit of time to run, I also tried the second example,  that is the same parsing function that is instead tested, the test is written using  some advanced functionalities.  The splact container in other specs generic, it gets an object and it makes it's a proxy  object that when you use the infunction, it tries really hard to see if the item that  you're looking for is somewhere in the object.  So for example, it looks into the attributes, into the properties, even if it's an iterable,  it looks inside every item inside the iterable.  So it's extremely thorough.  Let's see if we manage to not kill the queue.  So apparently they're both running.  So on this screen, I have the naive test, the one that crashed before.  It was some kind of race condition because it's going okay now.  And now let's see what the results are.  Okay.  I got several results.  Now the first three results in collections doesn't look very good because keys view,  chain map, user string really look like false positives.  They're not trying to do anything with RFC or URL parsing but it's just their packaging  somehow, the string that you are giving them.  But then you have this RFC3986 API, you reference that looks very nice.  But also the URL parts, you are parsed.  That means you hit function that are able to do this both in a package and also in the  standard library.  Now what is interesting is that in both cases, both URL, the URL parts inside the RFC3988  package and also the one in the standard library, they don't return lists.  They return classes.  So how exactly this worked with the class?  The point is that a lot of people are quite smart and they give you some way to get to  access stuff or to test stuff in an implementation independent way.  That is the two implementation that they used actually provide a underscore, underscore,  or contains underscore, underscore method that tests exactly if they managed to find  pieces that test exactly like a string, like a, sorry, a table or list.  So it's not a simple type but it's a class that behaves like a type.  Very nice.  So you can happily use this one for most of your need but if you need more features, you  may explore the code and you see this special package as more features.  For example, it's able to recognize the username which the standard library function doesn't.  Now there's something even more interesting.  The other test that one that uses a dedicated proxy object to do the containment test has  found one more object that matches.  And this is a class in the PIP product that actually does the right thing but doesn't  provide the nice containment helper functionality.  So we managed to get it as well because the helper function tried very hard to find if  the postgres and the path, the schema and the path were inside the class.  So this is a way to get to test results in an implementation-independent way but then  I want also to pass arguments in the independent way, in the implementation-independent way.  Still in this case, what helps me is the parameterized marker of Pyvest.  For example, in this case I'm looking for a function that just removes comments from  a stream.  And the main point is how do I represent the stream because this is my text and this is  the readout of my configuration file, for example, and I want to strip these comments  here.  So how do I do?  I use a parameterized argument so that I can say, okay, you have different functions  that will make this text, this comment, into different shapes.  I can pass it as it is.  I can pass it as a list of individual lines.  Or I can pass it as a list of individual lines with numbers.  This is how my application actually was doing this part.  Or I can pass it as a file.  Now in this case, since I have a lot of parameters, I will run not just 5,000 times, but this  will run 20,000 times.  So I prefer to restrict my search by just including any function whose name matches this regular  expression.  So I want something that has to do with comments.  This makes everything much, much faster.  And here it is.  So I find an ignore comments function in PIP, very good, because PIP is something that  I might assume it's a light dependency.  And this tells me that the text to stream that passes is the third one.  So I go back here.  It's 0, 1, 2.  So this is the way, which was exactly the way I preferred.  I could have worked with all the other trees, but this means that I don't even need to change  my application to use that function.  So by the way, extremely fast.  This is the ignore comments.  It's very simple.  And it has also the feature that keeps the line.  If it's empty.  It also returns the line number because it doesn't return all the lines that you pass.  And loop at the other function just below.  This function takes options, which is a special class.  And this class must have the skip requirements regx, otherwise it crashes.  Oh, God.  Even if I needed this, I would never, ever manage to pass the correct parameters to it  because these parameters is extremely tied to the implementation.  So I do tests that are quite loosely coupled with the implementation.  I try to be as implementable as possible, but I only find functions, callables, or classes  that are good code.  They are not, they don't mix implementation details uselessly.  It could just have you skip.  This skip requirements regx could just have been a keyword argument with the same default.  And the function would be as useful.  And I would have been able to search for it or to use it in general.  So when you search, you may get only relevant results, which means your query is just perfect.  Or you have to refine your query.  If you don't get any results at all, which happens quite often, it means that your test  is too strict.  And you probably need to remove test cases, edge cases, or probably just use a lower number  of normal cases.  If you find a lot of results, but they are not relevant, it means that your test is not,  it's too weak.  It's not strict enough, so you need to add more cases, more describe your feature better,  and probably add more corner cases.  If you appear to go from no result at all to no relevant result and back, it means that  you don't find anything you most probably are looking for a function that is not in  your environment.  Now this is the base of code test driven reuse, which is something that has been studied a  little bit in the Java community.  And the idea is to use test driven reuse, it's just that you start like test driven development,  you start your test, maybe you try to write it in a more independent way than you would  do if you already know what is the implementation that you are doing.  And then you try to search if you find a function that already works.  If any code, if any function pass your test, then you have three options.  So if you don't find any function, it's test driven development, you have to develop it,  so fine.  Otherwise, you may just import it, that means you get the dependency, and all you may fork  it, that is, you get exactly the same code, test, you accept the license, and copy it  to your project, or you may just have a look at it to see how many details you didn't think  already.  Another trick is that you may just use the test driven code search, which is a tool by  itself, unit test validation.  If you wrote a test, you think it's a good test, then you make a search with it, and  you find a couple of totally unrelated functions, it means that your test is too weak.  It finds false hit.  So limitation of future work.  The main point right now is performance.  Then you may do a lot of things like extending the search space and making more tools, but  then you get even more work to do, and so performance, performance, performance, and  validation, et cetera.  It would be very nice if this was not done on your machine, but on the web.  So what we are trying to do is to make kind of a search engine on the web.  If you want to know when things are starting to roll, write an email to the email here,  and we are looking for people who are willing to test.  Conclusions.  If you start using it, you will recognize much better what are good tests and what are  good codes, and you will tend, at least this is what we notice, we tend to write your code  so that all the implementation details are filed as way as possible, as simple or as  intuitive as possible.  Thank you for your attention.  Do you have any questions?  Do you filter somehow already on, for example, the number of arguments that can be passed  and similar things, because if a function doesn't take any argument and you need something  that takes one, then there's not a valid candidate, for example.  I didn't understand.  When you look for candidates of things that solve your problem, do you filter already  on the things that you've done?  Right now, no.  This is one of the reasons a web search, I mean, a curated index of objects would be  nice, but it's very difficult to do it on your machine.  I mean, with Python, you may tell how many arguments you are in a function, but not much  more because you do type, you might not actually want to be too strict.  So the idea behind having a web search engine is that you have a curated index of what kind  of function may fit a particular test or not.  Is there anything for timing out functions that could take a long time?  This is already taken into account.  Every test has a timeout of one second.  So a lot of the stuff that are tried are timeouts.  I use prompt, row input, et cetera, doesn't give any problem.  The real problem is when you call C extensions and they just crashed in Teppadur.  I have a long list of, I have a long blacklist for this kind of things.  Another question?  Yeah?  No?  Ah, see.  So how do you deal with multi-argument functions where I don't know what the order of the arguments  is going to be and what's the time complexity of that?  Okay.  So this is what he, in the first row, tried to do at the beginning.  There is an automatic permutation of arguments.  I refuse that because right now to get correct, it's more important than to a large search  space.  But as I was writing this talk, I noticed that you can easily use the parameter.  You can easily parameterize your function just switching arguments.  So it can be done right now with, by just passing the parameterized with switching.  And the complexity, it's very hard if you have two arguments, it's two, but then it's  n factorial.  So with four arguments, you are already very, very heavy.  It's very easy to go into the hundreds of thousands of tests.  Now it was really a small environment and for educational purposes, etc.  Thank you very much for everything.  See you next time.
Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_0_323.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_324_13739.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_13740_13871.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_13872_15431.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_15432_15707.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_15708_18071.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_18072_19703.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_19704_20291.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_20292_20747.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_20748_22355.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_22356_27947.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_27948_28271.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_28272_28967.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_28968_29135.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_29136_29675.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_29676_35171.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_35172_36227.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_36228_39983.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_39984_40391.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_40392_41603.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_41604_46427.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_46428_48539.png Alessandro_Amici_-_Test-driven_code_search_and_reuse_coming_to_Python_with_pytest-nodev_48540_55027.png