Testing microcontroller firmware with Python
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Steffen, Alexander
Testing microcontroller firmware with Python Last year's talk (-unit-tests-for-c-code-in-python) showed you how to use CFFI to write unit tests for C code in Python. This year we will take the concept one step further and create integration tests covering (almost) the whole firmware of a microcontroller, again leveraging the power of CFFI. But instead of running the firmware on the controller, it will be executed on the development machine (that is, a standard x86 architecture), allowing for much faster test execution, without requiring the target hardware. For this to work, all the hardware-dependent parts of the firmware code need to be replaced by Python code simulating the hardware functionality, so that all the firmware above this hardware abstraction layer can be executed unmodified. In addition, this allows to use advanced security testing tools like AddressSanitizer and american fuzzy lop that would not be able to run directly on the microcontroller
 Hello, I'm Alexander and I work as an embedded software developer. So I write firmware for microcontrollers. And today I'm going to show you how you can test such firmware much faster by not running it on the real device but integrating it with Python. And I already gave a talk in a similar direction last year. It was called writing unit tests for C code in Python where I used the CFFI library to just extract single functions or single modules from your C source code, build a Python module out of it so that you could load it into your Python process and then use all of Python's power to write unit tests. I'm building on this idea today. And you might have already seen in another context this hierarchy of tests. The unit tests last year we were at the bottom level where we only look at individual modules, individual functions and try to test them. But of course it's also important to test your code in integration with all the other parts. And so this year we're going to move up a layer into the integration tests and try to make sure that basically all of our firmware really works. And the motivation for that is that the firmware that we write is rather complex. So in the end we might have half a megabyte of compiled code on our microcontroller. For this we've written thousands of test cases. And when we run those against the real device it takes several hours for all those tests to complete. And as a developer that's not really what I want because when I make a change to the software I need to know fast whether this change is good or bad. The quarter of an hour is the upper limit for me. I don't want to wait hours before I can tell that. So what we did in the past was we just selected a subset of those test cases that tried to cover as much as possible. But of course you can't guarantee that it really gets every corner case that you have in your code base. So there might still be errors that slip through. This is what I want to avoid and how this project started. So I'm first going to show you the basic concept now and then afterwards give you a complete demonstration based on some firmware example to show you really the code that does all that. If you look at your typical microcontroller application it might look something like that. You've got a large application code base that's pretty standard C code that you could compile for any architecture. But of course you've also got hardware specific parts and if you've structured your firmware in some way you might have an hardware abstraction layer that really interfaces with the hardware and provides a nice and clean C interface to your application. And this is what we base this approach on because we want to make it look like this. We keep the application code and just replace the abstraction layer beneath it with some Python code. And the approach for this will be similar to what I showed last year with the CFFI library. But first when we are talking in the context of microcontroller firmware that's already written in C you might wonder why do we use Python at all? We could just replace this hardware abstraction layer with a different hardware abstraction layer for another machine that's faster and just use C for that by Python. We are at a Python conference here so I don't need to tell you much about the general advantages that Python has over other languages when you compare it with C code then you can easily see that you need to write less code to achieve the same results. And it's also usually easier to use. For example our microcontrollers have cryptographic functionality built in hardware so we have for example an AS peripheral in there where we can just pass in some data. It does the AS encryption in hardware and returns back the result. So this is something that we have to reimplement in our Python code for this to work. And there are libraries in C where you can do that, there are libraries in Python where you can do that but the Python ones are easier to use, easier to get around with. And in the end Python is also very powerful for this approach. The hardware abstraction layer might contain functions that for this simulation that we are going to build here can work similar and don't need a different implementation so you can just use a single template in Python and let Python generate the code for all those functions that you need. You don't need to specify each and every function in your C code just for the program to compile. And this is now what I'm going to show you. The general approach is that we'll collect all the application C source code, all the implementation of the application and we'll collect all the header files of this hardware abstraction layer. So everything that specifies the interface of the hardware abstraction layer. And when we've got both of those parts, we can pass them on to CFFI. CFFI will use this information to generate a Python loadable module that we can then run from our Python interpreter and then we have our application running inside a Python process on a normal machine, not on our microcontroller. And since the normal machine is much faster than the microcontroller, hopefully also our application will be much faster and our tests can execute faster. So as an example, I can unfortunately not show you our real code. So I looked for a different project and I chose the micro Python project because it's also very complex and a very complex project and has a lot of code. So it gets really an impression of a real-life application of this approach and not some artificial example that I just constructed for this talk. You might have already heard about the micro Python project. If not, a quick explanation is that it's a re-implementation of the Python programming language that can run directly on a microcontroller. It started several years ago, also with an hardware device. Maybe you've seen it in previous talk in this room where you've got a little board with a small controller on it, a lot of hardware peripherals that you can access from your Python code more or less directly. And it has basically full compatibility to the standard C Python 3.5 code. So they don't provide all the features, but most that you want to use. And first, we'll have a look at the structure of the source code. All the source code is open source. You can find it on GitHub. And if you look at the repository, then you'll find a structure that looks like this. So there are some files containing documentation and then a lot of folders. And many of those folders contain the code that is specific to one micro Python port. So micro Python already supports not only a single platform, but multiple platforms. There are, for example, some parts even for Windows for Unix systems. But the initial port was this one here, the ST port for an ST-based microcontroller. And in other folders, for example, the Py folder, there's the generic code that can run in every port. So the Py folder contains the Python interpreter, for example. And for this example, I will choose the minimal port with this similar to the ST port, but very stripped down in functionality. It just contains the bare essentials. It gives you a Python shell that can run code, but it doesn't give you any further hardware access. But for this demonstration, that should be sufficient. If we look at this minimal port, this is all the files that are contained in there. So you see only two C files. The main C file contains the basic application startup code that initializes everything. And you see this UART core file at the end. This is what the implementation of the hardware abstraction layer for this project is. So it contains some functions for input and some functions for output so that we can provide the Python shell. This is what the relevant functions look like from this file. You've got one function that reads a single character of input and does something with that. And you've got another function that can print strings to standard output. So in case of this minimal port, if you really run it on the Py board, then it just uses UART communication for that. So you see some accesses to the UART registers in this code. And if we try to compile this file for our normal machine, then this wouldn't work because there are no such registers where you could write to. So these are the functions that we want to replace with Python code so that we can execute them. All the rest of this code that is contained in the minimal port also that's imported from the Py folder, that should run on our architecture without any problems. So then there's another project that I need to talk about quickly. And that's called PyMake. That's a re-implementation of the make utility. And I want to use that in this demonstration to parse the make files that MicroPython uses for its build process. Because for this approach to work, we need to know which source code files to integrate into our binary. Where do we find the header files? Where do we find the source code files? And of course, I could just hard code that in this example. But if you wanted to use that productively, it makes more sense to keep this information in one place. And the place that already was chosen here is the make file. So I just want to parse the make file and extract the relevant information from there so that I can still keep all the information in this one place and don't have to adapt many places just for this whole process to work. And PyMake gives me such a make file parser in Python. So I'll build on that. When we look at the MicroPython make file, one bit of interesting information in there are the compiler options, for example, for the include directories. So it just builds a list of those here where it specifies some directories where we can find the include files, the header files. And in order to extract that using PyMake, I can tell PyMake just to parse the make file that I have without executing it really. It just parses all the data structures. And afterwards, I can ask PyMake, well, give me the contents of this variable, Inc, where the include directories are contained. And what I get back is not a string, but is an object, the representation you can see here. It's actually not bad to get back an object like this and not the raw contents because if you look at the beginning, then there is contained this value here that contains a reference to another variable. So I don't, I'm not interested in this, in the string value, but I need to have this value resolved to its actual value in order for this process to work. And this is what can be done with the expansion object that the last call here returns. There's a resolve string method on there, and this then returns the final string value that I'm interested in. So I can hide just all this code in a simple function so that I can use that to resolve other things. And now looking at the cleaned up example, we can just call this function, get back the string that were declared in the make file. Everything seems to work. So we store this value in a variable for later use and start with the real process now, collecting the source code. So for collecting the source code, we'll just change into the micropython minimal port directory, so all paths are relative to this directory. And again, look at the make file, there's a variable called source C that lists up all the source code files. And at the beginning, you see two that I've already shown to you, the main file, the UR core file, and then there are some references to other files in the lib directory, again, a directory that's shared by multiple ports. And so we can just extract this list of source codes again using the function that we've already created. And again, you can see here in the last variable again, contain the reference, the reference is resolved to the actual value. Now if we want to create a list of source files, we can use again the function, convert it into a set, then we need another variable from the make file that I haven't shown you so far. It contains a list of all the source code from the pi folder, not as C files, but as object files, so we just extract the name so that it matches to the file system location that we are interested in, and add that to the set. And in the end, there's one source file that we have to remove again, that's this UR core file that I showed you in the beginning, because this isn't really source code of the application, that's the source code of this hardware abstraction layer. We don't need that now, so we remove it here. And then there's one more thing that's special about mycopython here, if you look again at the path that are contained in here, the last one refers to a directory called build. And if you try to find that in the source code, you won't find it at GitHub because it's not contained in any of the commits, it's just a file that's generated during the build process. And contains information that mycopython extracts from its own source code. So we just tell the mycopython build environment, hey, please build this file for us, so we can compile it also into our extension module. So then we have a list of all the files, so we can just open all those files, collect the source code into one large string that we later pass on to CFFI. And before we do that, we make one more modification, the last line here just renames the existing main function to MP main. Of course, the mycopython port assumes that it is the only application that is running on this machine, so it has its own main function. When we import it into the Python interpreter, there is already a main function, so we rename it just to avoid any name conflicts here. So with this, step one is complete, we have collected all the application source code. Now step two is to collect all the hardware abstraction layer header files. And for this minimal port, that's rather easy, there's only one header file that we need to include, there were only those two functions I showed you in the beginning, the header file defines some more functions that are not really used by the code, so we only need this header file. But unfortunately, we cannot parse it directly to CFFI because CFFI's parser for this information doesn't understand everything that the C code or the C standard allows, it just understands a subset. For example, it has no idea of preprocessor directors, it doesn't understand some attribute annotations on the source code, so we need to clean up the source code in order to make CFFI understand it. This is something that I've already shown last year in the example with the unit tests, and I'm going to use similar code this year. What we're going to do is this year, we add some definitions for the C preprocessor to the contents of the header files, for example, for this attribute definition, it just tells the C preprocessor to discard all this information, CFFI doesn't need to know about it, and if it's not there, it can't get confused by it. And afterwards, we run the C preprocessor over the source code so that it takes care of everything that's included of all if-defs and other things, and then CFFI can understand the results. So this preprocess function that's used here looks like this, it just calls the GCC's preprocessor and uses its output for the further steps. And you can also see here a reference to this include options variable from the beginning, where we specify all the include directories, of course, the preprocessor needs to know about that, and afterwards everything is contained in the string we get here. So this is now an extract from the string that we've produced so far. There are three function prototypes, one of them I showed you the implementation for, for this, for the one in the middle, that can output a string of arbitrary length, and we can pass this code to CFFI, but what we want to do is we want to have Python implementations for those functions. So we want to tell CFFI, hey, these are functions that C code can call, but that we want to implement in Python. And in order for that, we need to prefix those prototypes with external Python plus C, then CFFI knows, okay, I need to generate some glue code in order to make that work. And again, the simple solution that you might come up with in the beginning might be to just use search and replace and add the string there, but depending on how complex your code gets, it's better again to use a real parter that understands the C code and can just make this modification. This is based on the implementation that I showed last year. It uses the PyC parser that's also used by CFFI internally, and it parses all your C code into a Python data structure, then you can modify that data structure and write it out again. And in this case, we do that in two functions here. We have one function that's called for every declaration that we find in the C source code. So that's the first one. And whenever we hit a function declaration, and it's for a function that we haven't seen already, then we will prefix it with external Python plus C and return the complete result. Otherwise, we'll just ignore it. And the second function takes care of all the function definitions that we might hit. So there might be inline functions that are specified in the header files. Of course, we don't want to create a Python implementation for something that's already there, so we just remove them from the output as well. So we can simply run that on the header content that we've collected so far, get back a new string, and if we look at that string, then we find the same functions as before, but now prefixed with external Python plus C. So CFFI should be happy with that. But there's one more modification that we need to make, and this is this. We had this MP main function already renamed in the C source code. Since we want to call it later from the Python code, we need to tell CFFI that this function exists and that it should provide some way for Python code to call this function. So in this case, it's the same function prototype as before, but there's no external Python plus C prefix. So CFFI will assume that it's an existing C function that we want to call from Python and not something new. And with that, step two is complete. We have collected all the header contents, and now can move on to CFFI. And the CFFI source code is this. It's only four lines. So we first create the CFFI object to build our module. We pass in the header content that we collected before, and CFFI will generate the Python interface out of this header content information. And we pass in all the source code that we collected, and CFFI will pass that on to a compiler to build our extension module that in this case will be called MPCM. Then we pass in the include directories that we had collected in the beginning, and afterwards we tell CFFI to compile all this into a loadable module. And then next steps are completed, and we have a loadable module. So now we can run it. And to run it, we simply import that module. And then we need to define the functions that we wanted to replace with Python code. And CFFI provides a decorator for that. It will just match on the function name. So if we define a function that has the same name as one of those external Python plus C functions, CFFI will know to call this implementation whenever the C code calls the function of this name. This is the implementation that reads a single character from standard input. And this then is the implementation in order to write out the contents of a string. And with that, our implementation is complete. We have everything we need. So I'm going to try to show you now that this really works. I've prepared a small script that contains basically this code. I can run it. And then I'm dropped into a micro Python shell. And I can execute micro Python code in here. I have the usual features of tab completion that micro Python provides. I can call some of those functions. Can look at the objects. Everything seems to work as it should. And in order to demonstrate you that this really uses the functions that we've defined before, I can just modify that code and tell it to print everything twice. And then you can see, okay, every output that we get is there twice. Everything that I type is printed twice and it really executes our Python implementation of those C-level functions. Okay. Then I want to talk about some of the challenges that you might face and that we faced when we invented this approach for our source code. First of all, your code should follow a certain structure in order for this to work easily. If you've just got a single file that contains everything, it's hard to separate the hardware dependent parts from the general source code. So what you really want to have is a clear distinction between the hardware abstraction layer and the application code. Then you can just match on the folders, for example, collect the one part from the one folder and the other part from the other folder. This is what we do in our example or have some other mechanisms like the make files that I showed you before. Then there's the problem of namespaces. It's perfectly valid C code to have two files that contain functions, static functions with the same name. But since this example collects all the source code into one large string, everything ends up in the same namespace. And this won't really work. So you need something like that where you prefix every function, for example, with the name of the module so that you end up with a unique name. And another problem is platform dependent code. I've prepared a small example that looks innocent but contains multiple problems when you try to run it on different architectures. So what we do here is we have defined a structure. We fill in some values into this structure. Afterwards, calculate a checksum over that structure. And of course, the checksum should always be the same no matter on what platform this code runs if the data in the structure is the same. The problems that you have here, I'll show you the corrected version already. As first, the data types in the structure, if you just use shorts or ints, there's no specification that defines what size you have here. So you should use types that really specify that. Then you might get problems with padding that the compiler inserts into your structure. So we tell it to avoid this padding with the attribute packed. And last but not least, you need to consider the end-ness of your data. So the byte order of your data if you've got multibyte values. So in the second example, I use some standard functions just to convert the end-ness of those values always to a network byte order, which is big byte, big-end-byte order. And so the structure always should contain the same values here. And the checksum should really be identical. Another problem you might get with code that relies on interrupts because that's not really supported on this platform. You might get something like this if you use threads to really achieve some parallel events, but we didn't have the use for that now, so I haven't tried that. And last but not least, let me talk about the external interface for your code. If we look again at this picture, what's beneath your hardware abstraction layer in your usual application is the actual hardware. And when we take away the abstraction layer, we also take away the hardware. So you need to replace that with something else. One solution would be to use just Python code running against your application, or what we use in our environment is just a network interface that can be used by our existing test cases. So they deliver their input there and get their output back. And so the test case doesn't even need to know whether it talks to the real device or our simulation device. Okay. Now you've done all of that. You'll also get some benefits out of it. And the first benefit and why we did all that was the fast execution. So I've collected all the test cases that I can run against the simulation. And they were executed in roughly five minutes. And if I run the same set of test cases against the real device, it takes one and a half hours. So that was already a huge speed up. In fact, these are the numbers from the first prototype that could execute everything. We didn't invest any more effort in optimizing that any further because it was already fast enough for everything we wanted. Another benefit that you get out of this is dynamic program analysis. You might know about static analysis tools. The warnings that the compiler gives you or that special linters give you. But there are also dynamic program analysis tools that look at your code or that don't look at your source code, but that look at your binary code while it's being run and can give you more information. One tool that we've integrated easily is the address sanitizer. That's just some extra compile options that you include into your calls. And then the compiler will add extra code that checks for invalid memory accesses out of bound accesses. And if it detects something like that, it will just abort at this point. And a second tool that we use is a fuzzer. It's called American Fuzzy Lop that tries to be a bit more intelligent than other fuzzers by trying to find new code path automatically. And you can use that with Python code as well. In our case, we just use a wrapper provided by AFL to compile the extension module. It's called AFL GCC. It calls internally to GCC, but in a way that AFL support is integrated. So this is all that you need in your code for the AFL support to be present. And then there's another nice tool called Python AFL that's actually intended to run Python code with this fuzzer, not the extension code, but Python code. But it also supports this use case. And then there's a small script that in this case reads fuzzer input from standard input and runs it against the application in a loop. We did this with our code for some, I don't know, 7 billion executions that fortunately or unfortunately didn't find any problems. But it works not with the highest speed, but you can use it. And the last benefit that you gained from this approach is a certain kind of hardware independence. You can do your development without having access to the real hardware. So maybe in the beginning of your project, when the real hardware isn't really available right now, or even later on when the real hardware is just too expensive or you just have a few of them, with this approach, you can easily scale and do your tests in parallel on many devices because you just need a standard PC. You don't need any complex setup for your hardware. And with that, my talk ends. And thank you for your attention. Thank you. There we go. So I think we've got time for maybe one very quick question if somebody would like to ask something. Thanks. In general, thanks for the talk, first time. And in general, you have to simulate your outside world with embedded systems. So there are some inputs you're waiting on in your C code, and maybe you check something or you have a control loop or something like this, and you have to simulate this. So you did simulate this in Python, the outside world, or how did you manage this? Yeah, though in our case, the outside world is really just a communication channel. We get some input there. We have to process that and generate the correct output. Of course, you could do something like the, I said in the interrupt example, that you use some threads that every five minutes, you have to simulate the value of some sensor or do whatever you want there, but this wasn't necessary for our use case. Okay. Thank you. Thanks very much.
 Hello, I'm Alexander and I work as an embedded software developer.  So I write firmware for microcontrollers.  And today I'm going to show you how you can test such firmware much faster by not running  it on the real device but integrating it with Python.  And I already gave a talk in a similar direction last year.  It was called writing unit tests for C code in Python where I used the CFFI library to  just extract single functions or single modules from your C source code, build a Python module  out of it so that you could load it into your Python process and then use all of Python's  power to write unit tests.  I'm building on this idea today.  And you might have already seen in another context this hierarchy of tests.  The unit tests last year we were at the bottom level where we only look at individual modules,  individual functions and try to test them.  But of course it's also important to test your code in integration with all the other  parts.  And so this year we're going to move up a layer into the integration tests and try to  make sure that basically all of our firmware really works.  And the motivation for that is that the firmware that we write is rather complex.  So in the end we might have half a megabyte of compiled code on our microcontroller.  For this we've written thousands of test cases.  And when we run those against the real device it takes several hours for all those tests  to complete.  And as a developer that's not really what I want because when I make a change to the  software I need to know fast whether this change is good or bad.  The quarter of an hour is the upper limit for me.  I don't want to wait hours before I can tell that.  So what we did in the past was we just selected a subset of those test cases that tried to  cover as much as possible.  But of course you can't guarantee that it really gets every corner case that you have  in your code base.  So there might still be errors that slip through.  This is what I want to avoid and how this project started.  So I'm first going to show you the basic concept now and then afterwards give you a complete  demonstration based on some firmware example to show you really the code that does all  that.  If you look at your typical microcontroller application it might look something like that.  You've got a large application code base that's pretty standard C code that you could compile  for any architecture.  But of course you've also got hardware specific parts and if you've structured your firmware  in some way you might have an hardware abstraction layer that really interfaces with the hardware  and provides a nice and clean C interface to your application.  And this is what we base this approach on because we want to make it look like this.  We keep the application code and just replace the abstraction layer beneath it with some  Python code.  And the approach for this will be similar to what I showed last year with the CFFI library.  But first when we are talking in the context of microcontroller firmware that's already  written in C you might wonder why do we use Python at all?  We could just replace this hardware abstraction layer with a different hardware abstraction  layer for another machine that's faster and just use C for that by Python.  We are at a Python conference here so I don't need to tell you much about the general advantages  that Python has over other languages when you compare it with C code then you can easily  see that you need to write less code to achieve the same results.  And it's also usually easier to use.  For example our microcontrollers have cryptographic functionality built in hardware so we have  for example an AS peripheral in there where we can just pass in some data.  It does the AS encryption in hardware and returns back the result.  So this is something that we have to reimplement in our Python code for this to work.  And there are libraries in C where you can do that, there are libraries in Python where  you can do that but the Python ones are easier to use, easier to get around with.  And in the end Python is also very powerful for this approach.  The hardware abstraction layer might contain functions that for this simulation that we  are going to build here can work similar and don't need a different implementation so you  can just use a single template in Python and let Python generate the code for all those  functions that you need.  You don't need to specify each and every function in your C code just for the program to compile.  And this is now what I'm going to show you.  The general approach is that we'll collect all the application C source code, all the  implementation of the application and we'll collect all the header files of this hardware  abstraction layer.  So everything that specifies the interface of the hardware abstraction layer.  And when we've got both of those parts, we can pass them on to CFFI.  CFFI will use this information to generate a Python loadable module that we can then  run from our Python interpreter and then we have our application running inside a Python  process on a normal machine, not on our microcontroller.  And since the normal machine is much faster than the microcontroller, hopefully also our  application will be much faster and our tests can execute faster.  So as an example, I can unfortunately not show you our real code.  So I looked for a different project and I chose the micro Python project because it's  also very complex and a very complex project and has a lot of code.  So it gets really an impression of a real-life application of this approach and not some  artificial example that I just constructed for this talk.  You might have already heard about the micro Python project.  If not, a quick explanation is that it's a re-implementation of the Python programming  language that can run directly on a microcontroller.  It started several years ago, also with an hardware device.  Maybe you've seen it in previous talk in this room where you've got a little board with  a small controller on it, a lot of hardware peripherals that you can access from your  Python code more or less directly.  And it has basically full compatibility to the standard C Python 3.5 code.  So they don't provide all the features, but most that you want to use.  And first, we'll have a look at the structure of the source code.  All the source code is open source.  You can find it on GitHub.  And if you look at the repository, then you'll find a structure that looks like this.  So there are some files containing documentation and then a lot of folders.  And many of those folders contain the code that is specific to one micro Python port.  So micro Python already supports not only a single platform, but multiple platforms.  There are, for example, some parts even for Windows for Unix systems.  But the initial port was this one here, the ST port for an ST-based microcontroller.  And in other folders, for example, the Py folder, there's the generic code that can  run in every port.  So the Py folder contains the Python interpreter, for example.  And for this example, I will choose the minimal port with this similar to the ST port, but  very stripped down in functionality.  It just contains the bare essentials.  It gives you a Python shell that can run code, but it doesn't give you any further hardware  access.  But for this demonstration, that should be sufficient.  If we look at this minimal port, this is all the files that are contained in there.  So you see only two C files.  The main C file contains the basic application startup code that initializes everything.  And you see this UART core file at the end.  This is what the implementation of the hardware abstraction layer for this project is.  So it contains some functions for input and some functions for output so that we can provide  the Python shell.  This is what the relevant functions look like from this file.  You've got one function that reads a single character of input and does something with  that.  And you've got another function that can print strings to standard output.  So in case of this minimal port, if you really run it on the Py board, then it just uses  UART communication for that.  So you see some accesses to the UART registers in this code.  And if we try to compile this file for our normal machine, then this wouldn't work because  there are no such registers where you could write to.  So these are the functions that we want to replace with Python code so that we can execute  them.  All the rest of this code that is contained in the minimal port also that's imported  from the Py folder, that should run on our architecture without any problems.  So then there's another project that I need to talk about quickly.  And that's called PyMake.  That's a re-implementation of the make utility.  And I want to use that in this demonstration to parse the make files that MicroPython uses  for its build process.  Because for this approach to work, we need to know which source code files to integrate  into our binary.  Where do we find the header files?  Where do we find the source code files?  And of course, I could just hard code that in this example.  But if you wanted to use that productively, it makes more sense to keep this information  in one place.  And the place that already was chosen here is the make file.  So I just want to parse the make file and extract the relevant information from there  so that I can still keep all the information in this one place and don't have to adapt  many places just for this whole process to work.  And PyMake gives me such a make file parser in Python.  So I'll build on that.  When we look at the MicroPython make file, one bit of interesting information in there  are the compiler options, for example, for the include directories.  So it just builds a list of those here where it specifies some directories where we can  find the include files, the header files.  And in order to extract that using PyMake, I can tell PyMake just to parse the make file  that I have without executing it really.  It just parses all the data structures.  And afterwards, I can ask PyMake, well, give me the contents of this variable, Inc, where  the include directories are contained.  And what I get back is not a string, but is an object, the representation you can see  here.  It's actually not bad to get back an object like this and not the raw contents because  if you look at the beginning, then there is contained this value here that contains a  reference to another variable.  So I don't, I'm not interested in this, in the string value, but I need to have this  value resolved to its actual value in order for this process to work.  And this is what can be done with the expansion object that the last call here returns.  There's a resolve string method on there, and this then returns the final string value  that I'm interested in.  So I can hide just all this code in a simple function so that I can use that to resolve  other things.  And now looking at the cleaned up example, we can just call this function, get back the  string that were declared in the make file.  Everything seems to work.  So we store this value in a variable for later use and start with the real process now, collecting  the source code.  So for collecting the source code, we'll just change into the micropython minimal port directory,  so all paths are relative to this directory.  And again, look at the make file, there's a variable called source C that lists up all  the source code files.  And at the beginning, you see two that I've already shown to you, the main file, the UR  core file, and then there are some references to other files in the lib directory, again,  a directory that's shared by multiple ports.  And so we can just extract this list of source codes again using the function that we've  already created.  And again, you can see here in the last variable again, contain the reference, the reference  is resolved to the actual value.  Now if we want to create a list of source files, we can use again the function, convert  it into a set, then we need another variable from the make file that I haven't shown you  so far.  It contains a list of all the source code from the pi folder, not as C files, but as  object files, so we just extract the name so that it matches to the file system location  that we are interested in, and add that to the set.  And in the end, there's one source file that we have to remove again, that's this UR core  file that I showed you in the beginning, because this isn't really source code of the application,  that's the source code of this hardware abstraction layer.  We don't need that now, so we remove it here.  And then there's one more thing that's special about mycopython here, if you look again at  the path that are contained in here, the last one refers to a directory called build.  And if you try to find that in the source code, you won't find it at GitHub because it's not  contained in any of the commits, it's just a file that's generated during the build process.  And contains information that mycopython extracts from its own source code.  So we just tell the mycopython build environment, hey, please build this file for us, so we  can compile it also into our extension module.  So then we have a list of all the files, so we can just open all those files, collect  the source code into one large string that we later pass on to CFFI.  And before we do that, we make one more modification, the last line here just renames the existing  main function to MP main.  Of course, the mycopython port assumes that it is the only application that is running  on this machine, so it has its own main function.  When we import it into the Python interpreter, there is already a main function, so we rename  it just to avoid any name conflicts here.  So with this, step one is complete, we have collected all the application source code.  Now step two is to collect all the hardware abstraction layer header files.  And for this minimal port, that's rather easy, there's only one header file that we need  to include, there were only those two functions I showed you in the beginning, the header  file defines some more functions that are not really used by the code, so we only need  this header file.  But unfortunately, we cannot parse it directly to CFFI because CFFI's parser for this information  doesn't understand everything that the C code or the C standard allows, it just understands  a subset.  For example, it has no idea of preprocessor directors, it doesn't understand some attribute  annotations on the source code, so we need to clean up the source code in order to make  CFFI understand it.  This is something that I've already shown last year in the example with the unit tests,  and I'm going to use similar code this year.  What we're going to do is this year, we add some definitions for the C preprocessor to  the contents of the header files, for example, for this attribute definition, it just tells  the C preprocessor to discard all this information, CFFI doesn't need to know about it, and if  it's not there, it can't get confused by it.  And afterwards, we run the C preprocessor over the source code so that it takes care of everything  that's included of all if-defs and other things, and then CFFI can understand the results.  So this preprocess function that's used here looks like this, it just calls the GCC's  preprocessor and uses its output for the further steps.  And you can also see here a reference to this include options variable from the beginning,  where we specify all the include directories, of course, the preprocessor needs to know about  that, and afterwards everything is contained in the string we get here.  So this is now an extract from the string that we've produced so far.  There are three function prototypes, one of them I showed you the implementation for,  for this, for the one in the middle, that can output a string of arbitrary length, and  we can pass this code to CFFI, but what we want to do is we want to have Python implementations  for those functions.  So we want to tell CFFI, hey, these are functions that C code can call, but that we want to  implement in Python.  And in order for that, we need to prefix those prototypes with external Python plus C, then  CFFI knows, okay, I need to generate some glue code in order to make that work.  And again, the simple solution that you might come up with in the beginning might be to  just use search and replace and add the string there, but depending on how complex your code  gets, it's better again to use a real parter that understands the C code and can just make  this modification.  This is based on the implementation that I showed last year.  It uses the PyC parser that's also used by CFFI internally, and it parses all your C  code into a Python data structure, then you can modify that data structure and write it  out again.  And in this case, we do that in two functions here.  We have one function that's called for every declaration that we find in the C source code.  So that's the first one.  And whenever we hit a function declaration, and it's for a function that we haven't seen  already, then we will prefix it with external Python plus C and return the complete result.  Otherwise, we'll just ignore it.  And the second function takes care of all the function definitions that we might hit.  So there might be inline functions that are specified in the header files.  Of course, we don't want to create a Python implementation for something that's already  there, so we just remove them from the output as well.  So we can simply run that on the header content that we've collected so far, get back a new  string, and if we look at that string, then we find the same functions as before, but  now prefixed with external Python plus C. So CFFI should be happy with that.  But there's one more modification that we need to make, and this is this.  We had this MP main function already renamed in the C source code.  Since we want to call it later from the Python code, we need to tell CFFI that this function  exists and that it should provide some way for Python code to call this function.  So in this case, it's the same function prototype as before, but there's no external Python plus  C prefix.  So CFFI will assume that it's an existing C function that we want to call from Python  and not something new.  And with that, step two is complete.  We have collected all the header contents, and now can move on to CFFI.  And the CFFI source code is this.  It's only four lines.  So we first create the CFFI object to build our module.  We pass in the header content that we collected before, and CFFI will generate the Python  interface out of this header content information.  And we pass in all the source code that we collected, and CFFI will pass that on to a  compiler to build our extension module that in this case will be called MPCM.  Then we pass in the include directories that we had collected in the beginning, and afterwards  we tell CFFI to compile all this into a loadable module.  And then next steps are completed, and we have a loadable module.  So now we can run it.  And to run it, we simply import that module.  And then we need to define the functions that we wanted to replace with Python code.  And CFFI provides a decorator for that.  It will just match on the function name.  So if we define a function that has the same name as one of those external Python plus  C functions, CFFI will know to call this implementation whenever the C code calls the function of this  name.  This is the implementation that reads a single character from standard input.  And this then is the implementation in order to write out the contents of a string.  And with that, our implementation is complete.  We have everything we need.  So I'm going to try to show you now that this really works.  I've prepared a small script that contains basically this code.  I can run it.  And then I'm dropped into a micro Python shell.  And I can execute micro Python code in here.  I have the usual features of tab completion that micro Python provides.  I can call some of those functions.  Can look at the objects.  Everything seems to work as it should.  And in order to demonstrate you that this really uses the functions that we've defined  before, I can just modify that code and tell it to print everything twice.  And then you can see, okay, every output that we get is there twice.  Everything that I type is printed twice and it really executes our Python implementation  of those C-level functions.  Okay.  Then I want to talk about some of the challenges that you might face and that we faced when  we invented this approach for our source code.  First of all, your code should follow a certain structure in order for this to work easily.  If you've just got a single file that contains everything, it's hard to separate the hardware  dependent parts from the general source code.  So what you really want to have is a clear distinction between the hardware abstraction  layer and the application code.  Then you can just match on the folders, for example, collect the one part from the one  folder and the other part from the other folder.  This is what we do in our example or have some other mechanisms like the make files that  I showed you before.  Then there's the problem of namespaces.  It's perfectly valid C code to have two files that contain functions, static functions with  the same name.  But since this example collects all the source code into one large string, everything ends  up in the same namespace.  And this won't really work.  So you need something like that where you prefix every function, for example, with the name  of the module so that you end up with a unique name.  And another problem is platform dependent code.  I've prepared a small example that looks innocent but contains multiple problems when you try  to run it on different architectures.  So what we do here is we have defined a structure.  We fill in some values into this structure.  Afterwards, calculate a checksum over that structure.  And of course, the checksum should always be the same no matter on what platform this  code runs if the data in the structure is the same.  The problems that you have here, I'll show you the corrected version already.  As first, the data types in the structure, if you just use shorts or ints, there's no  specification that defines what size you have here.  So you should use types that really specify that.  Then you might get problems with padding that the compiler inserts into your structure.  So we tell it to avoid this padding with the attribute packed.  And last but not least, you need to consider the end-ness of your data.  So the byte order of your data if you've got multibyte values.  So in the second example, I use some standard functions just to convert the end-ness of  those values always to a network byte order, which is big byte, big-end-byte order.  And so the structure always should contain the same values here.  And the checksum should really be identical.  Another problem you might get with code that relies on interrupts because that's not really  supported on this platform.  You might get something like this if you use threads to really achieve some parallel events,  but we didn't have the use for that now, so I haven't tried that.  And last but not least, let me talk about the external interface for your code.  If we look again at this picture, what's beneath your hardware abstraction layer in  your usual application is the actual hardware.  And when we take away the abstraction layer, we also take away the hardware.  So you need to replace that with something else.  One solution would be to use just Python code running against your application, or what  we use in our environment is just a network interface that can be used by our existing  test cases.  So they deliver their input there and get their output back.  And so the test case doesn't even need to know whether it talks to the real device or  our simulation device.  Okay.  Now you've done all of that.  You'll also get some benefits out of it.  And the first benefit and why we did all that was the fast execution.  So I've collected all the test cases that I can run against the simulation.  And they were executed in roughly five minutes.  And if I run the same set of test cases against the real device, it takes one and a half hours.  So that was already a huge speed up.  In fact, these are the numbers from the first prototype that could execute everything.  We didn't invest any more effort in optimizing that any further because it was already fast  enough for everything we wanted.  Another benefit that you get out of this is dynamic program analysis.  You might know about static analysis tools.  The warnings that the compiler gives you or that special linters give you.  But there are also dynamic program analysis tools that look at your code or that don't  look at your source code, but that look at your binary code while it's being run and  can give you more information.  One tool that we've integrated easily is the address sanitizer.  That's just some extra compile options that you include into your calls.  And then the compiler will add extra code that checks for invalid memory accesses out of  bound accesses.  And if it detects something like that, it will just abort at this point.  And a second tool that we use is a fuzzer.  It's called American Fuzzy Lop that tries to be a bit more intelligent than other fuzzers  by trying to find new code path automatically.  And you can use that with Python code as well.  In our case, we just use a wrapper provided by AFL to compile the extension module.  It's called AFL GCC.  It calls internally to GCC, but in a way that AFL support is integrated.  So this is all that you need in your code for the AFL support to be present.  And then there's another nice tool called Python AFL that's actually intended to run  Python code with this fuzzer, not the extension code, but Python code.  But it also supports this use case.  And then there's a small script that in this case reads fuzzer input from standard input  and runs it against the application in a loop.  We did this with our code for some, I don't know, 7 billion executions that fortunately  or unfortunately didn't find any problems.  But it works not with the highest speed, but you can use it.  And the last benefit that you gained from this approach is a certain kind of hardware  independence.  You can do your development without having access to the real hardware.  So maybe in the beginning of your project, when the real hardware isn't really available  right now, or even later on when the real hardware is just too expensive or you just  have a few of them, with this approach, you can easily scale and do your tests in parallel  on many devices because you just need a standard PC.  You don't need any complex setup for your hardware.  And with that, my talk ends.  And thank you for your attention.  Thank you.  There we go.  So I think we've got time for maybe one very quick question if somebody would like to ask  something.  Thanks.  In general, thanks for the talk, first time.  And in general, you have to simulate your outside world with embedded systems.  So there are some inputs you're waiting on in your C code, and maybe you check something  or you have a control loop or something like this, and you have to simulate this.  So you did simulate this in Python, the outside world, or how did you manage this?  Yeah, though in our case, the outside world is really just a communication channel.  We get some input there.  We have to process that and generate the correct output.  Of course, you could do something like the, I said in the interrupt example, that you  use some threads that every five minutes, you have to simulate the value of some sensor  or do whatever you want there, but this wasn't necessary for our use case.  Okay.  Thank you.  Thanks very much.
Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_0_1115.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_1116_1727.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_1728_4763.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_4764_7079.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_7080_17699.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_17700_18815.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_18816_20951.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_20952_26399.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_26400_29555.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_29556_30635.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_30636_32651.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_32652_34931.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_34932_35603.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_35604_37355.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_37356_39203.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_39204_40163.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_40164_42383.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_42384_42755.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_42756_43127.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_43128_43523.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_43524_43655.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_43656_43763.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_43764_43931.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_43932_44363.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_44364_44411.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_44412_44459.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_44460_44663.png Alexander_Steffen_-_Testing_microcontroller_firmware_with_Python_-SvmjCWBX10_44664_46309.png