Coala Framework: Discover Static Code Analysis in Python
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Tsamba, Lionel Lonkap
We, as developer, aim to provide code that, almost matches our team code style, looks better and behaves right. Static code analysis (SCA) tools are one of the way to achieves that. But, with multi-programming languages projects and all kinds of code related needs, It's difficult to address all thoses usecases without dealing with a vast majority of SCA tools. Coala is a — language agnostic — static code analysis framework that provides a common command-line interface for linting and fixing all your code. It is written in Python and supports way over 50 languages in addition to language independent routines. So, instead of building new analysis tools from scratch you can now build your own custom logic and let let coala deal with the rest. This talk introduces the audience to the Coala Framework and guides them through how the can use it to build routines to do almost anything you want with your code.
 Let me introduce you to Yulia Nel, who's going to tell us more about the COLA framework. Thank you. So, hello everyone. My name is Lionel. I work as a DevOps engineer at Evo Azure SNCF. Evo Azure SNCF is a digital factory which addresses the SNCF group digital challenges. We are almost 1,500 employees working in three sites, Lille, Nantes and Lille. I work in Lille. We address two main challenges, digital distribution through we SNCF and right Europe internationally. And travel information via the mobile app that is called SNCF. We are delivering to them IT services. Today we are going to talk about static code analysis. We will start first by make and do some overview, a quick definition of static code analysis. So we will follow up with the COLA framework and COLA beers. How is it used to do static code analysis in Python? And we will see what are coming next into the framework to ease the use of Python to do static code analysis. And we will finish with the Q&A. If you have questions, of course. Static code analysis. We can define it as a method to extract facts, detect and also fix defects in source code without executing it. There are many use to do code quality, code reviews, also compliance. When you have to do compliance test for security, you just want to ensure that your code style into the team are well-respected. And of course to detect flows and try to fix them before running it. The COLA framework, we have a lot of tools today to do static code analysis. As you see, there is a bunch of tools. And the more you have tools, the more you have the way to configure those tools. And it's pretty hard. So let's categorize those tools into analyzers as you see up there. And you have also the way we use those tools via editors, tools and services. And also the way we consume the results produced by those tools. Like exporting your results into JSON or an HTML report. As you see, it's pretty complicated to deal with all those tools and the way we use them. So how can we know what if we can have one only tool to manage all this mess? That's why COLA has been built. COLA is not pretty much, it's just an API which is language agnostic. That means you can use COLA in Python, but you can analyze code from any languages. It supports more than 60 languages for now, programming languages. Let's have a closer look. That's the typical constitution of the static analysis tool. We start by using the code as data and you have some model extraction to just get data from the code. You then produce an intermediate representation. We can be... You have AST. You also get data structures. You could also have call graphs and also control flow graphs. If we zoom out into the main goal of COLA, as I said, you have data structures. For analysis, we have rules that also call routines. In COLA, they call it bears, it's the way you implement your algorithm to do static code analysis. After that, the produced result, we could be in two forms, outputs and actions. Output is like detecting the way you detect flows and errors. You could have also as an output a fixed recommendation, do how we can fix our code. You get as a result also compliance reports for flows and code styles. As I said before, you also have actions. There is a lot of actions there. Action could be apply a patch or actually fixing the flows or the defects. Let's see how to pick setup. You could use pip to install COLA bears, which install everything you need to start working with COLA. You could also use Docker, which I think is a recommended way because you don't have to deal with pip packages. You just use the container which is packaged up with all the things you need. You have another way, which is online, where you can just put the repository of your code and start working on it. Let's see how to use COLA. Let's see some code. First, you clone the project and as you see, you have a bears directory and the SRC directory. In this project, you have mainly two languages. We have C and Python. The way you approach the use of COLA is by saying, okay, I have a project with two languages. Let's list the available bears, the available rules that I could use to analyze my code. You run this and you see a bunch of bears available. For the next slides, I will just add this COLA to make the slide more short. You just run COLA and it means running COLA behind Docker. I want to analyze my code written in Python. I will list the bears available for Python. I choose the Pepe bear, for example. I don't know much about it, so I get some documentation on how to use it. I see also we have optional settings on how to set the configuration for the bears. What the bears can do? The bears can detect formatting and can fix formatting. You run your bears, the specifying the bears on your Python files. First, you will get a GIF output. It says how your code is not compliant to the Pepe 8 rows. After that, you have action as I said before. You can either do nothing or open the files, apply the patch, which means make the code compliant, and also ignore the command. You can also pass apply patches at the command line to do the apply patch action directly. You also have another option to specify which action you need to do. As you have noticed, the tools recommended you or suggested you to use dash dash save to save your configuration. That's mainly brought us to the configuration file. As I said, you have a lot of tools and a lot of configuration files to tell which out the tools should analyze your code. In Koala, you also have one file to deal with all bears. When you run with dash dash save, you produce a file called a Koa file. It looks like this. You have sections. It's an init file. You have sections and you have two, at least two mandatory settings, which is specifying the bears. You need to apply on the code and the file. You need to be applying on. You can also enter your settings at the command line as you see in the int. In the configuration file, you have a way to organize your bears. You need to avoid repeating yourself in the configuration. You have inheritance. You just have to prefix the section with a base setting, base section name. As you see, section one and two extend from the configuration set at the base section. You also have append operators, which you can append files from section to section. As you see, at the base, I'm only analyzing Python files. In section one, I want to analyze not only Python files, but also C files. That's an example. I have all the specify org section. At the example section, I want to check also if the space consistent in my code. I want to see how bears work and how to create your own bears. As I said, bears are only rows. But bears are the base construct when you need to write a row. You have to implement the run function. The run function is the one which is executed to run your algorithm. From the bears, you have two classes, local bears and global bears. Actually, your code should extend from those two ones. You also can have user input at the arcs, which is provided by the framework. Local bears run on every file of your project. The run function provides the file name and the file content for you to run your algorithm. Also, user input and settings. And global bears are to run analysis on the whole project. As you see in the run function, you don't have the file, but you can do some internal things to do whatever you want with your files. Let's see an example. I have a low-wall bear. And now I just print some logging things. So I extend from the local bear. That means I want to run it on every file and just output a user input which is provided by the user. And the rule is that you have to yield the result. Why? Because when you mixed bears, you could get the result from any bears in your bears. If you have dependent bears, you can get the result, provide the result, getting from the inner bears you will use, you have used in your bear. I run it and that's the output. It suggests me to enter my user input and I put my name and it suggests me, as you saw before, action to do on this. So bears. In bears, to write a bear, you have three main categories. You have native bears, linters to do linting and you have external bears. What are native bears? Native bears are the bears that extend for local bears and global bears. Simply like this. If you want an example, that's a native bear. As you saw, the yellow bear was the native bear. Then I just implement the run function and yield some results. And the global bears aside also do the analysis on the work project. Linter bears. Linter bears use your own tools and wrap your tools. Just imagine you have a linter, like just lint, and you want to use koala to wrap it. That's why you use a linter, a linter bear. You specify the executable and you have to say if these bears is a global or not. If it's global, it means your tools analyze the whole project. And if not, it means just your tools analyze each file of a file. The particularity there is that the linter bear has to implement the create arguments. That's how you pass the arguments to your executable. Let's see an example. And you have also the way to provide a configuration. If you have a tool that is a highly configurable file, you can just do everything in the generate config. You can produce the addition you need to execute your tools and it will be injected in the create argument. After that. Let's see an example to see how it works. I have filint that I need to wrap. And now create the create argument function. That's where I return a list, a top, a top all. The top all contains every options filint need to analyze my code. That's simply how it works. And the output specify there you specify how you want to analyze it and interpret the result. Xn or Beals is wrapping also your Beals, your tool, but written in any language. It will provide, Koala will provide you some data in JSON. And you have as a rule, you have to produce this result with, we will be analyzed or used by Koala. That's an example. I create a beer. I wrap my tool with using node, I create my script in Node.js. And after that, with my add result, I produce the JSON wanted for my tools to be considered as a beer. Like this. I should use console log to show, to output it as a JSON. Going further, the new thing that will come up is they are creating a way to provide to us the AST of any language. A new API using the aspect of granted programming with aspect and taste. And a new package manager to just specify your requirements in your beers and it will go fetch your NPM package or your PIP package inside your beers. To be sure that when your beers run, you have everything you need to analyze your code. Thank you. We still have time for a few questions. This gentleman here. Oh, yes. With Koala, are you able to do some kind of semantic analysis to produce some properties of your program or so on? The question was, can I, oh, the question was, can I do some semantic analysis? I answered that, yes, you have just a run function so you can do whatever you want. Oh, the intermediate representation in that case was just a Python data structure. The Koala for the local beers Koala produce gives you the file content and the file name. And you can work with this. And if you have the file content, you can do whatever you want with your tools that you semantic analysis on this. That's how it works. And they saw that that was a little complicated and you cannot do much with this simple construct. That's why I said they are working on bringing some real tools to give you more abilities. Yes. Any additional questions? You mentioned it's a language agnostic. Can you use it for Python 2.7 as well? Can you speak alone? Oh, wait. Yes? Can you use it for Python 2.7 as well? Python 2? I think I said that it was available for Python 3, I think. Yes? It supports 60 languages. It supports 60 languages. Yes? Oh, it's hard. It supports 60 languages. So you could analyze the C program in Python with the complete ASD. Yes? Yes. The question was Koala supports 60 languages. And you can use Python to analyze C code. And how it works? That's my question because it depends on the compiler options. Yes. You can use the linter, the linter beer, and provide your compiler option via the beers. You can see it as a wrapper. That's how I think it's very powerful because you can have just one tool and do whatever you want with what exists, what is out there to do, analyze it. Any question? OK, thank you. Thank you very much. EPISODE 1
 Let me introduce you to Yulia Nel, who's going to tell us more about the COLA framework.  Thank you.  So, hello everyone.  My name is Lionel.  I work as a DevOps engineer at Evo Azure SNCF.  Evo Azure SNCF is a digital factory which addresses the SNCF group digital challenges.  We are almost 1,500 employees working in three sites, Lille, Nantes and Lille.  I work in Lille.  We address two main challenges, digital distribution through we SNCF and right Europe internationally.  And travel information via the mobile app that is called SNCF.  We are delivering to them IT services.  Today we are going to talk about static code analysis.  We will start first by make and do some overview, a quick definition of static code analysis.  So we will follow up with the COLA framework and COLA beers.  How is it used to do static code analysis in Python?  And we will see what are coming next into the framework to ease the use of Python to do static code analysis.  And we will finish with the Q&A. If you have questions, of course.  Static code analysis.  We can define it as a method to extract facts, detect and also fix defects in source code without executing it.  There are many use to do code quality, code reviews, also compliance.  When you have to do compliance test for security, you just want to ensure that your code style into the team are well-respected.  And of course to detect flows and try to fix them before running it.  The COLA framework, we have a lot of tools today to do static code analysis.  As you see, there is a bunch of tools.  And the more you have tools, the more you have the way to configure those tools.  And it's pretty hard.  So let's categorize those tools into analyzers as you see up there.  And you have also the way we use those tools via editors, tools and services.  And also the way we consume the results produced by those tools.  Like exporting your results into JSON or an HTML report.  As you see, it's pretty complicated to deal with all those tools and the way we use them.  So how can we know what if we can have one only tool to manage all this mess?  That's why COLA has been built.  COLA is not pretty much, it's just an API which is language agnostic.  That means you can use COLA in Python, but you can analyze code from any languages.  It supports more than 60 languages for now, programming languages.  Let's have a closer look.  That's the typical constitution of the static analysis tool.  We start by using the code as data and you have some model extraction to just get data from the code.  You then produce an intermediate representation.  We can be...  You have AST.  You also get data structures.  You could also have call graphs and also control flow graphs.  If we zoom out into the main goal of COLA, as I said, you have data structures.  For analysis, we have rules that also call routines.  In COLA, they call it bears, it's the way you implement your algorithm to do static code analysis.  After that, the produced result, we could be in two forms, outputs and actions.  Output is like detecting the way you detect flows and errors.  You could have also as an output a fixed recommendation, do how we can fix our code.  You get as a result also compliance reports for flows and code styles.  As I said before, you also have actions.  There is a lot of actions there.  Action could be apply a patch or actually fixing the flows or the defects.  Let's see how to pick setup.  You could use pip to install COLA bears, which install everything you need to start working with COLA.  You could also use Docker, which I think is a recommended way because you don't have to deal with pip packages.  You just use the container which is packaged up with all the things you need.  You have another way, which is online, where you can just put the repository of your code and start working on it.  Let's see how to use COLA.  Let's see some code.  First, you clone the project and as you see, you have a bears directory and the SRC directory.  In this project, you have mainly two languages.  We have C and Python.  The way you approach the use of COLA is by saying, okay, I have a project with two languages.  Let's list the available bears, the available rules that I could use to analyze my code.  You run this and you see a bunch of bears available.  For the next slides, I will just add this COLA to make the slide more short.  You just run COLA and it means running COLA behind Docker.  I want to analyze my code written in Python.  I will list the bears available for Python.  I choose the Pepe bear, for example.  I don't know much about it, so I get some documentation on how to use it.  I see also we have optional settings on how to set the configuration for the bears.  What the bears can do?  The bears can detect formatting and can fix formatting.  You run your bears, the specifying the bears on your Python files.  First, you will get a GIF output.  It says how your code is not compliant to the Pepe 8 rows.  After that, you have action as I said before.  You can either do nothing or open the files, apply the patch, which means make the code compliant, and also ignore the command.  You can also pass apply patches at the command line to do the apply patch action directly.  You also have another option to specify which action you need to do.  As you have noticed, the tools recommended you or suggested you to use dash dash save to save your configuration.  That's mainly brought us to the configuration file.  As I said, you have a lot of tools and a lot of configuration files to tell which out the tools should analyze your code.  In Koala, you also have one file to deal with all bears.  When you run with dash dash save, you produce a file called a Koa file.  It looks like this.  You have sections.  It's an init file.  You have sections and you have two, at least two mandatory settings, which is specifying the bears.  You need to apply on the code and the file.  You need to be applying on.  You can also enter your settings at the command line as you see in the int.  In the configuration file, you have a way to organize your bears.  You need to avoid repeating yourself in the configuration.  You have inheritance.  You just have to prefix the section with a base setting, base section name.  As you see, section one and two extend from the configuration set at the base section.  You also have append operators, which you can append files from section to section.  As you see, at the base, I'm only analyzing Python files.  In section one, I want to analyze not only Python files, but also C files.  That's an example.  I have all the specify org section.  At the example section, I want to check also if the space consistent in my code.  I want to see how bears work and how to create your own bears.  As I said, bears are only rows.  But bears are the base construct when you need to write a row.  You have to implement the run function.  The run function is the one which is executed to run your algorithm.  From the bears, you have two classes, local bears and global bears.  Actually, your code should extend from those two ones.  You also can have user input at the arcs, which is provided by the framework.  Local bears run on every file of your project.  The run function provides the file name and the file content for you to run your algorithm.  Also, user input and settings.  And global bears are to run analysis on the whole project.  As you see in the run function, you don't have the file, but you can do some internal things to do whatever you want with your files.  Let's see an example.  I have a low-wall bear.  And now I just print some logging things.  So I extend from the local bear.  That means I want to run it on every file and just output a user input which is provided by the user.  And the rule is that you have to yield the result.  Why?  Because when you mixed bears, you could get the result from any bears in your bears.  If you have dependent bears, you can get the result, provide the result, getting from the inner bears you will use, you have used in your bear.  I run it and that's the output.  It suggests me to enter my user input and I put my name and it suggests me, as you saw before, action to do on this.  So bears.  In bears, to write a bear, you have three main categories.  You have native bears, linters to do linting and you have external bears.  What are native bears?  Native bears are the bears that extend for local bears and global bears.  Simply like this.  If you want an example, that's a native bear.  As you saw, the yellow bear was the native bear.  Then I just implement the run function and yield some results.  And the global bears aside also do the analysis on the work project.  Linter bears.  Linter bears use your own tools and wrap your tools.  Just imagine you have a linter, like just lint, and you want to use koala to wrap it.  That's why you use a linter, a linter bear.  You specify the executable and you have to say if these bears is a global or not.  If it's global, it means your tools analyze the whole project.  And if not, it means just your tools analyze each file of a file.  The particularity there is that the linter bear has to implement the create arguments.  That's how you pass the arguments to your executable.  Let's see an example.  And you have also the way to provide a configuration.  If you have a tool that is a highly configurable file, you can just do everything in the generate config.  You can produce the addition you need to execute your tools and it will be injected in the create argument.  After that.  Let's see an example to see how it works.  I have filint that I need to wrap.  And now create the create argument function.  That's where I return a list, a top, a top all.  The top all contains every options filint need to analyze my code.  That's simply how it works.  And the output specify there you specify how you want to analyze it and interpret the result.  Xn or Beals is wrapping also your Beals, your tool, but written in any language.  It will provide, Koala will provide you some data in JSON.  And you have as a rule, you have to produce this result with, we will be analyzed or used by Koala.  That's an example.  I create a beer.  I wrap my tool with using node, I create my script in Node.js.  And after that, with my add result, I produce the JSON wanted for my tools to be considered as a beer.  Like this.  I should use console log to show, to output it as a JSON.  Going further, the new thing that will come up is they are creating a way to provide to us the AST of any language.  A new API using the aspect of granted programming with aspect and taste.  And a new package manager to just specify your requirements in your beers and it will go fetch your NPM package or your PIP package inside your beers.  To be sure that when your beers run, you have everything you need to analyze your code.  Thank you.  We still have time for a few questions.  This gentleman here.  Oh, yes.  With Koala, are you able to do some kind of semantic analysis to produce some properties of your program or so on?  The question was, can I, oh, the question was, can I do some semantic analysis?  I answered that, yes, you have just a run function so you can do whatever you want.  Oh, the intermediate representation in that case was just a Python data structure. The Koala for the local beers Koala produce gives you the file content and the file name.  And you can work with this.  And if you have the file content, you can do whatever you want with your tools that you semantic analysis on this.  That's how it works. And they saw that that was a little complicated and you cannot do much with this simple construct.  That's why I said they are working on bringing some real tools to give you more abilities.  Yes.  Any additional questions?  You mentioned it's a language agnostic. Can you use it for Python 2.7 as well?  Can you speak alone?  Oh, wait.  Yes?  Can you use it for Python 2.7 as well?  Python 2?  I think I said that it was available for Python 3, I think.  Yes?  It supports 60 languages.  It supports 60 languages.  Yes?  Oh, it's hard.  It supports 60 languages.  So you could analyze the C program in Python with the complete ASD.  Yes?  Yes.  The question was Koala supports 60 languages.  And you can use Python to analyze C code.  And how it works?  That's my question because it depends on the compiler options.  Yes.  You can use the linter, the linter beer, and provide your compiler option via the beers.  You can see it as a wrapper.  That's how I think it's very powerful because you can have just one tool and do whatever  you want with what exists, what is out there to do, analyze it.  Any question?  OK, thank you.  Thank you very much.  EPISODE 1
python2020_coala_0_143.png python2020_coala_144_731.png python2020_coala_732_1979.png python2020_coala_1980_3131.png python2020_coala_3132_4343.png python2020_coala_4344_4391.png python2020_coala_4392_4895.png python2020_coala_4896_6839.png python2020_coala_6840_7835.png python2020_coala_7836_10139.png python2020_coala_10140_10415.png python2020_coala_10416_11339.png python2020_coala_11340_11471.png python2020_coala_11472_12143.png python2020_coala_12144_13031.png python2020_coala_13032_13451.png python2020_coala_13452_14075.png python2020_coala_14076_14771.png python2020_coala_14772_15719.png python2020_coala_15720_16043.png python2020_coala_16044_19691.png python2020_coala_19692_19991.png python2020_coala_19992_20111.png python2020_coala_20112_21047.png python2020_coala_21048_21215.png python2020_coala_21216_22403.png python2020_coala_22404_23747.png python2020_coala_23748_23831.png python2020_coala_23832_24299.png python2020_coala_24300_25667.png python2020_coala_25668_28007.png python2020_coala_28008_29243.png python2020_coala_29244_29495.png python2020_coala_29496_29759.png python2020_coala_29760_30011.png python2020_coala_30012_30839.png python2020_coala_30840_31871.png python2020_coala_31872_34979.png python2020_coala_34980_39443.png python2020_coala_39444_39580.png