Extending and embedding Ansible with Python
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Rodríguez, Alejandro Guirao
Alejandro Guirao Rodríguez - Extending and embedding Ansible with Python [Ansible] is the _new cool kid in town_ in the configuration management world. It is easy to learn, fast to setup and works great! In the first part of the talk, I will do a super-fast introduction to Ansible for the newcomers. If you are a Pythonista, you can hack and leverage Ansible in many ways. In the second part of the talk, I will describe some options to extend and embed Ansible with Python: - Embedding Ansible with the Python API - Extending Ansible: creating modules, dynamic inventory scripts and plugins Previous experience with Ansible is advised in order to get the most of this talk, but beginners to the tool will also get an overview of the capabilities of this kind of integration.
 Well, good morning and thanks everybody for attending this. As Pina said, my name is Alec Han-Roydow and I'm not a double... Let me introduce you to Fabian Croetz. He's going to give his speech on... So I was saying I'm not a top notch developer, neither a top notch system administrator. So that's the reason that my job title is the option engineer. And today I'm going to talk about extending and embedding Ansible with Python. Please raise your hands. Any one of you who has used Ansible before? I'll have a brief introduction, like five minutes, just talking about what Ansible is for people that are not familiar. I'll continue explaining how to do some hacks and how to do some leverages, improvements and integrations, which is the part of extending and embedding. First, the introduction, the 101 to Ansible. What Ansible is? It's a configuration management tool. You may know many of those tools. There's also Puppet, Chef, Saltstack, CF Engine. The basic idea is to be able to automate the configuration of hosts. By hosts, I mean virtual machines, bare metal, containers, everything can be managed. There are many alternatives. Basically, the strong point of Ansible is that it works over SSH out of the box, and it is agentless. That means that you do not have to install anything in the host system, apart from Python 2.x. Another very strong point is that it is quite readable, and it has a smooth learning curve. It's not steep because it's DSL, its language is Jamel-based, so it's quite readable, and it's easy, very easy to get on and start it. This could be the example of an Ansible architecture in which there's a central node called the management node of a controller, which could be a laptop, could be a server, in which you run the Ansible software. Then you will have some inventory files that describe the IPs and DNS names of the hosts and the groups of the hosts that you are managing. And then there are the hosts, which are remote machines that can be accessed over SSH, and there are a series of playbooks that are like scripts that the Ansible machine executes in each one of the hosts that is managing. So this could be a very basic host file in which we define two groups of servers, one of web, another of dv-servers. We can define variables that are common to a group of servers, for example the username of the dv-group, and we can define meta-groups, that group, another group, like this one, which is infrastructure group, that joins the web and dv-group. There are two ways to use Ansible, basically there are other commands and using playbooks. So other commands is the easiest way, it's the hello world that everyone does, and they are just throw away one-time executions in which you use the Ansible command in order to execute a module in a group of hosts. And the syntax is very simple, just use the Ansible, name of the group, as described in your inventory file, minus m, name of the module, and minus i, the options. This could be the hello world that everyone has done. We are using the module called ping, the only thing that does is respond with a ping in the server. So if we issue ansible all minus m ping, and all is a magical group that groups everything in the inventory file, then we would have this response in each one of the servers, we would have the answer that is ponged, and also there's a value that says that stays if anything has changed in the remote host. And the other way to use Ansible is to use playbooks. Playbooks are a much more structured way to do configuration management. They are basically jumble files that specify a list of plays, and each plays a series of tasks that are applied to a group of hosts. And each one of those tasks is an execution of a module with some parameters and a description. This could be one task that is ensuring that the NGINX server is started. There's the description, this is the module name, and those are the parameters saying that the service is NGINX, and the state we decide is started. This is an example of configuration management concept that is that we define the desired states and not the actions to be performed, so that this module is clever enough to know that sometimes it has to start the service and sometimes just make a no-op and go on. This could be a playbook to deploy the NGINX server. In this playbook we are targeting the group of hosts named web. We define some variables, and this is the list of tasks. As you can see there are modules to add repositories, APT repositories, modules to install APT package, modules to manage services, modules to create or remove files. In this case we are removing a sim link, and modules to template out things like a Jinja 2 template that we are going to template out in the remote host. Also there are handlers that are special kind of tasks that are run once independently of the number of times that they have been notified during the playbook. This could be the execution of the playbook. We will have an output that sequentially shows the results of the task in each one of the hosts, and finally there's a nice recap. Just to finish this introduction, the way to manage things is using a kind of encapsulation that is roles. Roles is a little bit more advanced concept in Ansible that is a group of variables, of tasks, of handlers, files and templates, and maybe even dependencies to other roles. So we have finished our introduction. Now we will be talking about how to hack Ansible. There are two ways, embedding and extending Ansible. So what do I mean by embedding? I mean calling Ansible modules and playbooks from your Python code. This is basically possible because Ansible is based on Python. It's written in Python, so it has some kind of a Python API. It's not very well documented, but it's quite easy to use. If you read the source codes, you will see how to use it, although you can just continue and see how we can use it in this presentation. Before we start, a few disclaimers. This code is valid for Ansible version 1.9, which is the current stable one. Scroll in version 2, which is the current development one. Things are going to change. This API is going to change. And probably even things like the plugin mechanism is going to change. So this is only valid. I can only guarantee that it's valid for the current stable version. And everything in Ansible is Python 2.x only. It hasn't been ported to 3. You can find the examples in this GitHub repo. So first thing that we may want to do, we may want to run an Ansible task. This is a simpler kind of automation. It's just calling a module from your Ansible code. How do we do it? Very simple. Just import some classes, basically run around the inventory from the Ansible library. You will build your inventory, for example, directly in code. We may be just targeting our local host. And then making an instance of runner and calling the run method. Just passing the module name, the arguments, the inventory that we have just created, and the pattern of host that we want to target. And just a little bit, a little discretion. I'm talking about facts. What are the Ansible facts? Ansible facts are information that is retrieved at the beginning of the execution of the playbook in each one of the hosts. And there's a bunch of information available for you to make things in your playbook. So you have the host name, IP addresses, the hardware information, even the installed version of the software. So I'm going to make an example, which is the Flask factor, in which there's a proof of concept of embedding a module. I'm running a program that will create a REST API using the Flask RESTful module that parses URLs in this format, getting a fact that we want to know about the system. And it will run the module setup and then pass the JSON that we have showing the fact in our system. It could be just not very practical, but it could be an idea of how we can retrieve information from a remote server. So if we run it in localhost, I can retrieve the version of the software, the base system that I'm running, the version of the kernel, or things a bit more complex like the network card information. Sometimes we need something more complex than running a module. We need some kind of orchestration. So we may need to run a playbook, which is a more structured way. So you have to import a little bit more classes, basically classes for the callbacks and UTIMS. You have to build your inventory. It's similar to the other case. In this case, we're also using host variables, because in this example, I'm going to attach some information to the inventory, the list of users and APD packets that we want to install and create. You have to put some boilerplate code just to set the velocity of the execution of the playbook and also to register the typical callbacks for the runner and the playbook. And finally, creating a playbook instance and using the module run, specifying the playbook. In this case, the playbook is the installer.gamel. In this example, what I'm going to run... I'm going to run a playbook that install some APD packages and create some users, because it's a typical task when you arrive at a system to begin creating users and installing software packages. So I've just created a proof of concept script in which I get... First, I get via the interactive console the information of the name of the users and the list of packages, and then I call the interval playbook, which is called installer.gamel, and I pass the list of packages and users via inventory of arrivals. So that could be an example of this kind of integration if I run it. I can create users... and I can install packages. And when I finish, it runs in localhost, and it has changed that... It has shown that although the current user Alex is created, it has done anything, but for the new user that I've created, which is called George, it has changed the system, also installing another package. So I have called an Ansible playbook from my Pethen code. Well, one way to hack Ansible is by embedding, another way is by extending Ansible. And by extending, I mean adding more functionality or customizing its behavior. Basically, it's done by three ways, creating modules, creating dynamic inventory scripts, and creating plugins. So let's start with creating an Ansible module. Ansible ships with tons of modules ranging from creating user names, managing databases, spinning cloud servers. But sometimes we need something more specific to our business. So we have to create a module. And what is an Ansible module? It's just an executable file that you will find in the... some folders in the dot library relative to your playbook or in the Ansible library path. And basically, it has a JSON interface, so we expect a JSON for the input, and also it emits a JSON for the output. So it's language agnostic. You can do it in BASH, you can do it in every language that you want. But if you are using Python, then it's easier because there are some helper functions that will make it really easier. This could be the structure of a typical module. I borrowed some parts from the existing modules. And basically, it's a file in which you have two strings. We will come later to them that are the documentation and examples. Then this part of the code is what I call the Python part, in which we do the hard work for the libraries, and we run our business logic. And then from that point, this is like a template, let's say, in which we find a main module. We do some instantiation of the Ansible module class. We call our Python part from this function, and then we emit a different kind of JSON depending on the result. So let's get more detail. Commutation and examples are two strings variables that are very important because they are used by Mac WebDocs to generate the HTML of the documentation. And some important parts are the version option part, which will specify the values of the arguments. And it's important to keep it in sync with the real code, which is the argument spec dictionary. You have to specify the requirements, even some more Python libraries to be installed in the controller. And you have to use the node section if you need that some environment variable, for example, is present at the controller. And in the examples, just put code, please, that is tested and worked. Especially if you want to submit it to the Ansible repository. In the part that I call the Python part, it's usually a good idea to put pure Python code in all the libraries that you may need for dealing with the problem. But try not to use the Ansible part because it will make your code more robust to change this in the Ansible API. However, there are some helper functions, for example this one coming from the Ansible module, from the Ansible library that allows you to easily run commands and find executables in the remote machine. And just some tips. Try to return a value and a message, a meaningful message, and code every information on those values. And don't print to standard output or error because this mechanism won't work. And in the main function, you have to create an Ansible module instance. And when you instantiate it, you pass an argument spec dictionary which defines the module arguments, defining the series of modules that are required, the ones that are optional, the default values, possible choices, aliases, and then there's a section option which you specify the module exclusion of the parameters. And also if you support check mode, this is the mode in which you do a drive run and you do not mess with the real system. It's just a test. So when you have created that, you magically have a dictionary in which you have the parameters of the invocation of the module. You have just to take those parameters, call the Python part of the module with those parameters, and then pass the results depending on the status and the message. You have to create some kind of JSON. And if you use some helper functions like this one, exe-jation and fail-jation, then your life will be easier because they manage everything and create a JSON for you. And finally, probably you have seen those two lines at the end of the file. If you are a Pythonist, that you are. Probably you are now in tears and in pain because this is the Pythonic anti-pattern, the import asterisk. But please resist the temptation to make an explicit import because really it's like a define, a preprocessional in C because Ansible will substitute those lines with the code of the helper functions. So if you do not put it, it won't find it and you won't have an import. And if you do not put it at the end of the file, it will change the line numbering and debugging will be a hell. So some creation tips. To make a module you will love to use, a module that is edam.pdent that supports the check-mode. Test your module. There's a very handy tool included that is the test module script. And if you want to submit it, please follow the module creation checklist that is available online and it's much more comprehensive. I've created a module, an example, which is called the Tiger issue. Tiger is a agile project management system that has a REST API and there's a library called Python Tiger that allows to manage it and to play with it. So I've created a very simple module. That is called Tiger issue. This module is just for creating issues in this platform. For example, if we are deploying a system by an anteval and we find some problem during the automation of the task, we can create an issue so that the team will see it, will have the information, will have the logs, will have everything. So I have submitted it to Ansible extra module repository. And basically this is the documentation part. You have to specify the different options that the module supports. And you have to put some examples. This is what I call the Python part. I import the Python Tiger and I'll do the issue management here. It should be the pure Python code. And finally, this is the main part in which I specify the restrictions and the parameters. I part them and I call the Python part with arguments and depending on the return status, I issue one kind of JSON or the other. And if we see the playbook, we can just use this module to create Tiger issue in our project. We can manage the description. We can put also Ansible variables like the host name and distribution. We can attach to the issue a file. For example, we can attach the playbook. Then I will pause and when I continue, it will delete the Tiger issue because the module supports the state present and absent. So this is the test project that I've created with no issues. And then if I run the demo, it calls the Ansible playbook of the playbook that we have seen. First, it has run the creation of the Tiger issue and it has stopped. If I take here that I can see the issue. This issue has some tags. This is my Ansible distribution and this is the playbook also. So if I resume, it will go and delete it when possible. It's not there. So creating a dynamic inventory script. There's another way to hack it. If you are managing cloud servers and cloud infrastructure, probably you know about this. Dynamic inventory scripts are a way not to have to deal with a long list of servers that probably are changing. Their IPs are changing. You can scale things. So dynamic inventory scripts are a way to deal with that complexity. And basically, they are just an executable file that supports those command line features. Minus, minus list that will return a JSON dictionary with the name of the groups. And each group is a list of hosts and minus, minus host in which you specify a host name. And then you get a dictionary of the host variables. Just pure JSON. So just for the fun of it, I've created an example which is a shelf inventory. Shelf files are basically a key value store that Python supports natively in the serialization. So in the example, we are just using a shelf file that I've created. And we can open it and set the groups on the host version of it. I can get past through the demo because we are running out of time. I want to run another demo. Plugins. Plugins are a way to hack Ansible in very different ways. Basically, the common thing is that they run in the controller node and there are different kind of plugins. We're going to see some of them. And basically, the way to add them is just drop them in a folder. And if you want to use another folder, just treat the Ansible config file. So the callback plugin is just a kind of plugin that reacts in the controller to the Playbook and Runner events. In order to create them, you just have to define a callback module class and then override the method you want to. And this is an example of the list of methods available for this class in the repository. So just a brief example, I'm going to use to create a callback plugin that reacts to the failed event of a module. And then it will call the notify send binary, which will create a popup in my system with the name of the module and the result of the module. So, I'm going to use the callback plugin. I'm running this Playbook with just output message. The first task will go up. The second one will fail. So, I'll get a popup saying that the second one has failed. It's just an example of things that can be done. You can make Slack notifications. You can make HipTag, everything you see it useful. You can create connection plugins. Basically, the connection plugins allows the controller to connect to the remote host and Ansible ships with lots of them. But you can create much more. You'll just have to define a connection class and override the basic methods that are just connecting, disconnecting, executing commands, putting and affecting files, which are basically things that Ansible is doing in SSH mode. You can create lookup plugins. Lookup plugins allow to expand the functionality by accessing information in external sources from the controller, for example, in databases, in file systems. And the same task is just calling the lookup function. And also, if you define a lookup, you'll get a width expression that allows you to loop over some results. So, you just have to create a lookup module class and basically define an init method and a run method, which will pass the terms, which are the functions, the arguments of the lookup, and then return a list of results. Following the example of the shell file, I can define a shell file lookup that will search for a shell file and retrieve a key from that shell file. Basically, I just implemented the run method. And just add demo. In this paybook, I'm opening the book.dv, retrieving the current book name, page and book author of the book I'm reading now and print them. It's a good book, if you haven't read it. And you can create filter plugins. And you can define some other filters. The syntax for using them is very simple. Variable followed by a pipe and the name of the filter. Those are example filters that are simple comes with them. But you can define many other. So, it's very simple. Just create a filter module class that has a filter's method that returns a mapping from the name of the filter to the Python function that implements it. For example, we can create a filter to rotate 13 positions. This is the C-star rotation. And just if we see an example of playbook, we can apply once and then apply twice. And we can see that C-star becomes PNRFNA and then becomes C-star. So, it's working. Finally, just other plugins. Very fast action plugins that are plugins that allow separation of actions between the controller and the remote host. For example, when you're templating out a file, there are some actions that need to be performing the controller and some other. Different in the remote so that you can just implement an action module class with a method run. And when you're dealing, doing things in the controller, then you execute this method, run your execute module to call the real module on the remote host. You can define bar's plugins. Quite an undocumented part, possibly going to change with sensible version two. And it's the way that we can retrieve more information about the host by some external source like the host vars and group vars directory. Basically, this is the template that you will find in the repo. And you have to implement the get host vars and get group vars for a new plugin. You can create cache plugins, very modern functionality from mcvill version 1.8 in which we can retrieve the information of hosts that have not been contacted in the current playbook execution. Just using a backend like a redis or memcast, you can run once, get the facts and then runs without having to retrieve the facts of the rest of the host. You have to tweak your answalt config file. And basically, if you create a new one, you want to create a new one, you just have to implement this template. And finally, those are the references that I have used. These are very good books, not just for developing answalt but just for using them, especially the first two ones. And some articles talking about that. So that's it. Thank you very much. And now if you have any questions. Thank you very much. Well, thanks for the talk. And my question is, you mentioned that while using Ansible as embedded module, it's very hard to find the documentation of API or is it very poorly documented? So actually, how do I get this information to use it as embedded module? Okay. If you want to embed a module like in the first example that I have said, I have done. I had to do some research. I had to go into the source code and see how the Ansible command is using the API and try to mimic it because we are using the same API that the command line version that is just an icon file is executing. Also, I have seen some other examples which are here in the reference that are much more pragmatic examples. So I have been inspired from them. I think that in version 2.0, they are going to document it much better. They know in the Ansible project that there's a lack of documentation in that part. I don't know if intentionally, but there's a lack of it. And it's changing. So when Ansible 2.0 comes as a stable one probably, it will have changed. It's not going to be, I think, not so simple, but more flexible. So what about the way of testing modules? Do you need to write tests in some specific language, like or library for testing the code or standard tests in Python? Well, testing things in Ansible is quite a tricky aspect because even the Ansible code base is not very well covered with tests. If you see the project, there are just like 100 and something assertions on the base code. So they are not very concerned about the testing. And testing a module is even much more difficult. I would suggest you to test the Python part separately. Just those kind of part, which is pure Python. You can use PyTest. You can put the tests in another files. But there's no standard way to run programmatically tests to a module. And you do not have even to make tests to put tests in a full request to get the module accepted. They are not very strict with this part. So I would suggest using testing only this Python part, the business logic of your module, using external library like PyTest, something like that, putting in a different file and importing the functions and the classes that you are defining. So maybe it's easier using something like Test Kitchen. Like? Sorry? Like Test Kitchen. Test Kitchen. Well, with Test Kitchen you are not using, like, I think it's unit testing. It's more like, I don't know if it's functional testing or something like that. Test Kitchen, just to focus, is it a tool developed in Ruby for testing a puppet or things like that? Actually, I don't know. I was wondering if you can do a test of it. I see that it has a lot of parts. I didn't know that it was possible to reuse. Thank you. I'll have a keep an eye on that. Any more questions? Okay. If there are no more questions, then please thank Alejandro.
 Well, good morning and thanks everybody for attending this.  As Pina said, my name is Alec Han-Roydow and I'm not a double...  Let me introduce you to Fabian Croetz. He's going to give his speech on...  So I was saying I'm not a top notch developer, neither a top notch system administrator.  So that's the reason that my job title is the option engineer.  And today I'm going to talk about extending and embedding Ansible with Python.  Please raise your hands. Any one of you who has used Ansible before?  I'll have a brief introduction, like five minutes, just talking about what Ansible is for people that are not familiar.  I'll continue explaining how to do some hacks and how to do some leverages, improvements and integrations, which is the part of extending and embedding.  First, the introduction, the 101 to Ansible. What Ansible is?  It's a configuration management tool. You may know many of those tools. There's also Puppet, Chef, Saltstack, CF Engine.  The basic idea is to be able to automate the configuration of hosts.  By hosts, I mean virtual machines, bare metal, containers, everything can be managed.  There are many alternatives. Basically, the strong point of Ansible is that it works over SSH out of the box, and it is agentless.  That means that you do not have to install anything in the host system, apart from Python 2.x.  Another very strong point is that it is quite readable, and it has a smooth learning curve.  It's not steep because it's DSL, its language is Jamel-based, so it's quite readable, and it's easy, very easy to get on and start it.  This could be the example of an Ansible architecture in which there's a central node called the management node of a controller,  which could be a laptop, could be a server, in which you run the Ansible software.  Then you will have some inventory files that describe the IPs and DNS names of the hosts and the groups of the hosts that you are managing.  And then there are the hosts, which are remote machines that can be accessed over SSH, and there are a series of playbooks that are like scripts  that the Ansible machine executes in each one of the hosts that is managing.  So this could be a very basic host file in which we define two groups of servers, one of web, another of dv-servers.  We can define variables that are common to a group of servers, for example the username of the dv-group,  and we can define meta-groups, that group, another group, like this one, which is infrastructure group, that joins the web and dv-group.  There are two ways to use Ansible, basically there are other commands and using playbooks.  So other commands is the easiest way, it's the hello world that everyone does, and they are just throw away one-time executions  in which you use the Ansible command in order to execute a module in a group of hosts.  And the syntax is very simple, just use the Ansible, name of the group, as described in your inventory file, minus m, name of the module, and minus i, the options.  This could be the hello world that everyone has done.  We are using the module called ping, the only thing that does is respond with a ping in the server.  So if we issue ansible all minus m ping, and all is a magical group that groups everything in the inventory file,  then we would have this response in each one of the servers, we would have the answer that is ponged,  and also there's a value that says that stays if anything has changed in the remote host.  And the other way to use Ansible is to use playbooks. Playbooks are a much more structured way to do configuration management.  They are basically jumble files that specify a list of plays, and each plays a series of tasks that are applied to a group of hosts.  And each one of those tasks is an execution of a module with some parameters and a description.  This could be one task that is ensuring that the NGINX server is started.  There's the description, this is the module name, and those are the parameters saying that the service is NGINX,  and the state we decide is started.  This is an example of configuration management concept that is that we define the desired states and not the actions to be performed,  so that this module is clever enough to know that sometimes it has to start the service and sometimes just make a no-op and go on.  This could be a playbook to deploy the NGINX server.  In this playbook we are targeting the group of hosts named web.  We define some variables, and this is the list of tasks.  As you can see there are modules to add repositories, APT repositories, modules to install APT package,  modules to manage services, modules to create or remove files.  In this case we are removing a sim link, and modules to template out things like a Jinja 2 template that we are going to template out in the remote host.  Also there are handlers that are special kind of tasks that are run once independently of the number of times that they have been notified during the playbook.  This could be the execution of the playbook.  We will have an output that sequentially shows the results of the task in each one of the hosts, and finally there's a nice recap.  Just to finish this introduction, the way to manage things is using a kind of encapsulation that is roles.  Roles is a little bit more advanced concept in Ansible that is a group of variables, of tasks, of handlers, files and templates,  and maybe even dependencies to other roles.  So we have finished our introduction.  Now we will be talking about how to hack Ansible. There are two ways, embedding and extending Ansible.  So what do I mean by embedding?  I mean calling Ansible modules and playbooks from your Python code.  This is basically possible because Ansible is based on Python.  It's written in Python, so it has some kind of a Python API.  It's not very well documented, but it's quite easy to use.  If you read the source codes, you will see how to use it, although you can just continue and see how we can use it in this presentation.  Before we start, a few disclaimers.  This code is valid for Ansible version 1.9, which is the current stable one.  Scroll in version 2, which is the current development one.  Things are going to change. This API is going to change.  And probably even things like the plugin mechanism is going to change.  So this is only valid.  I can only guarantee that it's valid for the current stable version.  And everything in Ansible is Python 2.x only.  It hasn't been ported to 3.  You can find the examples in this GitHub repo.  So first thing that we may want to do, we may want to run an Ansible task.  This is a simpler kind of automation.  It's just calling a module from your Ansible code.  How do we do it? Very simple.  Just import some classes, basically run around the inventory from the Ansible library.  You will build your inventory, for example, directly in code.  We may be just targeting our local host.  And then making an instance of runner and calling the run method.  Just passing the module name, the arguments, the inventory that we have just created,  and the pattern of host that we want to target.  And just a little bit, a little discretion.  I'm talking about facts.  What are the Ansible facts?  Ansible facts are information that is retrieved at the beginning of the execution of the playbook  in each one of the hosts.  And there's a bunch of information available for you to make things in your playbook.  So you have the host name, IP addresses, the hardware information,  even the installed version of the software.  So I'm going to make an example, which is the Flask factor,  in which there's a proof of concept of embedding a module.  I'm running a program that will create a REST API using the Flask RESTful module  that parses URLs in this format,  getting a fact that we want to know about the system.  And it will run the module setup and then pass the JSON that we have  showing the fact in our system.  It could be just not very practical,  but it could be an idea of how we can retrieve information from a remote server.  So if we run it in localhost, I can retrieve the version of the software,  the base system that I'm running, the version of the kernel,  or things a bit more complex like the network card information.  Sometimes we need something more complex than running a module.  We need some kind of orchestration.  So we may need to run a playbook, which is a more structured way.  So you have to import a little bit more classes,  basically classes for the callbacks and UTIMS.  You have to build your inventory.  It's similar to the other case.  In this case, we're also using host variables,  because in this example, I'm going to attach some information to the inventory,  the list of users and APD packets that we want to install and create.  You have to put some boilerplate code just to set the velocity of the execution of the playbook  and also to register the typical callbacks for the runner and the playbook.  And finally, creating a playbook instance and using the module run,  specifying the playbook.  In this case, the playbook is the installer.gamel.  In this example, what I'm going to run...  I'm going to run a playbook that install some APD packages and create some users,  because it's a typical task when you arrive at a system to begin creating users  and installing software packages.  So I've just created a proof of concept script in which I get...  First, I get via the interactive console the information of the name of the users  and the list of packages, and then I call the interval playbook,  which is called installer.gamel, and I pass the list of packages and users  via inventory of arrivals.  So that could be an example of this kind of integration if I run it.  I can create users...  and I can install packages.  And when I finish, it runs in localhost,  and it has changed that...  It has shown that although the current user Alex is created,  it has done anything, but for the new user that I've created, which is called George,  it has changed the system, also installing another package.  So I have called an Ansible playbook from my Pethen code.  Well, one way to hack Ansible is by embedding,  another way is by extending Ansible.  And by extending, I mean adding more functionality or customizing its behavior.  Basically, it's done by three ways, creating modules,  creating dynamic inventory scripts, and creating plugins.  So let's start with creating an Ansible module.  Ansible ships with tons of modules ranging from creating user names,  managing databases, spinning cloud servers.  But sometimes we need something more specific to our business.  So we have to create a module.  And what is an Ansible module?  It's just an executable file that you will find in the...  some folders in the dot library relative to your playbook  or in the Ansible library path.  And basically, it has a JSON interface,  so we expect a JSON for the input,  and also it emits a JSON for the output.  So it's language agnostic.  You can do it in BASH, you can do it in every language that you want.  But if you are using Python, then it's easier  because there are some helper functions that will make it really easier.  This could be the structure of a typical module.  I borrowed some parts from the existing modules.  And basically, it's a file in which you have two strings.  We will come later to them that are the documentation and examples.  Then this part of the code is what I call the Python part,  in which we do the hard work for the libraries,  and we run our business logic.  And then from that point, this is like a template, let's say,  in which we find a main module.  We do some instantiation of the Ansible module class.  We call our Python part from this function,  and then we emit a different kind of JSON depending on the result.  So let's get more detail.  Commutation and examples are two strings variables  that are very important because they are used by Mac WebDocs  to generate the HTML of the documentation.  And some important parts are the version option part,  which will specify the values of the arguments.  And it's important to keep it in sync with the real code,  which is the argument spec dictionary.  You have to specify the requirements,  even some more Python libraries to be installed in the controller.  And you have to use the node section if you need  that some environment variable, for example, is present at the controller.  And in the examples, just put code, please, that is tested and worked.  Especially if you want to submit it to the Ansible repository.  In the part that I call the Python part,  it's usually a good idea to put pure Python code  in all the libraries that you may need for dealing with the problem.  But try not to use the Ansible part  because it will make your code more robust  to change this in the Ansible API.  However, there are some helper functions,  for example this one coming from the Ansible module,  from the Ansible library that allows you to easily run commands  and find executables in the remote machine.  And just some tips.  Try to return a value and a message, a meaningful message,  and code every information on those values.  And don't print to standard output or error  because this mechanism won't work.  And in the main function, you have to create an Ansible module instance.  And when you instantiate it, you pass an argument spec dictionary  which defines the module arguments,  defining the series of modules that are required,  the ones that are optional, the default values,  possible choices, aliases, and then there's a section option  which you specify the module exclusion of the parameters.  And also if you support check mode,  this is the mode in which you do a drive run  and you do not mess with the real system.  It's just a test.  So when you have created that, you magically have a dictionary  in which you have the parameters of the invocation of the module.  You have just to take those parameters,  call the Python part of the module with those parameters,  and then pass the results depending on the status and the message.  You have to create some kind of JSON.  And if you use some helper functions like this one,  exe-jation and fail-jation,  then your life will be easier  because they manage everything and create a JSON for you.  And finally, probably you have seen those two lines  at the end of the file.  If you are a Pythonist, that you are.  Probably you are now in tears and in pain  because this is the Pythonic anti-pattern, the import asterisk.  But please resist the temptation to make an explicit import  because really it's like a define, a preprocessional in C  because Ansible will substitute those lines  with the code of the helper functions.  So if you do not put it, it won't find it  and you won't have an import.  And if you do not put it at the end of the file,  it will change the line numbering and debugging will be a hell.  So some creation tips.  To make a module you will love to use,  a module that is edam.pdent that supports the check-mode.  Test your module.  There's a very handy tool included that is the test module script.  And if you want to submit it,  please follow the module creation checklist  that is available online and it's much more comprehensive.  I've created a module, an example, which is called the Tiger issue.  Tiger is a agile project management system  that has a REST API and there's a library called Python Tiger  that allows to manage it and to play with it.  So I've created a very simple module.  That is called Tiger issue.  This module is just for creating issues in this platform.  For example, if we are deploying a system by an anteval  and we find some problem during the automation of the task,  we can create an issue so that the team will see it,  will have the information, will have the logs,  will have everything.  So I have submitted it to Ansible extra module repository.  And basically this is the documentation part.  You have to specify the different options that the module supports.  And you have to put some examples.  This is what I call the Python part.  I import the Python Tiger and I'll do the issue management here.  It should be the pure Python code.  And finally, this is the main part in which I specify the restrictions  and the parameters.  I part them and I call the Python part with arguments  and depending on the return status,  I issue one kind of JSON or the other.  And if we see the playbook,  we can just use this module to create Tiger issue in our project.  We can manage the description.  We can put also Ansible variables like the host name and distribution.  We can attach to the issue a file.  For example, we can attach the playbook.  Then I will pause and when I continue,  it will delete the Tiger issue because the module supports the state present and absent.  So this is the test project that I've created with no issues.  And then if I run the demo, it calls the Ansible playbook of the playbook that we have seen.  First, it has run the creation of the Tiger issue and it has stopped.  If I take here that I can see the issue.  This issue has some tags.  This is my Ansible distribution and this is the playbook also.  So if I resume, it will go and delete it when possible.  It's not there.  So creating a dynamic inventory script.  There's another way to hack it.  If you are managing cloud servers and cloud infrastructure,  probably you know about this.  Dynamic inventory scripts are a way not to have to deal with a long list of servers that probably are changing.  Their IPs are changing.  You can scale things.  So dynamic inventory scripts are a way to deal with that complexity.  And basically, they are just an executable file that supports those command line features.  Minus, minus list that will return a JSON dictionary with the name of the groups.  And each group is a list of hosts and minus, minus host in which you specify a host name.  And then you get a dictionary of the host variables.  Just pure JSON.  So just for the fun of it, I've created an example which is a shelf inventory.  Shelf files are basically a key value store that Python supports natively in the serialization.  So in the example, we are just using a shelf file that I've created.  And we can open it and set the groups on the host version of it.  I can get past through the demo because we are running out of time.  I want to run another demo.  Plugins.  Plugins are a way to hack Ansible in very different ways.  Basically, the common thing is that they run in the controller node and there are different kind of plugins.  We're going to see some of them.  And basically, the way to add them is just drop them in a folder.  And if you want to use another folder, just treat the Ansible config file.  So the callback plugin is just a kind of plugin that reacts in the controller to the Playbook and Runner events.  In order to create them, you just have to define a callback module class and then override the method you want to.  And this is an example of the list of methods available for this class in the repository.  So just a brief example, I'm going to use to create a callback plugin that reacts to the failed event of a module.  And then it will call the notify send binary, which will create a popup in my system with the name of the module and the result of the module.  So, I'm going to use the callback plugin.  I'm running this Playbook with just output message.  The first task will go up.  The second one will fail.  So, I'll get a popup saying that the second one has failed.  It's just an example of things that can be done.  You can make Slack notifications.  You can make HipTag, everything you see it useful.  You can create connection plugins.  Basically, the connection plugins allows the controller to connect to the remote host and Ansible ships with lots of them.  But you can create much more.  You'll just have to define a connection class and override the basic methods that are just connecting, disconnecting, executing commands, putting and affecting files, which are basically things that Ansible is doing in SSH mode.  You can create lookup plugins.  Lookup plugins allow to expand the functionality by accessing information in external sources from the controller, for example, in databases, in file systems.  And the same task is just calling the lookup function.  And also, if you define a lookup, you'll get a width expression that allows you to loop over some results.  So, you just have to create a lookup module class and basically define an init method and a run method, which will pass the terms, which are the functions, the arguments of the lookup, and then return a list of results.  Following the example of the shell file, I can define a shell file lookup that will search for a shell file and retrieve a key from that shell file.  Basically, I just implemented the run method.  And just add demo.  In this paybook, I'm opening the book.dv, retrieving the current book name, page and book author of the book I'm reading now and print them.  It's a good book, if you haven't read it.  And you can create filter plugins.  And you can define some other filters.  The syntax for using them is very simple.  Variable followed by a pipe and the name of the filter.  Those are example filters that are simple comes with them.  But you can define many other.  So, it's very simple.  Just create a filter module class that has a filter's method that returns a mapping from the name of the filter to the Python function that implements it.  For example, we can create a filter to rotate 13 positions.  This is the C-star rotation.  And just if we see an example of playbook, we can apply once and then apply twice.  And we can see that C-star becomes PNRFNA and then becomes C-star.  So, it's working.  Finally, just other plugins.  Very fast action plugins that are plugins that allow separation of actions between the controller and the remote host.  For example, when you're templating out a file, there are some actions that need to be performing the controller and some other.  Different in the remote so that you can just implement an action module class with a method run.  And when you're dealing, doing things in the controller, then you execute this method, run your execute module to call the real module on the remote host.  You can define bar's plugins.  Quite an undocumented part, possibly going to change with sensible version two.  And it's the way that we can retrieve more information about the host by some external source like the host vars and group vars directory.  Basically, this is the template that you will find in the repo.  And you have to implement the get host vars and get group vars for a new plugin.  You can create cache plugins, very modern functionality from mcvill version 1.8 in which we can retrieve the information of hosts that have not been contacted in the current playbook execution.  Just using a backend like a redis or memcast, you can run once, get the facts and then runs without having to retrieve the facts of the rest of the host.  You have to tweak your answalt config file.  And basically, if you create a new one, you want to create a new one, you just have to implement this template.  And finally, those are the references that I have used.  These are very good books, not just for developing answalt but just for using them, especially the first two ones.  And some articles talking about that.  So that's it. Thank you very much.  And now if you have any questions.  Thank you very much.  Well, thanks for the talk. And my question is, you mentioned that while using Ansible as embedded module, it's very hard to find the documentation of API or is it very poorly documented?  So actually, how do I get this information to use it as embedded module?  Okay.  If you want to embed a module like in the first example that I have said, I have done.  I had to do some research.  I had to go into the source code and see how the Ansible command is using the API and try to mimic it because we are using the same API that the command line version that is just an icon file is executing.  Also, I have seen some other examples which are here in the reference that are much more pragmatic examples. So I have been inspired from them.  I think that in version 2.0, they are going to document it much better.  They know in the Ansible project that there's a lack of documentation in that part.  I don't know if intentionally, but there's a lack of it.  And it's changing. So when Ansible 2.0 comes as a stable one probably, it will have changed.  It's not going to be, I think, not so simple, but more flexible.  So what about the way of testing modules? Do you need to write tests in some specific language, like or library for testing the code or standard tests in Python?  Well, testing things in Ansible is quite a tricky aspect because even the Ansible code base is not very well covered with tests.  If you see the project, there are just like 100 and something assertions on the base code.  So they are not very concerned about the testing.  And testing a module is even much more difficult.  I would suggest you to test the Python part separately.  Just those kind of part, which is pure Python.  You can use PyTest. You can put the tests in another files.  But there's no standard way to run programmatically tests to a module.  And you do not have even to make tests to put tests in a full request to get the module accepted.  They are not very strict with this part.  So I would suggest using testing only this Python part, the business logic of your module, using external library like PyTest, something like that,  putting in a different file and importing the functions and the classes that you are defining.  So maybe it's easier using something like Test Kitchen.  Like? Sorry?  Like Test Kitchen.  Test Kitchen.  Well, with Test Kitchen you are not using, like, I think it's unit testing.  It's more like, I don't know if it's functional testing or something like that.  Test Kitchen, just to focus, is it a tool developed in Ruby for testing a puppet or things like that?  Actually, I don't know.  I was wondering if you can do a test of it.  I see that it has a lot of parts.  I didn't know that it was possible to reuse.  Thank you. I'll have a keep an eye on that.  Any more questions?  Okay. If there are no more questions, then please thank Alejandro.
Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_0_2087.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_2088_8759.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_8760_10247.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_10248_10919.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_10920_14963.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_14964_16739.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_16740_19679.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_19680_22391.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_22392_24551.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_24552_25787.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_25788_28775.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_28776_30959.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_30960_31055.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_31056_32087.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_32088_32531.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_32532_32795.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_32796_34271.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_34272_34943.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_34944_37679.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_37680_38315.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_38316_38675.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_38676_39107.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_39108_40799.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_40800_41183.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_41184_41591.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_41592_41903.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_41904_44459.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_44460_44975.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_44976_46223.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_46224_47471.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_47472_47723.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_47724_48659.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_48660_49031.png Alejandro_Guirao_Rodriguez_-_Extending_and_embedding_Ansible_with_Python_49032_56224.png