Packaging in packaging: dh-virtualenv
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Pulliainen, Jyrki
Jyrki Pulliainen - Packaging in packaging: dh-virtualenv Deploying your software can become a tricky task, regardless of the language. In the spirit of the Python conferences, every conference needs at least one packaging talk. This talk is about dh-virtualenv. It's a Python packaging tool aimed for Debian-based systems and for deployment flows that already take advantage of Debian packaging with Python virtualenvs - [Dh-virtualenv] is an open source tool developed at Spotify. We use it to ease deploying our Python software to production. We built dh-virtualenv as a tool that fits our existing continuous integration flow with a dedicated sbuild server. As we were already packaging software in Debian packages, the aim of dh-virtualenv was to make transition to virtualenv based installations as smooth as possible. This talk covers how you can use dh-virtualenv to help you deploy your software to production, where you are already running a Debian-based system, such as Ubuntu, and what are the advantages and disadvantages of the approach over other existing and popular techniques. We will discuss the deploying as a problem in general, look into building a dh-vritualenv-backed package, and in the end, look into how dh-virtualenv was actually made. Goal is that after this presentation you know how to make your Debian/Ubuntu deployments easier! [dh-virtualenv] if fully open sourced, production tested software, licensed under GPLv2+ and available in Debian testing and unstable. More information of it is also available in our [blogpost]. Talk outline: 1. Introduction & overview (3min) * Who am I? * Why am I fiddling with Python packaging? * What do you get out of this talk? 2. Different shortcomings of Python deployments (5min) * Native system packages * Virtualenv based installations * Containers, virtual machine images 3. dh-virtualenv (10 min) * What is dh-virtualenv? * Thought behind dh-virtualenv * Advantages over others * Requirements for your deployment flow * Short intro to packaging Sentry with dh-virtualenv 4. How is it built? (10 min) * Debian package building flow primer * How dh-virtualenv fits that flow * What does it do build time and why?
 Okay, no, Euro Python or any Python conference is complete without at least one talk on packaging and this must be at least a second. So let's see what your key has to tell us. Thank you. If you don't mind, I'm going to wait for a tiny bit for clock to hit three, four and then just depressed. No one comes. Before you can run away, I'm going to take a photo of you. This is a great in a zero chance that it happens when I start speaking. Okay. Good to talk, Euro Python or Helsinki will come in. That's my outcome of two years of German studies while in elementary school. Unfortunately I can't do better. But welcome everyone to a talk about a tool called DH virtual end or as I label it packaging in packaging. But before we look into that deeper, let me introduce myself. So my name is Jurke Poulienen. I'm from Finland but I'm living in Stockholm, Sweden nowadays. I work for a music streaming company called Spotify. In there I do kind of do two stuff. I am a content engineer. So I build like a pipeline of new music to the service and on the other hand I also fiddle around a lot with our internal Python stack and answer people's questions about Python. If you want to reach out to me, there's my email address and there's my Twitter handle. So please do if you have any questions. Now about this talk. This talk will be in three different sections. First we're going to look into some of the existing deployment strategies you can use on the PyNU-based machines. Then we're going to look into what actually is DH virtual environment and how does it differ from the existing deployment strategies. And in the end we're going to go through an example of how you package software and we're going to package Sentry mostly because it's not the simple piece of software. So I'm going to show you by example how you can use DH virtual environment to package something like Sentry in production. Now let's start with that tiny quiz. Who here runs a Debian or Ubuntu-based system? Good. You're from the correct token. Who here deploy stuff using the so-called native packages on Debian system, like relying on the native libraries on those systems? Who here is kind of frustrated with that? Yeah. Okay. Who here uses virtual environment? Like you set up a virtual environment and then you install everything in there on your production host. Cool. Now these both have their good sides too. If we take for example the native Debian packages. So this table, the stuff that gets in the Debian, especially stuff that's in main, is stable, it's well tested. People usually don't change them so that the backwards compatibility breaks. So if your Debian system gets an update for let's say Python requests, you know that that request is something that's backwards and compatible with the previous version. Now Debian has enough or Debian packaging has another nice bone side. You can declare non-Python dependencies. So say your software requires SQLite to be installed or your software requires MySQL to be installed on the same machine. When you create the Debian packages, you can say that your package depends on MySQL and you get that one installed on the machine. Everything nicely contained in one package. It also has pretty neat existing infrastructure. So not only you have like dedicated build tools or separated build environments like sbuild and other CH root solutions, but you also have the possibility of running your own app through repository which means that you will have your own way, or like own network of deploying your stuff and production. Plus all kinds of CI tools like Jenkins or Team City have at least some rudimentary support for dealing with Debian packages. And the last good thing I think in the Debian packaging is that you have a quite nice scripting support. What that means is that if you need to remove a cache when you upgrade your package version, you can write the script that before you upgrade your package, it will clear out the cache. Or if you want to restart your service after you've installed it, you can write the script that after installation, it will restart your service. You can do crazy borderline stupid stuff like database migrations and postings. I don't necessarily recommend them, but you can do all kinds of stuff there. It gives you quite a lot of power. Now that was the good parts. Then if we start talking about the bad parts, you probably have run into this case where you see that, oh, let's say back in the days, can network hard and push out request 1.3, and that has just the feature you need, but unfortunately your ancient web-filled box is running request 1.2. And what do you think? Are you going to wait for Debian to package that? What happens, you're just going to rot in front of your computer waiting for the newer request, especially if it's a backward incompatible to come on the current system you're running. So a lot of stuff in Debian, even at the release of that particular Debian or Ubuntu release, is already outdated. The packaging itself is kind of complex. So what we're talking here is that someone created a packaging system that is built for building a whole operating system. So it covers all the possible corner cases, all the possible scenarios, if you want to deploy Perl, Haskell, Python, you name it, it has everything covered. That means that the whole system is really, really complex and the documentation is far from simple. In addition to that, all the documentation that you usually can find is also geared towards Debian package maintenance. So okay, you want to ship this thing within the operating system. So this is how you should do. And it does not necessarily resonate to how you yourself would deploy your tiny service in the host. And what I think is the worst part in Debian packages is that you get a global state. So if you're only deploying all your libraries as Debian packages and then using them, eventually you'll end up having a case where you would like to upgrade one library, but it's also used by some other software on the same box and you don't know if you can do it without breaking anything. We've had this at Spotify a few times when we have rolled out like new common Python utils libraries on the host and we're just too afraid of deploying that because we might or kind of afraid that something down the line might break even we have been testing it for months. So it kind of slows you down and it's really, really annoying. Now if you think about virtual environments then, so what you get in the virtual environment is somewhat the opposite. So you get the whole new stuff. You just do pip install and you get whatever is available in PyPI, like the latest release. And you can go to the extent of like you can open your Git or Mercurial and pull the stuff in your virtual environment and run it in there. So you can always get the new, hottest stuff. It has become kind of a de facto method in the Python world. So every guide usually contains a word or two how you run stuff inside the virtual environment. You can do the same virtual environment stuff on your laptop as you can do on your servers. It kind of works. It's also battle tested. So nowadays so many people are running it in production so it's fairly safe to use that one. But the best part I think is that it's contained. So if you take a package and install it inside the virtual environment, a Python package, you can be sure that it won't affect anything outside the virtual environment. So updating a simple packages doesn't mean that your whole system crumbles because something was relying on all this older version or that you would interfere with the underlying operating system. No, you're only poking the actual virtual environment and you get this nice contained fuzzy feeling when dealing with that. Now on the downside, I bet some of you might have seen this line. That means that you can't have any native dependencies if you're using virtual environments and people install. You need to know what MySQL libraries you have to have available. Like in this case, you need to know that you have to install MySQL client to find this MySQL config on your host. So it requires you to do some manual digging through things. Even more, you end up doing source installs. So you can have wheels or even eggs to avoid source installs. But if you don't have wheels for your platform or you haven't set up your own wheel riposter, you end up doing source installations. When you have your virtual environment in your production server, you end up installing all the dependencies for that source installation on that production environment, which is not necessarily bad in the sense that you could break something, but you'll just clutter your production environment with developer headers and other unused files. But the worst part, what I think is with the PIP installs is that you're basically executing a bunch of random scripts. So sure, wheel gets around this, but if you run set up the PIP install, you probably haven't looked into what all those files do or what all those files that those packages depend on do. So you're just relying on a good faith of people and you're running random stuff in your production environment. It doesn't even need to be malicious to hurt you. Someone might just accidentally release a package that wipes your whole ETC or your whole home directory or something and you by accident end up crippling your system. Now this brings us to question then what is DH VirtualN. So DH VirtualN was about two years ago my attempt to combine the best of the two worlds. So it is a virtual environment that is placed inside the PN package. It supports both Python 2 and 3. It is kind of version agnostic. I won't say that you can't install Python 1 stuff with that, but it doesn't execute or import any Python code. So it doesn't really care if your tool is written with Python 2 or Python 3. You can use it anyway. It even supports using the new virtual environment package with 3.3 so you don't even need to install virtual N to run it. It's also open source. So it's GPL like all the different build tools. It has a good documentation. Now I'm the guy who wrote it. So I might be a bit biased here. But I think the documentation is good. It's at least better than the average open source documentation. But the best part which seems to be actually very functional is that it has a simple tutorial. So if you go to the DH VirtualN.readthedocs.org you find there's a four step tutorial that you can run through and boom, your package is inside the virtual environment inside the Debian package. Under the surface it is a depth helper extension. So depth helper is this... How would you make it pretty? Like a pile of purl scripts that Debian executes when you're building packages. It's a certain fixed sequence of purl scripts that you install and then there are different extensions for Debian to know how to build packages for Python packages or how you build purl packages or how you build bash completion stuff. So what the virtual environment does, it just injects itself into that flow. So there's 12 lines per included there that does the magic to inject the virtual environment there and then it just runs as a part of the sequence. Now this is kind of like the implementation details but for you who already have existing Debian build environments, like if you're using sbuild or something else or just playing the build or the package build package on the command line, this means that using the virtual N is just going to work in your existing workflow. So what I basically did back in the days was that I found a great blog post by Hinek Slavak. I adapted the idea a bit to fit our built environment and that's why I ended up writing depth helper extensions. Hinek system works well too but it uses a thing called FPM which back in the days didn't fit our build system. Now in practice, the DH virtual environment is a packaging builder that it creates a virtual environment, you can define what Python you want to use with it. So if you have multiple pythons installed in your machine, you just pick the one you want. It installs everything you have listed in requirements of TXC and this is the exact same format you get with pip freeze. It installs those inside the virtual environment, then it takes your project and runs set up the py install on that. So it just doesn't dump your sources in there, actually installs your project inside the virtual environment and then it does a bunch of magic which is like set scripts where big thanks actually goes to Hinek about those and other stuff like rewriting, activate, so that you can actually run like instead of having all your build system paths, it will contain your production system paths and you can use things like activate or activate this in the production and end up with the same virtual environment. So that's nice and cool. But let's see. Let's take a project. Let's package something with DHR term. Let's package sentry. So who here knows what sentry is? Cool. It's a really good exception tracking tool. We use it in our production systems. It works like a charm. The best part for the example part is that when you install sentry, if you have ever done pip install sentry, it pulls down half the Python package index. It pulls like what ever you can find. It depends on a lot of stuff. It's not because it's bad software. It's because it's a complex software in a good way. That sounds a bit bad. Well, anyway, let's see how we do the DHR term. So first step is that we need to install DHR term. If you're running a modern operating system like Ubuntu trustee or the Debian testing, the DHR term is actually available inside those repos. You can just say apt-get install DHR term. And all of a sudden you have the DHR term available in your system. It's as previously discussed, a bit old on Debian and Ubuntu, but it still works. Then you need to create a Debian directory inside your sentry installation. So Debian directory is a custom directory Debian uses to figure out what stuff should it, like from that it figures out what package should it build, what stuff should it run on the build time. In that directory you have to create a few files. There's a minimum set of four files that you need to create there. And don't be afraid. All of these are covered in a tutorial also. So when you start packaging your sentry, you add a control file. This is the place which Debian uses to figure out what does it need for building. So you can see that sentry requires the Python development headers for building. But for running it, it doesn't require anything special. So in this place you have to just build dependent DHR to run and then write the required dependencies like Python and stuff like that in there. But this is basically how I did this today when I built the sentry with the DHR challenge that I just copied over the tutorial stuff and changed the fields that I felt need to be changed, like package names. Then you need a change log. The change log is a, well, it's a change log, but it's the file that is required for the Debian package to figure out some version. And it just tells Debian that we are packaging sentries 6.4.4. Cool. And the third one. This is why the packaging is the complex black magic thingy. The third one you need to define what is called the compatibility level so that Debian knows how we should build the package. What's relevant to you guys is probably just the echo 9 in that combat file you're done with that. If you don't do that, it will pick up some engine compatibility level and won't build your package. So that's pretty much it. And the last part is the glorified make file, aka rules file, which just tells how you build stuff. Now, if you build Debian packages before for Python, you probably recognize this file, and you can just see that we changed the Python 2 to be Python virtual end. This basically tells Debian that built this package using DH virtual end instead of the default way of building Python stuff. And that's it. If I are D package, build package, and it rolls through, you get the nice matrix-like output of stuff building, and all of a sudden you see, hey, look at that. It's actually great to virtual environment, puts pip, whatnot in there, and starts pulling half the internet down into your package. And once that's done, all that's left for you is just take the Debian package, copy it on your production host, and install it there. Now if you have defined some build dependencies in the control file, they get installed at the same time. And the best part is that you haven't executed any random scripts on your production system because all of them were done in your build system. You end up with deploying the whole thing without cluttering any production system with any development here, stuff like that. And you have a nice contained virtual environment in your production host. Okay. So once you've done that, let's look at the kind of nice parts of the DH virtual end then. So what it gives you is that it gives you the non-Python dependencies or the possibility to define non-Python dependencies just the way you could do with normal Debian packages. It also leverages on the existing infrastructure of Debian building. So you can use your existing build agents, you can use your existing CI systems if you're already using them for Debian, or you can have your own app triple still in use. It has the new hotness. So it's not limited by what you can find on Debian. You can actually just install what's the newest stuff available. And as I told before, it's contained. So we end up with the virtual environment in certain place in your production system and that's it. Of course, like any solution, there's also some negative sides to this. The build times can be slow. So especially if you're not running your own PyPI mirror and you're not using wheels, you're basically downloading all the requirements from the internet and then building them, which means that, yes, your build time will become longer. But it can be also substantially mitigated by running your own mirror which cuts down the network latency and using wheels which cuts down the build times. It still requires you to dig some requirements. So you need to know what requirements you need, what native system requirements you need to have on those systems. It doesn't let you out of that loophole. So if you're running, let's say, you're parsing XML using LXML, you need to make sure that your control file depends on installing LIP XML on your production system as well as having the development headers for the build part. And the build system needs to have the exact same Python. So like, because it's virtual environment, what virtual end does, it actually links outside the virtual environment for, well, link or link stuff on the production system. So you need to have the same Python available on your build system as you have on your production system. But it rarely is a problem if you are already having an existing build that's pitching stuff. For the future of the virtual end, I'm at least trying to add some of this stuff. Like I'm looking into cookie cutter templates. Unfortunately, I didn't have time to prepare them before this talk. But it would be sweet if you could just use cookie cutter and boom, you would have your DH virtual end packaging done without you, you don't need to go and echo nine on random files. I'm planning to add a trigger support, like if you get a minor update of your Python on your production host, it refreshes the virtual environment to make sure it still runs. And I'm also, this was actually a tip from, damn, I'm so bad with names, from Adam, from this, because when he was rehearsing his talk and we talked about this and he gave me a tip, like what if I could actually break out the dependency on system Python, like use PyEnd or something to incorporate the whole Python into the virtual environment. But that's pretty much it. If you want to find out more, the source is available, open source, under the Spotify umbrella in GitHub, and there is the good, emphasis on the word good, documentation, on the read the docs, and then there's a blog post, we posted when we released this. So with that, I thank you for your time. Thank you, Joachim, you've just saved me a lot of time. Any questions to the microphones, please? Thanks very much for that. The other time I was looking at your presentation, not sure, but your colleague from Spotify about using Docker for the deployment and managing the infrastructure, I myself am right now struggling between Docker for dev and production, and right now in my company we are building the Debian packages, actually, so this would be really, really cool to use. Do you actually do that you install some software based on this Debian packages and some other deployments based on Docker, or do you mix it? The current plan with Python stuff is to mix it. While Docker is great, it still doesn't solve the problem of defining dependencies, so it basically boils down to two options. Either you write a Docker recipe that says, like, pip install this, pip install that, apt-get install this, which kind of does the trick, but then you have the same problem that you would probably need two different Docker images, one for building your package somehow then extracting that out and putting it into the other Docker image. So the benefit of the Docker is that you don't necessarily need the virtual environment because the Docker already provides the isolation. But we are leaning more and more towards to build Python software with this and then use the Debian package to dump that one into the Docker images. And my other question is, what do you use for your local APT repositories? I have no idea, but it's something fairly off the shelf. I looked at it, but it's like, it's ancient installation. Hi. Oh. Yeah, I've got a question. Does that support any great mechanism as well? Can you repeat the question? Yeah. Does your system have a kind of great mechanism? If you want to, I know, like, let's take the example of Sentry and let's say you database needs some migration and you want your package to be able to provide some, I don't know, whatever type of a great, your packaging need to have some changes from one version to another. So, because it's a Debian package, so it will have all the post installation scripts and stuff like that that you can run. So if you want to run scripts before removal, after removal, before installation and so forth, you can use the existing Debian infrastructure for that one. So it's not too complicated to do those, but of course, we require some knowledge of the Debian packaging at that point. Okay. One question related to that. Do you use these post installation, pre-installation files for database migrations? So you said earlier you use something different, but what do you use there? We do have, we actually do have some project that used the post-instit files for database migration, but it's kind of, I feel it's kind of scary. Like if something accidentally triggers a package update and we get like an unwanted migration, even if it would be tested safe. So usually we do that in database migrations with some sort of manual steps, depending on the project. But yeah, you can do post-instit data. That project has been working. Fingers crossed with the future. Can you use system Python packages or everything is installed in the virtual world? For example, Alex and Pilo. Currently by design, everything is installed in the virtual environment. But it shouldn't be too hard to add a feature to the virtual world where you can shoot in your both legs at the same time to allow the system packages to be. Because yeah, I see your point because Alex or Pilo are kind of annoying. Alex is fairly easy, but Pilo is really annoying to install inside virtual environments. So yeah, it's probably something for future to add a flag that if you want to use the site packages or the existing ones, then why not? But currently no. Maybe you can remove from requirements TXT and just add to the pens. Yeah. But then if you add it to the pens, it gets installed on the system level. But the virtual environment is filled with no site packages flag. So it won't see them. One intermediate step you could take is that you could depend on Pilo, let that install on the system level, and then you could use Python path when you start your software to point that in addition to the virtual environment's Python paths. So that could work, but I wouldn't go for, I wouldn't say sure if it's viable solution or not. But nevertheless, like if you want that feature, please open a ticket in a GitHub. It should be fairly simple to implement, so I can just build that. Thank you. Or if you want to make a pull request, that's even better. Sorry. Did you hear about FPM, so-called FV package management tool? Yeah. So this is a Hinex blog post, which is actually doing this exact same stuff, but using FPM instead of injecting the depth helper sequence. I wanted to use that originally. It works. It gets the job done, but it didn't fit our build systems. So then I ended up building a depth helper sequence instead. Okay. But if someone starts from scratch and doesn't have an legacy that he wanted to use, then how, why he could use your product instead of FPM? What's the advantages for him? I don't know if there's any specific reason for it. In that case, I would say go read Hinex blog post, check out the excellent documentation of my project, and decide on which one seems to be simpler. So I've aimed to cater people who don't really know the PMPackaging at all, so that it should be easy. But it's the same with Hinex blog post. You just follow steps and do stuff. So it's rather a matter of personal preference rather than features. Yeah, pretty much. Okay. Thanks. Just a quick one. Is there any plan on your side to port it to Weezy, for example, using back ports that even the log? Yeah, that could be done. I'm already planning to, like, because trust is having 0.6, so I'm planning at least to set up a PPA for trusty so that you can, the newer releases on trusty. This stuff builds fine on Weezy, so I've built it on Weezy, but Weezy was already stable at that point when I released this. So it shouldn't be too hard to add it to the back ports for Weezy. Okay, so there are no technical reasons not to do that? No. Should work? No, this is really simple. And, like, we build stuff with DH version on top of squeeze. So if it doesn't work on Weezy, then we've done something really bad wrong in that case. Okay, thanks. Thank you. I think that's all the questions. Thank you for a very clear, very useful talk, Joachim. Thank you. Thank you.
 Okay, no, Euro Python or any Python conference is complete without at least one talk on  packaging and this must be at least a second.  So let's see what your key has to tell us.  Thank you.  If you don't mind, I'm going to wait for a tiny bit for clock to hit three, four and  then just depressed.  No one comes.  Before you can run away, I'm going to take a photo of you.  This is a great in a zero chance that it happens when I start speaking.  Okay.  Good to talk, Euro Python or Helsinki will come in.  That's my outcome of two years of German studies while in elementary school.  Unfortunately I can't do better.  But welcome everyone to a talk about a tool called DH virtual end or as I label it packaging  in packaging.  But before we look into that deeper, let me introduce myself.  So my name is Jurke Poulienen.  I'm from Finland but I'm living in Stockholm, Sweden nowadays.  I work for a music streaming company called Spotify.  In there I do kind of do two stuff.  I am a content engineer.  So I build like a pipeline of new music to the service and on the other hand I also fiddle  around a lot with our internal Python stack and answer people's questions about Python.  If you want to reach out to me, there's my email address and there's my Twitter handle.  So please do if you have any questions.  Now about this talk.  This talk will be in three different sections.  First we're going to look into some of the existing deployment strategies you can use  on the PyNU-based machines.  Then we're going to look into what actually is DH virtual environment and how does it  differ from the existing deployment strategies.  And in the end we're going to go through an example of how you package software and we're  going to package Sentry mostly because it's not the simple piece of software.  So I'm going to show you by example how you can use DH virtual environment to package  something like Sentry in production.  Now let's start with that tiny quiz.  Who here runs a Debian or Ubuntu-based system?  Good.  You're from the correct token.  Who here deploy stuff using the so-called native packages on Debian system, like relying  on the native libraries on those systems?  Who here is kind of frustrated with that?  Yeah.  Okay.  Who here uses virtual environment?  Like you set up a virtual environment and then you install everything in there on your  production host.  Cool.  Now these both have their good sides too.  If we take for example the native Debian packages.  So this table, the stuff that gets in the Debian, especially stuff that's in main, is  stable, it's well tested.  People usually don't change them so that the backwards compatibility breaks.  So if your Debian system gets an update for let's say Python requests, you know that that  request is something that's backwards and compatible with the previous version.  Now Debian has enough or Debian packaging has another nice bone side.  You can declare non-Python dependencies.  So say your software requires SQLite to be installed or your software requires MySQL  to be installed on the same machine.  When you create the Debian packages, you can say that your package depends on MySQL and  you get that one installed on the machine.  Everything nicely contained in one package.  It also has pretty neat existing infrastructure.  So not only you have like dedicated build tools or separated build environments like  sbuild and other CH root solutions, but you also have the possibility of running your  own app through repository which means that you will have your own way, or like own network  of deploying your stuff and production.  Plus all kinds of CI tools like Jenkins or Team City have at least some rudimentary support  for dealing with Debian packages.  And the last good thing I think in the Debian packaging is that you have a quite nice scripting  support.  What that means is that if you need to remove a cache when you upgrade your package version,  you can write the script that before you upgrade your package, it will clear out the cache.  Or if you want to restart your service after you've installed it, you can write the script  that after installation, it will restart your service.  You can do crazy borderline stupid stuff like database migrations and postings.  I don't necessarily recommend them, but you can do all kinds of stuff there.  It gives you quite a lot of power.  Now that was the good parts.  Then if we start talking about the bad parts, you probably have run into this case where  you see that, oh, let's say back in the days, can network hard and push out request 1.3,  and that has just the feature you need, but unfortunately your ancient web-filled box  is running request 1.2.  And what do you think?  Are you going to wait for Debian to package that?  What happens, you're just going to rot in front of your computer waiting for the newer  request, especially if it's a backward incompatible to come on the current system you're running.  So a lot of stuff in Debian, even at the release of that particular Debian or Ubuntu release,  is already outdated.  The packaging itself is kind of complex.  So what we're talking here is that someone created a packaging system that is built for  building a whole operating system.  So it covers all the possible corner cases, all the possible scenarios, if you want to  deploy Perl, Haskell, Python, you name it, it has everything covered.  That means that the whole system is really, really complex and the documentation is far  from simple.  In addition to that, all the documentation that you usually can find is also geared towards  Debian package maintenance.  So okay, you want to ship this thing within the operating system.  So this is how you should do.  And it does not necessarily resonate to how you yourself would deploy your tiny service  in the host.  And what I think is the worst part in Debian packages is that you get a global state.  So if you're only deploying all your libraries as Debian packages and then using them, eventually  you'll end up having a case where you would like to upgrade one library, but it's also  used by some other software on the same box and you don't know if you can do it without  breaking anything.  We've had this at Spotify a few times when we have rolled out like new common Python  utils libraries on the host and we're just too afraid of deploying that because we might  or kind of afraid that something down the line might break even we have been testing  it for months.  So it kind of slows you down and it's really, really annoying.  Now if you think about virtual environments then, so what you get in the virtual environment  is somewhat the opposite.  So you get the whole new stuff.  You just do pip install and you get whatever is available in PyPI, like the latest release.  And you can go to the extent of like you can open your Git or Mercurial and pull the stuff  in your virtual environment and run it in there.  So you can always get the new, hottest stuff.  It has become kind of a de facto method in the Python world.  So every guide usually contains a word or two how you run stuff inside the virtual environment.  You can do the same virtual environment stuff on your laptop as you can do on your servers.  It kind of works.  It's also battle tested.  So nowadays so many people are running it in production so it's fairly safe to use that  one.  But the best part I think is that it's contained.  So if you take a package and install it inside the virtual environment, a Python package,  you can be sure that it won't affect anything outside the virtual environment.  So updating a simple packages doesn't mean that your whole system crumbles because something  was relying on all this older version or that you would interfere with the underlying operating  system.  No, you're only poking the actual virtual environment and you get this nice contained  fuzzy feeling when dealing with that.  Now on the downside, I bet some of you might have seen this line.  That means that you can't have any native dependencies if you're using virtual environments  and people install.  You need to know what MySQL libraries you have to have available.  Like in this case, you need to know that you have to install MySQL client to find this  MySQL config on your host.  So it requires you to do some manual digging through things.  Even more, you end up doing source installs.  So you can have wheels or even eggs to avoid source installs.  But if you don't have wheels for your platform or you haven't set up your own wheel riposter,  you end up doing source installations.  When you have your virtual environment in your production server, you end up installing  all the dependencies for that source installation on that production environment, which is not  necessarily bad in the sense that you could break something, but you'll just clutter your  production environment with developer headers and other unused files.  But the worst part, what I think is with the PIP installs is that you're basically executing  a bunch of random scripts.  So sure, wheel gets around this, but if you run set up the PIP install, you probably haven't  looked into what all those files do or what all those files that those packages depend  on do.  So you're just relying on a good faith of people and you're running random stuff in  your production environment.  It doesn't even need to be malicious to hurt you.  Someone might just accidentally release a package that wipes your whole ETC or your whole  home directory or something and you by accident end up crippling your system.  Now this brings us to question then what is DH VirtualN.  So DH VirtualN was about two years ago my attempt to combine the best of the two worlds.  So it is a virtual environment that is placed inside the PN package.  It supports both Python 2 and 3.  It is kind of version agnostic.  I won't say that you can't install Python 1 stuff with that, but it doesn't execute or  import any Python code.  So it doesn't really care if your tool is written with Python 2 or Python 3.  You can use it anyway.  It even supports using the new virtual environment package with 3.3 so you don't even need to  install virtual N to run it.  It's also open source.  So it's GPL like all the different build tools.  It has a good documentation.  Now I'm the guy who wrote it.  So I might be a bit biased here.  But I think the documentation is good.  It's at least better than the average open source documentation.  But the best part which seems to be actually very functional is that it has a simple tutorial.  So if you go to the DH VirtualN.readthedocs.org you find there's a four step tutorial that  you can run through and boom, your package is inside the virtual environment inside the  Debian package.  Under the surface it is a depth helper extension.  So depth helper is this...  How would you make it pretty?  Like a pile of purl scripts that Debian executes when you're building packages.  It's a certain fixed sequence of purl scripts that you install and then there are different  extensions for Debian to know how to build packages for Python packages or how you build  purl packages or how you build bash completion stuff.  So what the virtual environment does, it just injects itself into that flow.  So there's 12 lines per included there that does the magic to inject the virtual environment  there and then it just runs as a part of the sequence.  Now this is kind of like the implementation details but for you who already have existing  Debian build environments, like if you're using sbuild or something else or just playing  the build or the package build package on the command line, this means that using the  virtual N is just going to work in your existing workflow.  So what I basically did back in the days was that I found a great blog post by Hinek Slavak.  I adapted the idea a bit to fit our built environment and that's why I ended up writing  depth helper extensions.  Hinek system works well too but it uses a thing called FPM which back in the days didn't  fit our build system.  Now in practice, the DH virtual environment is a packaging builder that it creates a  virtual environment, you can define what Python you want to use with it.  So if you have multiple pythons installed in your machine, you just pick the one you  want.  It installs everything you have listed in requirements of TXC and this is the exact  same format you get with pip freeze.  It installs those inside the virtual environment, then it takes your project and runs set up  the py install on that.  So it just doesn't dump your sources in there, actually installs your project inside the  virtual environment and then it does a bunch of magic which is like set scripts where big  thanks actually goes to Hinek about those and other stuff like rewriting, activate, so  that you can actually run like instead of having all your build system paths, it will  contain your production system paths and you can use things like activate or activate this  in the production and end up with the same virtual environment.  So that's nice and cool.  But let's see.  Let's take a project.  Let's package something with DHR term.  Let's package sentry.  So who here knows what sentry is?  Cool.  It's a really good exception tracking tool.  We use it in our production systems.  It works like a charm.  The best part for the example part is that when you install sentry, if you have ever  done pip install sentry, it pulls down half the Python package index.  It pulls like what ever you can find.  It depends on a lot of stuff.  It's not because it's bad software.  It's because it's a complex software in a good way.  That sounds a bit bad.  Well, anyway, let's see how we do the DHR term.  So first step is that we need to install DHR term.  If you're running a modern operating system like Ubuntu trustee or the Debian testing,  the DHR term is actually available inside those repos.  You can just say apt-get install DHR term.  And all of a sudden you have the DHR term available in your system.  It's as previously discussed, a bit old on Debian and Ubuntu, but it still works.  Then you need to create a Debian directory inside your sentry installation.  So Debian directory is a custom directory Debian uses to figure out what stuff should  it, like from that it figures out what package should it build, what stuff should it run  on the build time.  In that directory you have to create a few files.  There's a minimum set of four files that you need to create there.  And don't be afraid.  All of these are covered in a tutorial also.  So when you start packaging your sentry, you add a control file.  This is the place which Debian uses to figure out what does it need for building.  So you can see that sentry requires the Python development headers for building.  But for running it, it doesn't require anything special.  So in this place you have to just build dependent DHR to run and then write the required dependencies  like Python and stuff like that in there.  But this is basically how I did this today when I built the sentry with the DHR challenge  that I just copied over the tutorial stuff and changed the fields that I felt need to  be changed, like package names.  Then you need a change log.  The change log is a, well, it's a change log, but it's the file that is required for the  Debian package to figure out some version.  And it just tells Debian that we are packaging sentries 6.4.4.  Cool.  And the third one.  This is why the packaging is the complex black magic thingy.  The third one you need to define what is called the compatibility level so that Debian knows  how we should build the package.  What's relevant to you guys is probably just the echo 9 in that combat file you're done  with that.  If you don't do that, it will pick up some engine compatibility level and won't build  your package.  So that's pretty much it.  And the last part is the glorified make file, aka rules file, which just tells how you build  stuff.  Now, if you build Debian packages before for Python, you probably recognize this file,  and you can just see that we changed the Python 2 to be Python virtual end.  This basically tells Debian that built this package using DH virtual end instead of the  default way of building Python stuff.  And that's it.  If I are D package, build package, and it rolls through, you get the nice matrix-like output  of stuff building, and all of a sudden you see, hey, look at that.  It's actually great to virtual environment, puts pip, whatnot in there, and starts pulling  half the internet down into your package.  And once that's done, all that's left for you is just take the Debian package, copy it  on your production host, and install it there.  Now if you have defined some build dependencies in the control file, they get installed at  the same time.  And the best part is that you haven't executed any random scripts on your production system  because all of them were done in your build system.  You end up with deploying the whole thing without cluttering any production system with  any development here, stuff like that.  And you have a nice contained virtual environment in your production host.  Okay.  So once you've done that, let's look at the kind of nice parts of the DH virtual end then.  So what it gives you is that it gives you the non-Python dependencies or the possibility  to define non-Python dependencies just the way you could do with normal Debian packages.  It also leverages on the existing infrastructure of Debian building.  So you can use your existing build agents, you can use your existing CI systems if you're  already using them for Debian, or you can have your own app triple still in use.  It has the new hotness.  So it's not limited by what you can find on Debian.  You can actually just install what's the newest stuff available.  And as I told before, it's contained.  So we end up with the virtual environment in certain place in your production system  and that's it.  Of course, like any solution, there's also some negative sides to this.  The build times can be slow.  So especially if you're not running your own PyPI mirror and you're not using wheels, you're  basically downloading all the requirements from the internet and then building them,  which means that, yes, your build time will become longer.  But it can be also substantially mitigated by running your own mirror which cuts down  the network latency and using wheels which cuts down the build times.  It still requires you to dig some requirements.  So you need to know what requirements you need, what native system requirements you need  to have on those systems.  It doesn't let you out of that loophole.  So if you're running, let's say, you're parsing XML using LXML, you need to make sure that  your control file depends on installing LIP XML on your production system as well as  having the development headers for the build part.  And the build system needs to have the exact same Python.  So like, because it's virtual environment, what virtual end does, it actually links outside  the virtual environment for, well, link or link stuff on the production system.  So you need to have the same Python available on your build system as you have on your production  system.  But it rarely is a problem if you are already having an existing build that's pitching stuff.  For the future of the virtual end, I'm at least trying to add some of this stuff.  Like I'm looking into cookie cutter templates.  Unfortunately, I didn't have time to prepare them before this talk.  But it would be sweet if you could just use cookie cutter and boom, you would have your  DH virtual end packaging done without you, you don't need to go and echo nine on random  files.  I'm planning to add a trigger support, like if you get a minor update of your Python on  your production host, it refreshes the virtual environment to make sure it still runs.  And I'm also, this was actually a tip from, damn, I'm so bad with names, from Adam, from  this, because when he was rehearsing his talk and we talked about this and he gave me a  tip, like what if I could actually break out the dependency on system Python, like use  PyEnd or something to incorporate the whole Python into the virtual environment.  But that's pretty much it.  If you want to find out more, the source is available, open source, under the Spotify  umbrella in GitHub, and there is the good, emphasis on the word good, documentation,  on the read the docs, and then there's a blog post, we posted when we released this.  So with that, I thank you for your time.  Thank you, Joachim, you've just saved me a lot of time.  Any questions to the microphones, please?  Thanks very much for that.  The other time I was looking at your presentation, not sure, but your colleague from Spotify  about using Docker for the deployment and managing the infrastructure, I myself am right now  struggling between Docker for dev and production, and right now in my company we are building  the Debian packages, actually, so this would be really, really cool to use.  Do you actually do that you install some software based on this Debian packages and some other  deployments based on Docker, or do you mix it?  The current plan with Python stuff is to mix it.  While Docker is great, it still doesn't solve the problem of defining dependencies, so it  basically boils down to two options.  Either you write a Docker recipe that says, like, pip install this, pip install that,  apt-get install this, which kind of does the trick, but then you have the same problem  that you would probably need two different Docker images, one for building your package  somehow then extracting that out and putting it into the other Docker image.  So the benefit of the Docker is that you don't necessarily need the virtual environment because  the Docker already provides the isolation.  But we are leaning more and more towards to build Python software with this and then use  the Debian package to dump that one into the Docker images.  And my other question is, what do you use for your local APT repositories?  I have no idea, but it's something fairly off the shelf.  I looked at it, but it's like, it's ancient installation.  Hi.  Oh.  Yeah, I've got a question.  Does that support any great mechanism as well?  Can you repeat the question?  Yeah.  Does your system have a kind of great mechanism?  If you want to, I know, like, let's take the example of Sentry and let's say you database  needs some migration and you want your package to be able to provide some, I don't know,  whatever type of a great, your packaging need to have some changes from one version to another.  So, because it's a Debian package, so it will have all the post installation scripts and  stuff like that that you can run.  So if you want to run scripts before removal, after removal, before installation and so  forth, you can use the existing Debian infrastructure for that one.  So it's not too complicated to do those, but of course, we require some knowledge of the  Debian packaging at that point.  Okay.  One question related to that.  Do you use these post installation, pre-installation files for database migrations?  So you said earlier you use something different, but what do you use there?  We do have, we actually do have some project that used the post-instit files for database  migration, but it's kind of, I feel it's kind of scary.  Like if something accidentally triggers a package update and we get like an unwanted  migration, even if it would be tested safe.  So usually we do that in database migrations with some sort of manual steps, depending  on the project.  But yeah, you can do post-instit data.  That project has been working.  Fingers crossed with the future.  Can you use system Python packages or everything is installed in the virtual world?  For example, Alex and Pilo.  Currently by design, everything is installed in the virtual environment.  But it shouldn't be too hard to add a feature to the virtual world where you can shoot in  your both legs at the same time to allow the system packages to be.  Because yeah, I see your point because Alex or Pilo are kind of annoying.  Alex is fairly easy, but Pilo is really annoying to install inside virtual environments.  So yeah, it's probably something for future to add a flag that if you want to use the  site packages or the existing ones, then why not?  But currently no.  Maybe you can remove from requirements TXT and just add to the pens.  Yeah.  But then if you add it to the pens, it gets installed on the system level.  But the virtual environment is filled with no site packages flag.  So it won't see them.  One intermediate step you could take is that you could depend on Pilo, let that install  on the system level, and then you could use Python path when you start your software to  point that in addition to the virtual environment's Python paths.  So that could work, but I wouldn't go for, I wouldn't say sure if it's viable solution  or not.  But nevertheless, like if you want that feature, please open a ticket in a GitHub.  It should be fairly simple to implement, so I can just build that.  Thank you.  Or if you want to make a pull request, that's even better.  Sorry.  Did you hear about FPM, so-called FV package management tool?  Yeah.  So this is a Hinex blog post, which is actually doing this exact same stuff, but using FPM  instead of injecting the depth helper sequence.  I wanted to use that originally.  It works.  It gets the job done, but it didn't fit our build systems.  So then I ended up building a depth helper sequence instead.  Okay.  But if someone starts from scratch and doesn't have an legacy that he wanted to use, then  how, why he could use your product instead of FPM?  What's the advantages for him?  I don't know if there's any specific reason for it.  In that case, I would say go read Hinex blog post, check out the excellent documentation  of my project, and decide on which one seems to be simpler.  So I've aimed to cater people who don't really know the PMPackaging at all, so that it should  be easy.  But it's the same with Hinex blog post.  You just follow steps and do stuff.  So it's rather a matter of personal preference rather than features.  Yeah, pretty much.  Okay.  Thanks.  Just a quick one.  Is there any plan on your side to port it to Weezy, for example, using back ports that  even the log?  Yeah, that could be done.  I'm already planning to, like, because trust is having 0.6, so I'm planning at least to  set up a PPA for trusty so that you can, the newer releases on trusty.  This stuff builds fine on Weezy, so I've built it on Weezy, but Weezy was already stable  at that point when I released this.  So it shouldn't be too hard to add it to the back ports for Weezy.  Okay, so there are no technical reasons not to do that?  No.  Should work?  No, this is really simple.  And, like, we build stuff with DH version on top of squeeze.  So if it doesn't work on Weezy, then we've done something really bad wrong in that case.  Okay, thanks.  Thank you.  I think that's all the questions.  Thank you for a very clear, very useful talk, Joachim.  Thank you.  Thank you.
Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_0_659.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_660_2591.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_2592_4451.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_4452_4511.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_4512_6395.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_6396_6503.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_6504_9395.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_9396_9443.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_9444_9671.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_9672_12671.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_12672_12947.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_12948_14243.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_14244_15179.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_15180_15383.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_15384_15575.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_15576_16079.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_16080_17003.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_17004_17063.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_17064_17255.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_17256_17495.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_17496_17687.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_17688_18119.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_18120_18179.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_18180_19031.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_19032_19247.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_19248_19859.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_19860_19943.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_19944_20243.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_20244_21371.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_21372_21635.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_21636_23891.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_23892_24071.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_24072_24503.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_24504_25115.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_25116_26819.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_26820_27227.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_27228_27479.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_27480_29591.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_29592_29867.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_29868_29927.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_29928_30587.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_30588_31703.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_31704_32375.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_32376_32447.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_32448_33095.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_33096_42551.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_42552_42647.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_42648_42719.png Jyrki_Pulliainen_-_Packaging_in_packaging__dh-virtualenv_42720_47340.png