Difficulties of Python code development: packages,virtualenvs and package mangers
CC Attribution - NonCommercial - ShareAlike 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal and non-commercial purpose as long as the work is attributed to the author in the manner specified by the author or licensor and the work or content is shared also in adapted form only under the conditions of this
Wodyński, Michał
In this presentation I will show different packages that are used in Python. I will point out differences and explain the prons and cons of using them during code development. After that we will jump to the topic of virtualenvs and popular tools that are used for managing them. I will explain what is purpose of the virtualenvs in Python and why we should use them. Finally we will focus on the most important topic without which development is not possible – package mangers. Package managers it is wide topic in Python world. There is many package mangers and currently we have tools like pip, pipenv or poetry but it is not obvious which of them we should use in first place. Which of them are recommended to use? What are prons and cons of certain package mangers? Is there any other not well known packages mangers? How package mangers are packaging Python code? Which of the package mangers are good for data scientists? Is your package manger is resolving dependencies? This questions can appear especially when you are beginner and just to want start working with Python. Unfortunately form the beginning you must face the problem which of the package manger you should use. In this presentation I will answer all of this questions. I will list popular package mangers and some not known. I will show differences and which of them are best for which case. Also I will explain differences between packages that are used by different packages mangers. If you are interested in code development and package mangers or you are just confused which of them you should use this presentation is definitely for you. During this presentation I will answer all questions and I will wipe out all doubts about packages, package mangers and virtualenvs.
 All right, okay. And what are you going to talk about? Today I will be talking about difficulties that we can face during development of Python code. I will share my screen. All right, so give a big hand to Nihal. Cool. Okay, so before I start, maybe a few words about me. I have eight years experience in Python, five years commercial experience in developing Python code. And today I will share my knowledge and best practices and best tools that I'm using during project development. My presentation will be divided in three main sections. Each of them will be covered with a small story. And first of all, we will start with the packages that are available in Python world. We will have a little bit about the history of Python and confusions related to that. Then we will switch to the virtual ads. We will tell what are for and some great tools that are allowing, developing our Python code in virtual and then we finally focus on package managers. We will try to clarify all what are, which are the best and what are the advantages and disadvantages of certain package managers. But before that, we have to say that the Python world packages, virtual ads and package managers is looking like this. Because of the history, we have some old documentation that is pointing for packages that are not used. The same thing is for package managers, which we, which is not obvious which of them we should use. And today I will try to clarify what is the right path. And I hope that this picture will be much simpler. So in the beginning, we will start with the story related to AWS. Recently, I have a few projects related to the, that were stored on AWS and mainly I was working with Lambda, AWS Lambda functions. And before we start our work, we was choosing out of many tools how to deploy the Lambda. We could use Terraform, serverless framework, but those three cloud formation and many, many, many other tools. Finally we decided for Terraform. To use Lambda, the deployment of Lambda is looking like this. We have to prepare the zip file with our Python code. And we also have to add the libraries that were required for example to communication with database, like a school alchemy, because Lambda itself contains only base libraries of Python. And during development, we noticed that we have a problem that Terraform is constantly building our code on Lambdas. And the reason of that is, was using different wheels, because we, in the beginning, we was doing this by hand and each developer was using different platform. And what is wheel, I will tell in within few slides. And also I will tell how we fixed this issue. But let's start from the beginning. We have Python files and we can share them between community. For example in the framework, it is possible to create a small web page for deployment and we can share this file. But it is not convenient to use that. We have to use something more convenient and reliable. And first thing that came across is a source code. And for that, it is just simply the archive that is containing the Python code. It may contain other source files like C source files for C Python. We can define the scripts that will be available for our terminal. We can define, of course, Ritme, the setup.py or Cfg. I will tell a little bit about setup.py later. And also we can define some external packages, package data like, for example, the trained model of artificial neural network. And of course, metadata to know what we have installed in our system. And here we are coming to the conclusion that each time we want to use that, especially if we have the C source files, we have to compile that. This takes time and we would like to have the instant package that we will be ready to use. And this can be offered by binary packages. The first approach was X in Python. They were introduced with the setup tools in 2004. There was no need for building our compilation. It was a distribution format and runtime installation format. And the first approach of listing what we have installed in our system was a infotainment directory where we had all necessary formations, what we have installed in our system. The funny thing about X, it was possible to install many versions of certain library. And of course, it was the possible to import directly the X. So what was the problem with them? There actually wasn't official PEP for that. And each package manager, package maintainer has its own directories inside X. And we in Python community, we had to make something with that and make some standardization. And for that, Wills came across. This also is a binary format. It was introduced in 2012 together with PEP 427. With Wills, we have also standard for distribution of the Wills and standardization of the metadata. So instead of a Gimli4, we now have different directory that contains different metadata. And this is the current standard for building the binary. Wills are not containing the Python bytecodes, but may contain other precompiled code, for example, for C-Python C libraries. As I mentioned, instead of egg info, it is used in this directory. And the very big advantage of Wills is that they are versions. They have the name convention. In the name, we can find the Python version that is supported. The certain implementation, because the Python can have different implementations. Also there is information about application binary interface, which is different in Macintosh, Linux and Windows, and for which the architecture is built, certain Wills. For example, Intel and AMD or other architectures, and of course, whether it's for 32 or 64 bits. For now, some Wills are importable directly, for example, for our rep. But it's not supported by PEP 427 itself. And here we can see the example Wills. The first is a new version of the PEP library, which is for Python 2 and 3. And this is one of the NumPy Wills, which is for C-Python 3.6, for Macintosh 10.9, for both Intel architectures, 32 and 64 bits. Currently, there are no builder on the PyPy. The maintainers just have to create all necessary Wills. We can create our Wills by this command. But there is a great talk about Wills by Elena Hashman, if you want to know about application binary interface in Linux, the ELF format, and how they are maintained and built. You can just watch it by clicking on the links provided here. And she's also supporting the projects, the Manelinox and AuditLit, the Manelinox project is actually the group of the most popular Linux distribution in Docker that are building Wills for this certain distribution. And AuditWill is a command line tool that is used for checking whether this will is proper for certain distribution and nothing is missing. So if you want to know more, just click on the links. And this is the end of the first section. We'll move to the virtual ends. Oh, okay. One more thing. Just to show how the package is built. In the most deep layer, we have the standalone model, which are Python file. We have more Python files. We can have a source code. And about that, if we compile that, we have the wheel for certain platform. So this is the summary of the packages. Then we can switch to the virtual ends and next story. During my journey of the development of the Python code on the studies, I have many projects. And I installed Ubuntu on my laptop and I started developing many projects at once. I started installing many libraries to the global system Python and I noticed that suddenly I noticed that such picture like here. And the problem is that when I was developing and installing many packages, I reinstalled one very important dependency that was used for a graphical engine of Ubuntu and I had to reinstall everything from the scratch. But it was very quick lesson for me why we should use virtual end. So what is exactly the virtual end? It is isolated directory where we have all necessary libraries and Python, which is completely isolated from the global system, which is good. And we have few tools to do that. One is the built into virtual end, which is refarctored now. There is a great talk. We had it today by Bernard Gabber writing the virtual end. I fully recommend to see this talk. And together with virtual end, there is a useful thing to use the virtual end wrapper, which provides useful comments for creating virtual end, deleting it, switching between them. And it is easier to develop the Python code with that. How the virtual ends actually working, they are putting the path of the certain and virtual end in the directory to the path before the global system and every time we are typing the Python, it is first looking in the virtual end, not in the global system. From today's presentation, I have heard there will be possibility to create virtual ends for different versions, which is very good. But they must be installed on the system. They must be present for virtual end. So in Python world, we have many implementations of the Python. We have the main Python in C Python, which is written in C. We have a Python, which is written in Python. We have a JTOM, which is written in Java. We have an Ion Python, which is written in C sharp. We have a micro Python, which is used for certain embedded systems. We have the stackless Python, which is not containing the stack. If we, for example, want to develop our library and test it against many versions and many, let's say, distributions of certain Python, we have to have this certain Python installed on our system. And there is a tool that is doing this, which is called PyEmp. There is possibility for installing many versions of Python, many implementations of Python. It is offering switching between them. There is an automation of switching between certain virtual end ends, creating the virtual end ends. Here I prepared a very useful command that can be used with PyEmp. And before that, I just want to tell you that the installation of PyEmp is very simple. If we want to install it, it just can be done by the script provided on GitHub, which is called PyEmp installer. If you don't trust this script, you can use the command one by one in the script. And very important thing, you have to install dependencies that are required for compiling the Python versions. And there is all dependencies accordingly to the distributions, which can be found here in the frequently asked questions. And be aware of that. If you want to, for example, use the Jton, you have to have Java on your machine. If you want to use C sharp, if Ion Python, you have to have the C sharp on your machine. And after that, if you prepare everything, you can just list all available, the PyEmp is installing in your home folder. And all versions are isolated from the global Python system and are just put it on the home folder. And if you want to list all available Python versions, you have to use this command. Sometimes there is a need for check out the GitHub repo locally just to update the newest version that appeared in community. To install certain version, we are using such command. We can uninstall certain version. And the same command is used for uninstalling environment. To list all available Python versions, we can use this command. And very important thing is that if we want to know where exactly our virtual and F is stored, we have to use the PyEmp which Python command. Because the normal command which Python will show the same directory, which is an engine used by the PyEmp. And the real path will be shown by this. By this command. Also, we can activate or deactivate the environment by hand. And there is a possibility to set some namespaces in our terminal of availability of the certain version of the Python. The hierarchy of this command is like this. If we want to test some Python version, we can use the command PyEmp and this command. We can attach certain version or environment to the directory. When we switch inside to this directory, we will have our environment. We can set up the PyEmp global which will be covering the system Python which is very good. But it will be shown the last in this hierarchy. I will show it in a quick demonstration. So here, this is my PyEmp versions. I have installed micro Python. I have mini-conv3. There is also anaconda for that. This is all versions that are available by PyEmp. We have the version of anaconda and Ion Python, j-tone, micro Python, mini-conv, PyPy, and so on and so on. Here, currently, my global system is 382 which is covering my system Python. For example, when I change my directory to the PyPyX directory, automatically there is activation of my virtual ENF. If I, for example, want to test the micro Python, I'm using the shell command which will be covering the environment from the directory and my global Python. If I just stop testing, I can unset it and I will be back for my global Python. This is very convenient during development. I very recommend to use that. Actually, that was the last slide of the second part of PyEmp. Then we are now switching for package manager which is a big topic and big confusion in Python community because there is a lot of choices that we can make before we start our project. This year, I start with my short story about dependency hell. In one of my projects, I have many libraries. Each of these libraries have many dependencies. Most of these libraries was using the same dependencies but in different versions. In our project, we are using the PyP package manager and we were using the requirements TXT file for reproducibility of the environment. After a few executions, we noticed that sometimes our application is not working because of the wrong dependency version and we were just wondering why it is happening. We noticed that in some cases, the order in requirements TXT was changing our dependency because the first library, for example, was using the Python, the library version 1.00. The second one was using 103. What was happening, the PPP was not resolving this dependency versions. It is just installing the first version. In the second line, it is uninstalling this first version and it is installing again the second version. We just wanted to know how we can manage that. We switched for different package manager and I will show you plenty of them. But before that, the short story, what was first, in the beginning of Python, we have easy install, which was using X for installing, it was provided with setup tools in 2004. It was using installing packages and its dependencies and there was no PEP for the easy install. With the standardization, the PEP came across and we find that the PEP was released in 2008. It is using wheels or source code and introduced the requirements TXT for freezing our environment and knowing what we have installed in our virtual environment. The good news is that from certain versions of the PEP, the latest versions of PEP, there is a possibility of checking the hashes of the versions of Python, so we have more security. But it has to be explicitly used. But as I mentioned, PEP is not resolving dependencies. So here is a package, here is a table that is comparing easy install together with the PEP. The main difference is that easy install is using X. There was a possibility to install many versions of certain library. There is no PEP for easy install. And for PEP, of course, there is PEP for 3.8. It is using wheels. And this is the main difference between them. So we have, but as I mentioned, there is still a problem with the resulting of dependencies. So before I move to this tool, to these tools, I will tell a short introduction to setuppy script, which is used for our packages and libraries that we want to install in our environment. We are feeling necessary information. And the setuppy script is providing the possibility of building the packages, distributing them to the PyPy and installing in our virtual ends. This is the example, a piece of code of setuppy, where we have the version, description, authors, packages. We can define additional non-Python packages like SVG files or other data. I don't want to focus on the setuppy in this presentation. It is, I just wanted to mention it because it's used in the next tool that I will be talking. So the PEP tools, it's a package manager that is using two commands, PEP compile and PEP sync. And it is resolving dependencies in certain way. It is using the setuppy and requirements in file for knowing, for resolving all dependencies. And after that, after compiling, we are, we have, we are getting the requirements TXT, where we have all resolved environments, all resolved dependencies, and we can reproduce our environment. The second command, PEP sync, is checking whether all necessary versions of libraries are installed in our environment. And it's just showing if there is some inconsistency. So here we are requiring three files for resolving dependencies. So if we would like, for example, to create it for a deaf test, UAT and Proth environments, we have to reproduce requirements in and requirements TXT, which is not so convenient. And then we have a different tool that is PEP-ANF. It is, it can manage the virtual environments. If we, for example, is installing something and we have not activated the virtual environment, it is automatically activating the random virtual ENF with the random name and installing it there. But it's, but when it's something, when the virtual ENF is activated, it is installing in the current virtual ENF. And it is using the PEP file and the PEP file lock files. And PEP file contains two sections. In the first section, there are development, local development packages that are used for creating the project and the production, the production libraries. And each time we are providing something, it is resolving automatically all necessary dependencies and it's telling whether there are conflicts that cannot be resolved. For example, one library is requiring the version 2.3 and the second library one dependency that will require version below two. So this is very useful tool for pointing out such situations. Of course, when we have the stable environment, we just lock in the environment, which then the PEP file lock is used just simply for installing our CI CDs, Docker images and so on, so on. Also, PEP is using our hashes, so we have the security. And there is an alternative for PEP ENF, which is also great, which is poetry. We use it for actually poetry has the same features as PEP ENF, but it's faster. I will show that. It's faster in resolving dependencies. So it is managing the different environments. It's also installing packages. It's resolving dependencies. But there is also the convenient way of creating packages and publishing into the PIPI, so it is more useful for libraries. And I will say a few words about a package manager that is not used, that is not pure Python, which is anaconda, which is the Python libraries plus some other data scientists libraries. The mini-conda is the smaller version of anaconda, and the conda is the package manager. The main problem with anaconda is that it's not supporting PIPI, it's supporting only their channels. They have different standards for their packages, so there is no compatibility between PIPI and channels, conda channels. And of course, conda package is not working with Python virtual elements. So be aware of that. It is mainly used for machine learning projects, and we have the possibility to use conda itself or conda with other package manager. If we use conda alone, we have to know that channels, we have to choose good channel because very often libraries are outdated. They are maintained by different people. And most safe is to use conda with other package manager, conda for some non-Python packages, and Python package manager for getting the latest libraries from Python. And this is mainly useful for machine learning. Also recently, there was a release of a PDM package manager, which is implementing PEP582. The PEP582 is talking about storing the libraries inside the project. There is no need for the virtual elements. Everything is with the project directory. And also it's implementing PEP5817, which is installing libraries from the directory. So this is a great feature. But it's on early stages. And finally, we can use sometimes the PEPX, which actually is not resolving the dependencies. It's installing one package per environment. And if you want to install a second package, it is installing in different in virtual elements. And also there is a possibility to install two packages in the same environment, but it will finish with not resolving the dependency. So it is resolving dependency because it's installing on one package. So here, I've just gathered all the information that I have already said. You can check it. It is separated in three tables and annotations to these tables. I will skip that. This is just the reference. And finally, I've made some benchmark. And surprisingly, poetry is the fastest from chosen package manager by me. And I'm very often using it because of that. And finally, I will tell a few words about my setup. I'm mainly installing PEP locally on my home folder. If I something install from my user, it's installing locally on my line-line system. Then I'm installing PEP and poetry and other tools that I'm requiring for most of the projects that are common for projects. And then I'm installing PyEmp for managing many Python versions and VirtualEmp. And then I start project by creating VirtualEmp and developing. So here are all resources that I used for presentation. Feel free to see everything, what I used for this presentation. And that's the end of my presentation. Feel free to ask me questions. Thank you, Michal. See, people loved your talk. See those claps. So, okay, so we got two minutes. I think I can only go through one or two questions. So Matthew is asking, how did Wills help you solve your problem with Terraform and AWS? Actually we just had to create a Docker image that was common for every developer and we solved that. And then we created AWS layers where we have all libraries. Right, okay. Oliver is asking, how is PyEmp in comparison to VirtualEmp? Any preference and why? I love to use PyEmp because I can easily switch between versions. I currently work in a project where I have five repositories and five different Python versions. I don't have to install it in different. I don't have to install it globally. And I just can switch it easily. Okay. What's the best way to maintain package versions between requirements.txt, pip file and set up of Py? So what's the best way to maintain package versions between requirements, pip file and set up of Py? I think I'm commonly using the PyFile lock or PoE3 lock because it's resolving my dependencies. PIP tools is okay but it's too big to have many requirements for many environments, I think. I don't know whether I answered your question but we can talk after. So we reached the end of the time. There's still three questions here. I will urge people to have questions to go to the talk on this board, the channel of this talk. Thanks for coming and thanks Mihai for the talk. We're going for a short break.
 All right, okay. And what are you going to talk about?  Today I will be talking about difficulties that we can face during development of Python  code. I will share my screen.  All right, so give a big hand to Nihal.  Cool.  Okay, so before I start, maybe a few words about me.  I have eight years experience in Python, five years commercial experience in developing  Python code. And today I will share my knowledge and best practices and best tools that I'm  using during project development.  My presentation will be divided in three main sections.  Each of them will be covered with a small story. And first of all, we will start with  the packages that are available in Python world.  We will have a little bit about the history of Python and confusions related to that.  Then we will switch to the virtual ads.  We will tell what are for and some great tools that are allowing, developing our Python code  in virtual and then we finally focus on package managers.  We will try to clarify all what are, which are the best and what are the advantages and  disadvantages of certain package managers.  But before that, we have to say that the Python world packages, virtual ads and package managers  is looking like this.  Because of the history, we have some old documentation that is pointing for packages  that are not used.  The same thing is for package managers, which we, which is not obvious which of them we  should use.  And today I will try to clarify what is the right path.  And I hope that this picture will be much simpler.  So in the beginning, we will start with the story related to AWS.  Recently, I have a few projects related to the, that were stored on AWS and mainly I  was working with Lambda, AWS Lambda functions.  And before we start our work, we was choosing out of many tools how to deploy the Lambda.  We could use Terraform, serverless framework, but those three cloud formation and many,  many, many other tools.  Finally we decided for Terraform.  To use Lambda, the deployment of Lambda is looking like this.  We have to prepare the zip file with our Python code.  And we also have to add the libraries that were required for example to communication  with database, like a school alchemy, because Lambda itself contains only base libraries  of Python.  And during development, we noticed that we have a problem that Terraform is constantly  building our code on Lambdas.  And the reason of that is, was using different wheels, because we, in the beginning, we was  doing this by hand and each developer was using different platform.  And what is wheel, I will tell in within few slides.  And also I will tell how we fixed this issue.  But let's start from the beginning.  We have Python files and we can share them between community.  For example in the framework, it is possible to create a small web page for deployment  and we can share this file.  But it is not convenient to use that.  We have to use something more convenient and reliable.  And first thing that came across is a source code.  And for that, it is just simply the archive that is containing the Python code.  It may contain other source files like C source files for C Python.  We can define the scripts that will be available for our terminal.  We can define, of course, Ritme, the setup.py or Cfg.  I will tell a little bit about setup.py later.  And also we can define some external packages, package data like, for example, the trained  model of artificial neural network.  And of course, metadata to know what we have installed in our system.  And here we are coming to the conclusion that each time we want to use that, especially  if we have the C source files, we have to compile that.  This takes time and we would like to have the instant package that we will be ready  to use.  And this can be offered by binary packages.  The first approach was X in Python.  They were introduced with the setup tools in 2004.  There was no need for building our compilation.  It was a distribution format and runtime installation format.  And the first approach of listing what we have installed in our system was a infotainment  directory where we had all necessary formations, what we have installed in our system.  The funny thing about X, it was possible to install many versions of certain library.  And of course, it was the possible to import directly the X.  So what was the problem with them?  There actually wasn't official PEP for that.  And each package manager, package maintainer has its own directories inside X.  And we in Python community, we had to make something with that and make some standardization.  And for that, Wills came across.  This also is a binary format.  It was introduced in 2012 together with PEP 427.  With Wills, we have also standard for distribution of the Wills and standardization of the metadata.  So instead of a Gimli4, we now have different directory that contains different metadata.  And this is the current standard for building the binary.  Wills are not containing the Python bytecodes, but may contain other precompiled code, for  example, for C-Python C libraries.  As I mentioned, instead of egg info, it is used in this directory.  And the very big advantage of Wills is that they are versions.  They have the name convention.  In the name, we can find the Python version that is supported.  The certain implementation, because the Python can have different implementations.  Also there is information about application binary interface, which is different in Macintosh,  Linux and Windows, and for which the architecture is built, certain Wills.  For example, Intel and AMD or other architectures, and of course, whether it's for 32 or 64 bits.  For now, some Wills are importable directly, for example, for our rep.  But it's not supported by PEP 427 itself.  And here we can see the example Wills.  The first is a new version of the PEP library, which is for Python 2 and 3.  And this is one of the NumPy Wills, which is for C-Python 3.6, for Macintosh 10.9, for  both Intel architectures, 32 and 64 bits.  Currently, there are no builder on the PyPy.  The maintainers just have to create all necessary Wills.  We can create our Wills by this command.  But there is a great talk about Wills by Elena Hashman, if you want to know about application  binary interface in Linux, the ELF format, and how they are maintained and built.  You can just watch it by clicking on the links provided here.  And she's also supporting the projects, the Manelinox and AuditLit, the Manelinox project  is actually the group of the most popular Linux distribution in Docker that are building  Wills for this certain distribution.  And AuditWill is a command line tool that is used for checking whether this will is proper  for certain distribution and nothing is missing.  So if you want to know more, just click on the links.  And this is the end of the first section.  We'll move to the virtual ends.  Oh, okay.  One more thing.  Just to show how the package is built.  In the most deep layer, we have the standalone model, which are Python file.  We have more Python files.  We can have a source code.  And about that, if we compile that, we have the wheel for certain platform.  So this is the summary of the packages.  Then we can switch to the virtual ends and next story.  During my journey of the development of the Python code on the studies, I have many projects.  And I installed Ubuntu on my laptop and I started developing many projects at once.  I started installing many libraries to the global system Python and I noticed that suddenly  I noticed that such picture like here.  And the problem is that when I was developing and installing many packages, I reinstalled  one very important dependency that was used for a graphical engine of Ubuntu and I had  to reinstall everything from the scratch.  But it was very quick lesson for me why we should use virtual end.  So what is exactly the virtual end?  It is isolated directory where we have all necessary libraries and Python, which is completely  isolated from the global system, which is good.  And we have few tools to do that.  One is the built into virtual end, which is refarctored now.  There is a great talk.  We had it today by Bernard Gabber writing the virtual end.  I fully recommend to see this talk.  And together with virtual end, there is a useful thing to use the virtual end wrapper,  which provides useful comments for creating virtual end, deleting it, switching between  them.  And it is easier to develop the Python code with that.  How the virtual ends actually working, they are putting the path of the certain and virtual  end in the directory to the path before the global system and every time we are typing  the Python, it is first looking in the virtual end, not in the global system.  From today's presentation, I have heard there will be possibility to create virtual ends  for different versions, which is very good.  But they must be installed on the system.  They must be present for virtual end.  So in Python world, we have many implementations of the Python.  We have the main Python in C Python, which is written in C. We have a Python, which is  written in Python.  We have a JTOM, which is written in Java.  We have an Ion Python, which is written in C sharp.  We have a micro Python, which is used for certain embedded systems.  We have the stackless Python, which is not containing the stack.  If we, for example, want to develop our library and test it against many versions and many,  let's say, distributions of certain Python, we have to have this certain Python installed  on our system.  And there is a tool that is doing this, which is called PyEmp.  There is possibility for installing many versions of Python, many implementations of Python.  It is offering switching between them.  There is an automation of switching between certain virtual end ends, creating the virtual  end ends.  Here I prepared a very useful command that can be used with PyEmp.  And before that, I just want to tell you that the installation of PyEmp is very simple.  If we want to install it, it just can be done by the script provided on GitHub, which is  called PyEmp installer.  If you don't trust this script, you can use the command one by one in the script.  And very important thing, you have to install dependencies that are required for compiling  the Python versions.  And there is all dependencies accordingly to the distributions, which can be found here  in the frequently asked questions.  And be aware of that.  If you want to, for example, use the Jton, you have to have Java on your machine.  If you want to use C sharp, if Ion Python, you have to have the C sharp on your machine.  And after that, if you prepare everything, you can just list all available, the PyEmp  is installing in your home folder.  And all versions are isolated from the global Python system and are just put it on the home  folder.  And if you want to list all available Python versions, you have to use this command.  Sometimes there is a need for check out the GitHub repo locally just to update the newest  version that appeared in community.  To install certain version, we are using such command.  We can uninstall certain version.  And the same command is used for uninstalling environment.  To list all available Python versions, we can use this command.  And very important thing is that if we want to know where exactly our virtual and F is  stored, we have to use the PyEmp which Python command.  Because the normal command which Python will show the same directory, which is an engine  used by the PyEmp.  And the real path will be shown by this.  By this command.  Also, we can activate or deactivate the environment by hand.  And there is a possibility to set some namespaces in our terminal of availability of the certain  version of the Python.  The hierarchy of this command is like this.  If we want to test some Python version, we can use the command PyEmp and this command.  We can attach certain version or environment to the directory.  When we switch inside to this directory, we will have our environment.  We can set up the PyEmp global which will be covering the system Python which is very  good.  But it will be shown the last in this hierarchy.  I will show it in a quick demonstration.  So here, this is my PyEmp versions.  I have installed micro Python.  I have mini-conv3.  There is also anaconda for that.  This is all versions that are available by PyEmp.  We have the version of anaconda and Ion Python, j-tone, micro Python, mini-conv, PyPy, and  so on and so on.  Here, currently, my global system is 382 which is covering my system Python.  For example, when I change my directory to the PyPyX directory, automatically there is  activation of my virtual ENF.  If I, for example, want to test the micro Python, I'm using the shell command which will be  covering the environment from the directory and my global Python.  If I just stop testing, I can unset it and I will be back for my global Python.  This is very convenient during development.  I very recommend to use that.  Actually, that was the last slide of the second part of PyEmp.  Then we are now switching for package manager which is a big topic and big confusion in  Python community because there is a lot of choices that we can make before we start our  project.  This year, I start with my short story about dependency hell.  In one of my projects, I have many libraries.  Each of these libraries have many dependencies.  Most of these libraries was using the same dependencies but in different versions.  In our project, we are using the PyP package manager and we were using the requirements  TXT file for reproducibility of the environment.  After a few executions, we noticed that sometimes our application is not working because of  the wrong dependency version and we were just wondering why it is happening.  We noticed that in some cases, the order in requirements TXT was changing our dependency  because the first library, for example, was using the Python, the library version 1.00.  The second one was using 103.  What was happening, the PPP was not resolving this dependency versions.  It is just installing the first version.  In the second line, it is uninstalling this first version and it is installing again the  second version.  We just wanted to know how we can manage that.  We switched for different package manager and I will show you plenty of them.  But before that, the short story, what was first, in the beginning of Python, we have  easy install, which was using X for installing, it was provided with setup tools in 2004.  It was using installing packages and its dependencies and there was no PEP for the easy install.  With the standardization, the PEP came across and we find that the PEP was released in 2008.  It is using wheels or source code and introduced the requirements TXT for freezing our environment  and knowing what we have installed in our virtual environment.  The good news is that from certain versions of the PEP, the latest versions of PEP, there  is a possibility of checking the hashes of the versions of Python, so we have more security.  But it has to be explicitly used.  But as I mentioned, PEP is not resolving dependencies.  So here is a package, here is a table that is comparing easy install together with the  PEP.  The main difference is that easy install is using X.  There was a possibility to install many versions of certain library.  There is no PEP for easy install.  And for PEP, of course, there is PEP for 3.8.  It is using wheels.  And this is the main difference between them.  So we have, but as I mentioned, there is still a problem with the resulting of dependencies.  So before I move to this tool, to these tools, I will tell a short introduction to setup py script, which is used for our packages and libraries that we want to install in our environment.  We are feeling necessary information.  And the setuppy script is providing the possibility of building the packages, distributing them  to the PyPy and installing in our virtual ends.  This is the example, a piece of code of setuppy, where we have the version, description, authors,  packages.  We can define additional non-Python packages like SVG files or other data.  I don't want to focus on the setuppy in this presentation.  It is, I just wanted to mention it because it's used in the next tool that I will be  talking.  So the PEP tools, it's a package manager that is using two commands, PEP compile and PEP  sync.  And it is resolving dependencies in certain way.  It is using the setuppy and requirements in file for knowing, for resolving all dependencies.  And after that, after compiling, we are, we have, we are getting the requirements TXT,  where we have all resolved environments, all resolved dependencies, and we can reproduce  our environment.  The second command, PEP sync, is checking whether all necessary versions of libraries  are installed in our environment.  And it's just showing if there is some inconsistency.  So here we are requiring three files for resolving dependencies.  So if we would like, for example, to create it for a deaf test, UAT and Proth environments,  we have to reproduce requirements in and requirements TXT, which is not so convenient.  And then we have a different tool that is PEP-ANF.  It is, it can manage the virtual environments.  If we, for example, is installing something and we have not activated the virtual environment,  it is automatically activating the random virtual ENF with the random name and installing  it there.  But it's, but when it's something, when the virtual ENF is activated, it is installing  in the current virtual ENF.  And it is using the PEP file and the PEP file lock files.  And PEP file contains two sections.  In the first section, there are development, local development packages that are used for  creating the project and the production, the production libraries.  And each time we are providing something, it is resolving automatically all necessary  dependencies and it's telling whether there are conflicts that cannot be resolved.  For example, one library is requiring the version 2.3 and the second library one dependency  that will require version below two.  So this is very useful tool for pointing out such situations.  Of course, when we have the stable environment, we just lock in the environment, which then  the PEP file lock is used just simply for installing our CI CDs, Docker images and so  on, so on.  Also, PEP is using our hashes, so we have the security.  And there is an alternative for PEP ENF, which is also great, which is poetry.  We use it for actually poetry has the same features as PEP ENF, but it's faster.  I will show that.  It's faster in resolving dependencies.  So it is managing the different environments.  It's also installing packages.  It's resolving dependencies.  But there is also the convenient way of creating packages and publishing into the PIPI, so  it is more useful for libraries.  And I will say a few words about a package manager that is not used, that is not pure  Python, which is anaconda, which is the Python libraries plus some other data scientists  libraries.  The mini-conda is the smaller version of anaconda, and the conda is the package manager.  The main problem with anaconda is that it's not supporting PIPI, it's supporting only  their channels.  They have different standards for their packages, so there is no compatibility between PIPI and  channels, conda channels.  And of course, conda package is not working with Python virtual elements.  So be aware of that.  It is mainly used for machine learning projects, and we have the possibility to use conda itself  or conda with other package manager.  If we use conda alone, we have to know that channels, we have to choose good channel because  very often libraries are outdated.  They are maintained by different people.  And most safe is to use conda with other package manager, conda for some non-Python packages,  and Python package manager for getting the latest libraries from Python.  And this is mainly useful for machine learning.  Also recently, there was a release of a PDM package manager, which is implementing PEP582.  The PEP582 is talking about storing the libraries inside the project.  There is no need for the virtual elements.  Everything is with the project directory.  And also it's implementing PEP5817, which is installing libraries from the directory.  So this is a great feature.  But it's on early stages.  And finally, we can use sometimes the PEPX, which actually is not resolving the dependencies.  It's installing one package per environment.  And if you want to install a second package, it is installing in different in virtual elements.  And also there is a possibility to install two packages in the same environment, but  it will finish with not resolving the dependency.  So it is resolving dependency because it's installing on one package.  So here, I've just gathered all the information that I have already said.  You can check it.  It is separated in three tables and annotations to these tables.  I will skip that.  This is just the reference.  And finally, I've made some benchmark.  And surprisingly, poetry is the fastest from chosen package manager by me.  And I'm very often using it because of that.  And finally, I will tell a few words about my setup.  I'm mainly installing PEP locally on my home folder.  If I something install from my user, it's installing locally on my line-line system.  Then I'm installing PEP and poetry and other tools that I'm requiring for most of the projects  that are common for projects.  And then I'm installing PyEmp for managing many Python versions and VirtualEmp.  And then I start project by creating VirtualEmp and developing.  So here are all resources that I used for presentation.  Feel free to see everything, what I used for this presentation.  And that's the end of my presentation.  Feel free to ask me questions.  Thank you, Michal.  See, people loved your talk.  See those claps.  So, okay, so we got two minutes.  I think I can only go through one or two questions.  So Matthew is asking, how did Wills help you solve your problem with Terraform and AWS?  Actually we just had to create a Docker image that was common for every developer and we  solved that.  And then we created AWS layers where we have all libraries.  Right, okay.  Oliver is asking, how is PyEmp in comparison to VirtualEmp?  Any preference and why?  I love to use PyEmp because I can easily switch between versions.  I currently work in a project where I have five repositories and five different Python  versions.  I don't have to install it in different.  I don't have to install it globally.  And I just can switch it easily.  Okay.  What's the best way to maintain package versions between requirements.txt, pip file and set  up of Py?  So what's the best way to maintain package versions between requirements, pip file and  set up of Py?  I think I'm commonly using the PyFile lock or PoE3 lock because it's resolving my dependencies.  PIP tools is okay but it's too big to have many requirements for many environments, I  think.  I don't know whether I answered your question but we can talk after.  So we reached the end of the time.  There's still three questions here.  I will urge people to have questions to go to the talk on this board, the channel of  this talk.  Thanks for coming and thanks Mihai for the talk.  We're going for a short break.
Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_0_227.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_228_875.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_876_1559.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_1560_3407.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_3408_4739.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_4740_11411.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_11412_13067.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_13068_13919.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_13920_16931.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_16932_18179.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_18180_21707.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_21708_26963.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_26964_29867.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_29868_29975.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_29976_30083.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_30084_30155.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_30156_30299.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_30300_30719.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_30720_30767.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_30768_31271.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_31272_34763.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_34764_35783.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_35784_36827.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_36828_38423.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_38424_38555.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_38556_43499.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_43500_46595.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_46596_46667.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_46668_55823.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_55824_60971.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_60972_61415.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_61416_61463.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_61464_63407.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63408_63467.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63468_63551.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63552_63719.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63720_63767.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63768_63863.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63864_63995.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_63996_64211.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_64212_64811.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_64812_64871.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_64872_65051.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_65052_65399.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_65400_65447.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_65448_65759.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_65760_65807.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_65808_66275.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66276_66527.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66528_66599.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66600_66647.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66648_66791.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66792_66839.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66840_66983.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_66984_67043.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_67044_67163.png Micha__Wodynski_-_Difficulties_of_Python_code_deve_NZ8K1GXI94M_67164_67285.png