Reproducible packaging and distribution of software with GNU Guix
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Prins, Pjotr
In this talk I will share the great experience we have of packaging,deploying, publishing and distributing software via GNU Guix of a complex webservice that has multiple servers under I will alsodiscuss the work we are now executing on creating 'channels', reproduciblebuild-systems and non-root installations and moving forward on putting Guix incontainers, using work flow engines, so that jobs can run on distributedsystems, such as Arvados. Room: K.4.601 Scheduled start: 2017-02-05
 Okay, we can start. It's time. So, I'm actually one of those people who is using geeks in the production environment. The website is called genenetwork.org. It's hosted by University of Tennessee Health Science Center in Memphis. And this is a site that has been developed over the last 20 years. So, even though it's essentially a simple setup, you know, we're running Python and MySQL to offer these services, I'm going to explain to you how it got out of hand a bit. Anyway, what is Geeks to me? It's all about sleeping at night. Yeah, I mean, the last 20 years I've been managing systems. And sometimes, you know, running 40 servers or something in that order. And you would get a call in the middle of the night and somebody says, you know, hey, you got to run to the data center. You know, that's something I really want to avoid. I could have somebody else go now, I think. But even so, you know, we are doing better. So it's about control software deployment. And it's about controlling the full dependency graph. And I'll explain that in a bit. And also, you know, what you want to do is as Ludo so nicely pointed out, you want your system to be deterministic, you know. Once you deploy it, you want to be, you want to really know it's the one that you actually deployed. Oh, sorry. So I'm going to talk about four things. One is deployment in a developing testing, staging and production environment. The second point I want to discuss is easy installation. And I'll get to, yeah, that will be the longest point. Distributed workflows, I'll discuss shortly. And then orchestration of services is something we could see as the next step. So when you talk about deployment, you know it's essentially simple, but it's all about dependencies, you know. And there's a link here at the bottom. This slide you can find online. And I'm going to educate you a little bit what it looks like. So this is a graph that has been produced by Geeks, you know, when you put all the dependencies together. It can show you an SVG. And I hope I can scroll it. So somewhere at the top, there's G network, right? So this is actually the starting point. So G network depends on quite a few things. And let's go sideways. See what's here. So there's Redis, okay. And there's RQTL, which is an R library. There's Python parallel in there. So here are some of the R packages, many of which Ricardo has made, has packaged. Yeah, so it continues for a bit. And let's see. There's also somewhere, oops, how to config. Yeah, so false cups is even there. God knows why. So this is the Glipsy stuff. And then we're getting in XML parsing. We're getting into fonts. The X, some X stuff somehow. So how is this all possible? Let's see what's here. I'm missing out on the Python stuff because it's also quite huge. Anyway, LZ4 is there. Valgrind is even there. Glipsy is fortunately there. What's that? It's GliP. Anyway, so to build an assistant deterministically, you have the immediate dependencies, which are quite obvious. So maybe we're using SciPy or RQTL, which depends on a few other R packages. But you also, to get to this environment, to deploy it, these packages have their own dependencies. So when I showed this to my professor, he said, how come Ruby's in there? It's not supposed to be any Ruby in there. But there's a dependency on Ruby somewhere. One of these packages requires Ruby. So this is the stuff we're finding out. And the versioning is extremely important. When you want to have something deterministic, you don't want to be building on shifting sand. So we have actually one thing I didn't show you, but there are three partons in there. There's parton 2.4, parton 2.7, and parton 3.3, I think, or 3.1, wherever they are. So there are packages also depending on different partons. So let's go back to the slides. So essentially what is simple is actually not so simple. And there's also dependencies for OpenBLAS and Atlas, which are competing libraries. And then there's the Ruby dependency, and the number of dependencies is actually growing over time. Any time you add a package, you'll get more dependencies in. And I've so far avoided doing the JavaScript stuff. Because the JavaScript we're loading at Hocknow, which NPM is actually great at, the Node Package Manager. But it's such a mess. It's unbelievable. How am I doing for time? So, also in development, the developers were using different systems, essentially. I mean, they were doing it at Hock, so you say, okay, I want Python, so you install Python. You want to install SciPy, you will get a version of SciPy. And this raises real questions. If you find a bug, is it actually inside the code base itself, or is it inside the underlying dependencies? There's not a chance of reproducing these, unless you work on exactly the same machine. So, a year ago, heroically, I started packaging this stuff in the new geeks. And basically, essentially, what I did is I took a checkout of the geeks itself. So, I started from the trunk of geeks and started building from there. And this means that, at that point in time, I sort of fixated the environment. So, I was using exactly that version of Python that was already in geeks. The problem with geeks is that if you keep updating the geeks, the geertree of geeks itself, these packages change also. So, over time, you will be hitting issues with dependencies that are not working anymore. So, I fixated it in February, and next to that, I was using the geeks package part. So, I put in my own packages, right, or our own packages. And in this repository, the link is here, there are now some 100 plus packages. And some of them are duplicates now with the trunk, because the trunk has moved on. And I actually did synchronize in August last year, and it was not trivial, actually, to move from February to August, because there were a number of packages that were duplicates, but they were also packages that somehow were broken, so I had to revert on things. So, switching, you know, after half a year to a new, an updated geeks tree is not trivial, actually. I'm not looking forward to the next one, because it's going to cost me a day, maybe two days. It will be possible, but it's something that is worth that you're never looking forward to. So, another solution I put in is using shared profiles. So, essentially, what we do now is we have G network, we fixated the geeks tree, we fixated the geeks package path with our own packages, and then we deploy it. All right, so we deployed in something called user local shared, GN staging, so this is a staging branch, and then the version of the tree, right? So, when we get to the next stage, if we want to move it to production, what I should do is I'll mount the profile on the user local shared GN production with this particular version. So, this is how we move forward, and all the developers, the deployment people, they share the same structure. So, we know exactly at any point in time what is actually in GN network 2.0, A8, FC, FF4, based on the August checkout of geeks. So, geeks channels could be an improvement, and that's what I'm going to touch on in a bit. So, some discussion we have on the main list. Geeks channels do not exist at this point, right? But the idea is that we allow people to provide different versions of the geeks tree. So, for example, if you have a product like GN network, and I want to tell somebody else, please install this product at this version, I would provide them a channel, essentially. I would tell them, you know, tell geeks to use this channel, and from that point onwards, you know, he'll be installing software from that particular version of the tree. That's ideally what I want. And you can also roll backwards, you could say, okay, you know, Ruby 1.8.7 is a very old Ruby, and sometimes you need it, you could actually provide a channel for that, say, okay, I'm going to provide a Ruby 1.8.7 channel that other people can use, right? Which is disconnected from the main geeks branch, the main geeks trunk. Another thing that is a problem with, you know, the geeks package path at this point is disconnected, because it actually doesn't look like the geeks tree itself. So, if you create a package in your own geeks package path, and you want to migrate it into trunk, it's actually a bit of work. Work is not good, so if you had a channel, which is actually a reflection of the geeks tree, it would become much more trivial to merge patches with the main trunk. So, for now, I'm the one who is juggling branches, because I cannot even explain to others how to do this stuff. They're not ready for it. So, the second part of my talk is about installation, and I gave this also yesterday for the HPC group. We also have the problem that we are running on high-performance computing systems and supercomputers. So, we want to run geeks packages there, and these guys and girls, who manage these systems, they are highly resistant to giving you a route. They don't want to give you administrative privileges. Geeks itself requires a geeks demon to install stuff, so to start that you need to administer the privileges. It's a no-go in many HPC environments. There's a few exceptions now, but the rule there has got one. But it's rare. It'll come over time. The administrators will start to realize that we actually are saving them time, and they get better environments. But anyway, the alternative is people to circumvent this question in HPC environments. What they start to do is use stuff like brew, conda, easy build, which actually are built environments that work under a local user account. The downside of these systems, these build systems, is that they're non-reproducible. Well, they're reproducible as long as you're using the same home directory that you're building stuff in. But also, it's hard to share. It's hard to share these build tools. And you need to build from scratch every time, which is a lot of work. You need to bootstrap. Docker and container solutions in HPC environments are not an option. There are actually HPC environments that are trying to provide them now. But even Docker needs administrator privileges, and people are always worried that their systems are going to get screwed. So something I wrote in the last couple of months is called relocatable geeks. And it was based on an insight which I had with Elko Dostra. A few years back at FOSM, and Elko was here a few hours ago, but he left. He's the inventor of Nix. And Geeks was forked from Nix. So there are many shared similarities. And one of them is that we have a path slash new slash store, and then we have a hash value, clip C version, and then, for example, the name of the file. And the key insight is that if you look into files, this is quite recognizable. This hash value is unique. So what about just patching these files with something new? If you look at LDC2, which is a decompiler, and you look at the shared libraries that it uses, you can see this pattern of hash values. All fingerprints. Some of them are shared. So the clip C ones are shared. So how about relocating those? So replace them with something that is a target prefix. So if you have a home directly slash home slash user, we're going to replace them. And it will look like this. In this case, I installed it in the home directory, opt LDC test, and then I've rewritten the hash value to look like this. And these are the files. So after installation of this LDC2, it's actually sitting in the home directory instead of opt. And will this resolve? And the answer is yes. It will just work. So just to reiterate, we are replacing that value with that value. Let's see. Looks like I'm skipping something. Anyway. I did this as an experiment, and I used Elko. Elko also created a tool called patch elf, which actually allows you to rewrite elf files. And it worked. For textual files like Ruby, Perl and Bashkrips that have the same fingerprints, it also works to replace them. But some formats like compiled Python files and JVM files, you know, turned out to be a little less easy, mostly because they're not zero-terminated strings. You know, they're actually strings that have a length indicator in front of them. So one night I came up with the solution and said, why don't I keep the size of this path exactly the same? So I don't have to worry about what it actually looks like inside the file, you know, whether it's zero-terminated or some other length indicator. So that was the second insight. So if you see these two paths, you know, now they have the exact same length. I just passed it in, you know, you just slam them in, and it works. It's definitely the easy way to go. Yeah. So I replaced it in all files. There's some URLs on here where all these projects are so you can visit them at leisure if you want. Yeah, so this is the idea. You know, in this case I have home user opt in the first one, so I slam it in. In the second case I tell it to install and use a local share LDC 1.01, and you can see it says the exact same length. And what I do is actually I cannibalize the hash value. Yeah, so it becomes shorter in the second case because the prefix is longer. Yeah, you can cannibalize a long way. You know, I can cannibalize all the way back to it doesn't exist anymore, and then I can go a bit further too. And if you add it all up, you have about 40 letters you can use as a prefix. You have to come up with these unique paths. And cannibalizing the hash value is not that important in this case because in Geeks we use it to isolate, you know, all these, all these directories. In this case I tell it to install it somewhere unique. Yeah, so I'm telling it to install it in this folder or in the LDC test folder over there. Yeah, so what it in the end looks like internally doesn't matter because we're not sharing. Yeah. Okay, so for the LDC compiler, there's an example you can actually download this. It's online. It's a 40 megabyte download. Unpack it. It's about 140 megabytes. Installation of the binary takes three seconds on my laptop. So this is very different from, you know, these easy build ideas or even Docker installations. You know, this is very lightweight. Yeah, the only thing I'm doing is replacing all the paths. And there's many people who've tested this now for the D compiler. So yeah, they're happy. And also, yeah, there's two other things that are important. One is that the shared library that actually, you know, this package, I can call it a package now, but the shared libraries all the way down to Glypsy and even the Linux loader isn't there. Yeah, so with many systems, you have this problem that if you deploy binaries somewhere else, you will get conflicts for example, because they want to load the underlying distributions Glypsy or Lipsy or LZ4 library or whatever. And there can be problems. There can be mismatches. So you see a lot of internet forums, they discuss, you know, these problems about libraries not working. And that's the reason. We'll talk about it in 10 more minutes. So I also did the same thing with Ruby, SSL and NokoGiri. NokoGiri is infamous for installation. Yeah, because it's a Ruby Jam that depends on LIP XML2. And it's, you know, the internet is just, you know, I think half of the internet is filled with messages about people having problems installing NokoGiri. It just works. In some Bamba as a tool we developed, it's written in D and is used in many high sequencing, in sequencing high-performance computer centers around the world. And there was a very, very one hard to reproduce bug. It's sac folds occasionally, right? In some conditions, in some environments, but it's never reproduced. But we can see it pop up once in a while. So I created the binary distribution. It was deployed on a cluster in Australia. They ran it for a day. They saw some of these sac folds. They ran the GDB debugger against it. And we found the exact location where the sac fold happened and we fixed it. And it's actually not in some Bamba, but it's upstream. Yeah, so I'm actually doing remote debugging. Somebody else is running it. And I'm helping with the debugging. And we're doing more. We're going to add more. So the potential here is also that we can actually create one click installs for binary packages. And it's something I want to discuss also in the future of geeks. There are security concerns, of course. When you create a binary that's downloadable from the web and you just install it with your eyes closed, we need to work on that somehow. But I think it is really cool that we can actually use a geeks package that is well tested and has been possibly been used by already 100,000 people. And we just deploy it by rewriting the internal pass. So the second, the third part of this talk, and I hardly have time left, so it's going to be really short, is about workflows. We have a need for running tools in order sequentially and in parallel on these high performance computing systems. And essentially it's called the workflow. And there's been many standards that have been developed for workflows. At the moment, there's a project going on. It's called the Common Workflow Language, and it has a lot of momentum in our community. And it started nice. It started really as a good idea because it started as a descriptive document of the workflow. But soon they embedded JavaScript because they wanted to do more and they wanted to avoid repetition. And they also have looping now. So deterministic it no longer is. And when you look at these workflow standards, occasionally someone will pop up and create a new standard like this one. And it turns out it's actually quite hard to do. Because people want to put in the kitchen sink. That's really the problem. So geeks essentially is already a workflow engine. We can handle dependencies, which is serial execution. And we can also handle parallel execution in a way because of the built farm. The way the built farm has been designed. So Ruler is going to talk about workflows at 3.30 in this room. And he's been doing a lot of work in this area. I'm not saying that built farms are a great idea, but he's probably doing something else. The fourth part of the talk is orchestration. So making the rounds, visiting companies and academic institutions. This is something that pops up every time. People are using tools like Puppet and Chef and likewise. And Ludo describes services, which are services running inside a computer. But we also have services that run across computers now. So we have systems that depend on each other. And this is called orchestration. One service may need another service to run. How do you set up the order of these? And when the one service goes down, what do the other services do? So this is called orchestration. And I think with the functional paradigm that we have with geeks, we can actually push forward to make things happen here too. So this may be something for the final discussion today, which is about the future of geeks. Conclusion. Geeks allows for controlled and sane software deployment with profiles and Git. The way I'm doing it. And we should have channels to make it easier for me. Geeks has relocatable binary packages now. And we should continue experimenting with that and see how far it gets us. Geeks will handle workflows. I have no doubt because RUL is working hard on it. And orchestration should be an agenda. That's my conclusion. And there are acknowledgments for people who are working. So RUL, Ludo and Ricardo especially. The new geeks communities, which one of the things that really strikes me with geeks is that it's engulfed. It attracts so many really intelligent people. So I'm really happy you're here. And then I want to thank Professor Robert Williams who's paying my bills. Thank you. So Chris, you can set up. Have you got any questions? Yes? Yes. I'm just going to ask somebody who will ask you a bit more. The source code of what? Of the geeks. I missed the last bit only. If you are not afraid that somebody will ask you for source code. What will you provide? So the question is, should I be afraid about the source code? If somebody asks for it, then it's all online. I only do open source software. Well, but you have a special geeks package part. Your own processing leads. We do have a geeks package part, but it's also online. It's out in the open. I'm highly resistant to using any type of proprietary software. But also our own systems are completely in the open. It's all open source software. So the geeks package part is just a construct where you can import a second git tree into the packaging tree. But the second packaging tree is just, I need it because it's not on the main tree. But it is out in the open. Anyone can use it. Anyone can update it. And another question. How do you save the package? How do you save the package version in just copy and paste? Yes. How do we handle the versioning? You have to explicitly for every package say, this is the version I want to use. Or the git checkout or whatever. And that's you describe in the package. Have a look at the links that are in my talk because it's all there. Yes. What do you think I got? I got Python packages. Okay. So the question is how do we handle Python packages in geeks? There's really three ways to do this. To go about this. The best way is to actually write geeks packages for Python modules. Which describe their dependencies also. And you install the package with geeks. And it pulls it in with the dependencies. To make it easy, we also provide a generator. So you can, for example, if the package is already in pip or pipi or whatever they're called. It can pull that information and then create the package for you automatically in geeks. And for this I use the geeks package path now, for example. But you're not completely delivered to geeks because you can also, Geeks actually has a package called Python virtual environment. So you can start up a virtual environment for Python and use Python from there. So you can just use pip and pipi to pull in packages. And the third option is just to use a source tree and to provide that in the Python path. And so geeks does not force you to work in any way. You can gradually, incrementally work towards the perfect solution. So... Yeah, we also have two minutes left. Oh, yes, I'm not trying to steal it from you. That's all right.
 Okay, we can start.  It's time.  So, I'm actually one of those people who is using geeks in the production environment.  The website is called genenetwork.org.  It's hosted by University of Tennessee Health Science Center in Memphis.  And this is a site that has been developed over the last 20 years.  So, even though it's essentially a simple setup,  you know, we're running Python and MySQL to offer these services,  I'm going to explain to you how it got out of hand a bit.  Anyway, what is Geeks to me?  It's all about sleeping at night.  Yeah, I mean, the last 20 years I've been managing systems.  And sometimes, you know, running 40 servers or something in that order.  And you would get a call in the middle of the night and somebody says,  you know, hey, you got to run to the data center.  You know, that's something I really want to avoid.  I could have somebody else go now, I think.  But even so, you know, we are doing better.  So it's about control software deployment.  And it's about controlling the full dependency graph.  And I'll explain that in a bit.  And also, you know, what you want to do is as Ludo so nicely pointed out,  you want your system to be deterministic, you know.  Once you deploy it, you want to be, you want to really know it's the one that you actually deployed.  Oh, sorry.  So I'm going to talk about four things.  One is deployment in a developing testing, staging and production environment.  The second point I want to discuss is easy installation.  And I'll get to, yeah, that will be the longest point.  Distributed workflows, I'll discuss shortly.  And then orchestration of services is something we could see as the next step.  So when you talk about deployment,  you know it's essentially simple, but it's all about dependencies, you know.  And there's a link here at the bottom.  This slide you can find online.  And I'm going to educate you a little bit what it looks like.  So this is a graph that has been produced by Geeks, you know,  when you put all the dependencies together.  It can show you an SVG.  And I hope I can scroll it.  So somewhere at the top, there's G network, right?  So this is actually the starting point.  So G network depends on quite a few things.  And let's go sideways.  See what's here.  So there's Redis, okay.  And there's RQTL, which is an R library.  There's Python parallel in there.  So here are some of the R packages,  many of which Ricardo has made, has packaged.  Yeah, so it continues for a bit.  And let's see.  There's also somewhere, oops, how to config.  Yeah, so false cups is even there.  God knows why.  So this is the Glipsy stuff.  And then we're getting in XML parsing.  We're getting into fonts.  The X, some X stuff somehow.  So how is this all possible?  Let's see what's here.  I'm missing out on the Python stuff because it's also quite huge.  Anyway, LZ4 is there.  Valgrind is even there.  Glipsy is fortunately there.  What's that?  It's GliP.  Anyway, so to build an assistant deterministically,  you have the immediate dependencies, which are quite obvious.  So maybe we're using SciPy or RQTL,  which depends on a few other R packages.  But you also, to get to this environment, to deploy it,  these packages have their own dependencies.  So when I showed this to my professor,  he said, how come Ruby's in there?  It's not supposed to be any Ruby in there.  But there's a dependency on Ruby somewhere.  One of these packages requires Ruby.  So this is the stuff we're finding out.  And the versioning is extremely important.  When you want to have something deterministic,  you don't want to be building on shifting sand.  So we have actually one thing I didn't show you,  but there are three partons in there.  There's parton 2.4, parton 2.7, and parton 3.3,  I think, or 3.1, wherever they are.  So there are packages also depending on different partons.  So let's go back to the slides.  So essentially what is simple is actually not so simple.  And there's also dependencies for OpenBLAS and Atlas,  which are competing libraries.  And then there's the Ruby dependency,  and the number of dependencies is actually growing over time.  Any time you add a package, you'll get more dependencies in.  And I've so far avoided doing the JavaScript stuff.  Because the JavaScript we're loading at Hocknow,  which NPM is actually great at, the Node Package Manager.  But it's such a mess. It's unbelievable.  How am I doing for time?  So, also in development, the developers were using different systems, essentially.  I mean, they were doing it at Hock, so you say,  okay, I want Python, so you install Python.  You want to install SciPy, you will get a version of SciPy.  And this raises real questions.  If you find a bug, is it actually inside the code base itself,  or is it inside the underlying dependencies?  There's not a chance of reproducing these, unless you work on exactly the same machine.  So, a year ago, heroically, I started packaging this stuff in the new geeks.  And basically, essentially, what I did is I took a checkout of the geeks itself.  So, I started from the trunk of geeks and started building from there.  And this means that, at that point in time, I sort of fixated the environment.  So, I was using exactly that version of Python that was already in geeks.  The problem with geeks is that if you keep updating the geeks, the geertree of geeks itself,  these packages change also.  So, over time, you will be hitting issues with dependencies that are not working anymore.  So, I fixated it in February, and next to that, I was using the geeks package part.  So, I put in my own packages, right, or our own packages.  And in this repository, the link is here, there are now some 100 plus packages.  And some of them are duplicates now with the trunk, because the trunk has moved on.  And I actually did synchronize in August last year, and it was not trivial, actually,  to move from February to August, because there were a number of packages that were duplicates,  but they were also packages that somehow were broken, so I had to revert on things.  So, switching, you know, after half a year to a new, an updated geeks tree is not trivial, actually.  I'm not looking forward to the next one, because it's going to cost me a day, maybe two days.  It will be possible, but it's something that is worth that you're never looking forward to.  So, another solution I put in is using shared profiles.  So, essentially, what we do now is we have G network, we fixated the geeks tree,  we fixated the geeks package path with our own packages, and then we deploy it.  All right, so we deployed in something called user local shared, GN staging,  so this is a staging branch, and then the version of the tree, right?  So, when we get to the next stage, if we want to move it to production,  what I should do is I'll mount the profile on the user local shared GN production with this particular version.  So, this is how we move forward, and all the developers, the deployment people, they share the same structure.  So, we know exactly at any point in time what is actually in GN network 2.0, A8, FC, FF4, based on the August checkout of geeks.  So, geeks channels could be an improvement, and that's what I'm going to touch on in a bit.  So, some discussion we have on the main list.  Geeks channels do not exist at this point, right? But the idea is that we allow people to provide different versions of the geeks tree.  So, for example, if you have a product like GN network, and I want to tell somebody else,  please install this product at this version, I would provide them a channel, essentially.  I would tell them, you know, tell geeks to use this channel, and from that point onwards, you know,  he'll be installing software from that particular version of the tree.  That's ideally what I want.  And you can also roll backwards, you could say, okay, you know, Ruby 1.8.7 is a very old Ruby,  and sometimes you need it, you could actually provide a channel for that, say, okay, I'm going to provide a Ruby 1.8.7 channel that other people can use, right?  Which is disconnected from the main geeks branch, the main geeks trunk.  Another thing that is a problem with, you know, the geeks package path at this point is disconnected,  because it actually doesn't look like the geeks tree itself.  So, if you create a package in your own geeks package path, and you want to migrate it into trunk, it's actually a bit of work.  Work is not good, so if you had a channel, which is actually a reflection of the geeks tree,  it would become much more trivial to merge patches with the main trunk.  So, for now, I'm the one who is juggling branches, because I cannot even explain to others how to do this stuff.  They're not ready for it.  So, the second part of my talk is about installation, and I gave this also yesterday for the HPC group.  We also have the problem that we are running on high-performance computing systems and supercomputers.  So, we want to run geeks packages there, and these guys and girls, who manage these systems, they are highly resistant to giving you a route.  They don't want to give you administrative privileges.  Geeks itself requires a geeks demon to install stuff, so to start that you need to administer the privileges.  It's a no-go in many HPC environments. There's a few exceptions now, but the rule there has got one.  But it's rare. It'll come over time. The administrators will start to realize that we actually are saving them time, and they get better environments.  But anyway, the alternative is people to circumvent this question in HPC environments. What they start to do is use stuff like brew, conda, easy build, which actually are built environments that work under a local user account.  The downside of these systems, these build systems, is that they're non-reproducible.  Well, they're reproducible as long as you're using the same home directory that you're building stuff in. But also, it's hard to share.  It's hard to share these build tools. And you need to build from scratch every time, which is a lot of work. You need to bootstrap.  Docker and container solutions in HPC environments are not an option. There are actually HPC environments that are trying to provide them now.  But even Docker needs administrator privileges, and people are always worried that their systems are going to get screwed.  So something I wrote in the last couple of months is called relocatable geeks. And it was based on an insight which I had with Elko Dostra.  A few years back at FOSM, and Elko was here a few hours ago, but he left. He's the inventor of Nix. And Geeks was forked from Nix.  So there are many shared similarities. And one of them is that we have a path slash new slash store, and then we have a hash value, clip C version,  and then, for example, the name of the file. And the key insight is that if you look into files, this is quite recognizable.  This hash value is unique. So what about just patching these files with something new?  If you look at LDC2, which is a decompiler, and you look at the shared libraries that it uses, you can see this pattern of hash values.  All fingerprints. Some of them are shared. So the clip C ones are shared.  So how about relocating those? So replace them with something that is a target prefix. So if you have a home directly slash home slash user,  we're going to replace them. And it will look like this. In this case, I installed it in the home directory, opt LDC test,  and then I've rewritten the hash value to look like this. And these are the files. So after installation of this LDC2,  it's actually sitting in the home directory instead of opt. And will this resolve? And the answer is yes. It will just work.  So just to reiterate, we are replacing that value with that value.  Let's see. Looks like I'm skipping something. Anyway.  I did this as an experiment, and I used Elko. Elko also created a tool called patch elf, which actually allows you to rewrite elf files.  And it worked. For textual files like Ruby, Perl and Bashkrips that have the same fingerprints, it also works to replace them.  But some formats like compiled Python files and JVM files, you know, turned out to be a little less easy, mostly because they're not zero-terminated strings.  You know, they're actually strings that have a length indicator in front of them.  So one night I came up with the solution and said, why don't I keep the size of this path exactly the same?  So I don't have to worry about what it actually looks like inside the file, you know, whether it's zero-terminated or some other length indicator.  So that was the second insight. So if you see these two paths, you know, now they have the exact same length.  I just passed it in, you know, you just slam them in, and it works. It's definitely the easy way to go.  Yeah. So I replaced it in all files. There's some URLs on here where all these projects are so you can visit them at leisure if you want.  Yeah, so this is the idea. You know, in this case I have home user opt in the first one, so I slam it in.  In the second case I tell it to install and use a local share LDC 1.01, and you can see it says the exact same length.  And what I do is actually I cannibalize the hash value. Yeah, so it becomes shorter in the second case because the prefix is longer.  Yeah, you can cannibalize a long way. You know, I can cannibalize all the way back to it doesn't exist anymore, and then I can go a bit further too.  And if you add it all up, you have about 40 letters you can use as a prefix. You have to come up with these unique paths.  And cannibalizing the hash value is not that important in this case because in Geeks we use it to isolate, you know, all these, all these directories.  In this case I tell it to install it somewhere unique. Yeah, so I'm telling it to install it in this folder or in the LDC test folder over there.  Yeah, so what it in the end looks like internally doesn't matter because we're not sharing. Yeah.  Okay, so for the LDC compiler, there's an example you can actually download this. It's online. It's a 40 megabyte download.  Unpack it. It's about 140 megabytes. Installation of the binary takes three seconds on my laptop.  So this is very different from, you know, these easy build ideas or even Docker installations.  You know, this is very lightweight. Yeah, the only thing I'm doing is replacing all the paths.  And there's many people who've tested this now for the D compiler. So yeah, they're happy.  And also, yeah, there's two other things that are important. One is that the shared library that actually, you know, this package, I can call it a package now,  but the shared libraries all the way down to Glypsy and even the Linux loader isn't there. Yeah, so with many systems, you have this problem that if you deploy binaries somewhere else,  you will get conflicts for example, because they want to load the underlying distributions Glypsy or Lipsy or LZ4 library or whatever.  And there can be problems. There can be mismatches. So you see a lot of internet forums, they discuss, you know, these problems about libraries not working.  And that's the reason. We'll talk about it in 10 more minutes.  So I also did the same thing with Ruby, SSL and NokoGiri. NokoGiri is infamous for installation.  Yeah, because it's a Ruby Jam that depends on LIP XML2. And it's, you know, the internet is just, you know, I think half of the internet is filled with messages about people having problems installing NokoGiri.  It just works. In some Bamba as a tool we developed, it's written in D and is used in many high sequencing, in sequencing high-performance computer centers around the world.  And there was a very, very one hard to reproduce bug. It's sac folds occasionally, right? In some conditions, in some environments, but it's never reproduced.  But we can see it pop up once in a while. So I created the binary distribution. It was deployed on a cluster in Australia.  They ran it for a day. They saw some of these sac folds. They ran the GDB debugger against it. And we found the exact location where the sac fold happened and we fixed it.  And it's actually not in some Bamba, but it's upstream. Yeah, so I'm actually doing remote debugging. Somebody else is running it.  And I'm helping with the debugging. And we're doing more. We're going to add more.  So the potential here is also that we can actually create one click installs for binary packages.  And it's something I want to discuss also in the future of geeks. There are security concerns, of course.  When you create a binary that's downloadable from the web and you just install it with your eyes closed, we need to work on that somehow.  But I think it is really cool that we can actually use a geeks package that is well tested and has been possibly been used by already 100,000 people.  And we just deploy it by rewriting the internal pass.  So the second, the third part of this talk, and I hardly have time left, so it's going to be really short, is about workflows.  We have a need for running tools in order sequentially and in parallel on these high performance computing systems.  And essentially it's called the workflow. And there's been many standards that have been developed for workflows.  At the moment, there's a project going on. It's called the Common Workflow Language, and it has a lot of momentum in our community.  And it started nice. It started really as a good idea because it started as a descriptive document of the workflow.  But soon they embedded JavaScript because they wanted to do more and they wanted to avoid repetition.  And they also have looping now. So deterministic it no longer is.  And when you look at these workflow standards, occasionally someone will pop up and create a new standard like this one.  And it turns out it's actually quite hard to do. Because people want to put in the kitchen sink. That's really the problem.  So geeks essentially is already a workflow engine. We can handle dependencies, which is serial execution.  And we can also handle parallel execution in a way because of the built farm. The way the built farm has been designed.  So Ruler is going to talk about workflows at 3.30 in this room. And he's been doing a lot of work in this area.  I'm not saying that built farms are a great idea, but he's probably doing something else.  The fourth part of the talk is orchestration. So making the rounds, visiting companies and academic institutions.  This is something that pops up every time. People are using tools like Puppet and Chef and likewise.  And Ludo describes services, which are services running inside a computer.  But we also have services that run across computers now. So we have systems that depend on each other.  And this is called orchestration. One service may need another service to run. How do you set up the order of these?  And when the one service goes down, what do the other services do? So this is called orchestration.  And I think with the functional paradigm that we have with geeks, we can actually push forward to make things happen here too.  So this may be something for the final discussion today, which is about the future of geeks.  Conclusion.  Geeks allows for controlled and sane software deployment with profiles and Git.  The way I'm doing it. And we should have channels to make it easier for me.  Geeks has relocatable binary packages now. And we should continue experimenting with that and see how far it gets us.  Geeks will handle workflows. I have no doubt because RUL is working hard on it.  And orchestration should be an agenda. That's my conclusion.  And there are acknowledgments for people who are working. So RUL, Ludo and Ricardo especially.  The new geeks communities, which one of the things that really strikes me with geeks is that it's engulfed.  It attracts so many really intelligent people. So I'm really happy you're here.  And then I want to thank Professor Robert Williams who's paying my bills.  Thank you.  So Chris, you can set up.  Have you got any questions? Yes?  Yes.  I'm just going to ask somebody who will ask you a bit more.  The source code of what?  Of the geeks.  I missed the last bit only.  If you are not afraid that somebody will ask you for source code.  What will you provide?  So the question is, should I be afraid about the source code?  If somebody asks for it, then it's all online. I only do open source software.  Well, but you have a special geeks package part.  Your own processing leads.  We do have a geeks package part, but it's also online. It's out in the open.  I'm highly resistant to using any type of proprietary software.  But also our own systems are completely in the open. It's all open source software.  So the geeks package part is just a construct where you can import a second git tree into the packaging tree.  But the second packaging tree is just, I need it because it's not on the main tree.  But it is out in the open. Anyone can use it. Anyone can update it.  And another question. How do you save the package?  How do you save the package version in just copy and paste?  Yes. How do we handle the versioning?  You have to explicitly for every package say, this is the version I want to use.  Or the git checkout or whatever. And that's you describe in the package.  Have a look at the links that are in my talk because it's all there.  Yes.  What do you think I got? I got Python packages.  Okay. So the question is how do we handle Python packages in geeks?  There's really three ways to do this. To go about this.  The best way is to actually write geeks packages for Python modules.  Which describe their dependencies also. And you install the package with geeks.  And it pulls it in with the dependencies.  To make it easy, we also provide a generator.  So you can, for example, if the package is already in pip or pipi or whatever they're called.  It can pull that information and then create the package for you automatically in geeks.  And for this I use the geeks package path now, for example.  But you're not completely delivered to geeks because you can also,  Geeks actually has a package called Python virtual environment.  So you can start up a virtual environment for Python and use Python from there.  So you can just use pip and pipi to pull in packages.  And the third option is just to use a source tree and to provide that in the Python path.  And so geeks does not force you to work in any way.  You can gradually, incrementally work towards the perfect solution.  So...  Yeah, we also have two minutes left.  Oh, yes, I'm not trying to steal it from you.  That's all right.
Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_0_1139.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_1140_2675.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_2676_3839.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_3840_4403.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_4404_4835.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_4836_5027.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_5028_5171.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_5172_5387.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_5388_5459.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_5460_5627.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_5628_22055.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_22056_22331.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_22332_24143.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_24144_36707.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_36708_36767.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_36768_36923.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_36924_37031.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_37032_37619.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_37620_38195.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_38196_38591.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_38592_39047.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_39048_39143.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_39144_39731.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_39732_40007.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_40008_40391.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_40392_40643.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_40644_40751.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_40752_41363.png Reproducible_packaging_and_distribution_of_software_with_GNU_Guix_Distributing_software_with_Gui_41364_43366.png