Using Nix in production for the last two years
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kožar, Domen
Talk with go through lessons learned using Nix in last two years to automate software distribution. What are the common technical and process obstacles using Nix to ship software in production - mean that customers depend on the pipeline. What supportive tools and Nix features were used, what Nix features aren't useful, where do I see improvements are needed. What do I see as something that will boost Nix usage in companies and what's my plan to get there.
 And he's also known for being the only one who has access on the GitHub organization, apart from him. Two years ago, he decided to quit his job to become a next consultant, and he created the EM Lambda, and then he was managing two Nixxs releases in 2016, and by the end of 2016, he joined force with IOHK. Give it up. Hi, everyone. So first of all, of course, I lost my hair because of doing too much Nixx. No, I've started on Lambda. It's exactly one year now. And the only thing I would like to say, I get about every two weeks an email for help for consulting. So I do encourage everyone to start doing Nixx consulting if that's what you feel you would like to try and do. You can come to me and ask me any questions if you need some help. One of the things that Robin actually talked about, and I would like to add a bit on top of that, is this graph. Well, first of all, besides that it's kind of growing from this small group of people that they hack out something on their free times on weekends, it's obviously becoming a distribution which people use every day. And you can see that, of course, these big fluctuations between releases are not just security issues, but I think I would claim it's also people that have deadlines. And they want to push things upstream, but of course, they come at the very last minute. And I think that's a sign of maturity, although we have to rethink how to offload these last time release fixes, like we talked about before at previous talks. And with that in mind, I think one of the major things to reconsider is how we do contributions, and there is a very good blog post by GitHub can host the Linux kernel community, where it talks about when someone maintains a subfolder, for example, in Nixx packages, let's say Haskell, you would have a tree of forks that then you contribute, for example, to Peter Simons, and then he maintains the Haskell changes, and then those get pushed as a batch to Nixx packages. And then instead of putting everything into master, you have a fork of master and then Haskell changes. This is basically how kernel is developed. And we have this master workflow right now in the pull request workflow, and they're completely different, and I think most of confusion comes from there. So why does it matter? If you're using Nixx as a company, like splitting up everything into a coherent pull request is not that easy, right? You care about a subsystem, and you do a lot of changes, and then you want to push this to upstream to a maintainer, but we currently say, hey, you're pushing a lot of things right now, and this will never get reviewed, right? But we already know, for example, for Haskell that nobody upstreams one package, right? And for example, commit everything at one site. So there is a bit of contradiction there, and I think this is really important if we want to scale in the future to decide what are we going to do. So this talk is going to be mostly about the major pain points and some tips that I've gathered over the last two years. Part of the talk that Deonis did was, how do you bring Nixx to a company? I think one of the tiny bits missing, and I'm faulty as that probably more than anyone else, is how do you fast track someone to use Nixx, right? Because when you have deadlines, when you have production code out there, you cannot just say, hey, you know what, let's just go to a nice island and talk about Nixx for two weeks, and then we go back to real world, and everything's going to be fine, right? So there needs to be some incremental changes. How do you get there? I found, actually, this block was to be very, very nice for people. One disclaimer, I always work with people that have been previously exposed with functional programming, so that didn't need to be explained, the concepts behind it. But they still need to know how Nixx language works, but they're usually used to work with types and so on. Besides going from the language itself, how do you contribute to Nixx packages and the whole ecosystem, there are still some pieces missing there, and I'm interested, do you have some suggestions of your own approach to that? So I'm interested to talk about that if somebody has something to add. So I think that's one of the things that I would like to work on when I finally have some more free time. And this is actually part of this gradual process of Nixxifying, right? You go from this impure packaging slowly to completely Nixxify your company or whatever project that you have, right? And this is a process, and you cannot say, again, yeah, we're just going to do one month and it's going to be done, and then everything will be Nixx-based. And this was the talk that Janus was speaking about, and I think that there are some things that we need to fix in order to get this process a bit nicer. So OK, in Nixx itself, we have a couple of functions. How do you get a source? But the main problem is, although you might have the very same source that's, you know, this content is always the same, you will get completely different hashes. And this is quite painful once you move away from Nixx packages and you're using Nixx packages, right? Or in this case, it's just custom software living in a separate repo, right? And so, for example, Hidra Nixx Prefetch Git has a different kind of behavior than Nixx packages. So, we can merge the two and, you know, that's going to get fixed. But that's still, you have FetchGit and FetchTarble and dot slash dot, right? And the main problem is that all of these have a different derivation name and that is part of the inputs, right? So because these are kind of nice names for, you know, a human, then that means that you have to rebuild your whole package because this is an input to your derivation and it will change the hash. And particularly the last one is the worst because who knows where does this name come from if you use the dot slash dot? Sorry? Yeah, it's a parent directory. So what does that mean? If you build this package on Hidra, it's going to have a completely different parent directory than on your local machine or someone else will have a different parent directory. They might call it dev, they might call it, you know, temp or whatever, right? So to some extent it breaks reference and transparency because, for example, Nixxops default.nix uses dot slash dot. And if you build that on Hidra, you will never be able to fetch that binary because it will have a different name. And this is extremely, extremely painful, especially if you have big builds, right? So okay, so how do we fix that, right? Well, we have to sacrifice probably the human readable name and just fix to skip, you know, stick to something. But there is, as far as I know, and correct me if I'm wrong, there is no primitive Nixx which would allow you to switch the name but keep everything else. And I think that probably we could add some kind of a rename derivation name, but then you would have to, for example, wrap that using the dot slash dot dot slash dot and you would have a new primitive for accessing a current folder. And I think that without sacrificing backwards compatibility, that's probably the only same way. And we have to talk about this because it is, in practice, really painful. For example, right now we use a fork of Nixxops but we always rebuild it because it doesn't get the package from the binary cache. Okay. So, Jonas talked about pinning Nixx packages and, you know, as you create your custom repo with all the fixes, you know, with custom Nixxs modules, with custom packages and so on, you start to put the pinning, or at first you don't pin at all, right? And then suddenly it breaks. Okay. Then you start pinning and then you have these different places like in shells, you have to pin and you pin in Nixx and you pin in shell scripts and you pin on Hydra and so on, right? So then you have this, then you have to maintain these pins Nixx packages at six places, right? And then you're like, okay, now we have to clear this up, right? And you know, and another problem that happens is even if you pin, when you say Nixx packages, these will always, you can always leak the host if you don't set it correctly, right? And then you have a problem of using two different ones. And you know, we have different concepts how Nixx packages is, the search pad is populated, but we have found that the only way to really make this reliable is basically to force the Nixx path to be empty and then use and the pinning Nixx packages locally. So as Dr. Toa, this was actually a joint effort from, you know, multiple people. I don't take any credit for it. Actually, most of the things I talk about, it's not my work. It's just how do you get things done, right? And so this is, this works, right? So this is a small snippet you copy paste to every repo because I mean, it solves the problem of bootstrapping Nixx packages. So you cannot pull from somewhere and then bootstrap because that's kind of what you want to solve, right? So what it does is it uses the built-in derivation and the fetch URL and some local paths. And actually the trick is to use the store path so that you can actually make this available in sandboxes. And then you build this and you get Nixx packages at the valuation time and you can just reuse that, right? And we store the Nixx packages source in JSON file and then we have a small command that just bumps that. You know, set revision to something and it will update that. And it's very nice. And the best thing about this is you can have multiple of those files, right? Or you could generalize it that it would consume multiple JSON files so you could have multiple Nixx packages and use it. You know, for example, if there is a bug, you could have two different ones and one packages build with one Nixx packages version and the other one is with the first one, right? And actually we do that for the Haskell software itself. We use the different Nixx packages and then for the machines we use a more stable version. And that's mainly because of Darwin. We try to use the stable Nixx release for the servers but unstable so that we can get the Darwin packages. Oh, that's about to change. Okay, so how do you use that? Okay, first you set the Nixx, but of course it can be populated, but the best way to make sure that none of the other stuff leaks in, it's very good to enforce that. And then if you want to use it in Nixx scripts, you just say, you know, export Nixx pad and you can do it like that, although preferably if you're just using Nixx, you import it directly from the Nixx file. One warning though, if people are using, for example, Gen2 installed Nixx, the config that Nixx gets will have impure paths and this won't work. The work around is of course to install Nixx with Nixx and then you get the one that has the Nixx bands. So this works pretty reliably and in 1.12 hopefully this is all gone because you can add the SHA attribute to fetchTarbo. And I actually think that probably we might just use fetch URL and add an ability to unpack but yeah, it's just an API thingy. All right. So there is a special type in Nixx where you can actually have a URL without the quotes, but the side effect is due to the RFC column A is considered as a URL and it's converted into a string. So I think that we should probably disallow such things because they're not too useful and I don't think we're going to break anyone's code. Maybe we're going to make it work and this is really painful. Why is it painful? Well, because we don't have types. And then you get this kind of error where something is back to the function and well, it's really a string. I mean, not specifically this one, it would be probably a bit different. And this is really painful because it's not easy to debug. And this one line white space changes are a waste of time. Another example is multi-line string. So who has an idea what's wrong with this example? Yes, the indentation is wrong. So to put it in perspective, our monitoring system didn't work because somebody did something like that and it was offset. Of course, the parser didn't parse the other file and say, oh, I don't see any contents and the monitoring system just didn't have anything configured. This is the effect of these things. Because we kind of rely on indentation in Nixx and we assume it works, this is actually one carrier that we have found that is extremely painful. And of course, now if you want to fix that, it's not easy because you would essentially break compatibility. If you would change the semantics, there would be a white space change and you would get completely different stoplights. By using different Nixx versions, you would have different hashes. So if we ought to fix this, it's got to be Nixx 2.0, although talk to Python guys what happens when you change backwards compatibility. There's lessons to be learned, but it's not impossible. And there's actually a poor request by Elias as he was speaking before to fix that. So just to be clear, I'm not trying to say Nixx is bad. We have done great things with Nixx. I'm really happy. I'm trying to point out what are the anti-patterns in Nixx that you have to be careful about. So okay, then we have two types. One is string and one is path. And one pulls in a folder to Nixx and one just references it as a string. And okay, this is how you do it. If you want to go from path to string, you use the string function. Oh, and then you're like, okay, but let's go from string to path back. And there is a built-in called to path. But it doesn't do anything useful, really. You still get a string. So okay, so how do you actually do that? Well if you first assign something and concatenate with a path, then you get nothing useful. But actually the to path function is the second example where you first take a path and then concatenate the string. And then you can actually do that. All right. It's painful. It's doable. And even more, strings are not as simple as strings. There's actually strings and strings with context. So what does that mean? A context means that these strings refers to a derivation. And if you use a string somewhere, then that derivation will depend on that derivation, which is in the context of the string. So you can discard the string context or not. And for example, Nick Stripol doesn't make a difference between the two strings. So if you use two strings and then save this card, string context, a different order, you get different things, and they're both a string. So there is actually a StackOverflow question. I think that Nicholas answered. It probably needs a bit more work that it's going to be complete. But I would really like to have a really good explanation of how this works so that it's clear for people. I've just represented the problem. I didn't go and actually try to make a total sense of it. So OK, we have some bad parts in Nick's. That's fine. JavaScript has those as well. And it's widely used. So one of the approaches, how to fix that, is by Gabriel. And I really like his work because he's trying to be really pragmatic of how do we fix things. He did a Munich hack project to basically write a Nick's format, which would then basically some kind of aesthetic analyzer, which would say, ah, you're doing crazy things. Don't kind of coordinate multi-line strings because that's going to end not end well. It's not completely working because the power of a Nick's is not very well defined or it has some special cases. The anti-quoting needs work. And there needs to be a special mode at Implementer to handle that with the current puzzle. So if somebody wants to work with that, please let me know. I would really love to basically bend these things in the X and just say, don't use two patterns and so on. And of course, build a tool that tells you some kind of shell check or a H-link or something like that. And then actually, we could just get the habit of using this tool and we're pretty good. Of course, we can't fix all of those things with just static analyzing, but I think that most of those could be. Well, then another thing is the Nick strings are terminated by null. So if you use random data and you put it in Nick strings, you will get, well, you can see the error, but... Ah. Well, it basically says it's not a valid string. And that's because it contains a null and you can represent secret. So this is kind of painful. Okay, if you put a secret into a string and you put it into your store, then it's insecure, right? But sometimes that's okay. For example, if you have server deployments where you know, it's not just such a security threat or something like that. So we added, for example, key file to NickSops, which instead of taking a string that's then copied to a machine, it takes a path so it doesn't go to Nick strings at all and it's just copied over. But in general, if we ever want to support proper encrypted string, then if we want to represent them in Nick's as a text, then this will probably have to be solved. Okay. Enough of Nick's. Let's go to NickSops. So one of the things that I got bitten by is that actually, who knows how long it takes to evaluate this? It's about half a second, right? And one thing to be aware of that, if you evaluate a NickSops system, it is one thing like all the modules are evaluated. And if you have multiple machines, there is no sharing between them. You evaluate one machine, one NickSops machine, and another one, and so on. So it's basically completely linear. So okay. So for one of the clients, basically I misused or abused or whatever you want to call it, Nick's to run benchmarks, right? So we have this thing in Nick's, in a derivation, we can say this derivation has a feature of KVM. And that means it needs KVM to run. And then Hydra is able to say, oh, this machine actually can run KVM, or only this machine can only run KVM. So I used that basically because we were doing SNAP, high-speed networking protocol, and we would use, and we would use, and Hydra basically to GCU build benchmarks, one per machine, and with specialized software in a controlled, isolated environment, run those benchmarks. And what we wanted to know is that if we take different kind of kernel version, different kind of QEMU, and different kind of our software, and so on, build, build, build, build metrics of things, and see if there's any regressions. And we also had support to patch those. So you could like, for example, we had some patches, so we could compare them, patch software versus other software. And this is just what SNIP it. So basically it's a flat map, some sort, and then in the middle it selects benchmarks, and the benchmark name is a list of benchmarks you want to run, and then it puts all the inputs that those differ from, right? And then it basically builds one, and then just basically flat maps everything into one big attribute set, but you can also do a list. So the problem here was that inside there I was building, I was actually building a VM image that would then be used as a kind of a fixture, right? And the problem was if we actually built 20,000 packages like that, and it evaluated NixOS 20,000 times, right, and that took actually, the evaluation took like 20 minutes. So with this change, it only took 10 seconds. So because SNIX doesn't do any sharing or optimization, when you move out creation of the NixOS of this fixture only to be in the second loop, right, then it doesn't have to do so much work. And there is actually a paper by Erkut also, Maximum Laziness, and it's basically you can make Nix to share things, but then GC becomes a problem because it basically keeps track of all the inputs and everything, and it's not there, and it used to be part of Nix actually a couple of years ago. So how do you debug that? There is a Nix count calls which basically prints out how many times Nix called a pre-ump and how many times Nix called something else. But what you don't get is how long those calls took, and evaluating, in my case, the NixOS evil wasn't called too many times, right, so it was not useful. So I couldn't find a better way, and probably there's a better way, and I'm interested to hear that, but I basically just did bisect, so I pulled code out until I got to the point where I found out, oh, it's the fixtures, and then I figured out what's the problem. So there's some tooling probably to be developed around this, or we can just identify what are the things that are really painful here. One of the anti-patterns is that when you use NixOS modules, you wrap them into other functions, and this is terrible, don't do this. It's very, I did this, I'm faulty over this, so guilty of charge, but what happens then is that when you actually specify inputs, it's not a file-based thing, but it's actually a function that you pass, and that messes up with NixOS because it doesn't know that it, a few things it does are basically gone, for example, you might lose the line numbers, and then your error message is going to be an anonymous function that unknown file failed, and then you have no idea what happened. So no, don't do that. This is one of the worst things, and another trick is we have NixOS tests, right, and then it suddenly fails, and you want to know what's going on, but there is no way to go into that machine and inspect it, so I have this local page that I use from time to time, and I don't know how to really upstream it, is that you tell the video switch to basically expose the network in NixOS on tab zero interface, and then if you just add, you know, you have to start SSH and add some key to your NixOS machines, but then you can actually go and SSH into those NixOS tests, and you can, you know, I just create this big sleep thingy, sleep for two hours, and then I SSH into the test, and I can inspect what's going on there, and I think this is really useful because otherwise it's like Travis, you know, it runs there and you cannot access it, and why do that if we are able to run the VMs locally, right? So, another thing that probably needs to be upstreamed and done properly. How am I doing on time? 10 minutes. Oh, okay. That might do. Alright, so NixOps. My major problem with NixOps, it's really imperative, right? You have to run steps, you have to run set arguments, you have to run this thing and other thing and so on. So in our HK, we created this small wrapper in Haskell around it and make it a bit declarative. It's not general purpose, maybe one day it will be when we have time, but the idea is that you can specify what arguments are passed and they're tagged by Haskell type, and you know, for example, what files you want the deployment to be. So for example, in NixOps, if you add a file to the deployment, you have to call NixOps modify to modify that state and so on. And this does everything in the background. So every time you run, you know, deploy, it basically sets the arguments, it modifies things, so it's pretty much declarative. And then what we also can do then is, like we can say, oh, if you have, for example, explorer in there, we're going to do something a bit different, right? This is actually from our source code. Another very nice undocumented feature of NixOps is that you have this defaults thingy and this is a NixOps module which is then included into all machines in your NixOps configuration. So basically, I encourage everyone that in here, actually, you would have a list of all your custom modules and then they would be available to all of the machines, right? And you can set your, for example, deployment defaults and so on. Otherwise, you always have to add for each machine. And usually, you import this global NixOS module or something like that to share the global or common configuration, but that's what defaults does. And I've actually documented this and in this release, it's going to be there in the NixOps documentation. Another nice feature that NixOps has, you can actually get a node attribute and this is actually the whole network that you define. So you can go like nodes back into that config and then get anything. And for example, if you wanted to create a big cluster that gets all the IPs in your network, you can just basically map through all the attributes and nodes and then get the config, networking and public IPv4 and you can access those and then filter, for example, and stuff like that. It's another feature that's not too well known. Actually encourage everyone that use NixOps to go through examples because they do include this. But usually, you look at one, but for example, there are gems hidden in random files that are unrelated. I mean, they don't say we, this file basically tries to explain defaults, but it says, here's an example about EC2. So it's a good idea to go through all of them. So, we have been deploying many NixOps machines and the defaults once you go below 10 are a bit not, not right. For example, if you copy everything from your deployment machine to the deployment machines, deployed machines, and if you have 100, for example, by default, it will only copy four and then another four and another four and this will take a lot of time. So you have to set the max concurrent copy to like 20. If you set it too much, then it's going to hang everything. But even better is that there is now this has fast connection and there is a description of what does that mean. So it basically switches between does it copy from the deploy machine to all machines, the closures, or does it download it from the cache and then try to do the rest with just what the binary cache doesn't have, then it tries to copy that. So there is then a smaller diff of things that need to be copied. And then, Hydra, so Hydra, it thanks mostly to Shelly nowadays. We have declarative configuration. We have actually, we can test GitHub pull requests in a very hacky way and there is a way to update the standards. And I think only the first one is documented and not even that is totally clear. So that's something that I would maybe like to work with. I run my own Hydra for a couple of customers, but I would really like to document this. It would be easier for everyone to do that. But again, in a company, you always, as I've said before, you almost never have completely pure NICs built. There is always some part that is impure and parts that are pure. And the problem is, Hydra doesn't do impure stuff. It has this Perl plug-in system where you can write some of the impure stuff. But that's pretty much it. So then you have to kind of integrate. So basically, you end up with two CI systems again. One, one that, you know, what people do is either use Jenkins for everything and they skip Hydra completely. But I think Hydra has really nice features. Or use Hydra and then you end up with another CI. So at IOHK, for example, we use Travis and Hydra, but then the problem is you make a pull request and Hydra starts building binaries, but Travis as well, so it doesn't have the binary cache. And then you have these two competing who's going to be first and stuff like that. So I researched a bit. And to be honest, I'm really disappointed by Travis because they do all kinds of scary stuff that I don't think it's even considered moral because they claim that you get this many, this much CPU. But then their build queue becomes really big and they just throttle that. So your build that took previously 30 minutes takes an hour because they have just reduced your capacity for the CPU and then it times out. So I call that unfair because I mean, you're paying for that, for example, right? So don't do that, right? So okay, we want to control the CI machines. That's pretty clear by now. And as Yana said, we have found a build kite. So just to be clear, we haven't implemented it yet, but we have a pretty clear idea. And what it basically does, it runs an agent on your machine, a kind of a bashy agent. So you can actually run this on your Hydra slaves. So the idea is you would have a Hydra that then first builds the package and then triggers the build kite to run on the same machine, the impure stuff. So you don't have to copy stuff anymore from Hydra. And it has also things like you can secure these environments and so on and these nice pipelines and so on. So I think in the next couple of months, we're going to try this and replace Travis and hopefully have time to try a blog post and report back. And I think Yana will keep in touch. How does this work? But based on what I've seen so far, I have very good hopes. And there is Hercules CI that I've started, which is back and written in Haskell and front and in Elm. Unfortunately, I don't have time to work on it, but I do have a budget for that. So if somebody would be interested, please let me know. I estimate there is about two months of work to get something like what Hydra does. And we would basically reuse the C++ demons that are there, but mainly rewrite the web part so we can make the API a bit better. Because one of the nice things that we have in Nix that other CIs don't have is you could just say, Hercules, build me this local thing, it would contact Hercules and then that would build your thing and bring it back. So it would be basically a build. It's true Hydra, right? And then everyone else can already benefit from the binary cache. And other things like that. I have one minute, right? Okay. So, Haskell, we're doing mostly Haskell and there is a bunch of front end JavaScript stuff. I think Haskell infrastructure is amazing. I have no complaints. Thank you, Peter Simons. We added two things. One is multiple output support. That is really nice. For example, if you depend on EKG package and Haskell, it has static files and that then references the whole EKG package and then you have everything up to GAC in there, for example, if you see it statically. Now that multiple outputs are there, the static files are referenced separately and you don't bring the GAC in the whole closure anymore. And we have built stack tunics. They're actually stack tunics and stack tunics. They're two different approaches to this. We'll see which one is better. But it follows this philosophy. Let's use developer stack, which is the development tool for Haskell. But let's generate packets that use exactly the same version as developers did and then deploy that. And that's what we actually use and it works pretty well. There are a few things to be fixed like Mac OS support and so on. But I think it's shown to be pretty good. And the last thing, this is actually what I would probably like to work on Hackathon tomorrow is take all of this knowledge and create like a bootstrappy example of how do you basically run your company and how do you start from and use this pinnix packages, overlays, and these tips and tricks. And then you could just start using nixOS and nix and nixops as well. And of course, you start with nix shell like Anna says, but then the next thing is probably bringing these things in. And I think there is a couple of things we have learned so far. So if we join forces and we bring this into one maintain thing, of course, it's always opinionated, but I think we can work with that. It's going to be really useful for everyone. All right. So that's it. Thank you. I am two questions. One question and one request. The question is, I've already been bitten by the fact that two paths does not create a path, but a string. Is there any reason why not to fix this? Is there any reason why not to hack two paths in the C++ source to return a path in a string? That's probably more a question for Elko than me, but I think that two paths was added at one point and nobody is really using it. So I don't think there is too much hard, too much, I don't think there is a problem actually fixing it. But if somebody is using it, it could, of course, break backwards compatibility in that case. We aren't using it because it doesn't work. I mean, probably a better way would be just to deprecate two paths and create something similar that people would start to use. The other thing is why you answered a bit on the last slide, that this is going to happen tomorrow during the hackathon, but it would be really helpful to have this Nix packages pinning things somehow documented, blocked, whatever, because this is a quite useful piece of code and useful mindset. I think Gabrielle Gonzalez and their team actually created a pull request and they plan to move it to Wiki. But then it also depends if we need that if we are going to get Nix 1.12. But yeah, there is a pull request open for that already. But the main problem is you have to always copy paste this to every place, right, because it's a bootstrap process. Last question. I'm not really a questioner, I just wanted to make a remark. It seems like that in the beginning, you always think, ah, we're just building a simple language. It runs only at evaluation time. So features in there aren't really that important. And then a couple years later, you find that any simple language eventually turns into like a fully featured programming language. Then you have one guy who suddenly really needs something like a profiler, right, or maybe a debugger and these kind of things. And I think that is an interesting insight. Another thing is that from the same idea, you say, oh, okay, in the beginning, because it just runs at evaluation time before deployment machines or evaluate my OS and so on, you don't really need types. And if it crashes, it is not so bad because it crashes before something bad happens, right. But then you end up with like a million lines of code in the next packages and then you have multi hour time loss when you get exactly what you said with the anonymous function in the unknown file, right. So I think those are interesting insights. Yeah, I think that, well, first of all, Elko always says, you know, Nix is not the general purpose program language, right. So the fact that I recreated this whole framework of doing benchmark is probably, you know, our fault. But yeah, I think that we can fix most of these bumps and make this language actually good enough and live with the fact. But this white space thingies and so on don't make it too nice when you, you know, and yeah. But in general, I agree. The more you use it, the more tools you need to around it, right. And I think that at the end, we will end up with all of that at some point. Thank you very much. Thanks guys. You
 And he's also known for being the only one who has access on the GitHub organization,  apart from him.  Two years ago, he decided to quit his job to become a next consultant, and he created  the EM Lambda, and then he was managing two Nixxs releases in 2016, and by the end of 2016,  he joined force with IOHK.  Give it up.  Hi, everyone.  So first of all, of course, I lost my hair because of doing too much Nixx.  No, I've started on Lambda.  It's exactly one year now.  And the only thing I would like to say, I get about every two weeks an email for help  for consulting.  So I do encourage everyone to start doing Nixx consulting if that's what you feel you would  like to try and do.  You can come to me and ask me any questions if you need some help.  One of the things that Robin actually talked about, and I would like to add a bit on top  of that, is this graph.  Well, first of all, besides that it's kind of growing from this small group of people  that they hack out something on their free times on weekends, it's obviously becoming  a distribution which people use every day.  And you can see that, of course, these big fluctuations between releases are not just  security issues, but I think I would claim it's also people that have deadlines.  And they want to push things upstream, but of course, they come at the very last minute.  And I think that's a sign of maturity, although we have to rethink how to offload these last  time release fixes, like we talked about before at previous talks.  And with that in mind, I think one of the major things to reconsider is how we do contributions,  and there is a very good blog post by GitHub can host the Linux kernel community, where  it talks about when someone maintains a subfolder, for example, in Nixx packages, let's say Haskell,  you would have a tree of forks that then you contribute, for example, to Peter Simons,  and then he maintains the Haskell changes, and then those get pushed as a batch to Nixx  packages.  And then instead of putting everything into master, you have a fork of master and then  Haskell changes.  This is basically how kernel is developed.  And we have this master workflow right now in the pull request workflow, and they're completely  different, and I think most of confusion comes from there.  So why does it matter?  If you're using Nixx as a company, like splitting up everything into a coherent pull request  is not that easy, right?  You care about a subsystem, and you do a lot of changes, and then you want to push this  to upstream to a maintainer, but we currently say, hey, you're pushing a lot of things right  now, and this will never get reviewed, right?  But we already know, for example, for Haskell that nobody upstreams one package, right?  And for example, commit everything at one site.  So there is a bit of contradiction there, and I think this is really important if we  want to scale in the future to decide what are we going to do.  So this talk is going to be mostly about the major pain points and some tips that I've  gathered over the last two years.  Part of the talk that Deonis did was, how do you bring Nixx to a company?  I think one of the tiny bits missing, and I'm faulty as that probably more than anyone  else, is how do you fast track someone to use Nixx, right?  Because when you have deadlines, when you have production code out there, you cannot  just say, hey, you know what, let's just go to a nice island and talk about Nixx for  two weeks, and then we go back to real world, and everything's going to be fine, right?  So there needs to be some incremental changes.  How do you get there?  I found, actually, this block was to be very, very nice for people.  One disclaimer, I always work with people that have been previously exposed with functional  programming, so that didn't need to be explained, the concepts behind it.  But they still need to know how Nixx language works, but they're usually used to work with  types and so on.  Besides going from the language itself, how do you contribute to Nixx packages and the  whole ecosystem, there are still some pieces missing there, and I'm interested, do you  have some suggestions of your own approach to that?  So I'm interested to talk about that if somebody has something to add.  So I think that's one of the things that I would like to work on when I finally have  some more free time.  And this is actually part of this gradual process of Nixxifying, right?  You go from this impure packaging slowly to completely Nixxify your company or whatever  project that you have, right?  And this is a process, and you cannot say, again, yeah, we're just going to do one month  and it's going to be done, and then everything will be Nixx-based.  And this was the talk that Janus was speaking about, and I think that there are some things  that we need to fix in order to get this process a bit nicer.  So OK, in Nixx itself, we have a couple of functions.  How do you get a source?  But the main problem is, although you might have the very same source that's, you know,  this content is always the same, you will get completely different hashes.  And this is quite painful once you move away from Nixx packages and you're using Nixx  packages, right?  Or in this case, it's just custom software living in a separate repo, right?  And so, for example, Hidra Nixx Prefetch Git has a different kind of behavior than  Nixx packages.  So, we can merge the two and, you know, that's going to get fixed.  But that's still, you have FetchGit and FetchTarble and dot slash dot, right?  And the main problem is that all of these have a different derivation name and that  is part of the inputs, right?  So because these are kind of nice names for, you know, a human, then that means that you  have to rebuild your whole package because this is an input to your derivation and it  will change the hash.  And particularly the last one is the worst because who knows where does this name come  from if you use the dot slash dot?  Sorry?  Yeah, it's a parent directory.  So what does that mean?  If you build this package on Hidra, it's going to have a completely different parent directory  than on your local machine or someone else will have a different parent directory.  They might call it dev, they might call it, you know, temp or whatever, right?  So to some extent it breaks reference and transparency because, for example, Nixxops  default.nix uses dot slash dot.  And if you build that on Hidra, you will never be able to fetch that binary because  it will have a different name.  And this is extremely, extremely painful, especially if you have big builds, right?  So okay, so how do we fix that, right?  Well, we have to sacrifice probably the human readable name and just fix to skip, you know,  stick to something.  But there is, as far as I know, and correct me if I'm wrong, there is no primitive Nixx  which would allow you to switch the name but keep everything else.  And I think that probably we could add some kind of a rename derivation name, but then  you would have to, for example, wrap that using the dot slash dot dot slash dot and  you would have a new primitive for accessing a current folder.  And I think that without sacrificing backwards compatibility, that's probably the only same  way.  And we have to talk about this because it is, in practice, really painful.  For example, right now we use a fork of Nixxops but we always rebuild it because it doesn't  get the package from the binary cache.  Okay.  So, Jonas talked about pinning Nixx packages and, you know, as you create your custom repo  with all the fixes, you know, with custom Nixxs modules, with custom packages and so  on, you start to put the pinning, or at first you don't pin at all, right?  And then suddenly it breaks.  Okay.  Then you start pinning and then you have these different places like in shells, you have  to pin and you pin in Nixx and you pin in shell scripts and you pin on Hydra and so on, right?  So then you have this, then you have to maintain these pins Nixx packages at six places, right?  And then you're like, okay, now we have to clear this up, right?  And you know, and another problem that happens is even if you pin, when you say Nixx packages,  these will always, you can always leak the host if you don't set it correctly, right?  And then you have a problem of using two different ones.  And you know, we have different concepts how Nixx packages is, the search pad is populated,  but we have found that the only way to really make this reliable is basically to force the  Nixx path to be empty and then use and the pinning Nixx packages locally.  So as Dr. Toa, this was actually a joint effort from, you know, multiple people.  I don't take any credit for it.  Actually, most of the things I talk about, it's not my work.  It's just how do you get things done, right?  And so this is, this works, right?  So this is a small snippet you copy paste to every repo because I mean, it solves the  problem of bootstrapping Nixx packages.  So you cannot pull from somewhere and then bootstrap because that's kind of what you  want to solve, right?  So what it does is it uses the built-in derivation and the fetch URL and some local paths.  And actually the trick is to use the store path so that you can actually make this available  in sandboxes.  And then you build this and you get Nixx packages at the valuation time and you can just reuse  that, right?  And we store the Nixx packages source in JSON file and then we have a small command that  just bumps that.  You know, set revision to something and it will update that.  And it's very nice.  And the best thing about this is you can have multiple of those files, right?  Or you could generalize it that it would consume multiple JSON files so you could have multiple  Nixx packages and use it.  You know, for example, if there is a bug, you could have two different ones and one  packages build with one Nixx packages version and the other one is with the first one, right?  And actually we do that for the Haskell software itself.  We use the different Nixx packages and then for the machines we use a more stable version.  And that's mainly because of Darwin.  We try to use the stable Nixx release for the servers but unstable so that we can get  the Darwin packages.  Oh, that's about to change.  Okay, so how do you use that?  Okay, first you set the Nixx, but of course it can be populated, but the best way to make  sure that none of the other stuff leaks in, it's very good to enforce that.  And then if you want to use it in Nixx scripts, you just say, you know, export Nixx pad and  you can do it like that, although preferably if you're just using Nixx, you import it directly  from the Nixx file.  One warning though, if people are using, for example, Gen2 installed Nixx, the config that  Nixx gets will have impure paths and this won't work.  The work around is of course to install Nixx with Nixx and then you get the one that has  the Nixx bands.  So this works pretty reliably and in 1.12 hopefully this is all gone because you can  add the SHA attribute to fetchTarbo.  And I actually think that probably we might just use fetch URL and add an ability to unpack  but yeah, it's just an API thingy.  All right.  So there is a special type in Nixx where you can actually have a URL without the quotes,  but the side effect is due to the RFC column A is considered as a URL and it's converted  into a string.  So I think that we should probably disallow such things because they're not too useful  and I don't think we're going to break anyone's code.  Maybe we're going to make it work and this is really painful.  Why is it painful?  Well, because we don't have types.  And then you get this kind of error where something is back to the function and well,  it's really a string.  I mean, not specifically this one, it would be probably a bit different.  And this is really painful because it's not easy to debug.  And this one line white space changes are a waste of time.  Another example is multi-line string.  So who has an idea what's wrong with this example?  Yes, the indentation is wrong.  So to put it in perspective, our monitoring system didn't work because somebody did something  like that and it was offset.  Of course, the parser didn't parse the other file and say, oh, I don't see any contents  and the monitoring system just didn't have anything configured.  This is the effect of these things.  Because we kind of rely on indentation in Nixx and we assume it works, this is actually  one carrier that we have found that is extremely painful.  And of course, now if you want to fix that, it's not easy because you would essentially  break compatibility.  If you would change the semantics, there would be a white space change and you would get  completely different stoplights.  By using different Nixx versions, you would have different hashes.  So if we ought to fix this, it's got to be Nixx 2.0, although talk to Python guys what  happens when you change backwards compatibility.  There's lessons to be learned, but it's not impossible.  And there's actually a poor request by Elias as he was speaking before to fix that.  So just to be clear, I'm not trying to say Nixx is bad.  We have done great things with Nixx.  I'm really happy.  I'm trying to point out what are the anti-patterns in Nixx that you have to be careful about.  So okay, then we have two types.  One is string and one is path.  And one pulls in a folder to Nixx and one just references it as a string.  And okay, this is how you do it.  If you want to go from path to string, you use the string function.  Oh, and then you're like, okay, but let's go from string to path back.  And there is a built-in called to path.  But it doesn't do anything useful, really.  You still get a string.  So okay, so how do you actually do that?  Well if you first assign something and concatenate with a path, then you get nothing useful.  But actually the to path function is the second example where you first take a path and then  concatenate the string.  And then you can actually do that.  All right.  It's painful.  It's doable.  And even more, strings are not as simple as strings.  There's actually strings and strings with context.  So what does that mean?  A context means that these strings refers to a derivation.  And if you use a string somewhere, then that derivation will depend on that derivation,  which is in the context of the string.  So you can discard the string context or not.  And for example, Nick Stripol doesn't make a difference between the two strings.  So if you use two strings and then save this card, string context, a different order, you  get different things, and they're both a string.  So there is actually a StackOverflow question.  I think that Nicholas answered.  It probably needs a bit more work that it's going to be complete.  But I would really like to have a really good explanation of how this works so that it's  clear for people.  I've just represented the problem.  I didn't go and actually try to make a total sense of it.  So OK, we have some bad parts in Nick's.  That's fine.  JavaScript has those as well.  And it's widely used.  So one of the approaches, how to fix that, is by Gabriel.  And I really like his work because he's trying to be really pragmatic of how do we fix things.  He did a Munich hack project to basically write a Nick's format, which would then basically  some kind of aesthetic analyzer, which would say, ah, you're doing crazy things.  Don't kind of coordinate multi-line strings because that's going to end not end well.  It's not completely working because the power of a Nick's is not very well defined or it  has some special cases.  The anti-quoting needs work.  And there needs to be a special mode at Implementer to handle that with the current puzzle.  So if somebody wants to work with that, please let me know.  I would really love to basically bend these things in the X and just say, don't use two  patterns and so on.  And of course, build a tool that tells you some kind of shell check or a H-link or something  like that.  And then actually, we could just get the habit of using this tool and we're pretty good.  Of course, we can't fix all of those things with just static analyzing, but I think that  most of those could be.  Well, then another thing is the Nick strings are terminated by null.  So if you use random data and you put it in Nick strings, you will get, well, you can  see the error, but...  Ah.  Well, it basically says it's not a valid string.  And that's because it contains a null and you can represent secret.  So this is kind of painful.  Okay, if you put a secret into a string and you put it into your store, then it's insecure,  right?  But sometimes that's okay.  For example, if you have server deployments where you know, it's not just such a security  threat or something like that.  So we added, for example, key file to NickSops, which instead of taking a string that's then  copied to a machine, it takes a path so it doesn't go to Nick strings at all and it's  just copied over.  But in general, if we ever want to support proper encrypted string, then if we want to  represent them in Nick's as a text, then this will probably have to be solved.  Okay.  Enough of Nick's.  Let's go to NickSops.  So one of the things that I got bitten by is that actually, who knows how long it takes  to evaluate this?  It's about half a second, right?  And one thing to be aware of that, if you evaluate a NickSops system, it is one thing  like all the modules are evaluated.  And if you have multiple machines, there is no sharing between them.  You evaluate one machine, one NickSops machine, and another one, and so on.  So it's basically completely linear.  So okay.  So for one of the clients, basically I misused or abused or whatever you want to call it,  Nick's to run benchmarks, right?  So we have this thing in Nick's, in a derivation, we can say this derivation has a feature of  KVM.  And that means it needs KVM to run.  And then Hydra is able to say, oh, this machine actually can run KVM, or only this machine  can only run KVM.  So I used that basically because we were doing SNAP, high-speed networking protocol, and we  would use, and we would use, and Hydra basically to GCU build benchmarks, one per machine,  and with specialized software in a controlled, isolated environment, run those benchmarks.  And what we wanted to know is that if we take different kind of kernel version, different  kind of QEMU, and different kind of our software, and so on, build, build, build, build metrics  of things, and see if there's any regressions.  And we also had support to patch those.  So you could like, for example, we had some patches, so we could compare them, patch software  versus other software.  And this is just what SNIP it.  So basically it's a flat map, some sort, and then in the middle it selects benchmarks,  and the benchmark name is a list of benchmarks you want to run, and then it puts all the  inputs that those differ from, right?  And then it basically builds one, and then just basically flat maps everything into one  big attribute set, but you can also do a list.  So the problem here was that inside there I was building, I was actually building a VM  image that would then be used as a kind of a fixture, right?  And the problem was if we actually built 20,000 packages like that, and it evaluated  NixOS 20,000 times, right, and that took actually, the evaluation took like 20 minutes.  So with this change, it only took 10 seconds.  So because SNIX doesn't do any sharing or optimization, when you move out creation of  the NixOS of this fixture only to be in the second loop, right, then it doesn't have to  do so much work.  And there is actually a paper by Erkut also, Maximum Laziness, and it's basically you  can make Nix to share things, but then GC becomes a problem because it basically keeps  track of all the inputs and everything, and it's not there, and it used to be part of  Nix actually a couple of years ago.  So how do you debug that?  There is a Nix count calls which basically prints out how many times Nix called a pre-ump  and how many times Nix called something else.  But what you don't get is how long those calls took, and evaluating, in my case, the NixOS  evil wasn't called too many times, right, so it was not useful.  So I couldn't find a better way, and probably there's a better way, and I'm interested to  hear that, but I basically just did bisect, so I pulled code out until I got to the point  where I found out, oh, it's the fixtures, and then I figured out what's the problem.  So there's some tooling probably to be developed around this, or we can just identify what  are the things that are really painful here.  One of the anti-patterns is that when you use NixOS modules, you wrap them into other  functions, and this is terrible, don't do this.  It's very, I did this, I'm faulty over this, so guilty of charge, but what happens then  is that when you actually specify inputs, it's not a file-based thing, but it's actually  a function that you pass, and that messes up with NixOS because it doesn't know that  it, a few things it does are basically gone, for example, you might lose the line numbers,  and then your error message is going to be an anonymous function that unknown file failed,  and then you have no idea what happened.  So no, don't do that.  This is one of the worst things, and another trick is we have NixOS tests, right, and then  it suddenly fails, and you want to know what's going on, but there is no way to go into that  machine and inspect it, so I have this local page that I use from time to time, and I don't  know how to really upstream it, is that you tell the video switch to basically expose  the network in NixOS on tab zero interface, and then if you just add, you know, you have  to start SSH and add some key to your NixOS machines, but then you can actually go and  SSH into those NixOS tests, and you can, you know, I just create this big sleep thingy,  sleep for two hours, and then I SSH into the test, and I can inspect what's going on there,  and I think this is really useful because otherwise it's like Travis, you know, it runs  there and you cannot access it, and why do that if we are able to run the VMs locally,  right?  So, another thing that probably needs to be upstreamed and done properly.  How am I doing on time?  10 minutes.  Oh, okay.  That might do.  Alright, so NixOps.  My major problem with NixOps, it's really imperative, right?  You have to run steps, you have to run set arguments, you have to run this thing and  other thing and so on.  So in our HK, we created this small wrapper in Haskell around it and make it a bit declarative.  It's not general purpose, maybe one day it will be when we have time, but the idea is  that you can specify what arguments are passed and they're tagged by Haskell type, and you  know, for example, what files you want the deployment to be.  So for example, in NixOps, if you add a file to the deployment, you have to call NixOps  modify to modify that state and so on.  And this does everything in the background.  So every time you run, you know, deploy, it basically sets the arguments, it modifies  things, so it's pretty much declarative.  And then what we also can do then is, like we can say, oh, if you have, for example,  explorer in there, we're going to do something a bit different, right?  This is actually from our source code.  Another very nice undocumented feature of NixOps is that you have this defaults thingy  and this is a NixOps module which is then included into all machines in your NixOps configuration.  So basically, I encourage everyone that in here, actually, you would have a list of all  your custom modules and then they would be available to all of the machines, right?  And you can set your, for example, deployment defaults and so on.  Otherwise, you always have to add for each machine.  And usually, you import this global NixOS module or something like that to share the  global or common configuration, but that's what defaults does.  And I've actually documented this and in this release, it's going to be there in the NixOps  documentation.  Another nice feature that NixOps has, you can actually get a node attribute and this is  actually the whole network that you define.  So you can go like nodes back into that config and then get anything.  And for example, if you wanted to create a big cluster that gets all the IPs in your  network, you can just basically map through all the attributes and nodes and then get  the config, networking and public IPv4 and you can access those and then filter, for  example, and stuff like that.  It's another feature that's not too well known.  Actually encourage everyone that use NixOps to go through examples because they do include  this.  But usually, you look at one, but for example, there are gems hidden in random files that  are unrelated.  I mean, they don't say we, this file basically tries to explain defaults, but it says, here's  an example about EC2.  So it's a good idea to go through all of them.  So, we have been deploying many NixOps machines and the defaults once you go below 10 are  a bit not, not right.  For example, if you copy everything from your deployment machine to the deployment machines,  deployed machines, and if you have 100, for example, by default, it will only copy four  and then another four and another four and this will take a lot of time.  So you have to set the max concurrent copy to like 20.  If you set it too much, then it's going to hang everything.  But even better is that there is now this has fast connection and there is a description  of what does that mean.  So it basically switches between does it copy from the deploy machine to all machines, the  closures, or does it download it from the cache and then try to do the rest with just  what the binary cache doesn't have, then it tries to copy that.  So there is then a smaller diff of things that need to be copied.  And then, Hydra, so Hydra, it thanks mostly to Shelly nowadays.  We have declarative configuration.  We have actually, we can test GitHub pull requests in a very hacky way and there is a  way to update the standards.  And I think only the first one is documented and not even that is totally clear.  So that's something that I would maybe like to work with.  I run my own Hydra for a couple of customers, but I would really like to document this.  It would be easier for everyone to do that.  But again, in a company, you always, as I've said before, you almost never have completely  pure NICs built.  There is always some part that is impure and parts that are pure.  And the problem is, Hydra doesn't do impure stuff.  It has this Perl plug-in system where you can write some of the impure stuff.  But that's pretty much it.  So then you have to kind of integrate.  So basically, you end up with two CI systems again.  One, one that, you know, what people do is either use Jenkins for everything and they  skip Hydra completely.  But I think Hydra has really nice features.  Or use Hydra and then you end up with another CI.  So at IOHK, for example, we use Travis and Hydra, but then the problem is you make a  pull request and Hydra starts building binaries, but Travis as well, so it doesn't have the  binary cache.  And then you have these two competing who's going to be first and stuff like that.  So I researched a bit.  And to be honest, I'm really disappointed by Travis because they do all kinds of scary  stuff that I don't think it's even considered moral because they claim that you get this  many, this much CPU.  But then their build queue becomes really big and they just throttle that.  So your build that took previously 30 minutes takes an hour because they have just reduced  your capacity for the CPU and then it times out.  So I call that unfair because I mean, you're paying for that, for example, right?  So don't do that, right?  So okay, we want to control the CI machines.  That's pretty clear by now.  And as Yana said, we have found a build kite.  So just to be clear, we haven't implemented it yet, but we have a pretty clear idea.  And what it basically does, it runs an agent on your machine, a kind of a bashy agent.  So you can actually run this on your Hydra slaves.  So the idea is you would have a Hydra that then first builds the package and then triggers  the build kite to run on the same machine, the impure stuff.  So you don't have to copy stuff anymore from Hydra.  And it has also things like you can secure these environments and so on and these nice  pipelines and so on.  So I think in the next couple of months, we're going to try this and replace Travis and hopefully  have time to try a blog post and report back.  And I think Yana will keep in touch.  How does this work?  But based on what I've seen so far, I have very good hopes.  And there is Hercules CI that I've started, which is back and written in Haskell and front  and in Elm.  Unfortunately, I don't have time to work on it, but I do have a budget for that.  So if somebody would be interested, please let me know.  I estimate there is about two months of work to get something like what Hydra does.  And we would basically reuse the C++ demons that are there, but mainly rewrite the web  part so we can make the API a bit better.  Because one of the nice things that we have in Nix that other CIs don't have is you could  just say, Hercules, build me this local thing, it would contact Hercules and then that would  build your thing and bring it back.  So it would be basically a build.  It's true Hydra, right?  And then everyone else can already benefit from the binary cache.  And other things like that.  I have one minute, right?  Okay.  So, Haskell, we're doing mostly Haskell and there is a bunch of front end JavaScript  stuff.  I think Haskell infrastructure is amazing.  I have no complaints.  Thank you, Peter Simons.  We added two things.  One is multiple output support.  That is really nice.  For example, if you depend on EKG package and Haskell, it has static files and that then  references the whole EKG package and then you have everything up to GAC in there, for  example, if you see it statically.  Now that multiple outputs are there, the static files are referenced separately and you don't  bring the GAC in the whole closure anymore.  And we have built stack tunics.  They're actually stack tunics and stack tunics.  They're two different approaches to this.  We'll see which one is better.  But it follows this philosophy.  Let's use developer stack, which is the development tool for Haskell.  But let's generate packets that use exactly the same version as developers did and then  deploy that.  And that's what we actually use and it works pretty well.  There are a few things to be fixed like Mac OS support and so on.  But I think it's shown to be pretty good.  And the last thing, this is actually what I would probably like to work on Hackathon  tomorrow is take all of this knowledge and create like a bootstrappy example of how do  you basically run your company and how do you start from and use this pinnix packages,  overlays, and these tips and tricks.  And then you could just start using nixOS and nix and nixops as well.  And of course, you start with nix shell like Anna says, but then the next thing is probably  bringing these things in.  And I think there is a couple of things we have learned so far.  So if we join forces and we bring this into one maintain thing, of course, it's always  opinionated, but I think we can work with that.  It's going to be really useful for everyone.  All right.  So that's it.  Thank you.  I am two questions.  One question and one request.  The question is, I've already been bitten by the fact that two paths does not create  a path, but a string.  Is there any reason why not to fix this?  Is there any reason why not to hack two paths in the C++ source to return a path in a string?  That's probably more a question for Elko than me, but I think that two paths was added at  one point and nobody is really using it.  So I don't think there is too much hard, too much, I don't think there is a problem actually  fixing it.  But if somebody is using it, it could, of course, break backwards compatibility in that  case.  We aren't using it because it doesn't work.  I mean, probably a better way would be just to deprecate two paths and create something  similar that people would start to use.  The other thing is why you answered a bit on the last slide, that this is going to happen  tomorrow during the hackathon, but it would be really helpful to have this Nix packages  pinning things somehow documented, blocked, whatever, because this is a quite useful  piece of code and useful mindset.  I think Gabrielle Gonzalez and their team actually created a pull request and they plan  to move it to Wiki.  But then it also depends if we need that if we are going to get Nix 1.12.  But yeah, there is a pull request open for that already.  But the main problem is you have to always copy paste this to every place, right, because  it's a bootstrap process.  Last question.  I'm not really a questioner, I just wanted to make a remark.  It seems like that in the beginning, you always think, ah, we're just building a simple language.  It runs only at evaluation time.  So features in there aren't really that important.  And then a couple years later, you find that any simple language eventually turns into  like a fully featured programming language.  Then you have one guy who suddenly really needs something like a profiler, right, or  maybe a debugger and these kind of things.  And I think that is an interesting insight.  Another thing is that from the same idea, you say, oh, okay, in the beginning, because  it just runs at evaluation time before deployment machines or evaluate my OS and so on, you  don't really need types.  And if it crashes, it is not so bad because it crashes before something bad happens, right.  But then you end up with like a million lines of code in the next packages and then you  have multi hour time loss when you get exactly what you said with the anonymous function in  the unknown file, right.  So I think those are interesting insights.  Yeah, I think that, well, first of all, Elko always says, you know, Nix is not the general  purpose program language, right.  So the fact that I recreated this whole framework of doing benchmark is probably, you know, our  fault.  But yeah, I think that we can fix most of these bumps and make this language actually  good enough and live with the fact.  But this white space thingies and so on don't make it too nice when you, you know, and yeah.  But in general, I agree.  The more you use it, the more tools you need to around it, right.  And I think that at the end, we will end up with all of that at some point.  Thank you very much.  Thanks guys.  You
Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_0_407.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_408_1607.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_1608_1715.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_1716_6251.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_6252_7055.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_7056_7151.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_7152_9131.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_9132_12443.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_12444_14447.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_14448_14651.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_14652_15707.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_15708_19559.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_19560_20111.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_20112_20735.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_20736_20927.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_20928_21371.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_21372_22715.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_22716_23015.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_23016_23099.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_23100_23783.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_23784_24191.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_24192_24323.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_24324_25031.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_25032_27131.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_27132_27803.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_27804_28907.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_28908_29243.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_29244_29699.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_29700_29795.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_29796_30863.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_30864_31559.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_31560_31631.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_31632_32267.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_32268_32327.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_32328_32399.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_32400_32483.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_32484_36083.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_36084_44411.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_44412_44927.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_44928_45023.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_45024_45575.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_45576_45947.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_45948_46079.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_46080_47087.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_47088_47351.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_47352_49175.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_49176_49931.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_49932_50231.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_50232_50807.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_50808_55175.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_55176_55499.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_55500_55607.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_55608_55655.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_55656_55703.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_55704_56075.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_56076_56315.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_56316_56855.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_56856_57203.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_57204_57635.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_57636_60227.png Using_Nix_in_production_for_the_last_two_years_by_Domen_Kozar_dlRuMdm6pRU_60228_63526.png