NixWRT: purely functional firmware images for IoT
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Barlow, Daniel
Reflashing your broadband router with Linux (e.g. DD-WRT, OpenWRT, Tomato or variants) gives you unparalleled flexibility to do things that the manufacturer probably hadn't thought of. Remembering what you did, six months later, is often trickier. NixWRT is a (currently experimental) collection of derivations using the Nix package system and bits of NixOS to build router and IoT device firmware images using the principles of declarativity and composability that are why we love Nix. This talk will give you an overview of how it works, some war stories about the challenges faced, and hopefully the data you need to to decide whether to try it yourself on your own hardware. - Bio: Daniel Barlow has been using Linux since kernel 0.99 (Slackware and MCC-Interim), and has never really adjusted as computing has moved on. Playing with resource-limited systems like routers and IoT devices helps him pretend it's still 1995. Since then he's programmed professionally in Perl, Common Lisp and Ruby, and played with Clojure and Nix. Most likely to say: ""try looking at it with {strace, wireshark}?"" Least likely to say: ""just buy a closed-source solution from an enterprise vendor""
 12 o'clock, let's continue with the next talk, which is going to be given by Daniel Barlow and it's going to be about Nix WRT, which is a collection of derivations to basically build flash images from WS. Flash with a network driver and hopefully not brick it. So yeah, give me a round of applause and enjoy the talk. Thank you very much. Yeah, my name is Daniel Barlow. I'm here to talk about using the Nix Package Collection to build images for embedded routers, IoT devices, things like this little widget here, which you can't see, but don't worry there's pictures. I'd like to start by saying welcome to London. There's a nice iconic image here of Piccadilly Square with one of our London Black cabs in it. If you're not local, you might not know, they had to take a knowledge test to memorize every road name within six miles of Charing Cross before they're allowed to drive one of these cars. Unfortunately, you can't see the contrast quite so well there, this is my iconic Black X1 ThinkPad Carbon after a Black cab driver ran over it the other week. It doesn't boot any more surprisingly enough. So any problems with the quality or the polish in this talk, that's what I'm blaming you on. So yeah, Nix WRT or NixWirt, I came up with the name like six months ago and only just today realize I don't know how to pronounce it. I'm going to go with NixWirt, but I'm not going to be standard about it. So when I started out with it, it was an experiment to see if I could use the Nix Package Collection to build images for Wi-Fi routers of the kind usually run open-wirt or DDWRT or tomato, that kind of thing. Show of hands, I mean, who's got one of these things at home, who's tried flashing their router at home? Fair few people, yeah. Okay. So this isn't NixWirt on your router. The image you get out of it is just an image. It's immutable. You can't log into it and compile packages or anything like that. It's got no compilers. Yeah. So why? Last November, I built a new computer for my home office. You see it there. It's not actually on fire. It's got LEDs inside it. And I wanted to be able to back it up. I didn't want to build another computer to back up host because that feels a bit sort of recursive. So I've got a USB disk drive. I've got a spare router with USB ports. I know there's some kind of embedded Linux thing inside that router. Maybe I can repurpose that and get some use out of it instead of throwing it into eWaste. Why not use open-wirt? And the first thing I have to say here is, open-wirt is great. And there's so much work gone into it. So many people working on different random problems which you would never even hope to be able to replicate as a small developer. I didn't realize how great it was until I tried bringing the mainstream Linux kernel up on a MIPS device and found out I had no ethernet devider. And a router with no ethernet driver is a little bit. It doesn't root much. Yeah. However, I'm not going to be telling anyone anything you don't already know. But when you're talking about divergent, convergent, and congruent change management, open-wirt is on the left-hand end of the scale to a big extent. You have 60 billion different packages in open-wirt. You install them imperatively by typing commands on the thing itself. Then you configure it using a GUI. Six months later, you upgrade the firmware or the router blows up and you need to do hardware, or your ISP says, try factory-resetting it. Then you remember everything you did. Maybe some people are diligent enough to back these things up properly, but I'm certainly not. So I got to thinking, maybe there is a more general problem to be solved here than just somewhere safe to store my RIP CDs. So yeah, did some hacking, did some blogging, got distracted, got distracted again, got my laptop trashed, and managed to trash my own home network in several different fun ways. Here we are a year on. Is it done? There's a little bit of scope creep. I did the backup server. The backup server is working fine. That's great. I did what I came for. I repurposed the wireless range extender in my study upstairs to run the XWRT as well. The router downstairs, which is the one that actually connects to broadband, I'm still working on that, take over the world as kind of a stretch goal. The other question, I said it was an experiment, so what were the findings, I guess? Is Nix good for this stuff? And emphatically, yes, it is. Okay, I'm kind of preaching to the converted. But the Nix language, compared to anything else I've used in terms of configuration management or building is superb. Everyone knows that. The cross compilation stuff, you might not know unless you've been involved with it, has all sort of beginning of 2018. A lot of work you've done last year on cross compilation in Nix. And it's really made it easy to build Nix binaries from an X86 system. And the support for Muzzle, which is an alternative C library to the new C library, it's smaller and faster and more compliant and works better on embedded systems. I think that's pretty new as well. It's certainly the case that I've hardly ever hit problems trying to use a C library. It's not the new C library. So on those three counts, Nix is pretty awesome. Overlays, again, you could do various ways of customizing derivations before overlays came in, but having a consistent way of doing this reasonably principled is really useful and really helpful for making packages smaller. And I'll talk about it more in a minute. So, yeah, there are things I've learned along the way that if you wanted to get into it, you would also end up learning sooner or later, you know, how to read and write Nix derivations, stuff about Linux, stuff about how the kernel is put together, things about networks, switches, CTP, Ethernet, what's a Mac, what's a Fi, overlays, fixed points are awesome. I know enough to hand wave about them. I don't know enough to explain them, so I'm going to do some hand waving in a minute. If you have enough coffee, the rest of it is easy. So what do you actually need? Obviously, you need some kind of thing to run it on. This is the GLINET MT300N, which is like GLINET MT300A, but it's a different color. It's slightly cheaper, and the hardware inside it's slightly different. So, yeah, these things, these are not your typical PCs. They're smaller, they're slower. So graphics hardware, so you establish a console connection to it by attaching three wires to it, which is more or less complicated, depending on whether there are pin headers there to attach them to or you need to go and do some bad soldering. Obviously, the architecture is different. It runs some variety of MIPS, or the ones I've tried to say if I do anywhere, I guess some people are using ARM for this stuff. You don't boot from the BIOS or from UEFI. You don't use grub, users, and called Uboots, which I'll be showing you in a minute. And the way the particular board knows about all the bits that are in it, like where the GPI opens are, where the LEDs are, how to make the Ethernet, how to initialize the little network switch inside it. On a grown-up computer like a PC, you've got things like ACPI where it can sort of go and enumerate the bus and find out where all the things are because all the things will say, look, I'm over here. On these older, smaller systems, either that knowledge is compiled into the kernel for the particular board you're using or on some varieties, a device tree, which is basically a data file with the same information in it. So we extracted that code into data. The device tree is a better way of doing it, but not all boards have been updated to use it yet. Yeah, so in terms of mixed WR2, if you want to play with it, the best supported boards are the ones based on the AI9330 or the ones based on the MediaTek socks. And I say the blue one and the yellow one are both cheap and both easy to get hold of. And also no soldering required. You just pop the top off. Is there an emulator? QEMU works, but the hardware emulates not very much like the real hardware. Testing all the real hardware is actually not that bad. So obviously you need to connect it together to something. This is my test setup. It's a little bit more baroque than it needs to be. What you will see is the device itself with three cables coming out of it for the serial console. The serial console is TTL level logic. It runs into something which understands that. In my case at the moment, that's a Raspberry Pi because I blew up my serial cable. And then that, that is the most overkill use ever of an Arduino YUN, which basically exists to toggle the USPM power off and off. So when I wedge it solid, I can turn it off and on again without having to go upstairs and pull the USB cable out. So I'm going to do a little demonstration of what it looks like to build on or what the actual hardware looks like. I'm not going to demo on this thing here because it would take too long to plug everything together. I'm going to attempt the SSH system at home and show you it there. You ask what could go wrong, well, this guy here actually, my son has dropped to the age where he really likes to turn the power on and off in the front of the machine. So hopefully it's still there when we get there. We're just about to go find out. Right. Nope. That's not it. Either is it. Ah, it was that one. Okay. So let's have 80 columns as the good Lord intended. My handle on most online system is talent, which I chose a long time ago because I couldn't type telnet correctly without typoing it. So the host name theme is all typos as well. So here we are. I'm going to start by making it. It does nothing of course because I made it last night to check that everything worked and therefore there's nothing to be done. So there's a little make file just because there's an awful lot of parameters to give to Nick builds. So the particular derivation we're using, I'll show you that in a minute, it's called backup host. We're building a target called framework. Just parameters, we're passing it for things like SSH keys and an rsync password and other stuff and you'll see it is eventually built a file in that directory there and then it's passing it into my TFTP server directory. And I'm just going to show you that... Yeah, there it is. So that is my firmware file, which is about four and a half megabase in size, which is okay for that particular target device. Put that back in the stand. So I'm going to introduce some insignificant white space into one of these derivations. Oh, that's nice. No, I'll come back to that bit. Ah, here we are. So this is... Is that volume level okay for everyone? Yeah. So this is the device itself. I'm connected to it over an SSH section, over an SSH section, over a mini-com serial session over the three wires you saw in the picture into the actual hardware device. I'm just going to reset it just to show you it's there and I'm going to stop auto-boot. So this is this thing called U-boot. It's the universal boot loader. It's called the universal boot loader. What actually happens is that hardware manufacturers take it, fork it, hack it up, burn it onto their machines. Getting a new version of U-boot onto your device is a bit like trying to put core boot onto your laptop. It can be done, but if it goes wrong, you need JTAG debuggers and stuff like that. So generally speaking, the one you've got is the one you're going to live with unless you want to get more complicated than I've got. So anyway, you've got a connection of commands in U-boot here, which do things like print the environment variable, boots, change memory addresses, let you do things with flash and so on. So if I just do print-env, they are more or less defective depending on the manufacturer. So in theory, if you've done a proper good install of U-boot, then this environment will be writable and you can change parameters here. Then you will find that the hardware manufacturer has just hard-coded the environment and you can't actually save anything, which is annoying, but not insurable. So I'm going to boot this device, and I'm going to do it with a slightly cheesy way by inserting a file, empty boot.script, because there's about five or six lines I have to type to get it to boot from RAM. Okay, off it goes. Okay, and then it's doing exciting stuff. It's going back through the scroll back here in Tmux, quite a lot of it. So what did it do? These are all U-boot commands that it's typing in. It's booting the TFTP. It's boot... There's the TFTP command. It's copying it into a particular staff address, which is carefully chosen not to clash with anything else. And then the bootM command is boot from memory. So there you see the TFTP happening. There it's loading the thing down. That's where it's... Sorry, I'm pointing at the play screen instead of your screen. That's the actual where Linux actually starts happening and starts in kernel, and the rest of it is all Linux and veriteers. And just to show you that we're not in X86 anymore. Yes, it is running Yerut-Shell on the console. This could be considered as a security problem if your threat model includes people prying the covers off and attaching 3Ys to the... That can be configured, of course. So there we are. It's a MediaTek MT67620, GL3MZ10 today. It's running on a MIPS. It's 335 Bogga MIPS if they can't run anything anymore and various other stuff there. That concludes the demonstration, I think, probably. So how does it work? I don't know if any of that writing is readable from where anyone is sitting. It surprisingly took longer to draw that than I thought it was going to. So what I'd like you to focus on here is... This is a description of the build process. This is our output at the bottom, which is the firmware binary that we just saw being TFTP booted. In that image, you've got a kernel image, which is this UImage thing here. You've got a file system image for the root file system, and the two of them basically just splattered together with DD. The file system image is generated from a configuration, which is a Nix value I'll show you in a moment, which is an address set with various different things you want to go into your image, and the image builder then makes the image out of it. Some of the things... I'm going to talk about modules, and I know a lot of people talk about modules. These aren't NixOS modules. These are something else. I named them badly. So you start with an empty configuration. You apply modules to it until you've got the config you want, and then you send it into the builder, and it builds your image. Some of the things you've figured out are package references, so we also set the Nix package answer there. We've got an overlay to make them smaller and so on. The other half of the picture is the kernel build, which comes from the kernel.org, the upstream. It also comes from openwrt. They get merged together. It gets built. We get the VM Linux file out of it. We stick the device tree into it, which is the data file I was telling you about earlier. That gets you the bootable image, and so that's the other half of the output there. So I think the three things in that picture, which you would have to touch if you were hacking on this, are the package overlay for including packages and making sure your packages are going to build on it, the module system, which is badly named, and potentially the kernel build if you want to get detailed about it. We're going to have a look at each of those. So there's good news. There's lots of good news. Lots of Nix packages already just works, which is awesome. We quite often have to patch our packages because we're not using the standard C library. Some things don't cross compile, so we need to disable the do-check clauses where they're in if they're unconditional. Sometimes you get very big closures. Malco was talking about big closures yesterday, and I really feel it on this thing because we have very limited rest of Flash Space. It can be as little as four megabytes. You don't want to be getting more than a half gigabytes of Firefox in there. So there are various hacks we've done to remove library dependencies for optional features where the original derivation included them. Get rid of shell scripts that depend on bash because bash is a huge... Yeah, if all those fails just run strings and grep for Nix Store, that's a good way of finding leaky bits and new closures. Make sure you stripped everything. And we also slightly hacked up the squashFS generation to remove static libraries from the generated output because who is ever going to use those when you don't have compiler? So here's an example. This is one of the entries in our package overlay for the host app, Demon. As you see, it's based on the upstream 1 in 6 packages. We've overrided the SQLite attribute because we don't need it. Sorry, the SQLite parameter we've overridden. Then we want to override the attributes as well, and we said we don't want extra config, and we're going to use this configure file which we generated here instead of using what the Nix packages one does. And that's a huge saving on host app D, and that's how I've got a wireless extender running in for Megaflash. Modules not the NixOS module system. We've had a lot this couple of days about the module system. The chief reason for writing it differently was that it seemed like fun. Still where this might be a better name. A module is a function which applies to configuration and generates new configuration, and we apply them repeatedly in the same kind of pattern as overlays do, except we're applying them to the configuration rather than applying them to the Nix package derivation. So your configuration object, that's really bad. I don't have a picture of the configuration object. Wow. Let's get forward a little bit there. What's it doing? Hey, right, okay. As revealed on JS, it's three dimensional side navigation. It's funky. So the essence of it is that in your derivation, you'll start with the base configuration which has almost nothing in it. It has empty arrays of files, packages, some other stuff. You then apply each module in sequence to it. So that's the hardware module for the device. That's in going to include an asking server. That's going to include an SSH server. That's going to include busybox, which is pretty fundamental. This particular device is my backup host, so it's got a USB disk module in it with some parameters. There's some stuff in the kernel. There's some stuff for configuring the network switch which is integrated into the device. And we're running syslogd, we're running NTPD, we're running a DHCP client. And to build the firmware, we merge all the modules together and pass it to the firmware generation function and that's our firmware.bin. The modules themselves actually look like this. You see the whole fixed point pattern up here. It's the self and the super. So that's our parent module. We're adding a service called hostappd. We're adding the hostappd package and we're adding a, right down at the bottom, I don't know if anyone can see over here as his head, right down at the bottom we're adding a file called etc.hostappd.psk, which has also got some content and a mode in it. And those styles are written into Flash, the same as everything else. Oh, I'm running out of time to talk about adding new hardware devices, which is great because this is the complicated bit and I'm going to skim through it. New hardware devices, I think I've had it running on four different devices so far, which are all sort of one or two of the various families. It is much, much, much easier if it already works in openWRT. In fact, I wouldn't even bother trying it unless you want a project to try and build something that isn't in openWRT already. There's kind of things you'll expect to have to do is find out what SOC family it runs on, is it Atheros, is it an RA link, is it a media tech, is it something else? Because that's going to impact whether it is a device tree or something else. Find out how to attach a serial cable to it. I've burnt one router by doing some really bad soldering on it already. As I said, does it use a device tree? And then you finally find out things like where in memory is the flash? How do I configure the network switch such that the ethernet in the kernel actually gets out to the box? All of this information for all the sort of devices in the file called devices.nix, and there's a lot of commentary in there as well. If you are minded to do your own device, you would do that. And yeah, if you can cobble together something to turn its power on and off remotely, you'll find remote development a lot easier for the earliest ages to bring up. I can show you my Arduino. It's not pretty. But it was stuff I had around. So Future Plans, obviously the first thing to do is to finish the PPP over internet supports because then I can actually run it on my primary router at home, which the family will love, I'm sure, when it starts crashing. Better story for upgrades. I've got a plan for upgrading without having to flash it every time. Better story for the first time support, which doesn't involve attaching cables to the device. That's going to depend on the device because your vendor firmware may allow you to upgrade freely or may have restrictions on the format, so that's going to be a bit of fun working on it. And a better story for secrets. So at the moment, we don't put secrets in the next store on the build machine. I've been quite careful about that because I checked that into GitHub, but they do end up burning the image. And if you want to change your password, then you end up having to reflash the image, which doesn't feel quite right. So design something which will get them for a writable file system or over a network or something else. I can quite see the appeal of network-based secrets if you're managing a decent size for a few of these devices. And perhaps even get some users. At the moment, it's just me. So that's it. Two minutes for questions. Ask me anything. I might not know the answer. I'm probably hand-wave. All righty. Thank you for your talk. Who has got questions? Yes, we've got a question. I have a pile of WRT54GL routers up in my attic. Is the four megabytes of RAM, does that look like your image was a little bit bigger? Is there a little bit more that could be cut to get this running on them? That image is bigger than the smallest people have around it. It has four megabytes of flash. And that's the wireless extender upstairs. I had to work quite hard to get it into four megs. But things can be done with the four meg image, yeah. More people with stuff in their basement. Sorry. Hello. Thanks for the talk. I'm wondering about whether we can implement DMVarity onto this DMVarity, like the Merkle hash trees just to check the hashes of the file system. It's not something I thought about. It would be a good idea, yeah. Another question? Nope. Then I thank you very much again for your wonderful talk. Thank you. All right. Thanks. next
 12 o'clock, let's continue with the next talk, which is going to be given by Daniel Barlow  and it's going to be about Nix WRT, which is a collection of derivations to basically  build flash images from WS.  Flash with a network driver and hopefully not brick it.  So yeah, give me a round of applause and enjoy the talk.  Thank you very much.  Yeah, my name is Daniel Barlow.  I'm here to talk about using the Nix Package Collection to build images for embedded routers,  IoT devices, things like this little widget here, which you can't see, but don't worry  there's pictures.  I'd like to start by saying welcome to London.  There's a nice iconic image here of Piccadilly Square with one of our London Black cabs in  it.  If you're not local, you might not know, they had to take a knowledge test to memorize  every road name within six miles of Charing Cross before they're allowed to drive one  of these cars.  Unfortunately, you can't see the contrast quite so well there, this is my iconic Black  X1 ThinkPad Carbon after a Black cab driver ran over it the other week.  It doesn't boot any more surprisingly enough.  So any problems with the quality or the polish in this talk, that's what I'm blaming you  on.  So yeah, Nix WRT or NixWirt, I came up with the name like six months ago and only just  today realize I don't know how to pronounce it.  I'm going to go with NixWirt, but I'm not going to be standard about it.  So when I started out with it, it was an experiment to see if I could use the Nix Package Collection  to build images for Wi-Fi routers of the kind usually run open-wirt or DDWRT or tomato,  that kind of thing.  Show of hands, I mean, who's got one of these things at home, who's tried flashing their  router at home?  Fair few people, yeah.  Okay.  So this isn't NixWirt on your router.  The image you get out of it is just an image.  It's immutable.  You can't log into it and compile packages or anything like that.  It's got no compilers.  Yeah.  So why?  Last November, I built a new computer for my home office.  You see it there.  It's not actually on fire.  It's got LEDs inside it.  And I wanted to be able to back it up.  I didn't want to build another computer to back up host because that feels a bit sort  of recursive.  So I've got a USB disk drive.  I've got a spare router with USB ports.  I know there's some kind of embedded Linux thing inside that router.  Maybe I can repurpose that and get some use out of it instead of throwing it into eWaste.  Why not use open-wirt?  And the first thing I have to say here is, open-wirt is great.  And there's so much work gone into it.  So many people working on different random problems which you would never even hope to  be able to replicate as a small developer.  I didn't realize how great it was until I tried bringing the mainstream Linux kernel  up on a MIPS device and found out I had no ethernet devider.  And a router with no ethernet driver is a little bit.  It doesn't root much.  Yeah.  However, I'm not going to be telling anyone anything you don't already know.  But when you're talking about divergent, convergent, and congruent change management, open-wirt  is on the left-hand end of the scale to a big extent.  You have 60 billion different packages in open-wirt.  You install them imperatively by typing commands on the thing itself.  Then you configure it using a GUI.  Six months later, you upgrade the firmware or the router blows up and you need to do  hardware, or your ISP says, try factory-resetting it.  Then you remember everything you did.  Maybe some people are diligent enough to back these things up properly, but I'm certainly  not.  So I got to thinking, maybe there is a more general problem to be solved here than just  somewhere safe to store my RIP CDs.  So yeah, did some hacking, did some blogging, got distracted, got distracted again, got my  laptop trashed, and managed to trash my own home network in several different fun ways.  Here we are a year on.  Is it done?  There's a little bit of scope creep.  I did the backup server.  The backup server is working fine.  That's great.  I did what I came for.  I repurposed the wireless range extender in my study upstairs to run the XWRT as well.  The router downstairs, which is the one that actually connects to broadband, I'm still  working on that, take over the world as kind of a stretch goal.  The other question, I said it was an experiment, so what were the findings, I guess?  Is Nix good for this stuff?  And emphatically, yes, it is.  Okay, I'm kind of preaching to the converted.  But the Nix language, compared to anything else I've used in terms of configuration  management or building is superb.  Everyone knows that.  The cross compilation stuff, you might not know unless you've been involved with it,  has all sort of beginning of 2018.  A lot of work you've done last year on cross compilation in Nix.  And it's really made it easy to build Nix binaries from an X86 system.  And the support for Muzzle, which is an alternative C library to the new C library, it's smaller  and faster and more compliant and works better on embedded systems.  I think that's pretty new as well.  It's certainly the case that I've hardly ever hit problems trying to use a C library.  It's not the new C library.  So on those three counts, Nix is pretty awesome.  Overlays, again, you could do various ways of customizing derivations before overlays  came in, but having a consistent way of doing this reasonably principled is really useful  and really helpful for making packages smaller.  And I'll talk about it more in a minute.  So, yeah, there are things I've learned along the way that if you wanted to get into it,  you would also end up learning sooner or later, you know, how to read and write Nix derivations,  stuff about Linux, stuff about how the kernel is put together, things about networks, switches,  CTP, Ethernet, what's a Mac, what's a Fi, overlays, fixed points are awesome.  I know enough to hand wave about them.  I don't know enough to explain them, so I'm going to do some hand waving in a minute.  If you have enough coffee, the rest of it is easy.  So what do you actually need?  Obviously, you need some kind of thing to run it on.  This is the GLINET MT300N, which is like GLINET MT300A, but it's a different color.  It's slightly cheaper, and the hardware inside it's slightly different.  So, yeah, these things, these are not your typical PCs.  They're smaller, they're slower.  So graphics hardware, so you establish a console connection to it by attaching three  wires to it, which is more or less complicated, depending on whether there are pin headers  there to attach them to or you need to go and do some bad soldering.  Obviously, the architecture is different.  It runs some variety of MIPS, or the ones I've tried to say if I do anywhere, I guess  some people are using ARM for this stuff.  You don't boot from the BIOS or from UEFI.  You don't use grub, users, and called Uboots, which I'll be showing you in a minute.  And the way the particular board knows about all the bits that are in it, like where the  GPI opens are, where the LEDs are, how to make the Ethernet, how to initialize the little  network switch inside it.  On a grown-up computer like a PC, you've got things like ACPI where it can sort of go  and enumerate the bus and find out where all the things are because all the things will  say, look, I'm over here.  On these older, smaller systems, either that knowledge is compiled into the kernel for  the particular board you're using or on some varieties, a device tree, which is basically  a data file with the same information in it.  So we extracted that code into data.  The device tree is a better way of doing it, but not all boards have been updated to use  it yet.  Yeah, so in terms of mixed WR2, if you want to play with it, the best supported boards  are the ones based on the AI9330 or the ones based on the MediaTek socks.  And I say the blue one and the yellow one are both cheap and both easy to get hold of.  And also no soldering required.  You just pop the top off.  Is there an emulator?  QEMU works, but the hardware emulates not very much like the real hardware.  Testing all the real hardware is actually not that bad.  So obviously you need to connect it together to something.  This is my test setup.  It's a little bit more baroque than it needs to be.  What you will see is the device itself with three cables coming out of it for the serial  console.  The serial console is TTL level logic.  It runs into something which understands that.  In my case at the moment, that's a Raspberry Pi because I blew up my serial cable.  And then that, that is the most overkill use ever of an Arduino YUN, which basically exists  to toggle the USPM power off and off.  So when I wedge it solid, I can turn it off and on again without having to go upstairs  and pull the USB cable out.  So I'm going to do a little demonstration of what it looks like to build on or what  the actual hardware looks like.  I'm not going to demo on this thing here because it would take too long to plug everything  together.  I'm going to attempt the SSH system at home and show you it there.  You ask what could go wrong, well, this guy here actually, my son has dropped to the age  where he really likes to turn the power on and off in the front of the machine.  So hopefully it's still there when we get there.  We're just about to go find out.  Right.  Nope.  That's not it.  Either is it.  Ah, it was that one.  Okay.  So let's have 80 columns as the good Lord intended.  My handle on most online system is talent, which I chose a long time ago because I couldn't  type telnet correctly without typoing it.  So the host name theme is all typos as well.  So here we are.  I'm going to start by making it.  It does nothing of course because I made it last night to check that everything worked  and therefore there's nothing to be done.  So there's a little make file just because there's an awful lot of parameters to give  to Nick builds.  So the particular derivation we're using, I'll show you that in a minute, it's called  backup host.  We're building a target called framework.  Just parameters, we're passing it for things like SSH keys and an rsync password and other  stuff and you'll see it is eventually built a file in that directory there and then it's  passing it into my TFTP server directory.  And I'm just going to show you that...  Yeah, there it is.  So that is my firmware file, which is about four and a half megabase in size, which is  okay for that particular target device.  Put that back in the stand.  So I'm going to introduce some insignificant white space into one of these derivations.  Oh, that's nice.  No, I'll come back to that bit.  Ah, here we are.  So this is...  Is that volume level okay for everyone?  Yeah.  So this is the device itself.  I'm connected to it over an SSH section, over an SSH section, over a mini-com serial  session over the three wires you saw in the picture into the actual hardware device.  I'm just going to reset it just to show you it's there and I'm going to stop auto-boot.  So this is this thing called U-boot.  It's the universal boot loader.  It's called the universal boot loader.  What actually happens is that hardware manufacturers take it, fork it, hack it up, burn it onto  their machines.  Getting a new version of U-boot onto your device is a bit like trying to put core boot  onto your laptop.  It can be done, but if it goes wrong, you need JTAG debuggers and stuff like that.  So generally speaking, the one you've got is the one you're going to live with unless  you want to get more complicated than I've got.  So anyway, you've got a connection of commands in U-boot here, which do things like print  the environment variable, boots, change memory addresses, let you do things with flash and  so on.  So if I just do print-env, they are more or less defective depending on the manufacturer.  So in theory, if you've done a proper good install of U-boot, then this environment will  be writable and you can change parameters here.  Then you will find that the hardware manufacturer has just hard-coded the environment and you  can't actually save anything, which is annoying, but not insurable.  So I'm going to boot this device, and I'm going to do it with a slightly cheesy way by  inserting a file, empty boot.script, because there's about five or six lines I have to  type to get it to boot from RAM.  Okay, off it goes.  Okay, and then it's doing exciting stuff.  It's going back through the scroll back here in Tmux, quite a lot of it.  So what did it do?  These are all U-boot commands that it's typing in.  It's booting the TFTP.  It's boot...  There's the TFTP command.  It's copying it into a particular staff address, which is carefully chosen not to clash with  anything else.  And then the bootM command is boot from memory.  So there you see the TFTP happening.  There it's loading the thing down.  That's where it's...  Sorry, I'm pointing at the play screen instead of your screen.  That's the actual where Linux actually starts happening and starts in kernel, and the rest  of it is all Linux and veriteers.  And just to show you that we're not in X86 anymore.  Yes, it is running Yerut-Shell on the console.  This could be considered as a security problem if your threat model includes people prying  the covers off and attaching 3Ys to the...  That can be configured, of course.  So there we are.  It's a MediaTek MT67620, GL3MZ10 today.  It's running on a MIPS.  It's 335 Bogga MIPS if they can't run anything anymore and various other stuff there.  That concludes the demonstration, I think, probably.  So how does it work?  I don't know if any of that writing is readable from where anyone is sitting.  It surprisingly took longer to draw that than I thought it was going to.  So what I'd like you to focus on here is...  This is a description of the build process.  This is our output at the bottom, which is the firmware binary that we just saw being  TFTP booted.  In that image, you've got a kernel image, which is this UImage thing here.  You've got a file system image for the root file system, and the two of them basically  just splattered together with DD.  The file system image is generated from a configuration, which is a Nix value I'll show  you in a moment, which is an address set with various different things you want to go into  your image, and the image builder then makes the image out of it.  Some of the things...  I'm going to talk about modules, and I know a lot of people talk about modules.  These aren't NixOS modules.  These are something else.  I named them badly.  So you start with an empty configuration.  You apply modules to it until you've got the config you want, and then you send it into  the builder, and it builds your image.  Some of the things you've figured out are package references, so we also set the Nix  package answer there.  We've got an overlay to make them smaller and so on.  The other half of the picture is the kernel build, which comes from the kernel.org, the  upstream.  It also comes from openwrt.  They get merged together.  It gets built.  We get the VM Linux file out of it.  We stick the device tree into it, which is the data file I was telling you about earlier.  That gets you the bootable image, and so that's the other half of the output there.  So I think the three things in that picture, which you would have to touch if you were  hacking on this, are the package overlay for including packages and making sure your packages  are going to build on it, the module system, which is badly named, and potentially the  kernel build if you want to get detailed about it.  We're going to have a look at each of those.  So there's good news.  There's lots of good news.  Lots of Nix packages already just works, which is awesome.  We quite often have to patch our packages because we're not using the standard C library.  Some things don't cross compile, so we need to disable the do-check clauses where they're  in if they're unconditional.  Sometimes you get very big closures.  Malco was talking about big closures yesterday, and I really feel it on this thing because  we have very limited rest of Flash Space.  It can be as little as four megabytes.  You don't want to be getting more than a half gigabytes of Firefox in there.  So there are various hacks we've done to remove library dependencies for optional features  where the original derivation included them.  Get rid of shell scripts that depend on bash because bash is a huge...  Yeah, if all those fails just run strings and grep for Nix Store, that's a good way of finding  leaky bits and new closures.  Make sure you stripped everything.  And we also slightly hacked up the squashFS generation to remove static libraries from  the generated output because who is ever going to use those when you don't have compiler?  So here's an example.  This is one of the entries in our package overlay for the host app, Demon.  As you see, it's based on the upstream 1 in 6 packages.  We've overrided the SQLite attribute because we don't need it.  Sorry, the SQLite parameter we've overridden.  Then we want to override the attributes as well, and we said we don't want extra config,  and we're going to use this configure file which we generated here instead of using what  the Nix packages one does.  And that's a huge saving on host app D, and that's how I've got a wireless extender running  in for Megaflash.  Modules not the NixOS module system.  We've had a lot this couple of days about the module system.  The chief reason for writing it differently was that it seemed like fun.  Still where this might be a better name.  A module is a function which applies to configuration and generates new configuration, and we apply  them repeatedly in the same kind of pattern as overlays do, except we're applying them  to the configuration rather than applying them to the Nix package derivation.  So your configuration object, that's really bad.  I don't have a picture of the configuration object.  Wow.  Let's get forward a little bit there.  What's it doing?  Hey, right, okay.  As revealed on JS, it's three dimensional side navigation.  It's funky.  So the essence of it is that in your derivation, you'll start with the base configuration which  has almost nothing in it.  It has empty arrays of files, packages, some other stuff.  You then apply each module in sequence to it.  So that's the hardware module for the device.  That's in going to include an asking server.  That's going to include an SSH server.  That's going to include busybox, which is pretty fundamental.  This particular device is my backup host, so it's got a USB disk module in it with some  parameters.  There's some stuff in the kernel.  There's some stuff for configuring the network switch which is integrated into the device.  And we're running syslogd, we're running NTPD, we're running a DHCP client.  And to build the firmware, we merge all the modules together and pass it to the firmware  generation function and that's our firmware.bin.  The modules themselves actually look like this.  You see the whole fixed point pattern up here.  It's the self and the super.  So that's our parent module.  We're adding a service called hostappd.  We're adding the hostappd package and we're adding a, right down at the bottom, I don't  know if anyone can see over here as his head, right down at the bottom we're adding a file  called etc.hostappd.psk, which has also got some content and a mode in it.  And those styles are written into Flash, the same as everything else.  Oh, I'm running out of time to talk about adding new hardware devices, which is great  because this is the complicated bit and I'm going to skim through it.  New hardware devices, I think I've had it running on four different devices so far,  which are all sort of one or two of the various families.  It is much, much, much easier if it already works in openWRT.  In fact, I wouldn't even bother trying it unless you want a project to try and build  something that isn't in openWRT already.  There's kind of things you'll expect to have to do is find out what SOC family it runs  on, is it Atheros, is it an RA link, is it a media tech, is it something else?  Because that's going to impact whether it is a device tree or something else.  Find out how to attach a serial cable to it.  I've burnt one router by doing some really bad soldering on it already.  As I said, does it use a device tree?  And then you finally find out things like where in memory is the flash?  How do I configure the network switch such that the ethernet in the kernel actually gets  out to the box?  All of this information for all the sort of devices in the file called devices.nix,  and there's a lot of commentary in there as well.  If you are minded to do your own device, you would do that.  And yeah, if you can cobble together something to turn its power on and off remotely, you'll  find remote development a lot easier for the earliest ages to bring up.  I can show you my Arduino.  It's not pretty.  But it was stuff I had around.  So Future Plans, obviously the first thing to do is to finish the PPP over internet  supports because then I can actually run it on my primary router at home, which the family  will love, I'm sure, when it starts crashing.  Better story for upgrades.  I've got a plan for upgrading without having to flash it every time.  Better story for the first time support, which doesn't involve attaching cables to the device.  That's going to depend on the device because your vendor firmware may allow you to upgrade  freely or may have restrictions on the format, so that's going to be a bit of fun working  on it.  And a better story for secrets.  So at the moment, we don't put secrets in the next store on the build machine.  I've been quite careful about that because I checked that into GitHub, but they do end  up burning the image.  And if you want to change your password, then you end up having to reflash the image, which  doesn't feel quite right.  So design something which will get them for a writable file system or over a network or  something else.  I can quite see the appeal of network-based secrets if you're managing a decent size for  a few of these devices.  And perhaps even get some users.  At the moment, it's just me.  So that's it.  Two minutes for questions.  Ask me anything.  I might not know the answer.  I'm probably hand-wave.  All righty.  Thank you for your talk.  Who has got questions?  Yes, we've got a question.  I have a pile of WRT54GL routers up in my attic.  Is the four megabytes of RAM, does that look like your image was a little bit bigger?  Is there a little bit more that could be cut to get this running on them?  That image is bigger than the smallest people have around it.  It has four megabytes of flash.  And that's the wireless extender upstairs.  I had to work quite hard to get it into four megs.  But things can be done with the four meg image, yeah.  More people with stuff in their basement.  Sorry.  Hello.  Thanks for the talk.  I'm wondering about whether we can implement DMVarity onto this DMVarity, like the Merkle  hash trees just to check the hashes of the file system.  It's not something I thought about.  It would be a good idea, yeah.  Another question?  Nope.  Then I thank you very much again for your wonderful talk.  Thank you.  All right.  Thanks.  next
Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_0_731.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_732_815.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_816_1475.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_1476_1571.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_1572_4067.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_4068_4667.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_4668_4751.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_4752_6731.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_6732_9803.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_9804_10103.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_10104_17171.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_17172_21815.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_21816_24719.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_24720_25811.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_25812_30359.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_30360_34859.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_34860_35243.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_35244_35567.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_35568_35927.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_35928_36599.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_36600_37079.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_37080_38219.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_38220_42923.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_42924_43259.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_43260_45119.png Daniel_Barlow_-_NixWRT__purely_functional_firmware_images_for_IoT_0K1qn60X2HI_45120_45143.png