boot/loader: How to boot Linux and nothing else
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Stelmach, Łukasz
To boot Linux on a new ARM/ARM64 platform we have to port Linux to that platform and a bootloader. Aside from the platform setup code, we need to add and maintain some device drivers to both Linux and the bootloader. We decided to avoid the extra effort and get rid of the dedicated bootloader. With a few dozen lines of assembly code, Linux Kernel and a pinch of userland tools the "boot" kernel was running with all the cool features we wanted! Then we used kexec(2) to start a "full" Linux kernel.
 Hello everyone, it's nice to see all of you here. Some of you were on the previous talk and those of you who were then will probably better understand what I will say. My name is Łukasz, I work at Samsuk R&D Institute Poland. Today I would like to share with you an idea we've got in my team. We decided to replace a bootloader on one of our development platforms with Linux kernel and the handful of user-launch tools. This by no means is something new. We did it because others did it before us and here on FOSDEM I have already seen at least two talks about it. But we've got our approach which I would like to share with you. We think there are several good reasons to do that, one of them us being a little bit lazy as probably some of you here too and liking others to help us do our job. We want to avoid double effort that is required to develop the, to bring up a new platform because code for both Linux kernel and the bootloader needs to be developed to get a functional platform. And we think there is no need to create code for both Linux and bootloader. Because the talk is rather short and I really don't want to miss the ending, I will do my best to tell you everything I've got and then I will answer your questions. This presentation has got two parts, unlike what's on the screen. The first describes more or less the current state of publicly available bootloaders on ARM Cortex-A CPUs. Angelo said a lot more about that. The second is about our work, how we think it is possible to improve the situation. For those who are unfamiliar with the topic on ARM, there is no standard firmware like BIOS or UEFI on X86. That is why it is a bootloader's job to configure most important hardware components. Let me ask how many of you have touched a bootloader code or an architecture dependent Linux code that runs before start kernel function? Okay, we are on the same page. That's nice. In this, maybe some of you are more clever than I am. In this presentation, I assume the booting process comprises the following stages. The firmware is a piece of, this is what others call the RBL, Angelo called RBL. This is a piece of immutable code that comes with a CPU. It cannot be changed by any means. Then there is a bootloader, aka SPL, divided into several stages. And then, of course, there is an OS, the code that implements the application the device was designed for, like desktop environment or multimedia system or choose yours. An additional task bootloaders sometimes perform is an OS update, especially when the OS is broken. This means a bootloader should be very robust as it may be the only way to unbreak a device. We concluded that the available bootloaders for ARM have so significant drawbacks and we need to improve upon that. Following the experience of the Linux boot project, we decided to use Linux as a bootloader. These are the three most common bootloaders I could find for ARM, each of them support different architectures too. There is a lot more proprietary, commercial, custom ones. Until now, in our work, we have used Uboot on most development platforms and some custom bootloaders on products. Every new platform needs a bootloader to be ported to it. You build a PCB, you smash an SOC onto it, to boot it, you need a bootloader. Just to repeat, I mean all the code from that is loaded from ROM that runs until Linux kicks in. So that's what I consider a bootloader. Supporting a bootloader to a new platform requires writing platform setup code which will configure the platform to be usable. It turns on DRAM, some clocks, some voltages on board, some basic peripherals like console. Then there is a general purpose code which deals with loading OS kernel from a storage or from a network and starting it. In traditional bootloaders, the range of options here in this area is somewhat limited. On the other hand, for Linux, there are tons of device drivers, dozens of file systems, and more than one network stack too. And thanks to Kexec, Linux can act as a bootloader and start completely new kernel, load it from wherever you want and start it as a new system. This is Android XU4. It is based on Exynos 542 SoC and it has been a reference platform for Tizen OS for the past couple of years and almost everyone in our office has one in their drawer. That's why we chose it for this proof of concept. It's well supported in the mainline Linux and we knew that if it doesn't boot, then it is our fault, we're doing something wrong. It wasn't entirely true, but that's not the point. Now we fixed like probably two bugs somewhere in the compressor code that nobody looks into, it works and we did something different and it appeared a fix was needed. There have been a number of similar attempts to push Linux as close to hardware as possible, mostly on X86 platforms and as I learned today on RISC 5.2. We work with ARM platform and thus our approach is slightly different. Please do keep in mind also that this was kind of a side project and we focused on getting the most important functionality available and we're not bothered by any convenience feature too much. Okay, the boot on Android looks as follows. There are three blobs provided by the vendor which is hard kernel and they are flashed with appropriate script, they are flashed onto SD or EMMC card so the board can boot. BL1 does the very basic platform setup which is turning on DRAM and reads BL2. BL2 loads trust zone and unsigned UBOOT image. Trust zone blob appears to be doing nothing because there are a lot of null bytes so we don't have source code for it, it doesn't do anything I think. No, it doesn't behave wrong, we don't have any problems with it. Let's stay with that. All three blobs need to be signed. Null signs trust zone handler and BL2 upon request on their forum but this is a little bit cumbersome to change them so we decided we will replace UBOOT, just UBOOT. We didn't replace any of these. There is however as you might see, the name of the last node, the BL2, suggests the largest hurdle we faced that is one megabyte limit on the UBOOT image. We had to fit within that limit. After a quick recap, we found these four tasks were essential to boot the board with Linux kernel as a bootloader. Platform setup, as I said, this was rather quick. The BL1 and BL2 code does all of the setup except for console. We did console with a small, I don't know, 50, 60 bytes shim. That's easy. One megabyte is very, very little for a kernel, device tree and reasonable, whatever that means, user land. After disabling almost everything in kernel, what we could, our image took like 950 kilobytes so we were left with 50 or 60 kilobytes for the user land part. For the record, without such strict limit, we would build a separate bootloading kernel anyway with different configuration than the one we used for the OS. So two kernels are always within required. Maybe the first one could be not that much limited. Linux, luckily, Kexec worked. However, we had recently some other boards which had some problems with Kexec. It appears to be the problem with drivers of some devices that don't shut down devices properly and when a new kernel starts, drivers try to probe devices. The devices are in states that cannot be handled at the probing stage by the drivers. So this will need investigation. Although we weren't much into providing user land tools, just a few bits to load the kernel started, we found the U-root used in Linux boot, for example, to be a very nice piece of software for that. Much better than, in our opinion, much better than other such toolboxes. Alas, it doesn't support Kexec on-arm. We had to use Kexec tools, but that's just a minor problem. Thanks to TMPFS, which, unlike old-style RAM disk, is able to expand in RAM as new data is saved, we could split our init-RAM-FS archive into two. The first stage comprises only a tiny init we call half-stage init, which mounts FAT partition and packs stage two archive, or stage one full archive, execute slash init that is expected to be in the archive and replaces HSE init. There is no limit for the stage two archive, except, of course, the size of RAM, size and speed of storage and such things. U-root, short for universal root, is a really nice piece of software I'd like to recommend you if you build boot-loading environments. It is a set of basic tools. There is a shell, text editor, LS, cut, and stuff that you find in all the boxes out there. But unlike other boxes, it is written in Go. It can be deployed in two modes, Source and Busybox mode. The former is need because the only binary that is on the init-RAM-FS is Go compiler, and every tool that you use is compiled on demand. Unfortunately, it requires much more space and rather strong CPU. On ARM, it was too slow to build the commands. The Busybox mode is a single binary with multiple sim links, calling multiple functions. At the moment, what we've got, we can configure the kernel to build an image for a selected board, in our case it is XU4, to boot without U-boot and start another fully-fledged kernel. Although XU4 cannot act as a USB device, to use Linux on other our boards, we need to develop function FS-based tool to receive files, to receive and download to the Flash. We also need to make Kexec tool in U-root work on ARM. Why? You may ask. From our perspective, there are following advantages to this approach. Let's effort during platform bring up. Because we avoid code duplication, this extends also to maintenance. You don't end up with, I don't know for example, PMIC driver into repositories in Linux kernel where you need it for power management and in bootloader where you need it to bring up the platform. Linux more flexibility. Linux both the kernel and the user land is much more capable and flexible than bootloaders available. Two areas we consider crucial are network support and security. So it is possible to boot over a network today. Only TFTP is available for file transfer which is perfectly fine from a client perspective. However, without an authentication, it is not a protocol I would deploy outside any local network. Support for crypto libraries is very new and quite limited as far as I can tell. And finally, from our experience, the general purpose code like file systems, like network stack, like, I don't know, USB stack, not the driver for the hardware, but the rest of the stack is somewhat in better shape in the Linux kernel and is maintained more regularly. That is our experience. In summary, the key features of our approach of what we did, we are definitely arm centric because that's the platform we work with. We assume, however, that it applies directly to other platforms which don't employ standards based firmware like BIOS or UEFI. Our motivation to push Linux as close to the hardware as possible is a little different than on X86. We are not concerned much about security like management engine on X86 that affect platform user, but rather we want to help vendors, ourselves included, to minimize the effort to bring up the platform, to avoid the unnecessary effort to prepare two versions of device drivers and keep file systems up to date and so on and so on. Our goal is to find a place and means in the Linux kernel source tree to keep platform, board dependent code, which can do platform setup before the kernel starts and can be built together with the kernel to form a single binary that the firmware, the Rambut loader can load, alternatively signed SPL. Although this might look like we try to resurrect board files, we don't. The platform specific code that we need is not the same as the board files that were used before device tree became popular. The code needs to bring up only several components, which are not accessed, which are not controlled later on. With this in place, porting Linux would require less effort, less amount of work, and you get really, really powerful boot environment. These are the guys that helped me a lot by either providing knowledge or allowing me to work on this as a site project. Special thanks go to Mateusz, who wrote the small init, the HS init, and squeezed it really hard to fit in the limit. Okay. I managed somehow. Any questions? Any questions? I guess it's more. I guess it's more common than a question, but I guess the question comes along. First off, I think it's an amazing idea to use Linux as your driver toolkit because then you're only right to drive us once, right? Doing those in U-boot and in Linux at the same time is a terrible idea either way, simply because you always get this typical fork effect where you have diverging drivers sooner or later. The biggest you're I see is your, how do I put this? The platform init code is something that the ARM world was really proud of to finally get rid of. I don't see us reinventing that or reintroducing it in a 64-bit ARM world. It just is not going to happen. We need to find a different way to get this tiny platform init code in between your basically a BL2 in the actual Linux kernel. This is where I really would like to brainstorm with you on cool ideas we can do. I think we have a couple of things on the plate of what is possible to create a tiny shim that runs before Linux so that you don't have to put the platform code into the Linux kernel itself and the tree itself. I'm very sorry, but there was something. Could you come and repeat me the question without the mic? The platform init code you're putting into Linux kernel. Before the Linux kernel. So is it actually separate standalone code? It's a single file. But it's in Linux. In the source tree. Yes. I think we need to work on untangling this. The question was is the shim we prepend to Linux kernel inside the kernel tree? Yes, we want to be inside the kernel tree. And I haven't mentioned that. I have posted RFC patches to the list and they are available for commenting. So it's not like we want to push something. We want to discuss it because Linux kernel, the source tree can become a single stop shop for vendors, for people to boot code that boots. We're definitely open for discussion. You can walk with their food, not with their mouth. So please, if you are going out of the room, please keep somehow silent to let the speaker finish the talk. Thank you. Okay, it's up here. I finished. Yeah. Thank you. Thank you, too.
 Hello everyone, it's nice to see all of you here.  Some of you were on the previous talk and those of you who were then will probably better  understand what I will say.  My name is Łukasz, I work at Samsuk R&D Institute Poland.  Today I would like to share with you an idea we've got in my team.  We decided to replace a bootloader on one of our development platforms with Linux kernel  and the handful of user-launch tools.  This by no means is something new.  We did it because others did it before us and here on FOSDEM I have already seen at  least two talks about it.  But we've got our approach which I would like to share with you.  We think there are several good reasons to do that, one of them us being a little bit  lazy as probably some of you here too and liking others to help us do our job.  We want to avoid double effort that is required to develop the, to bring up a new platform  because code for both Linux kernel and the bootloader needs to be developed to get a  functional platform.  And we think there is no need to create code for both Linux and bootloader.  Because the talk is rather short and I really don't want to miss the ending, I will do my  best to tell you everything I've got and then I will answer your questions.  This presentation has got two parts, unlike what's on the screen.  The first describes more or less the current state of publicly available bootloaders on  ARM Cortex-A CPUs.  Angelo said a lot more about that.  The second is about our work, how we think it is possible to improve the situation.  For those who are unfamiliar with the topic on ARM, there is no standard firmware like  BIOS or UEFI on X86.  That is why it is a bootloader's job to configure most important hardware components.  Let me ask how many of you have touched a bootloader code or an architecture dependent  Linux code that runs before start kernel function?  Okay, we are on the same page.  That's nice.  In this, maybe some of you are more clever than I am.  In this presentation, I assume the booting process comprises the following stages.  The firmware is a piece of, this is what others call the RBL, Angelo called RBL.  This is a piece of immutable code that comes with a CPU.  It cannot be changed by any means.  Then there is a bootloader, aka SPL, divided into several stages.  And then, of course, there is an OS, the code that implements the application the device  was designed for, like desktop environment or multimedia system or choose yours.  An additional task bootloaders sometimes perform is an OS update, especially when the  OS is broken.  This means a bootloader should be very robust as it may be the only way to unbreak a device.  We concluded that the available bootloaders for ARM have so significant drawbacks and  we need to improve upon that.  Following the experience of the Linux boot project, we decided to use Linux as a bootloader.  These are the three most common bootloaders I could find for ARM, each of them support  different architectures too.  There is a lot more proprietary, commercial, custom ones.  Until now, in our work, we have used Uboot on most development platforms and some custom  bootloaders on products.  Every new platform needs a bootloader to be ported to it.  You build a PCB, you smash an SOC onto it, to boot it, you need a bootloader.  Just to repeat, I mean all the code from that is loaded from ROM that runs until Linux kicks  in.  So that's what I consider a bootloader.  Supporting a bootloader to a new platform requires writing platform setup code which  will configure the platform to be usable.  It turns on DRAM, some clocks, some voltages on board, some basic peripherals like console.  Then there is a general purpose code which deals with loading OS kernel from a storage  or from a network and starting it.  In traditional bootloaders, the range of options here in this area is somewhat limited.  On the other hand, for Linux, there are tons of device drivers, dozens of file systems,  and more than one network stack too.  And thanks to Kexec, Linux can act as a bootloader and start completely new kernel, load it from  wherever you want and start it as a new system.  This is Android XU4.  It is based on Exynos 542 SoC and it has been a reference platform for Tizen OS for the  past couple of years and almost everyone in our office has one in their drawer.  That's why we chose it for this proof of concept.  It's well supported in the mainline Linux and we knew that if it doesn't boot, then  it is our fault, we're doing something wrong.  It wasn't entirely true, but that's not the point.  Now we fixed like probably two bugs somewhere in the compressor code that nobody looks into,  it works and we did something different and it appeared a fix was needed.  There have been a number of similar attempts to push Linux as close to hardware as possible,  mostly on X86 platforms and as I learned today on RISC 5.2.  We work with ARM platform and thus our approach is slightly different.  Please do keep in mind also that this was kind of a side project and we focused on getting  the most important functionality available and we're not bothered by any convenience  feature too much.  Okay, the boot on Android looks as follows.  There are three blobs provided by the vendor which is hard kernel and they are flashed with  appropriate script, they are flashed onto SD or EMMC card so the board can boot.  BL1 does the very basic platform setup which is turning on DRAM and reads BL2.  BL2 loads trust zone and unsigned UBOOT image.  Trust zone blob appears to be doing nothing because there are a lot of null bytes so we  don't have source code for it, it doesn't do anything I think.  No, it doesn't behave wrong, we don't have any problems with it.  Let's stay with that.  All three blobs need to be signed.  Null signs trust zone handler and BL2 upon request on their forum but this is a little  bit cumbersome to change them so we decided we will replace UBOOT, just UBOOT.  We didn't replace any of these.  There is however as you might see, the name of the last node, the BL2, suggests the largest  hurdle we faced that is one megabyte limit on the UBOOT image.  We had to fit within that limit.  After a quick recap, we found these four tasks were essential to boot the board with Linux  kernel as a bootloader.  Platform setup, as I said, this was rather quick.  The BL1 and BL2 code does all of the setup except for console.  We did console with a small, I don't know, 50, 60 bytes shim.  That's easy.  One megabyte is very, very little for a kernel, device tree and reasonable, whatever that  means, user land.  After disabling almost everything in kernel, what we could, our image took like 950 kilobytes  so we were left with 50 or 60 kilobytes for the user land part.  For the record, without such strict limit, we would build a separate bootloading kernel  anyway with different configuration than the one we used for the OS.  So two kernels are always within required.  Maybe the first one could be not that much limited.  Linux, luckily, Kexec worked.  However, we had recently some other boards which had some problems with Kexec.  It appears to be the problem with drivers of some devices that don't shut down devices  properly and when a new kernel starts, drivers try to probe devices.  The devices are in states that cannot be handled at the probing stage by the drivers.  So this will need investigation.  Although we weren't much into providing user land tools, just a few bits to load the kernel  started, we found the U-root used in Linux boot, for example, to be a very nice piece  of software for that.  Much better than, in our opinion, much better than other such toolboxes.  Alas, it doesn't support Kexec on-arm.  We had to use Kexec tools, but that's just a minor problem.  Thanks to TMPFS, which, unlike old-style RAM disk, is able to expand in RAM as new data  is saved, we could split our init-RAM-FS archive into two.  The first stage comprises only a tiny init we call half-stage init, which mounts FAT  partition and packs stage two archive, or stage one full archive, execute slash init  that is expected to be in the archive and replaces HSE init.  There is no limit for the stage two archive, except, of course, the size of RAM, size and  speed of storage and such things.  U-root, short for universal root, is a really nice piece of software I'd like to recommend  you if you build boot-loading environments.  It is a set of basic tools.  There is a shell, text editor, LS, cut, and stuff that you find in all the boxes out there.  But unlike other boxes, it is written in Go.  It can be deployed in two modes, Source and Busybox mode.  The former is need because the only binary that is on the init-RAM-FS is Go compiler,  and every tool that you use is compiled on demand.  Unfortunately, it requires much more space and rather strong CPU.  On ARM, it was too slow to build the commands.  The Busybox mode is a single binary with multiple sim links, calling multiple functions.  At the moment, what we've got, we can configure the kernel to build an image for a selected  board, in our case it is XU4, to boot without U-boot and start another fully-fledged kernel.  Although XU4 cannot act as a USB device, to use Linux on other our boards, we need to  develop function FS-based tool to receive files, to receive and download to the Flash.  We also need to make Kexec tool in U-root work on ARM.  Why?  You may ask.  From our perspective, there are following advantages to this approach.  Let's effort during platform bring up.  Because we avoid code duplication, this extends also to maintenance.  You don't end up with, I don't know for example, PMIC driver into repositories in Linux kernel  where you need it for power management and in bootloader where you need it to bring up  the platform.  Linux more flexibility.  Linux both the kernel and the user land is much more capable and flexible than bootloaders  available.  Two areas we consider crucial are network support and security.  So it is possible to boot over a network today.  Only TFTP is available for file transfer which is perfectly fine from a client perspective.  However, without an authentication, it is not a protocol I would deploy outside any local  network.  Support for crypto libraries is very new and quite limited as far as I can tell.  And finally, from our experience, the general purpose code like file systems, like network  stack, like, I don't know, USB stack, not the driver for the hardware, but the rest  of the stack is somewhat in better shape in the Linux kernel and is maintained more  regularly.  That is our experience.  In summary, the key features of our approach of what we did, we are definitely arm centric  because that's the platform we work with.  We assume, however, that it applies directly to other platforms which don't employ standards  based firmware like BIOS or UEFI.  Our motivation to push Linux as close to the hardware as possible is a little different  than on X86.  We are not concerned much about security like management engine on X86 that affect platform  user, but rather we want to help vendors, ourselves included, to minimize the effort  to bring up the platform, to avoid the unnecessary effort to prepare two versions of device drivers  and keep file systems up to date and so on and so on.  Our goal is to find a place and means in the Linux kernel source tree to keep platform,  board dependent code, which can do platform setup before the kernel starts and can be  built together with the kernel to form a single binary that the firmware, the Rambut  loader can load, alternatively signed SPL.  Although this might look like we try to resurrect board files, we don't.  The platform specific code that we need is not the same as the board files that were  used before device tree became popular.  The code needs to bring up only several components, which are not accessed, which are not controlled  later on.  With this in place, porting Linux would require less effort, less amount of work, and you  get really, really powerful boot environment.  These are the guys that helped me a lot by either providing knowledge or allowing me  to work on this as a site project.  Special thanks go to Mateusz, who wrote the small init, the HS init, and squeezed it really  hard to fit in the limit.  Okay.  I managed somehow.  Any questions?  Any questions?  I guess it's more.  I guess it's more common than a question, but I guess the question comes along.  First off, I think it's an amazing idea to use Linux as your driver toolkit because then  you're only right to drive us once, right?  Doing those in U-boot and in Linux at the same time is a terrible idea either way, simply  because you always get this typical fork effect where you have diverging drivers sooner or  later.  The biggest you're I see is your, how do I put this?  The platform init code is something that the ARM world was really proud of to finally  get rid of.  I don't see us reinventing that or reintroducing it in a 64-bit ARM world.  It just is not going to happen.  We need to find a different way to get this tiny platform init code in between your basically  a BL2 in the actual Linux kernel.  This is where I really would like to brainstorm with you on cool ideas we can do.  I think we have a couple of things on the plate of what is possible to create a tiny  shim that runs before Linux so that you don't have to put the platform code into the Linux  kernel itself and the tree itself.  I'm very sorry, but there was something.  Could you come and repeat me the question without the mic?  The platform init code you're putting into Linux kernel.  Before the Linux kernel.  So is it actually separate standalone code?  It's a single file.  But it's in Linux.  In the source tree.  Yes.  I think we need to work on untangling this.  The question was is the shim we prepend to Linux kernel inside the kernel tree?  Yes, we want to be inside the kernel tree.  And I haven't mentioned that.  I have posted RFC patches to the list and they are available for commenting.  So it's not like we want to push something.  We want to discuss it because Linux kernel, the source tree can become a single stop  shop for vendors, for people to boot code that boots.  We're definitely open for discussion.  You can walk with their food, not with their mouth.  So please, if you are going out of the room, please keep somehow silent to let the speaker  finish the talk.  Thank you.  Okay, it's up here.  I finished.  Yeah.  Thank you.  Thank you, too.
ema_boot_linux_only_0_143.png ema_boot_linux_only_144_2855.png ema_boot_linux_only_2856_4043.png ema_boot_linux_only_4044_6851.png ema_boot_linux_only_6852_7535.png ema_boot_linux_only_7536_8903.png ema_boot_linux_only_8904_10163.png ema_boot_linux_only_10164_12671.png ema_boot_linux_only_12672_15695.png ema_boot_linux_only_15696_18155.png ema_boot_linux_only_18156_18251.png ema_boot_linux_only_18252_19355.png ema_boot_linux_only_19356_20327.png ema_boot_linux_only_20328_21539.png ema_boot_linux_only_21540_23411.png ema_boot_linux_only_23412_24575.png ema_boot_linux_only_24576_25631.png ema_boot_linux_only_25632_26891.png ema_boot_linux_only_26892_27743.png ema_boot_linux_only_27744_28979.png ema_boot_linux_only_28980_29555.png ema_boot_linux_only_29556_30035.png ema_boot_linux_only_30036_31571.png ema_boot_linux_only_31572_32219.png ema_boot_linux_only_32220_36971.png ema_boot_linux_only_36972_37019.png ema_boot_linux_only_37020_37199.png ema_boot_linux_only_37200_37427.png ema_boot_linux_only_37428_37552.png