Secure and Safe Updates for Your Embedded Device
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Jörns, Enrico
Recent DDoS attacks powered by embedded devices have finally discredited theold excuse that security is not important: security support (and thus softwareupdates) is suddenly a required feature. Often, physical access to these devices is limited and there is noadministrator who can fix issues manually. Thus, performing updates is anoperation with a critical design goal: Never brick the device! This talk gives an overview of the surprisingly complex requirements andcommon pitfalls for a generic update mechanism by comparing several existingapproaches. Also, our reasons for implementing (yet another) tool and thereasoning behind the design choices are explained. Using RAUC and other open source update tools as examples, requirements,limitations and possible pitfalls in the process of designing and implementinga redundancy and update infrastructure will be presented. You will also get abrief overview over RAUCs design and abstraction of the underlying system thatallows to manage both simple asymmetric setups consisting of a full system andan initramfs-based recovery system as well as complex setups with multipleroot filesystems, application and data storage partitions. * How to make updates atomic? * How to manage both simple and complex redundancy concepts? * How to allow secure and trusted updates? * How to schedule updates to a large number of devices?
 OK. Sorry for the short delay. Now the setup is working. So welcome, everyone, to my presentation, secure and save updates for your embedded device with the small subtitle and how to solve it with the Roug update tool. My name is Enrique Jerns. I'm an embedded system developer and system integrator working at Pangotronics. So embedded IoT stuff, well, do we really need updates for this kind of topic? And the answer, of course, is yes. This is why we are all here. And at least the events that took place in October last year, when those small IoT devices broke down parts of the internet that showed the last person that security, especially for these devices, that become more and more connected to the internet and infrastructure, that security is a topic of high priority. So connect with security. There's always the need for updating. Because if we detect a security hole somewhere, we have to update the device. And what other reasons do we have for updating it's obviously for fixing bugs if the software does not perform as we assume it to, or if we want to add new features to an existing device? So a short structure of the talk. The first chapter will be a bit about general challenges in updating and give a short overview of that. Then I will introduce the Rauch Update tool. And then hopefully the time is not over then yet. I'll show a short demonstration, a live demonstration, so I hope it will work. And at the end, I give a short, very short conclusion and summary of what you've heard here. So updating, you might think, is very simple. So you just have a few steps. You have an update and load it on your system. Then you perform the update right. It changes to your system. And maybe you reboot the system, and everything works and is fine. Well, as usual, reality is not that easy. So especially on embedded systems, you have different devices you want to update, for example, a North Flash or an SD card with different petitions on it. And you have to find out what to update there. And then you don't know who is the author of your update. Is he allowed to update your device? And are you updating the right device with the right update? And you also need some interaction with the boot loader. And if you've done all this step by step, then suddenly someone plugs the power cable off and everything breaks down completely. And you don't have real direct access to this. So the main question we try to answer here is, what is so special on updating embedded devices? And why don't we simply write it in a short Bash script? So updates you may know from your daily use, you use some kind of packet manager like aptitude or something like this, and you update your system. And in most cases, it will work. And if it does not work, OK, you fix it. So what is the difference when you have an embedded system? Well, on embedded systems, you don't have direct access to the device. In most cases, you don't have any access to this. And the only way you can start your update is either by plugging in a USB stick or by starting a remote deployment via a network. And then all you can do is watch the progress bar, and at the end, it either succeeds or it will fail. And if it fails, you really get in trouble because you don't have any chance to recover it. You don't have any easy access to the device and will cost your time and maybe money to recover the device. So what are the key requirements for updating an embedded system? First of all, the update process itself should not fail. And in case it does fail, the system itself should still in a state where it can be updated. And you also want to recover from runtime issues when your system suddenly stops, and then there should be some mechanism that reboots the device and brings it back into a state where it's still operating. Apart from the safety aspects, you also want some security aspects as these devices normally operate in some areas where you don't have direct access and you can't control who accesses the device. And you have to prevent the device from unauthorized access or updates. And there are some further limitations like you have some sort of storage that is not that easy to handle and limited resources, limited bandwidth, et cetera. So one good rule for updating an embedded system to not make a fail is to only install well-defined states. So this means the entire state with all components tested together. And the major rule we have when updating an embedded system, we do not want to break the device. We have to do everything to avoid this. So how to avoid this? So let's talk about redundancy. The simplest redundancy mechanism is when you run from a system that has no redundancy. So you perform an update. And if there is a power loss or some issue during the update, then the device might not be able to boot or the update system might not work again. And you have no chance to update it again. So you have to cause a service technician. Better approach is to use an asymmetric recovery redundancy mechanism. This is especially good for devices where you have limited storage capacity. So you have a small, dedicated recovery system where you boot into when you want to perform an update. And you can perform the update. And even if the update of the system itself fails, you can still boot into the recovery system and perform a further update. So it's updatable in case of a failure, but you don't have an operating system if the update fails. So what we really want to have is a full symmetric redundancy system where you have a system A that may update system B and a system B that is able to update system A. And even in case of failure, then you have maybe an old, but you have one system that is still operating. So another thing that is really important when updating is automacy. The result of the update should either be the update succeeded or the update failed. And you don't want to leave it in an intermediate state where it's not defined. So how can we achieve this? Here's an example. You start your update from your running partition A. And you have to see the entire update as a critical operation. If something fails during this, then you have to roll back or you don't have to break your device. So what you normally have is some kind of switch where you can say, OK, boot from system A. And then you turn the switch over and say, now boot from system B. This is normally done in the boot loader. And this operation has to be done at the very, very end of the update process. That's the last operation. Only there you can be true that your update was performed successfully. And it should be itself an atomic operation, of course. And you also want to security. You don't want someone unauthorized to be able to update your system. There are two basic ways to achieve this. The first of this, you use a trusted transport, like in webpads, for example, SSL, HTTPS. Then it does not matter, really, if you trust the update itself, you know it comes from a trusted source. This works well for the download case, but it will fail in the USB stick case. If you plug in a USB stick, you don't have easy mechanisms to trust the USB stick. So a more general approach is to put the trust in the update itself and sign the update. Then you don't have to trust your connection. And this will work both for downloading and for the USB stick case. And another thing is that you also don't want to inattentively update a device with the wrong software, for example. You don't want to turn your toaster into a printer by an update. And one could think, OK, I just check the hardware if it's the right hardware I write to, but this will not work in all cases. As you'll see, basically from the out of view, the same hardware, but it works as a different product. So a very simple and most times the best approach is to use an explicit identification that you place on the target and on the update. So here, this is Pytos 2000 and Pytos 2000. And if I say it's an update for Pytos 2000, then it's clear it's for this. You can check this and not for this. And another task that your update system has to handle is that you have to find out where you update. So you have an image B and you have to say, OK, is it a bootloader image? Is it a bit stream image? And other image, is it for the first or is it for the second redundant partition and the same for the root file system? And this works only if you have some kind of configuration, either only in the update itself or in the update and the system itself so that you can match these partitions. And there are a lot of other things you have to take care of. You have to do good error handling to be sure that nothing in the update process failed. You have to think about, do I store my data? Do I store it in a single partition? Or in a redundant partition, this matters if you deal with fallback or data migration because this system A and B use slightly different data formats and watch the handling and much more. So as a conclusion, you can say updating is not that easy. And this is why we developed Raucy Update Tool in firmware. So short overview, it's licensed under the LGPL 2.1. So it should be easy to use it in your application or in your product. It's a C code built based on a G-Lib and packaged with out-of-tools so it should easily build. And Raucy, you can compile as a host tool. Then it is used to create, to sign, to resign, and to inspect updates for your device. Updates in Raucy have called bundles. And you can compile it for the target. Then it operates as the update service on the target that controls the entire update process, the installation, communication with the bootloader. And it also provides a small comment line tool that allows you to easily install a bundle from the comment line or inspect the system or updates. Here's a short overview of how bundles look. You have your images and bit streams, for example, that you want to install. There's some information extracted and put in the manifest for verification. And these all are packed together in a squashFS. We have used a squashFS as it provides good compression and as it is mountable so you don't have to unpack data and store data capacity. And these are signed with X509 signatures you know well from SSL. And the idea of a bundle is that the bundle describes the entire intended state the system should have after the update, and then the Raucy service takes care of how to achieve this. This is a bit about handling in Raucy. Everything that could be a target for installing something we called a slot. And then you can create in your configuration classes of slots, for example, application file system, a root file system, and instances of the slot. And this is used by Rauc to match what you have in your update bundle, where you have an image for a specific class to the real device on your target. That's one of the core algorithms that Rauc provides. Here's a sample configuration. This one is the system configuration. This is normally stored on the target's root file system. We have some basic information about the system, about curing, and here define the slots that you may update. And then you have here the update manifest. You can here see the compatible that has to match between the target and the update itself, and the image description of those images you want to install. For example, here, this root file system image would be installed on one of these slots, depending on which is detected as currently inactive by Rauc. If you want to reuse Rauc in your next device, we have Yoccha integration in our meta-pdx layer, which will soon become meta-Rauc. We have pdx-dist integration currently only on the mailing list, not in pdx-dist mainline. And you can also integrate it manually. And here's a short overview to give a basic idea how it's used. This is the Rauc bundle command you use on your host to create the bundle with directory that includes all the content you want to bundle. And on the target, you simply execute Rauc install. Or if you have an application, you have a debus interface where you can trigger the installation from. So let's check the time. How much may I exceed? Two minutes. Two minutes. The previous talk was exceeded, too. So a bit, I hope. Yes, thanks. So the demonstration, you may all see are these are six Raspberry Pi currently running a pdx-based image. And this includes the Rauc update tool. And I try to trigger the update before continuing the talk. Thanks. Yes, I know. But I'm not sure if they. So these devices are currently connected to a Hockpit update. Now I know what you're talking about. So I shortly described the setup. We have currently a pdx-based image running. And these devices here are connected to a Hockpit software deployment server. And on the devices, the Rauc services running and a little Python script that shows these small pdx-disk images. And the presentation, or the demo, should now show that I deploy a Rauc to update on these devices. Yeah, this is what I wanted to show here. But I think the setup did not entirely finished because of the irritation in the beginning of the talk. So if you want to see the demo working, maybe you would like to join me after the talk. I tried in a few minutes. Yeah, this is the basic layout of the root file system on the Raspberry Pi. You have an easy redundant system and the barebox boot loader with its boot state framework running on it. And this shows that the update system is really independent from the integration. And it can easily be integrated into two different needs. And it's generic, so it is designed to be adaptable to exactly your needs. Can't switch to the next slide. Should be the conclusion. So I have to try it manually. So what are the basic facts that we have learned? Updates, especially on embedded devices that are connected, are mandatory. The second one is that updates are not really trivial. There are some pitfalls I've shown and some more I have not shown yet. And if you do updates, only install, well, tested software sets. So this is the best way to not break your device. And of course, use free and open software update frameworks. For example, I give you an idea, Raoq would be a good choice. So thanks to everyone. I think that time for questions is coming directly to me, and we will discuss it.
 OK.  Sorry for the short delay.  Now the setup is working.  So welcome, everyone, to my presentation, secure and save  updates for your embedded device with the small subtitle  and how to solve it with the Roug update tool.  My name is Enrique Jerns.  I'm an embedded system developer and system  integrator working at Pangotronics.  So embedded IoT stuff, well, do we really need updates for  this kind of topic?  And the answer, of course, is yes.  This is why we are all here.  And at least the events that took place in October last  year, when those small IoT devices broke down parts of  the internet that showed the last person that security,  especially for these devices, that become more and more  connected to the internet and infrastructure, that security  is a topic of high priority.  So connect with security.  There's always the need for updating.  Because if we detect a security hole somewhere, we have to  update the device.  And what other reasons do we have for updating it's  obviously for fixing bugs if the software does not perform as  we assume it to, or if we want to add new features to an  existing device?  So a short structure of the talk.  The first chapter will be a bit about general challenges in  updating and give a short overview of that.  Then I will introduce the Rauch Update tool.  And then hopefully the time is not over then yet.  I'll show a short demonstration, a live  demonstration, so I hope it will work.  And at the end, I give a short, very short conclusion and  summary of what you've heard here.  So updating, you might think, is very simple.  So you just have a few steps.  You have an update and load it on your system.  Then you perform the update right.  It changes to your system.  And maybe you reboot the system, and everything works and  is fine.  Well, as usual, reality is not that easy.  So especially on embedded systems, you have different  devices you want to update, for example, a North Flash or an  SD card with different petitions on it.  And you have to find out what to update there.  And then you don't know who is the author of your update.  Is he allowed to update your device?  And are you updating the right device with the right update?  And you also need some interaction with the boot loader.  And if you've done all this step by step, then suddenly  someone plugs the power cable off and everything  breaks down completely.  And you don't have real direct access to this.  So the main question we try to answer here is, what is so  special on updating embedded devices?  And why don't we simply write it in a short Bash script?  So updates you may know from your daily use, you use some  kind of packet manager like aptitude or something like  this, and you update your system.  And in most cases, it will work.  And if it does not work, OK, you fix it.  So what is the difference when you have an embedded system?  Well, on embedded systems, you don't have direct access to  the device.  In most cases, you don't have any access to this.  And the only way you can start your update is either by  plugging in a USB stick or by starting a remote deployment  via a network.  And then all you can do is watch the progress bar, and at  the end, it either succeeds or it will fail.  And if it fails, you really get in trouble because you don't  have any chance to recover it.  You don't have any easy access to the device and will cost  your time and maybe money to recover the device.  So what are the key requirements for updating an  embedded system?  First of all, the update process itself should not fail.  And in case it does fail, the system itself should still  in a state where it can be updated.  And you also want to recover from runtime issues when your  system suddenly stops, and then there should be some  mechanism that reboots the device and brings it back  into a state where it's still operating.  Apart from the safety aspects, you also want some security  aspects as these devices normally operate in some  areas where you don't have direct access and you can't  control who accesses the device.  And you have to prevent the device from unauthorized  access or updates.  And there are some further limitations like you have some  sort of storage that is not that easy to handle and limited  resources, limited bandwidth, et cetera.  So one good rule for updating an embedded system to not make  a fail is to only install well-defined states.  So this means the entire state with all components  tested together. And the major rule we have when updating an  embedded system, we do not want to break the device.  We have to do everything to avoid this.  So how to avoid this?  So let's talk about redundancy.  The simplest redundancy mechanism is when you run  from a system that has no redundancy.  So you perform an update.  And if there is a power loss or some issue during the  update, then the device might not be able to boot or the  update system might not work again.  And you have no chance to update it again.  So you have to cause a service technician.  Better approach is to use an asymmetric recovery  redundancy mechanism.  This is especially good for devices where you have limited  storage capacity.  So you have a small, dedicated recovery system where you  boot into when you want to perform an update.  And you can perform the update.  And even if the update of the system itself fails, you can  still boot into the recovery system and perform a  further update.  So it's updatable in case of a failure, but you don't have an  operating system if the update fails.  So what we really want to have is a full symmetric  redundancy system where you have a system A that may  update system B and a system B that is able to update system  A. And even in case of failure, then you have maybe an old,  but you have one system that is still operating.  So another thing that is really important when updating is  automacy.  The result of the update should either be the update  succeeded or the update failed.  And you don't want to leave it in an intermediate state where  it's not defined.  So how can we achieve this?  Here's an example.  You start your update from your running partition A. And you  have to see the entire update as a critical operation.  If something fails during this, then you have to roll back  or you don't have to break your device.  So what you normally have is some kind of switch where you  can say, OK, boot from system A. And then you turn the  switch over and say, now boot from system B. This is  normally done in the boot loader.  And this operation has to be done at the very, very end of  the update process.  That's the last operation.  Only there you can be true that your update was performed  successfully.  And it should be itself an atomic operation, of course.  And you also want to security.  You don't want someone unauthorized to be able to  update your system.  There are two basic ways to achieve this.  The first of this, you use a trusted transport, like in  webpads, for example, SSL, HTTPS.  Then it does not matter, really, if you trust the update  itself, you know it comes from a trusted source.  This works well for the download case, but it will  fail in the USB stick case.  If you plug in a USB stick, you don't have easy  mechanisms to trust the USB stick.  So a more general approach is to put the trust in the  update itself and sign the update.  Then you don't have to trust your connection.  And this will work both for downloading and for the  USB stick case.  And another thing is that you also don't want to  inattentively update a device with the wrong software,  for example.  You don't want to turn your toaster into a printer by an  update.  And one could think, OK, I just check the hardware if it's  the right hardware I write to, but this will not work in  all cases.  As you'll see, basically from the out of view, the same  hardware, but it works as a different product.  So a very simple and most times the best approach is to use  an explicit identification that you place on the target  and on the update.  So here, this is Pytos 2000 and Pytos 2000.  And if I say it's an update for Pytos 2000, then it's  clear it's for this.  You can check this and not for this.  And another task that your update system has to handle  is that you have to find out where you update.  So you have an image B and you have to say, OK, is it a  bootloader image?  Is it a bit stream image?  And other image, is it for the first or is it for the  second redundant partition and the same for the root file  system?  And this works only if you have some kind of  configuration, either only in the update itself or in the  update and the system itself so that you can match these  partitions.  And there are a lot of other things you have to take care  of.  You have to do good error handling to be sure that  nothing in the update process failed.  You have to think about, do I store my data?  Do I store it in a single partition?  Or in a redundant partition, this matters if you deal  with fallback or data migration because this system A and B  use slightly different data formats and watch  the handling and much more.  So as a conclusion, you can say updating is not that easy.  And this is why we developed Raucy Update Tool in firmware.  So short overview, it's licensed under the LGPL 2.1.  So it should be easy to use it in your application or in your  product.  It's a C code built based on a G-Lib and packaged with  out-of-tools so it should easily build.  And Raucy, you can compile as a host tool.  Then it is used to create, to sign, to resign, and to  inspect updates for your device.  Updates in Raucy have called bundles.  And you can compile it for the target.  Then it operates as the update service on the target that  controls the entire update process, the installation,  communication with the bootloader.  And it also provides a small comment line tool that allows  you to easily install a bundle from the comment line or  inspect the system or updates.  Here's a short overview of how bundles look.  You have your images and bit streams, for example, that  you want to install.  There's some information extracted and put in the  manifest for verification.  And these all are packed together in a squashFS.  We have used a squashFS as it provides good compression and  as it is mountable so you don't have to unpack data and  store data capacity.  And these are signed with X509 signatures you know  well from SSL.  And the idea of a bundle is that the bundle describes the  entire intended state the system should have after the  update, and then the Raucy service takes care of how to  achieve this.  This is a bit about handling in Raucy.  Everything that could be a target for installing  something we called a slot.  And then you can create in your configuration classes of  slots, for example, application file system, a root file  system, and instances of the slot.  And this is used by Rauc to match what you have in your  update bundle, where you have an image for a specific class  to the real device on your target.  That's one of the core algorithms that Rauc provides.  Here's a sample configuration.  This one is the system configuration.  This is normally stored on the target's root file system.  We have some basic information about the system, about  curing, and here define the slots that you may update.  And then you have here the update manifest.  You can here see the compatible that has to match  between the target and the update itself, and the image  description of those images you want to install.  For example, here, this root file system image would be  installed on one of these slots, depending on which is  detected as currently inactive by Rauc.  If you want to reuse Rauc in your next device, we have  Yoccha integration in our meta-pdx layer, which will  soon become meta-Rauc.  We have pdx-dist integration currently only on the mailing  list, not in pdx-dist mainline.  And you can also integrate it manually.  And here's a short overview to give a basic idea how it's  used.  This is the Rauc bundle command you use on your host to  create the bundle with directory that includes all the  content you want to bundle.  And on the target, you simply execute Rauc install.  Or if you have an application, you have a debus  interface where you can trigger the installation from.  So let's check the time.  How much may I exceed?  Two minutes.  Two minutes.  The previous talk was exceeded, too.  So a bit, I hope.  Yes, thanks.  So the demonstration, you may all see are these are six  Raspberry Pi currently running a pdx-based image.  And this includes the Rauc update tool.  And I try to trigger the update before continuing the talk.  Thanks.  Yes, I know.  But I'm not sure if they.  So these devices are currently connected to a  Hockpit update.  Now I know what you're talking about.  So I shortly described the setup.  We have currently a pdx-based image running.  And these devices here are connected to a Hockpit  software deployment server.  And on the devices, the Rauc services running and a little  Python script that shows these small pdx-disk images.  And the presentation, or the demo, should now show that I  deploy a Rauc to update on these devices.  Yeah, this is what I wanted to show here.  But I think the setup did not entirely finished because of  the irritation in the beginning of the talk.  So if you want to see the demo working, maybe you would like  to join me after the talk.  I tried in a few minutes.  Yeah, this is the basic layout of the root file system on  the Raspberry Pi.  You have an easy redundant system and the barebox boot  loader with its boot state framework running on it.  And this shows that the update system is really  independent from the integration.  And it can easily be integrated into two different  needs.  And it's generic, so it is designed to be adaptable to  exactly your needs.  Can't switch to the next slide.  Should be the conclusion.  So I have to try it manually.  So what are the basic facts that we have learned?  Updates, especially on embedded devices that are  connected, are mandatory.  The second one is that updates are not really trivial.  There are some pitfalls I've shown and some more I have  not shown yet.  And if you do updates, only install, well, tested  software sets.  So this is the best way to not break your device.  And of course, use free and open software update frameworks.  For example, I give you an idea, Raoq would be a good choice.  So thanks to everyone.  I think that time for questions is coming directly to me,  and we will discuss it.
Secure_and_Safe_Updates_for_Your_Embedded_Device_0_131.png Secure_and_Safe_Updates_for_Your_Embedded_Device_132_755.png Secure_and_Safe_Updates_for_Your_Embedded_Device_756_2519.png Secure_and_Safe_Updates_for_Your_Embedded_Device_2520_2891.png Secure_and_Safe_Updates_for_Your_Embedded_Device_2892_3395.png Secure_and_Safe_Updates_for_Your_Embedded_Device_3396_3947.png Secure_and_Safe_Updates_for_Your_Embedded_Device_3948_5087.png Secure_and_Safe_Updates_for_Your_Embedded_Device_5088_5435.png Secure_and_Safe_Updates_for_Your_Embedded_Device_5436_6119.png Secure_and_Safe_Updates_for_Your_Embedded_Device_6120_6935.png Secure_and_Safe_Updates_for_Your_Embedded_Device_6936_9167.png Secure_and_Safe_Updates_for_Your_Embedded_Device_9168_11423.png Secure_and_Safe_Updates_for_Your_Embedded_Device_11424_13103.png Secure_and_Safe_Updates_for_Your_Embedded_Device_13104_14459.png Secure_and_Safe_Updates_for_Your_Embedded_Device_14460_14951.png Secure_and_Safe_Updates_for_Your_Embedded_Device_14952_15863.png Secure_and_Safe_Updates_for_Your_Embedded_Device_15864_16775.png Secure_and_Safe_Updates_for_Your_Embedded_Device_16776_17627.png Secure_and_Safe_Updates_for_Your_Embedded_Device_17628_19343.png Secure_and_Safe_Updates_for_Your_Embedded_Device_19344_20711.png Secure_and_Safe_Updates_for_Your_Embedded_Device_20712_21563.png Secure_and_Safe_Updates_for_Your_Embedded_Device_21564_22631.png Secure_and_Safe_Updates_for_Your_Embedded_Device_22632_23723.png Secure_and_Safe_Updates_for_Your_Embedded_Device_23724_24239.png Secure_and_Safe_Updates_for_Your_Embedded_Device_24240_24587.png Secure_and_Safe_Updates_for_Your_Embedded_Device_24588_24659.png Secure_and_Safe_Updates_for_Your_Embedded_Device_24660_24815.png Secure_and_Safe_Updates_for_Your_Embedded_Device_24816_25595.png Secure_and_Safe_Updates_for_Your_Embedded_Device_25596_25679.png Secure_and_Safe_Updates_for_Your_Embedded_Device_25680_25763.png Secure_and_Safe_Updates_for_Your_Embedded_Device_25764_28499.png Secure_and_Safe_Updates_for_Your_Embedded_Device_28500_29795.png Secure_and_Safe_Updates_for_Your_Embedded_Device_29796_29915.png