Updating Embedded Systems - Putting it all Together
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Olbrich, Michael
Updating embedded systems reliably requires more than just the actual update process. This presentation gives an overview of the overall design and components needed for successful system updates.
 So, my name is Michael, Michael Ulbricht, or Michael Ulbricht if you wanted to do it in German, but Michael is okay too. So I'll be talking a bit about updating embedded systems. Let's start about me. I've been working on embedded Linux for quite some time, working for Pengatronics the whole time and this is mostly support kind of tasks. Getting words, hardware to start running Linux, helping with things, Windows, bugs, basically to allow our customers to do their own applications, right? So we do the base Linux operating system and they kind of focus on their own applications, the actual stuff that they want to do with their hardware. So I mean, updating is not something that you want to care about. This is something that's supposed to work. So I've seen that for quite a few projects over the last few years, it's getting more. In the beginning it was like updating, do we have to do that? Maybe we don't need to update our software, it's okay. And nowadays updating comes more and more important and as I've said, I've worked with a lot of different projects. So my focus is less on how the individual, the updating process itself works. But I want to focus here today on more, what do you have to do to integrate an updating tool and updating software, product, whatever, with your own application, your own product? Because I mean, we've had a talk yesterday about updating and if you look at other conferences and look at the talks there, the videos, you'll find several talks about updating software that describes the software itself and it's important and it's not easy to get right, but we're getting there. And that's the stuff that can be shared. But there are a lot of parts of the updating process itself on a particular device that is device specific or in ways where you have to add your own features, your own stuff to make it really work. And that's the parts where I want to focus on. So a bit about updating first. I'd like mostly to focus on the standard simple A, B setup kind of updates where you have one running system and then you have a separate partition where you can write the new version of your operating system into it while your system is still running. And then when all that process is done, then you say, okay, now I'm finished and I can start the other operating system, the new version of my operating system. And if that fails, that's an important part. Maybe there's some bug in there. Maybe the rating of the operating system failed in some way and didn't detect while it was flashed. Or maybe in this special use case, there's some outside circumstances that make the startup or whatever fail. So in that case, we want to go back to the original version, the previous version that worked. And if you look at that, writing the update itself, that's mostly generic, right? I mean, there are a few ways to say I write a file system image to the disk or I format a new file system and extract a top all into that or these kind of things. But this is not really project specific. That's something you can implement once it can be tested really well. But when it comes to checking, okay, so my system is not starting. What does that mean my new system is not starting? There are a lot of ways that that can fail and many of those are project specific. So that's part of where I want to go, right? We need to detect those failures and then we need to say, okay, something happened here and I want to go back to the old version. And then maybe there's some user interaction there. Do I say explicitly, okay, something's wrong, can we go back? It depends, really, it's really on the devices. Sometimes it's some embedded IoT device, then this has no user interaction and it has to be automatic. Often there are devices and an operator is in front of them and says this is not working. I've just applied an update and now it's not working anymore. Can we go back to the last one? These kind of things and that's when you need to integrate the update mechanism with your own system. So let's start with the failures. I'd like to classify at first what kind of different categories there are. So and depending on those, how we can react and how difficult they are to work with that. Let's say start up failure, this is simple ones. And that's typically what you see when you slide. I remember from yesterday there was this, okay, we have installed a new one and there was three lines on one slide where we said, okay, we start a new image and then there's our own, we talk to our update server again and then you can hook in something else to check if the new system is okay. That's the start. A lot of stuff can be detected on the first start up if it comes up and it works. But some errors come later either because some user interaction comes afterwards and only with that user interaction things start to fail or there's memory corruption or there are memory leaks that take some time to occur. So maybe the system starts but it starts to break down afterwards. And then there is called this update preparation errors. Means you want to install a new update and that's not what, you don't get to the point when you actually write something, right? You can try it again or it still doesn't work and maybe you can go to the old system and then there's this, I call it update errors and if they are permanent, that's a problem because if you say, okay, I have a system that is running, I try to write an update and it fails because the update process is broken and I cannot go back anymore to the previous one. So we have to be really careful there to make sure that that doesn't happen or there's even more fallback. But at the end of the day, at some point, the effort to work around or to handle those issues gets pretty high and the probability that they occur pretty low. So at some point you need to stop and say, okay, this is enough. These are the things I check for and for everything else, okay, the device is bricked and we have 0.001% boards that fail and then we can replace those and it's okay. So for inter-detection on startup, the most important thing there is first the hardware watchdog. If you're doing these kind of things, it's pretty mandatory that you have a hardware watchdog that can detect problems and booting because at some point it might be the Linux kernel that's basically just not starting or gets stuck somewhere in hardware initialization and the only way to recover from that is when a hardware watchdog triggers and reboots the system and say, okay, here, something went wrong. So we start the hardware watchdog in the bootloader. That's important too. The bootloader is the first step here. And keep it running. That's many, many drivers in the kernel by default will reset the watchdog hardware when the driver initializes. And it will not activate the watchdog until the user space activates the watchdog again. So there are ways that you can basically tell the driver not to touch the watchdog at the beginning or the state of the watchdog. But you need to make sure that that happens, actually. And then systemD comes up. So that's one part I'm always using, actually. SystemD has really great infrastructure for that. So systemD starts opening the watchdog and starts triggering that. So as long as systemD itself is in a useful state, it will continue to trigger the watchdog. And as long as that happens, we say, okay, systemD is okay, and systemD will handle all the other errors and propagate those correctly. That brings me to the next part, systemD. Don't forget to handle all cases, because systemD will happily trigger the watchdog while it's stuck in a situation where basically nothing works. For example, emergency and rescue targets. SystemD by default is configured in a way for a desktop user or a server where an administrator can log in and say, hey, what's going on? What went wrong here? And so if the FAS check of the root file system fails or at the critical file systems, or if you cannot reach your default target system, they will start these rescue targets or emergency targets, and then you may get a shell there. But for an embedded system, the shell will help nothing, because there is no one in front of it that can actually do something with it. So you need to put some service there that will actually do something useful like restart your system. And then we come to the actual applications, and that's where most of the work usually is for the individual devices, for the individual projects, is you need to figure out what do I need to check to verify that it's still running to make sure, yes, my system started and yes, it's still working and it's doing its job. SystemD will give you a lot of tools for that. There's a watchdog per application, basically the SD-Nautify hooks where you can say, yes, I've started, and then later on, you start pinging. Yes, I'm still here. I'm still here. I'm still here. So if you can stop, then SystemD will either restart your application or kill it completely or act on that depending on the option. There is the failure action, basically, or there is the start limit stuff where you can say, OK, if the service restarted three times in five minutes, then maybe it's not going to start again properly. So let's do something, maybe reboot the system. Or if you prefer to fail immediately, it's the failure action to say, OK, this service failed in some way. Let's immediately reboot the system. These kind of things help to propagate errors from the individual applications to SystemD and from there, you can then restart your system and say, OK, now the bootlogger is again here to make the next decision which system is going to start. And that's pretty tricky, actually, because when do we say the current system that's running is not working out for us? So let's go to the last one. Basically from a high-level perspective, if the error is transient, well, let's try again. Maybe it works out afterwards. If it's persistent, then we can try again and again. At some point, we have to say, OK, well, it's still not working, so let's go back to the previous system. And that really depends on what your individual requirements are. Maybe this one part of the system, well, OK, it fails too bad, but let's say running. Or the other part that's really, really important. So we need to reboot into the old version. And that really can be some kind of conflicting requirements, because if you say, OK, I cannot reach my update server, but the actual application is working. I want to have my new service running, but I need to go back somewhere to do an update for later on. So it's really a decision process where you have to look at all the possible failures and not decide, this is where I go back and this is where I stay with the current system and how often do I try to reboot and to restart individual services or restart the whole system. And to actually implement that, I'm using here Bearbox as a bootloader as an example, that's basically because that's what I'm working with most of the time. And it has this boot chooser. Basically what it does is you have a slot. This is one version of your A-B scenario, is one slot. A is one slot, B is one slot. It says, OK, both of them have a priority and a boot counter. And then Bearbox comes and says, OK, the priority is, for this one is higher, so I try this one and look at the boot counter. Boot counter is rated in zero, minus one, and then start this one. And then in the user space, you have to reset that boot counter. And there are various ways where you can use that to implement different kind of scenarios where you say, OK, I tried this multiple times, or a failure at startup is more important than a failure later on, and these kind of things. The important thing is that we can reset those boot counters again. Because, I mean, let's say there's an outside effect that causes failure. Say the power is really low and the system starts. And as soon as you start powering on additional components, it breaks down and resets the system. And then you do that in a loop and say, OK, three times, four times, my system failed. OK, let's follow over to the previous version. And then you try it here three times, four times, and then, yeah, both boot counters are zero. And what do we do now? We have no system that's seemingly working. And then I can either say, OK, get stuck here and don't do anything, or we can say, maybe there was some outside effect. So if all systems say boot counter zero can't do anything, we start again from the beginning with reset the boot counters to some value, to three or five or whatever you want, and then try again. And maybe now the power is better, and then it works again. And also important, atomic updates. This is something where I like it's built into the barebox boot loader. It's something where a lot of people spend time fixing things, because writing multiple values on an embedded system in an atomic way that when any time the power fails, it's still there in a consistent state. That's pretty tricky. Because if you have an ant flash, for example, then doing, OK, I need to erase this block and I need to write here and there and do things in a way that at any time the power can fail, that's pretty hard. And so there's just basically a framework there that says, OK, set these values and then now sync it to whatever spec and storage we have. And then it's always an atomic update. So and now let's look at actually how we use that. Just a few examples of what you can do. And this is basically one end of the range of options. You probably want to do something in the middle. But this is OK for many cases. Actually, not what many people actually do right now. Is this, well, it's started up. Maybe things can fail. But afterwards, everything errors that come there, that just trends in. And it works afterwards again after we reboot. It's probably true in many cases because it will work for time if it's an error that's say a memory leak, right? Memory leak that accumulates memory over time. So it runs for three days and then runs out of memory. We reset the system. It runs again for three days. That's OK under some circumstances because the system is running for three days, which means at the very least, it's running for three days in which time we can actually continue an update in your system. So we can write a new update in there. And that's always the most important thing that you're always in a situation where you can write a new update so you can fix things. In this case, we just reset the boot counter when the system has successfully booted. Somebody will help us with that because we can just pick a target and say, OK, we have this target. And any application, any service is required by this target. So if any of those fail, then we never reach this target. And then we have one service that waits for this target and says, OK, when we reach this target, we have the next service in the chain. And this one will simply just reset the boot counters for the running system. If you're a bit more paranoid, you'll say, well, any errors may be permanent. So we don't actually reset the boot counter after booting. So the system starts, the boot loader counts down the boot counter, and then it's OK. And then it's running. And only if we do an explicit reboot of the system, if we trigger a normal reboot, for example, after an update or for whatever reason else, then we reset the boot counter. Say, OK, it's running for some time, and then it's good. And everything works. And maybe we need to reboot for whatever reason or not. And this is kind of like any failure later on, it triggers to reset. The boot counter was not reset. So it's counting down. However, well, if you're developing or something, it's often this, well, let's switch off the device and on again and do work and retry. Some things, you will never actually reset the boot counter. It happens pretty easily that you run into the case that you accidentally start the old system because you didn't actually never reach the situation where the boot counters were reset. So this is the other extreme. So there must be, in most cases, probably somewhere in between that you say, maybe it's running for five days and now it looks good, or you have an outside operator actually monitoring your devices and says, hey, this device was running for 10 days and this looks good, so let's reset the boot counter. It's probably working properly now. It's really application-specific, device-specific, what kind of choices you make for that. But what I like here for the system we have with system D, doing the monitoring and barebox with its state and relatively isolated, is the state you can modify with any tool here. You can choose when you update these states, when you reset the counters, and you can modify how it reacts to your requirements. So let's go to a nice topic. When we're always talking about failures, let's talk a bit of user interaction because update has a lot to do with users as soon as there is actually someone watching. It can be someone local. A lot of devices are starting to get displaced, more and more devices with displaced and more and more. This plays an info and see the progress of an update and these kind of things. It's a remote operator that says, okay, I installed this update here and I want to see something. And then it helps actually because if you can ask a user to, or the user can decide to turn the device completely off and on again, you can use that to handle errors as well. Because if you cannot detect it in your software, maybe because it's a high level user interaction stuff, you can still trigger a reset of the whole system by turning the power off and on and you can use that, okay, the power was turned off. We didn't reset the boot counter so it's one less. And if you try it multiple times, then we start the old system at some point or immediately. We can reset the boot counters after power off and have this all counters for zero and after power doing a power cycle once, we start again trying to do things. It really depends on the situation. You can also say you have something in your UI to do, yeah, switch to the old system because this is not working. Or yeah, these kind of things. This is faster, I think, I don't have that much time. And what's really important, what I think is that they take time. Even if it's running more or less in the background because you can still do the actual job. If it's a device where a user is standing in front of it, puts the USB again, say I do run my update, and it says nothing for ten minutes, that's going to be, yeah, the update is not working. Let's power off and try again. So adding information there to update is really important, I think. And also in case of error handling, your customer says it's not working and then says, well, yeah, and it says the customer is gone. I don't know why. And if you don't have, can pull out some information that's integrated with your application where you can pull in your log files, that's pretty hard to debug afterwards. So not just the black box, I put here the route, it's one update system, I know it because my colleagues are working on it. That just has a debug API and says, hey, I'm doing this and this and this step. So yeah, that's it. Few links there, but just a few questions. Getting pretty late. No questions anymore? Okay. I think that's it for today. Thank you.
 So, my name is Michael, Michael Ulbricht, or Michael Ulbricht if you wanted to do it  in German, but Michael is okay too.  So I'll be talking a bit about updating embedded systems.  Let's start about me.  I've been working on embedded Linux for quite some time, working for Pengatronics the whole  time and this is mostly support kind of tasks.  Getting words, hardware to start running Linux, helping with things, Windows, bugs, basically  to allow our customers to do their own applications, right?  So we do the base Linux operating system and they kind of focus on their own applications,  the actual stuff that they want to do with their hardware.  So I mean, updating is not something that you want to care about.  This is something that's supposed to work.  So I've seen that for quite a few projects over the last few years, it's getting more.  In the beginning it was like updating, do we have to do that?  Maybe we don't need to update our software, it's okay.  And nowadays updating comes more and more important and as I've said, I've worked with a lot of  different projects.  So my focus is less on how the individual, the updating process itself works.  But I want to focus here today on more, what do you have to do to integrate an updating  tool and updating software, product, whatever, with your own application, your own product?  Because I mean, we've had a talk yesterday about updating and if you look at other conferences  and look at the talks there, the videos, you'll find several talks about updating software  that describes the software itself and it's important and it's not easy to get right, but  we're getting there.  And that's the stuff that can be shared.  But there are a lot of parts of the updating process itself on a particular device that  is device specific or in ways where you have to add your own features, your own stuff to  make it really work.  And that's the parts where I want to focus on.  So a bit about updating first.  I'd like mostly to focus on the standard simple A, B setup kind of updates where you have  one running system and then you have a separate partition where you can write the new version  of your operating system into it while your system is still running.  And then when all that process is done, then you say, okay, now I'm finished and I can  start the other operating system, the new version of my operating system.  And if that fails, that's an important part.  Maybe there's some bug in there.  Maybe the rating of the operating system failed in some way and didn't detect while it was  flashed.  Or maybe in this special use case, there's some outside circumstances that make the  startup or whatever fail.  So in that case, we want to go back to the original version, the previous version that  worked.  And if you look at that, writing the update itself, that's mostly generic, right?  I mean, there are a few ways to say I write a file system image to the disk or I format  a new file system and extract a top all into that or these kind of things.  But this is not really project specific.  That's something you can implement once it can be tested really well.  But when it comes to checking, okay, so my system is not starting.  What does that mean my new system is not starting?  There are a lot of ways that that can fail and many of those are project specific.  So that's part of where I want to go, right?  We need to detect those failures and then we need to say, okay, something happened here  and I want to go back to the old version.  And then maybe there's some user interaction there.  Do I say explicitly, okay, something's wrong, can we go back?  It depends, really, it's really on the devices.  Sometimes it's some embedded IoT device, then this has no user interaction and it has to  be automatic.  Often there are devices and an operator is in front of them and says this is not working.  I've just applied an update and now it's not working anymore.  Can we go back to the last one?  These kind of things and that's when you need to integrate the update mechanism with your  own system.  So let's start with the failures.  I'd like to classify at first what kind of different categories there are.  So and depending on those, how we can react and how difficult they are to work with that.  Let's say start up failure, this is simple ones.  And that's typically what you see when you slide.  I remember from yesterday there was this, okay, we have installed a new one and there  was three lines on one slide where we said, okay, we start a new image and then there's  our own, we talk to our update server again and then you can hook in something else to  check if the new system is okay.  That's the start.  A lot of stuff can be detected on the first start up if it comes up and it works.  But some errors come later either because some user interaction comes afterwards and only  with that user interaction things start to fail or there's memory corruption or there  are memory leaks that take some time to occur.  So maybe the system starts but it starts to break down afterwards.  And then there is called this update preparation errors.  Means you want to install a new update and that's not what, you don't get to the point  when you actually write something, right?  You can try it again or it still doesn't work and maybe you can go to the old system and  then there's this, I call it update errors and if they are permanent, that's a problem  because if you say, okay, I have a system that is running, I try to write an update  and it fails because the update process is broken and I cannot go back anymore to the  previous one.  So we have to be really careful there to make sure that that doesn't happen or there's even  more fallback.  But at the end of the day, at some point, the effort to work around or to handle those  issues gets pretty high and the probability that they occur pretty low.  So at some point you need to stop and say, okay, this is enough.  These are the things I check for and for everything else, okay, the device is bricked  and we have 0.001% boards that fail and then we can replace those and it's okay.  So for inter-detection on startup, the most important thing there is first the hardware  watchdog.  If you're doing these kind of things, it's pretty mandatory that you have a hardware  watchdog that can detect problems and booting because at some point it might be the Linux  kernel that's basically just not starting or gets stuck somewhere in hardware initialization  and the only way to recover from that is when a hardware watchdog triggers and reboots the  system and say, okay, here, something went wrong.  So we start the hardware watchdog in the bootloader.  That's important too.  The bootloader is the first step here.  And keep it running.  That's many, many drivers in the kernel by default will reset the watchdog hardware when  the driver initializes.  And it will not activate the watchdog until the user space activates the watchdog again.  So there are ways that you can basically tell the driver not to touch the watchdog at the  beginning or the state of the watchdog.  But you need to make sure that that happens, actually.  And then systemD comes up.  So that's one part I'm always using, actually.  SystemD has really great infrastructure for that.  So systemD starts opening the watchdog and starts triggering that.  So as long as systemD itself is in a useful state, it will continue to trigger the watchdog.  And as long as that happens, we say, okay, systemD is okay, and systemD will handle all  the other errors and propagate those correctly.  That brings me to the next part, systemD.  Don't forget to handle all cases, because systemD will happily trigger the watchdog  while it's stuck in a situation where basically nothing works.  For example, emergency and rescue targets.  SystemD by default is configured in a way for a desktop user or a server where an administrator  can log in and say, hey, what's going on?  What went wrong here?  And so if the FAS check of the root file system fails or at the critical file systems, or  if you cannot reach your default target system, they will start these rescue targets or emergency  targets, and then you may get a shell there.  But for an embedded system, the shell will help nothing, because there is no one in front  of it that can actually do something with it.  So you need to put some service there that will actually do something useful like restart  your system.  And then we come to the actual applications, and that's where most of the work usually  is for the individual devices, for the individual projects, is you need to figure out what do  I need to check to verify that it's still running to make sure, yes, my system started  and yes, it's still working and it's doing its job.  SystemD will give you a lot of tools for that.  There's a watchdog per application, basically the SD-Nautify hooks where you can say, yes,  I've started, and then later on, you start pinging.  Yes, I'm still here.  I'm still here.  I'm still here.  So if you can stop, then SystemD will either restart your application or kill it completely  or act on that depending on the option.  There is the failure action, basically, or there is the start limit stuff where you  can say, OK, if the service restarted three times in five minutes, then maybe it's not  going to start again properly.  So let's do something, maybe reboot the system.  Or if you prefer to fail immediately, it's the failure action to say, OK, this service  failed in some way.  Let's immediately reboot the system.  These kind of things help to propagate errors from the individual applications to SystemD  and from there, you can then restart your system and say, OK, now the bootlogger is  again here to make the next decision which system is going to start.  And that's pretty tricky, actually, because when do we say the current system that's running  is not working out for us?  So let's go to the last one.  Basically from a high-level perspective, if the error is transient, well, let's try again.  Maybe it works out afterwards.  If it's persistent, then we can try again and again.  At some point, we have to say, OK, well, it's still not working, so let's go back to the  previous system.  And that really depends on what your individual requirements are.  Maybe this one part of the system, well, OK, it fails too bad, but let's say running.  Or the other part that's really, really important.  So we need to reboot into the old version.  And that really can be some kind of conflicting requirements, because if you say, OK, I cannot  reach my update server, but the actual application is working.  I want to have my new service running, but I need to go back somewhere to do an update  for later on.  So it's really a decision process where you have to look at all the possible failures  and not decide, this is where I go back and this is where I stay with the current system  and how often do I try to reboot and to restart individual services or restart the whole system.  And to actually implement that, I'm using here Bearbox as a bootloader as an example,  that's basically because that's what I'm working with most of the time.  And it has this boot chooser.  Basically what it does is you have a slot.  This is one version of your A-B scenario, is one slot.  A is one slot, B is one slot.  It says, OK, both of them have a priority and a boot counter.  And then Bearbox comes and says, OK, the priority is, for this one is higher, so I try this  one and look at the boot counter.  Boot counter is rated in zero, minus one, and then start this one.  And then in the user space, you have to reset that boot counter.  And there are various ways where you can use that to implement different kind of scenarios  where you say, OK, I tried this multiple times, or a failure at startup is more important  than a failure later on, and these kind of things.  The important thing is that we can reset those boot counters again.  Because, I mean, let's say there's an outside effect that causes failure.  Say the power is really low and the system starts.  And as soon as you start powering on additional components, it breaks down and resets the system.  And then you do that in a loop and say, OK, three times, four times, my system failed.  OK, let's follow over to the previous version.  And then you try it here three times, four times, and then, yeah, both boot counters  are zero.  And what do we do now?  We have no system that's seemingly working.  And then I can either say, OK, get stuck here and don't do anything, or we can say,  maybe there was some outside effect.  So if all systems say boot counter zero can't do anything, we start again from the beginning  with reset the boot counters to some value, to three or five or whatever you want, and  then try again.  And maybe now the power is better, and then it works again.  And also important, atomic updates.  This is something where I like it's built into the barebox boot loader.  It's something where a lot of people spend time fixing things, because writing multiple  values on an embedded system in an atomic way that when any time the power fails, it's  still there in a consistent state.  That's pretty tricky.  Because if you have an ant flash, for example, then doing, OK, I need to erase this block  and I need to write here and there and do things in a way that at any time the power  can fail, that's pretty hard.  And so there's just basically a framework there that says, OK, set these values and  then now sync it to whatever spec and storage we have.  And then it's always an atomic update.  So and now let's look at actually how we use that.  Just a few examples of what you can do.  And this is basically one end of the range of options.  You probably want to do something in the middle.  But this is OK for many cases.  Actually, not what many people actually do right now.  Is this, well, it's started up.  Maybe things can fail.  But afterwards, everything errors that come there, that just trends in.  And it works afterwards again after we reboot.  It's probably true in many cases because it will work for time if it's an error that's  say a memory leak, right?  Memory leak that accumulates memory over time.  So it runs for three days and then runs out of memory.  We reset the system.  It runs again for three days.  That's OK under some circumstances because the system is running for three days, which  means at the very least, it's running for three days in which time we can actually continue  an update in your system.  So we can write a new update in there.  And that's always the most important thing that you're always in a situation where you  can write a new update so you can fix things.  In this case, we just reset the boot counter when the system has successfully booted.  Somebody will help us with that because we can just pick a target and say, OK, we have  this target.  And any application, any service is required by this target.  So if any of those fail, then we never reach this target.  And then we have one service that waits for this target and says, OK, when we reach this  target, we have the next service in the chain.  And this one will simply just reset the boot counters for the running system.  If you're a bit more paranoid, you'll say, well, any errors may be permanent.  So we don't actually reset the boot counter after booting.  So the system starts, the boot loader counts down the boot counter, and then it's OK.  And then it's running.  And only if we do an explicit reboot of the system, if we trigger a normal reboot, for  example, after an update or for whatever reason else, then we reset the boot counter.  Say, OK, it's running for some time, and then it's good.  And everything works.  And maybe we need to reboot for whatever reason or not.  And this is kind of like any failure later on, it triggers to reset.  The boot counter was not reset.  So it's counting down.  However, well, if you're developing or something, it's often this, well, let's switch off the  device and on again and do work and retry.  Some things, you will never actually reset the boot counter.  It happens pretty easily that you run into the case that you accidentally start the old  system because you didn't actually never reach the situation where the boot counters were  reset.  So this is the other extreme.  So there must be, in most cases, probably somewhere in between that you say, maybe it's  running for five days and now it looks good, or you have an outside operator actually monitoring  your devices and says, hey, this device was running for 10 days and this looks good, so  let's reset the boot counter.  It's probably working properly now.  It's really application-specific, device-specific, what kind of choices you make for that.  But what I like here for the system we have with system D, doing the monitoring and barebox  with its state and relatively isolated, is the state you can modify with any tool here.  You can choose when you update these states, when you reset the counters, and you can modify  how it reacts to your requirements.  So let's go to a nice topic.  When we're always talking about failures, let's talk a bit of user interaction because  update has a lot to do with users as soon as there is actually someone watching.  It can be someone local.  A lot of devices are starting to get displaced, more and more devices with displaced and  more and more.  This plays an info and see the progress of an update and these kind of things.  It's a remote operator that says, okay, I installed this update here and I want to see  something.  And then it helps actually because if you can ask a user to, or the user can decide  to turn the device completely off and on again, you can use that to handle errors as well.  Because if you cannot detect it in your software, maybe because it's a high level user interaction  stuff, you can still trigger a reset of the whole system by turning the power off and  on and you can use that, okay, the power was turned off.  We didn't reset the boot counter so it's one less.  And if you try it multiple times, then we start the old system at some point or immediately.  We can reset the boot counters after power off and have this all counters for zero and  after power doing a power cycle once, we start again trying to do things.  It really depends on the situation.  You can also say you have something in your UI to do, yeah, switch to the old system because  this is not working.  Or yeah, these kind of things.  This is faster, I think, I don't have that much time.  And what's really important, what I think is that they take time.  Even if it's running more or less in the background because you can still do the actual job.  If it's a device where a user is standing in front of it, puts the USB again, say I  do run my update, and it says nothing for ten minutes, that's going to be, yeah, the  update is not working.  Let's power off and try again.  So adding information there to update is really important, I think.  And also in case of error handling, your customer says it's not working and then says, well,  yeah, and it says the customer is gone.  I don't know why.  And if you don't have, can pull out some information that's integrated with your application where  you can pull in your log files, that's pretty hard to debug afterwards.  So not just the black box, I put here the route, it's one update system, I know it because  my colleagues are working on it.  That just has a debug API and says, hey, I'm doing this and this and this step.  So yeah, that's it.  Few links there, but just a few questions.  Getting pretty late.  No questions anymore?  Okay.  I think that's it for today.  Thank you.
2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_0_59.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_60_155.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_156_467.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_468_551.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_552_899.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_900_1055.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_1056_3059.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_3060_4055.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_4056_6659.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_6660_8063.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_8064_11939.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_11940_14567.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_14568_18683.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_18684_21299.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_21300_25967.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_25968_29015.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_29016_32771.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_32772_33551.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_33552_33599.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_33600_33719.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_33720_35735.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_35736_35783.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_35784_36107.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_36108_36155.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_36156_36503.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_36504_36875.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_36876_36935.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_36936_37259.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_37260_37307.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_37308_37655.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_37656_38495.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_38496_38555.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_38556_38603.png 2017_Updating_Embedded_Systems_--_Putting_it_all_Together_kMZQtRGT1Yc_38604_38636.png