Solo5: A sandboxed, re-targetable execution environment for unikernels
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Lucina, Martin
Koller, Ricardo
Solo5 is a microkernel friendly, sandboxed, re-targetable execution environment for unikernels, with a taste for minimalism. We will start with an overview of core Solo5 concepts and present the interfaces it offers to the unikernel/library operating system/application developer. Using existing library operating systems, such as MirageOS, we will demonstrate the developer experience for various Solo5 targets, going on to show how rigorously applying minimalist principles to interface design is used to our advantage, blurring traditional lines between unikernels, processes, kernels and hypervisors. We will conclude with some lessons learned during development of Solo5 thus far, and present ideas and challenges for future development. The target audiences for this talk are application developers with an interest in operating systems and unikernels, language runtime and toolchain developers interested in porting their language to a unikernel environment such as Solo5, and operating systems developers in general. No {uni,micro,separation,monolithic,}kernels or hypervisors will be harmed during this talk.
 Hi everyone, welcome. My name is Martin and this is my collaborator, Ricardo, and we're going to be talking about Solo 5, which is something we're calling a retargetable sandbox execution environment for Unicernals. That's a lot of words. So we'll go and try and explain some of the concepts. If I can switch to the next slide. So actually Ricardo told me that in the Unicraft talk, which was just before a bunch of people were asking what is a Unicernal, so it's good that we have some background to this. There are two separate terms that people use here. There are two separate terms that people use that are kind of similar related to each other. But I think it's important to show some distinction between them. So a library operating system, these are things that have been around since the early 90s. They actually come from even older technologies, Butler-Lampson had some papers I think in the 70s that were kind of using similar concepts. A library operating system is essentially just a collection of libraries except those libraries do traditional OSC kind of things that you'd normally expect to have in your monolithic kernel. It's in the sacred kernel space and you have file systems, TCP stacks and so on. With micro kernels, that's a bit different, but it's still part of, I mean, a micro kernel component. Whereas here, it really is just a collection of libraries. Generally, there is no concept of process isolation with a Libos. That's not really its job. And they also use cooperative scheduling. So for the non-OSC people in the room, that basically means applications are more like event loops rather than getting preempted internally. So you take this collection of libraries and then you combine it at compile time and out pops a unikernel. That's the distinction I'd like to make between these terms. The unikernel is the art effect which is made up of your library operating system and your application code. Unikernels, then due to this process of being built out of this collection of libraries, we can minimize the code that goes into them. So you have minimal code size which implies a minimal kind of horizontal attack surface. There's not very much stuff that goes into this thing. You can think of them as single purpose, single application operating systems. Though then people tend to think, oh, it's actually a kernel. It has to run in kernel space. That's not true. Conceptually, I would, my favorite way of looking at it is it's just a process that does more. So it does more operating system type things. So what is solo5? This is the shape of solo5 and the top bit. So is where your unikernel and library OS go? And the bottom triangle pointing up which is much bigger in this case is some host thing. And solo5 occupies the space that's the blue circle in the middle. Okay, what is this then? So there are three important parts to it. One is a minimalist legacy free interface. So really, really small and by interface, as I'll go to show later, it's basically a bunch of C function calls. Then we have some bindings which are the implementation of that interface. And in this picture here, the bindings would be roughly the top bits, the top part of the intersection of the circle with the triangle. And we have bindings for micro kernels, for separation kernels, and so for a vertio based hypervisors, so cloud hypervisors basically, and existing monolithic kernels. Additionally, on a monolithic kernel, we have a third concept which is a component we call a tender. And that's used to strongly sandbox the unikernel further away from this big bad monolithic kernel with its huge interfaces as I'll go to show in a bit. And we have two tenders. There's a hardware virtualized tender and a sandbox process tender. From the library operating system and all the application and library operating system developers point of view, solo5 is just middleware. So ideally, it's integrated directly into the libos build system and you don't actually interact with it directly. An example for MirageOS is it is just a different parameter to Mirage Configure where you can say which of these targets I want and out pops a unikernel. That's the way it should be. Other libos is not quite up to that state yet, but hopefully we'll get there. So comparing the solo5 shape to other common isolation interfaces and the unites of execution. Again, the bottom side is always some host and the top part is the unites of execution that we're comparing solo5 against. Now, what I want to show here is both horizontally in the middle where they meet at this line, there is something that represents the relative width of these interfaces. Also vertically, I'm very roughly not entirely to scale comparing that unikernels are small, containers are generally larger and the VM, well a traditional VM can just be as big as your host or bigger. So the width of this interface in the container case, I'm actually showing that the interface is the entire host interface. Now, people may say, well, okay, but it's namespaced and we have C groups and whatever. Actually, that's just adding more logic to this interface which is just as wide as it was before. You can switch bits of it off which some people do with Sec-conf, we'll talk about that in a bit, but most of the time in your general container case you still have access to 350 calls. In the case with the traditional VM talking to some kind of host or hypervisor, the entire interface between the VM and the host is actually quite big. So you've got tons of stuff in here, you've got ACPI, device discovery, vertio, PCI, what about all of these things in themselves quite wide and complicated and stateful. So the guiding philosophy of Sol-05 is that the interface must be minimal, that I think is a fairly straightforward term. States list which is something I've appropriated for this purpose and I'll go on to explain what that means and it should also be portable. And then the implementation should be clearly layered, it should do one thing and do it well, it should be an engine for running unicernals, other things like orchestration, configuration management, monitoring, whatever, those are done by layers higher up the stack. So we're at the lowest layer, we try to integrate or we don't get in the way of other people integrating us into higher layers but that's not our job. So in terms of minimalism, we want to provide the simplest useful abstraction. Importantly that means we're not trying to be Linux, so this is not an exercise in re-implementing Linux or POSIX, it's let's try the smallest thing which is actually useful and still let's us get some work done. One thing that pops out of minimalism is also that we don't do any kind of device discovery. This leads to a very small implementation size, a typical configuration by that I mean Sol-05 compiled into one target that has a tender and some bindings, some of which go in your unicernals, some of which live on the host is about 3,000 lines of C. 12,000 lines of C is in total the whole code base with all the different combinations of tenders and bindings that we have. Minimalism also gives us clarity of implementation, so we don't try to do very much and we prefer to do what we do correctly rather than add more features. And it gives us very fast startup time, comparable to what a normal user process would have. So with Sol-05 HVT or SPT we get something of less than 50 milliseconds. A traditional QMU will typically take when starting a Linux VM up to a second before you get say to a first network packet. A lot of that time is spent precisely doing things like device discovery and then actually bootstrapping the kernel. But yeah, and then for cloud managed VMs with traditional hypervisors, so things like Google computer and general, it takes seconds. So stateless, minimalism by itself or a minimal interface by itself that's just saying okay, we don't do very much. But that is not necessarily all we want. Stateless, I mean that there's very little state inside this interface. So that means both there's little, there's very little state transferred in any of the calls. You could think of them as idempotent as possible, I suppose. And the guest, the unicernel cannot change host state. So there is no dynamic resource allocation. Conversely, the host cannot change guest state. So there are no interrupts going across this interface. This gives us a system that's very deterministic and easy to reason about. And it's also essentially a static system. So you get some resources, some things from the host up front, memory access to devices, so on. And that's it. You can never get more. And I think that the stateless property is also what very much helps us to actually strongly isolate this interface and do that both on existing monolithic kernels and on component-based and high assurance micro kernels. So it's this way of thinking about what the interface should do that helps us to get to strong isolation. So and the third part of the philosophy is that we want to be portable. That includes easily porting existing library OSs to Solif5. And we currently have MirageOS, which was the original library OS where Solif5 got started as a way of getting MirageOS to run on other things. And Riccardo here has also done support for IncludeOS, which is a C++-based library operating system, and RumpRun, which is a POSIX-like thing using NetBSD Rump kernels. So conversely, it should be easy to port Solif5 to new targets. And interestingly, since the project started, we've had OpenBSD VMM support contributed, which I'll show by a guy who's basically an OpenBSD user, definitely not a Solif5 expert. Adrian from the Moon Separation Kernel contributed a port to target Moon-native subjects as a Solif5 target. And Emery, who's here in this room, just last year contributed a port to run or implement the Solif5 interface, so bindings to the Solif5 interface as a genode component. So let's be honest, there are some limitations, or you might want to call this slide non-goals. I mean, these are kind of the inherent limitations. As I already said, it's not Linux. We don't intend to become Linux. If you want Linux, you know where to find it. But there are library OSs that let you get quite a lot of the way there. So RumpRun, LKL is something called, I think, the Linux kernel library, which is a similar project trying to do what Rump kernels did for NetBSD, but with Linux. So if you want, you can build your POSIX application against a LibOS. Semantically, it might be a bit different. You've got to be a bit careful, but generally, it's not a good idea. So keeping the interface stateless, well, no interrupts implies it's a single core, no SMP. The answer to that is run more than one instance of your thing and parallelize your work and pass messages around. It's not intended for interfacing to hardware. That is possibly a bigger limitation, but then we see CELF-5 as something which is components as part of a wider, disaggregated system, and then drivers become some other components problem. So you can do that both in the microkernel case, where you have other microkernel components providing access to hardware, or in the monolithic kernel case, you have the monolithic kernel itself in a kind of in the type two hypervisor approach providing access to the hardware, then it's not isolated. Or you have a disaggregated system like cubes or something based on passing hardware directly to isolated components. Portability implies that we have some copying semantics in the interface, and also we have things like one core per packet. That's not very nice, but well, actually, we found it's pretty good. But the main thing I'd point out there is it's not intended for scenarios where you're trying to do high performance computing or pass millions of packets per second. If you're doing that, generally, you will want hardware pass through anyway, and trying to implement an interface alongside of these goals for something like hardware pass through is, I think, just a fool's errand. You will end up with bits of the interface and typical hardware interface, ripple wind down the middle and you'll have bits sticking out. So having shown the philosophy, I'd like to show the actual interface. I'll leave this up on here for you for a bit to read through it. Originally, I had one slide with all of the interface because I thought it would be cool to just fit the whole interface in one slide, but then maybe that's too much text to read and one goes. So I've split it up into two groups. The top core, you can see, is basically an entry point into the unicernel guest. This also shows that the fact that there is a static system, so you get given some heap memory with some size and that's it. And you get told here it is, go for your life. There is an exit and an abort call. Those are separate calls. The abort call is something that's intended to say, oh, I'm giving up. Could the host thing, whatever it is, if it's configured to do things like core dumps or page someone or whatever, do something abnormal happened. If you care about that, then treat it differently. I don't like overloading exit and status codes for something which I think is conceptually slightly different, so that's why there are two. There's a console. The console is intended mainly for debugging-y type stuff or some simple logging, but generally you'd want to do that over protocols and networks or somewhere else, so it's a console. Nothing particularly interesting about that. We have monotonic time, we have wall clock time. Those are 64-bit types. They are nanoseconds, so they'll run out in 500-something years from now. Too bad. Then there is a yield function. What this does is it essentially says, please yield to the host until some deadline comes. Then, I'll show the second part where I'll repeat the yield. We actually want to talk to some devices. What we have here is we have a network API which is a very simple layer-to network with layer-to semantics. You push a package, packets can get robbed. Read is always non-blocking, so it will return again if there isn't anything to read. This ties into cello5 yield which will give you back a bool that says, do I have work to do or is there nothing to do and it's just the fact that the deadline was reached. Then we have a block interface which is basically block.io. Again, calls are item potent in themselves, so there is always an offset and a buffer together. The interface is intended to read and write only at some minimum block size that you get out of block info. You will either get back an okay and what you asked to write was written or not. There are a couple of things missing here. I'm trying to observe and people will be asking themselves right now which is where do you say which of these devices you want to talk to? Also, more obviously, in the block API, there is no flush. That is just missing. That is something that we need to add, so there will likely be a flush and a discard. The multiple devices thing at the moment, the implementation is rather naive and you can only have one or zero or one of these. This is something that we want to change. I'll go into a bit more detail later about why it hasn't happened yet, but fundamentally this interface will not change hugely even in the presence of multiple devices. The difference will probably be that block info and net info will give you back some kind of handle. You will pass in and identify some name of the device, you will get a handle out and then read and write except a handle. Yield, interestingly, will not change much. It might give you back a mask of bits saying which of the handles have worked for them, but I don't expect to have again with the not changing host states, you will always ask for is there any work at all? You don't go saying, oh, I'm no longer interested in net zero. Well, no, you asked for it in the first place in the beginning, so if there's any work, then I'll tell you there's work on it. So, I'm going to hijack the presentation for a couple of minutes. Concrete view. I need to give you a mic. Well, but no one will hear you on the stream. I can just kind of hold it. I can do it. Before that, I just wanted to add something to what Meito was saying, which is that even though the interface looks very, very minimalistic, it's surprisingly enough to run a lot of stuff. For example, with run, we can run Python, node, a lot of things which at least to me, it's pretty amazing that such a simple interface is able to do. So, now, again. Can I make the font size smaller? Sure. There you go. That's probably just about right there. Okay. Okay. So, let's say you have a, actually, let me show you this. Let's say you want to implement the small web server as a unicernel that just returns something like a pretty camel. And it stores, of course, camels. And it stores counter in persistent storage just to count the number of requests. So, you need storage, you need a file system with a key value store, and you need an HTTP server, basically. So, one way of implementing this would be to use something like Mirage. And because you want to experiment camels, you would use something like a camel, right? Makes sense. So, you would go into Mirage and you would write a bit of a camel and you would use an HTTP library and also a network library and also something like a key value store as a library. And then you would tell the Mirage build system that you want to construct a unicernel using some target. Let's say a solo5 target. And more specifically, let's say a solo5 micro-VMM target. So, in that case, you would do that, you will make and you will get something like this. You would build and you will get the unicernel, which will be this one dot HVT, which is actually it's not that small as I thought. And also the tender, which will be the equivalent of QMU, for example, if you were running a typical VM. And now the way you would run this is just like with QMU, you would say HVT. Now because we need a disk to store the key value store information, we need to specify the disk and we would say disk image net tap and HTTP HVT. Yeah, well, it's huge. Okay. And then just to show you that actually works. So we go. Cool. Another possible target is the process one, which is called SPT. And so you would again tell the Mirageable system to build another unicernel, you will tell it now I want SPT and it will output something like this. SPT. SPT, which is also pretty large. I don't know why it's so large. Yeah, makes a lot of sense. Really, size will give a different number. Yeah, still pretty large. Anyway, yeah, so you would use the SPT one, something like this, exactly the same. Just to show you as well that it works. Ta-da. 195 camels. So the counter is working. And now to give you a more in-depth view of what this thing is doing, let's take a look at the S-trace. So for example, for the HVT one, the SVT tender is kind of implementing the Sol of 5 interface by using Cisco. So we would map a Sol of 5 API call, for example, I yield into a poll into a specific file descriptor into the host. You don't need to explain that. That's on the following slide. True. Okay, sorry. Sorry, sorry. Okay, so let's do an S-trace. Yeah, well, just boot. And now it's just polling. It's just an event loop. And we'll take a look at what happens when we actually curl. We see a bunch of KVM stuff reading to the, this is the tab, and it pulls. I mean, it's really very, very minimal. It basically reads and writes into the, this file descriptor and reads and writes into the, into the tab file descriptor. And now to compare against the, the SPT one, let's do the same on SPT. Yeah, that's booting down the way. So that's pretty much exactly the same except that it doesn't have the KVM I-actiles. Yeah. And again, that's just to give you a rough idea of the minimalism. And also that we have an even more minimal interface which is called SPT that is pretty much like the KVM one, but without doing all the KVM stuff. And now back to Maitre to get, to tell you more about how this works. So, Ricardo showed a couple of the solo five targets that use tenders to run on monolithic kernels. What do these actually do? So there's the HVT, which is the hardware virtualized tender, which uses hardware virtualization, obviously, from the name. It runs on KVM on Linux, on FreeBSD on Behive, and on OpenBSD on the VMM APIs. So this is not a traditional VMM, not even close. It has about 10 hypercalls. It self-codebase is modular, again, a typical configuration. So a solo five HVT on Linux on x8664 would be about one and a half thousand lines of code. So again, right, we're talking about the tender, which is this part. It's the bottom intersection of the bottom triangle with the blue circle. Comparing to something like QMU, well, QMU, I tried to be nice to them. So I ignored the one and a half million lines of code that was just random blobs of firmware and stuff, and that still leaves another roughly million lines of code. Something like Crosvm, this may be a little bit unfair, but I didn't, S-Lock counts doesn't understand Rust, but I came up with some figure which looks like it's an order of magnitude smaller. HVT is also something that was the first tender that we implemented. So it's been around since 2015. It works. It then got FreeBSD added into it, and then OpenBSD contributed later as well. It was formerly known as UKVM, which some people here may be familiar with that name. So what is the tender responsible for? Well, it loads the Unicernal artifact into the VCPU's memory. It sets up access to the host resources that the Unicernal is asking for, so the network tap device or the disk block device. It does some VCPU setup, sets up page tables for the guest which are very simple, just one-to-one page tables. So there's no virtual memory is being used at the guest level. And then it needs to hang around to handle guest hypercalls, which handle VM exits to actually perform some IO. So hypercalls, what do they actually look like? What we do is we transfer a 32-bit pointer to a struct which contains the hypercalls parameters. The reason there is this hybrid approach is we actually want to run on existing deployed KVM systems. So I think Martin Dedzky, I don't know if he's here mentioned VMfunk. There is also an instruction called VM call, which you could use on x86 for this, except that KVM steals that for its own purposes. So too bad. So we figured out this approach which on x86 basically does an out to some port which causes the VM exit and on arm64 we do an MMIO to some region which also causes VM exit. At that point the tender just fishes the data out of the guest's memory and operates on it directly. So the HVT bindings then implement the cell F5 interface on using these hypercalls to talk to the tender. Apart from that they don't do very much. So there's some handling of VCP trap vectors which basically just say, oh, we got a trap abort, sorry. And the only other thing they do is provide monotonic time using a timestamp counter or an equivalent mechanism. So you can see an entire implementation of cell F5 net write for HVT which is basically just putting stuff into a structure and then invoking the hypercall. The mucking about with return codes here is because these two interfaces actually need to be aligned but we haven't done that yet. So Ricardo showed you HVT but we also have something called SPT which we're calling a sandbox process tender. The interesting thing about this, notice that the blue circle suddenly got much much smaller. So what SPT does is instead of using hardware virtualization which is great if you have it but you don't always have access to that but just for example you might be running in a cloud where you're already virtualized. While nested virtualization exists it's not something I would actually use to isolate things with. So instead we had this realization that, oh, we could just, we already know exactly the calls we need to implement this interface. So why not just use a system call filter, use a strict system call filter and turn off all the APIs that the host kernel is providing. And it turns out that this is exactly what SPT does. Currently we only have this implemented for Linux using seccomp.bpf but it should be possible to port to free BSD and open BSD. There's also a link to our paper from last year which talks a bit more about quantifying the difference in the TCB where we actually make some claims where we say they're using something like HVT which touches hardware virtualization and KVM and actually ends up using more of the host kernel code than the approach where we switch off all but these seven system calls. So similarly to HVT, SPT loads the unicolonal into its process address space, sets up access to the host resources, applies the sandbox and then basically gets out of the way. So the blue circle should actually disappear at this point but I don't have animations in there. So what this does is you've now taken your unicolonal recompiled the same code against a different target and you're now running as essentially a user processor. That's coming back to the original slide where I say just unicolonals are just like processors that do more and here we're basically treating the monolithic kernel as kind of as a hypervisor. So switch off everything. So some more about the binding side. Those who are interested, so the bindings basically end up directly in voting system calls but they are now inside the sandbox which they cannot change. There's no libc involved. It's in there. Yep, we have it for x8664 and now 64, for SPT it's pretty trivial to add basically anything that the host supports. For HVT, I'm not sure if I mentioned that. We also have these two architectures. It's a little bit more work but yeah, so possible. What this is. So as I mentioned, we have targets for micro kernels as well. So moon is a rather interesting, it's actually technically called a separation kernel. So for people that are not familiar with the distinction, a separation kernel gives you much, in a simplified way gives you much stronger guarantees about how your components can communicate. So about the data flow between the things in the system. It's also, I mean the original term is used as a way of saying that the componentized system should look exactly like as if it were a distributed system. You can look up the term and there's some more discussion about it there. So moon provides isolation using hardware virtualization. It's implemented in Ada Spark and it's formally proven to contain no runtime errors at the source code level. The shape is now a bit different. So the blue circle has moved more or less into the unicernel and moon is much, much smaller. There aren't any hypercalls, communication is done via shared memory. Verti was the first seller 5 implementation. Shape doesn't really match up. We still have it there because, well, people find it useful. I mean it runs on Google Compute Engine, that's great. But yeah, it's doing like two and a half thousand lines of code in there just to do virtio stuff. Does anyone want to implement virtio scuzzy? Not really. Debugging, this is the special slides. There is this myth that unicernels are undebuggable. Just use GDB. And you can do it with a unicernel hidden inside an HVTVM or with SPT as well. Some lessons learned. Okay, this is a bit complicated if I have only five minutes. But originally with UKVM we wanted to specialize the tender to the unicernel. It turns out that, and we did that at compile time, that's not practical. The supply chain doesn't make sense. You're not going to be supplying the tender and the unicernel at the same time. It's now gone, but we still want to, it's gone for SPT. We still haven't removed it for HVTV, but it needs to go, we still want to enforce a contract between the tender and the unicernel. So the unicernel wants some resources, tender should match those and not accidentally provide something else. Possible solution is to embed a manifest into the unicernel artifacts, but can we still specialize the tender? Some plans and challenges, security, we are gradually implementing more best practices like ASLR, stack smashing protection, write or execute, but again, stack smashing protection, the compiler ABI's are completely undocumented. You have to go and read the compiler source to find out what it's doing. But on x86, it always wants %fs, 0x28. Why isn't that specified somewhere? So there's some hypervisor support lacking on free BSD and open BSD, where if you do an M-protect on memory that's owned by the guest, by the hypervisor, that does not update the EBT tables, it's just a knob. KVM actually does this right. And some defense in depth. Portability, can we do dynamic linking safely and get a tender independent unicernel binary? So none of this ahead of time choosing of do I want this or that isolation mechanism. That's a very good question. Can we define an interface that allows asynchronousity but is consistent with our philosophy? If someone can define one for me on a back of a bare mat and this later this evening, then you are definitely God. Please contribute. We would like more languages and LibOS is ported. Also, more targets. HVT should run on Mac OS. You should be able to build an SGX enclave out of this, using, keeping the same interface. Web assembly. Okay. Related work. How much time do I have? Two minutes. This should be on a spectrum. Solo5 is somewhere here, like the most radical thing. QMU is somewhere here. Drawbridge is somewhere here. Cross VM, firecracker, the sort of going towards the right. G visor is just weird. That's up there. That's trying to reimplement the Linux syscall interface and go. I mean, yeah, sure. It's strange. So we can do a lot with minimalism. We have a tiny API. It's legacy free. It's ISC licensed. You can run 1500 HVT things on your three year old laptop. We want to apply you to kernels everywhere. Both on monolithic systems, because there are a lot of them around, but you should trivially move to recompile your code and run it and G node, run it on NUIN. And at the same time, no compromises. So you still get strong isolation across the spectrum of targets. Thank you. Any questions? Okay. So what you're saying is should we sandbox HVT even more? So one of the things that we will do is extend the set comp sandbox to the HVT tender itself. So I think that's kind of what you're touching on. Well, like providing a super minimal kernel that would run inside HVT and expose the SVT system calls to your SVT. So you have like Solar 5 and up to Solar 5. No, we haven't thought about that. Yeah, it's not recursive. Sorry. Yeah. Any other questions? Yes. Yes. That was the real code you will get. So there's some. There's no reason you should get into because there aren't any signals being handled, especially not in the set comp case because you don't even have SIG return in the set comp filter. So if you get a signal, then you can't handle it. Sorry, your process just dies. I haven't actually seen, so I haven't really seen E-inter happen in any other case in practice. I mean, if you're not handling signals, then why would, why should you get that? But a good point. Yeah. Any other questions? Kind of an out there question. I would love to use this for sandboxing desktop graphical applications with super minimal interface for UI stuff. Have you put, has there any thought or? You're not the first person that's asked about. So can this be used for sandboxing desktop applications with actual UI? The question is what would that interface be because then you run into the problem of, well, if you want performance, you want at least some kind of, at least a frame buffer, but then that's very low level. So it's like, how do you decide that abstraction? I mean, how would this run across this interface? I mean, you could, of course, certainly run like an X11 client inside this thing, right, and have that talking to an X server somewhere else. So that would be one way of accomplishing that, but yeah, I mean, possibly not the, not the cleanest approach. I'm not sure. I know that there are people that did things with, I mean, even the, even the genoed folks have been doing things with virtualizing graphics hardware, but it's really quite a different area which I don't know that much about. But yeah, you could do the X11 thing or, yeah. Okay, thank you. Thank you.
 Hi everyone, welcome. My name is Martin and this is my collaborator, Ricardo, and we're  going to be talking about Solo 5, which is something we're calling a retargetable sandbox  execution environment for Unicernals. That's a lot of words. So we'll go and try and explain  some of the concepts. If I can switch to the next slide. So actually Ricardo told me that  in the Unicraft talk, which was just before a bunch of people were asking what is a Unicernal,  so it's good that we have some background to this. There are two separate terms that  people use here. There are two separate terms that people use that are kind of similar related  to each other. But I think it's important to show some distinction between them. So a library  operating system, these are things that have been around since the early 90s. They actually  come from even older technologies, Butler-Lampson had some papers I think in the 70s that were  kind of using similar concepts. A library operating system is essentially just a collection of  libraries except those libraries do traditional OSC kind of things that you'd normally expect  to have in your monolithic kernel. It's in the sacred kernel space and you have file systems,  TCP stacks and so on. With micro kernels, that's a bit different, but it's still part of,  I mean, a micro kernel component. Whereas here, it really is just a collection of libraries.  Generally, there is no concept of process isolation with a Libos. That's not really its job. And  they also use cooperative scheduling. So for the non-OSC people in the room, that basically  means applications are more like event loops rather than getting preempted internally. So  you take this collection of libraries and then you combine it at compile time and out pops  a unikernel. That's the distinction I'd like to make between these terms. The unikernel  is the art effect which is made up of your library operating system and your application  code. Unikernels, then due to this process of being built out of this collection of libraries,  we can minimize the code that goes into them. So you have minimal code size which implies  a minimal kind of horizontal attack surface. There's not very much stuff that goes into  this thing. You can think of them as single purpose, single application operating systems.  Though then people tend to think, oh, it's actually a kernel. It has to run in kernel  space. That's not true. Conceptually, I would, my favorite way of looking at it is it's just  a process that does more. So it does more operating system type things. So what is solo5?  This is the shape of solo5 and the top bit. So is where your unikernel and library OS  go? And the bottom triangle pointing up which is much bigger in this case is some host thing.  And solo5 occupies the space that's the blue circle in the middle. Okay, what is this then?  So there are three important parts to it. One is a minimalist legacy free interface.  So really, really small and by interface, as I'll go to show later, it's basically a  bunch of C function calls. Then we have some bindings which are the implementation of that  interface. And in this picture here, the bindings would be roughly the top bits, the top part  of the intersection of the circle with the triangle. And we have bindings for micro kernels,  for separation kernels, and so for a vertio based hypervisors, so cloud hypervisors basically,  and existing monolithic kernels. Additionally, on a monolithic kernel, we have a third concept  which is a component we call a tender. And that's used to strongly sandbox the unikernel  further away from this big bad monolithic kernel with its huge interfaces as I'll go  to show in a bit. And we have two tenders. There's a hardware virtualized tender and  a sandbox process tender. From the library operating system and all the application and  library operating system developers point of view, solo5 is just middleware. So ideally,  it's integrated directly into the libos build system and you don't actually interact with  it directly. An example for MirageOS is it is just a different parameter to Mirage Configure  where you can say which of these targets I want and out pops a unikernel. That's the  way it should be. Other libos is not quite up to that state yet, but hopefully we'll  get there. So comparing the solo5 shape to other common isolation interfaces and the  unites of execution. Again, the bottom side is always some host and the top part is the  unites of execution that we're comparing solo5 against. Now, what I want to show here is  both horizontally in the middle where they meet at this line, there is something that  represents the relative width of these interfaces. Also vertically, I'm very roughly not entirely  to scale comparing that unikernels are small, containers are generally larger and the VM,  well a traditional VM can just be as big as your host or bigger. So the width of this  interface in the container case, I'm actually showing that the interface is the entire host  interface. Now, people may say, well, okay, but it's namespaced and we have C groups and  whatever. Actually, that's just adding more logic to this interface which is just as wide  as it was before. You can switch bits of it off which some people do with Sec-conf, we'll  talk about that in a bit, but most of the time in your general container case you still  have access to 350 calls. In the case with the traditional VM talking to some kind of  host or hypervisor, the entire interface between the VM and the host is actually quite big.  So you've got tons of stuff in here, you've got ACPI, device discovery, vertio, PCI, what  about all of these things in themselves quite wide and complicated and stateful. So the  guiding philosophy of Sol-05 is that the interface must be minimal, that I think is a fairly  straightforward term. States list which is something I've appropriated for this purpose  and I'll go on to explain what that means and it should also be portable. And then the  implementation should be clearly layered, it should do one thing and do it well, it should  be an engine for running unicernals, other things like orchestration, configuration management,  monitoring, whatever, those are done by layers higher up the stack. So we're at the lowest  layer, we try to integrate or we don't get in the way of other people integrating us  into higher layers but that's not our job. So in terms of minimalism, we want to provide  the simplest useful abstraction. Importantly that means we're not trying to be Linux, so  this is not an exercise in re-implementing Linux or POSIX, it's let's try the smallest  thing which is actually useful and still let's us get some work done. One thing that pops  out of minimalism is also that we don't do any kind of device discovery. This leads to  a very small implementation size, a typical configuration by that I mean Sol-05 compiled  into one target that has a tender and some bindings, some of which go in your unicernals,  some of which live on the host is about 3,000 lines of C. 12,000 lines of C is in total  the whole code base with all the different combinations of tenders and bindings that  we have. Minimalism also gives us clarity of implementation, so we don't try to do very  much and we prefer to do what we do correctly rather than add more features. And it gives  us very fast startup time, comparable to what a normal user process would have. So with  Sol-05 HVT or SPT we get something of less than 50 milliseconds. A traditional QMU will  typically take when starting a Linux VM up to a second before you get say to a first  network packet. A lot of that time is spent precisely doing things like device discovery  and then actually bootstrapping the kernel. But yeah, and then for cloud managed VMs with  traditional hypervisors, so things like Google computer and general, it takes seconds.  So stateless, minimalism by itself or a minimal interface by itself that's just saying okay,  we don't do very much. But that is not necessarily all we want. Stateless, I mean that there's  very little state inside this interface. So that means both there's little, there's very  little state transferred in any of the calls. You could think of them as idempotent as possible,  I suppose. And the guest, the unicernel cannot change host state. So there is no dynamic  resource allocation. Conversely, the host cannot change guest state. So there are no  interrupts going across this interface. This gives us a system that's very deterministic  and easy to reason about. And it's also essentially a static system. So you get some resources,  some things from the host up front, memory access to devices, so on. And that's it. You  can never get more. And I think that the stateless property is also what very much helps us to  actually strongly isolate this interface and do that both on existing monolithic kernels  and on component-based and high assurance micro kernels. So it's this way of thinking  about what the interface should do that helps us to get to strong isolation. So and the  third part of the philosophy is that we want to be portable. That includes easily porting  existing library OSs to Solif5. And we currently have MirageOS, which was the original library  OS where Solif5 got started as a way of getting MirageOS to run on other things. And Riccardo  here has also done support for IncludeOS, which is a C++-based library operating system,  and RumpRun, which is a POSIX-like thing using NetBSD Rump kernels. So conversely, it should  be easy to port Solif5 to new targets. And interestingly, since the project started,  we've had OpenBSD VMM support contributed, which I'll show by a guy who's basically an  OpenBSD user, definitely not a Solif5 expert. Adrian from the Moon Separation Kernel contributed  a port to target Moon-native subjects as a Solif5 target. And Emery, who's here in this  room, just last year contributed a port to run or implement the Solif5 interface, so  bindings to the Solif5 interface as a genode component. So let's be honest, there are some  limitations, or you might want to call this slide non-goals. I mean, these are kind of  the inherent limitations. As I already said, it's not Linux. We don't intend to become  Linux. If you want Linux, you know where to find it. But there are library OSs that let  you get quite a lot of the way there. So RumpRun, LKL is something called, I think,  the Linux kernel library, which is a similar project trying to do what Rump kernels did  for NetBSD, but with Linux. So if you want, you can build your POSIX application against  a LibOS. Semantically, it might be a bit different. You've got to be a bit careful, but generally,  it's not a good idea. So keeping the interface stateless, well, no interrupts implies it's  a single core, no SMP. The answer to that is run more than one instance of your thing  and parallelize your work and pass messages around. It's not intended for interfacing  to hardware. That is possibly a bigger limitation, but then we see CELF-5 as something which  is components as part of a wider, disaggregated system, and then drivers become some other  components problem. So you can do that both in the microkernel case, where you have other  microkernel components providing access to hardware, or in the monolithic kernel case,  you have the monolithic kernel itself in a kind of in the type two hypervisor approach  providing access to the hardware, then it's not isolated. Or you have a disaggregated system  like cubes or something based on passing hardware directly to isolated components.  Portability implies that we have some copying semantics in the interface, and also we have  things like one core per packet. That's not very nice, but well, actually, we found it's  pretty good. But the main thing I'd point out there is it's not intended for scenarios  where you're trying to do high performance computing or pass millions of packets per  second. If you're doing that, generally, you will want hardware pass through anyway, and  trying to implement an interface alongside of these goals for something like hardware  pass through is, I think, just a fool's errand. You will end up with bits of the interface  and typical hardware interface, ripple wind down the middle and you'll have bits sticking  out.  So having shown the philosophy, I'd like to show the actual interface. I'll leave this  up on here for you for a bit to read through it. Originally, I had one slide with all of  the interface because I thought it would be cool to just fit the whole interface in one  slide, but then maybe that's too much text to read and one goes. So I've split it up  into two groups. The top core, you can see, is basically an entry point into the unicernel  guest. This also shows that the fact that there is a static system, so you get given  some heap memory with some size and that's it. And you get told here it is, go for your  life.  There is an exit and an abort call. Those are separate calls. The abort call is something  that's intended to say, oh, I'm giving up. Could the host thing, whatever it is, if it's  configured to do things like core dumps or page someone or whatever, do something abnormal  happened. If you care about that, then treat it differently. I don't like overloading exit  and status codes for something which I think is conceptually slightly different, so that's  why there are two. There's a console. The console is intended mainly for debugging-y  type stuff or some simple logging, but generally you'd want to do that over protocols and networks  or somewhere else, so it's a console. Nothing particularly interesting about that. We have  monotonic time, we have wall clock time. Those are 64-bit types. They are nanoseconds, so  they'll run out in 500-something years from now. Too bad.  Then there is a yield function. What this does is it essentially says, please yield to  the host until some deadline comes. Then, I'll show the second part where I'll repeat  the yield. We actually want to talk to some devices. What we have here is we have a network  API which is a very simple layer-to network with layer-to semantics. You push a package,  packets can get robbed. Read is always non-blocking, so it will return again if there isn't anything  to read. This ties into cello5 yield which will give you back a bool that says, do I  have work to do or is there nothing to do and it's just the fact that the deadline was  reached. Then we have a block interface which is basically block.io. Again, calls are item  potent in themselves, so there is always an offset and a buffer together. The interface  is intended to read and write only at some minimum block size that you get out of block  info. You will either get back an okay and what you asked to write was written or not.  There are a couple of things missing here. I'm trying to observe and people will be  asking themselves right now which is where do you say which of these devices you want  to talk to? Also, more obviously, in the block API, there is no flush. That is just missing.  That is something that we need to add, so there will likely be a flush and a discard.  The multiple devices thing at the moment, the implementation is rather naive and you  can only have one or zero or one of these. This is something that we want to change.  I'll go into a bit more detail later about why it hasn't happened yet, but fundamentally  this interface will not change hugely even in the presence of multiple devices. The difference  will probably be that block info and net info will give you back some kind of handle. You  will pass in and identify some name of the device, you will get a handle out and then  read and write except a handle. Yield, interestingly, will not change much. It might give you back  a mask of bits saying which of the handles have worked for them, but I don't expect to  have again with the not changing host states, you will always ask for is there any work  at all? You don't go saying, oh, I'm no longer interested in net zero. Well, no, you asked  for it in the first place in the beginning, so if there's any work, then I'll tell you  there's work on it. So, I'm going to hijack the presentation for a couple of minutes.  Concrete view. I need to give you a mic. Well, but no one will hear you on the stream. I  can just kind of hold it. I can do it. Before that, I just wanted to add something to what  Meito was saying, which is that even though the interface looks very, very minimalistic,  it's surprisingly enough to run a lot of stuff. For example, with run, we can run Python,  node, a lot of things which at least to me, it's pretty amazing that such a simple interface  is able to do. So, now, again. Can I make the font size smaller? Sure. There you go.  That's probably just about right there. Okay. Okay. So, let's say you have a, actually,  let me show you this. Let's say you want to implement the small web server as a unicernel  that just returns something like a pretty camel. And it stores, of course, camels. And  it stores counter in persistent storage just to count the number of requests. So, you need  storage, you need a file system with a key value store, and you need an HTTP server,  basically. So, one way of implementing this would be to use something like Mirage. And  because you want to experiment camels, you would use something like a camel, right? Makes  sense. So, you would go into Mirage and you would write a bit of a camel and you would  use an HTTP library and also a network library and also something like a key value store  as a library. And then you would tell the Mirage build system that you want to construct  a unicernel using some target. Let's say a solo5 target. And more specifically, let's  say a solo5 micro-VMM target. So, in that case, you would do that, you will make and  you will get something like this. You would build and you will get the unicernel, which  will be this one dot HVT, which is actually it's not that small as I thought. And also  the tender, which will be the equivalent of QMU, for example, if you were running a typical  VM. And now the way you would run this is just like with QMU, you would say HVT. Now  because we need a disk to store the key value store information, we need to specify the  disk and we would say disk image net tap and HTTP HVT. Yeah, well, it's huge. Okay. And  then just to show you that actually works. So we go. Cool. Another possible target is  the process one, which is called SPT. And so you would again tell the Mirageable system  to build another unicernel, you will tell it now I want SPT and it will output something  like this. SPT. SPT, which is also pretty large. I don't know why it's so large.  Yeah, makes a lot of sense. Really, size will give a different number. Yeah, still pretty  large. Anyway, yeah, so you would use the SPT one, something like this, exactly the same.  Just to show you as well that it works. Ta-da. 195 camels. So the counter is working. And  now to give you a more in-depth view of what this thing is doing, let's take a look at the  S-trace. So for example, for the HVT one, the SVT tender is kind of implementing the  Sol of 5 interface by using Cisco. So we would map a Sol of 5 API call, for example, I yield  into a poll into a specific file descriptor into the host.  You don't need to explain that. That's on the following slide.  True. Okay, sorry. Sorry, sorry. Okay, so let's do an S-trace. Yeah, well, just boot. And  now it's just polling. It's just an event loop. And we'll take a look at what happens  when we actually curl. We see a bunch of KVM stuff reading to the, this is the tab, and  it pulls. I mean, it's really very, very minimal. It basically reads and writes into  the, this file descriptor and reads and writes into the, into the tab file descriptor. And  now to compare against the, the SPT one, let's do the same on SPT. Yeah, that's booting  down the way. So that's pretty much exactly the same except that it doesn't have the KVM  I-actiles. Yeah. And again, that's just to give you a rough idea of the minimalism. And  also that we have an even more minimal interface which is called SPT that is pretty much like  the KVM one, but without doing all the KVM stuff. And now back to Maitre to get, to tell  you more about how this works. So, Ricardo showed a couple of the solo five targets that  use tenders to run on monolithic kernels. What do these actually do? So there's the HVT,  which is the hardware virtualized tender, which uses hardware virtualization, obviously,  from the name. It runs on KVM on Linux, on FreeBSD on Behive, and on OpenBSD on the  VMM APIs. So this is not a traditional VMM, not even close. It has about 10 hypercalls.  It self-codebase is modular, again, a typical configuration. So a solo five HVT on Linux  on x8664 would be about one and a half thousand lines of code. So again, right, we're talking  about the tender, which is this part. It's the bottom intersection of the bottom triangle  with the blue circle. Comparing to something like QMU, well, QMU, I tried to be nice to  them. So I ignored the one and a half million lines of code that was just random blobs of  firmware and stuff, and that still leaves another roughly million lines of code. Something  like Crosvm, this may be a little bit unfair, but I didn't, S-Lock counts doesn't understand  Rust, but I came up with some figure which looks like it's an order of magnitude smaller.  HVT is also something that was the first tender that we implemented. So it's been around since  2015. It works. It then got FreeBSD added into it, and then OpenBSD contributed later  as well. It was formerly known as UKVM, which some people here may be familiar with that  name. So what is the tender responsible for? Well, it loads the Unicernal artifact into  the VCPU's memory. It sets up access to the host resources that the Unicernal is asking  for, so the network tap device or the disk block device. It does some VCPU setup, sets  up page tables for the guest which are very simple, just one-to-one page tables. So there's  no virtual memory is being used at the guest level. And then it needs to hang around to  handle guest hypercalls, which handle VM exits to actually perform some IO. So hypercalls,  what do they actually look like? What we do is we transfer a 32-bit pointer to a struct  which contains the hypercalls parameters. The reason there is this hybrid approach is  we actually want to run on existing deployed KVM systems. So I think Martin Dedzky, I don't  know if he's here mentioned VMfunk. There is also an instruction called VM call, which  you could use on x86 for this, except that KVM steals that for its own purposes. So too  bad. So we figured out this approach which on x86 basically does an out to some port  which causes the VM exit and on arm64 we do an MMIO to some region which also causes  VM exit. At that point the tender just fishes the data out of the guest's memory and operates  on it directly. So the HVT bindings then implement the cell F5 interface on using these hypercalls  to talk to the tender. Apart from that they don't do very much. So there's some handling  of VCP trap vectors which basically just say, oh, we got a trap abort, sorry. And the only  other thing they do is provide monotonic time using a timestamp counter or an equivalent  mechanism. So you can see an entire implementation of cell F5 net write for HVT which is basically  just putting stuff into a structure and then invoking the hypercall. The mucking about with  return codes here is because these two interfaces actually need to be aligned but we haven't  done that yet. So Ricardo showed you HVT but we also have something called SPT which  we're calling a sandbox process tender. The interesting thing about this, notice that  the blue circle suddenly got much much smaller. So what SPT does is instead of using hardware  virtualization which is great if you have it but you don't always have access to that  but just for example you might be running in a cloud where you're already virtualized.  While nested virtualization exists it's not something I would actually use to isolate  things with. So instead we had this realization that, oh, we could just, we already know exactly  the calls we need to implement this interface. So why not just use a system call filter,  use a strict system call filter and turn off all the APIs that the host kernel is providing.  And it turns out that this is exactly what SPT does. Currently we only have this implemented  for Linux using seccomp.bpf but it should be possible to port to free BSD and open BSD.  There's also a link to our paper from last year which talks a bit more about quantifying  the difference in the TCB where we actually make some claims where we say they're using  something like HVT which touches hardware virtualization and KVM and actually ends up  using more of the host kernel code than the approach where we switch off all but these  seven system calls. So similarly to HVT, SPT loads the unicolonal into its process address  space, sets up access to the host resources, applies the sandbox and then basically gets  out of the way. So the blue circle should actually disappear at this point but I don't  have animations in there. So what this does is you've now taken your unicolonal recompiled  the same code against a different target and you're now running as essentially a user  processor. That's coming back to the original slide where I say just unicolonals are just  like processors that do more and here we're basically treating the monolithic kernel as  kind of as a hypervisor. So switch off everything. So some more about the binding side. Those  who are interested, so the bindings basically end up directly in voting system calls but  they are now inside the sandbox which they cannot change. There's no libc involved. It's  in there. Yep, we have it for x8664 and now 64, for SPT it's pretty trivial to add basically  anything that the host supports. For HVT, I'm not sure if I mentioned that. We also  have these two architectures. It's a little bit more work but yeah, so possible. What  this is. So as I mentioned, we have targets for micro kernels as well. So moon is a rather  interesting, it's actually technically called a separation kernel. So for people that are  not familiar with the distinction, a separation kernel gives you much, in a simplified way  gives you much stronger guarantees about how your components can communicate. So about  the data flow between the things in the system. It's also, I mean the original term is used  as a way of saying that the componentized system should look exactly like as if it were  a distributed system. You can look up the term and there's some more discussion about  it there. So moon provides isolation using hardware virtualization. It's implemented  in Ada Spark and it's formally proven to contain no runtime errors at the source code  level. The shape is now a bit different. So the blue circle has moved more or less into  the unicernel and moon is much, much smaller. There aren't any hypercalls, communication  is done via shared memory. Verti was the first seller 5 implementation. Shape doesn't really  match up. We still have it there because, well, people find it useful. I mean it runs  on Google Compute Engine, that's great. But yeah, it's doing like two and a half thousand  lines of code in there just to do virtio stuff. Does anyone want to implement virtio scuzzy?  Not really. Debugging, this is the special slides. There is this myth that unicernels  are undebuggable. Just use GDB. And you can do it with a unicernel hidden inside an HVTVM  or with SPT as well. Some lessons learned. Okay, this is a bit complicated if I have  only five minutes. But originally with UKVM we wanted to specialize the tender to the  unicernel. It turns out that, and we did that at compile time, that's not practical. The  supply chain doesn't make sense. You're not going to be supplying the tender and the unicernel  at the same time. It's now gone, but we still want to, it's gone for SPT. We still haven't  removed it for HVTV, but it needs to go, we still want to enforce a contract between  the tender and the unicernel. So the unicernel wants some resources, tender should match  those and not accidentally provide something else. Possible solution is to embed a manifest  into the unicernel artifacts, but can we still specialize the tender? Some plans and challenges,  security, we are gradually implementing more best practices like ASLR, stack smashing protection,  write or execute, but again, stack smashing protection, the compiler ABI's are completely  undocumented. You have to go and read the compiler source to find out what it's doing.  But on x86, it always wants %fs, 0x28. Why isn't that specified somewhere? So there's  some hypervisor support lacking on free BSD and open BSD, where if you do an M-protect  on memory that's owned by the guest, by the hypervisor, that does not update the EBT tables,  it's just a knob. KVM actually does this right. And some defense in depth. Portability, can  we do dynamic linking safely and get a tender independent unicernel binary? So none of this  ahead of time choosing of do I want this or that isolation mechanism. That's a very good  question. Can we define an interface that allows asynchronousity but is consistent with  our philosophy? If someone can define one for me on a back of a bare mat and this later  this evening, then you are definitely God. Please contribute. We would like more languages  and LibOS is ported. Also, more targets. HVT should run on Mac OS. You should be able  to build an SGX enclave out of this, using, keeping the same interface. Web assembly.  Okay. Related work. How much time do I have? Two minutes. This should be on a spectrum.  Solo5 is somewhere here, like the most radical thing. QMU is somewhere here. Drawbridge is  somewhere here. Cross VM, firecracker, the sort of going towards the right. G visor  is just weird. That's up there. That's trying to reimplement the Linux syscall interface  and go. I mean, yeah, sure. It's strange. So we can do a lot with minimalism. We have  a tiny API. It's legacy free. It's ISC licensed. You can run 1500 HVT things on your three  year old laptop. We want to apply you to kernels everywhere. Both on monolithic systems, because  there are a lot of them around, but you should trivially move to recompile your code and  run it and G node, run it on NUIN. And at the same time, no compromises. So you still  get strong isolation across the spectrum of targets. Thank you. Any questions?  Okay. So what you're saying is should we sandbox HVT even more? So one of the things that we  will do is extend the set comp sandbox to the HVT tender itself. So I think that's kind  of what you're touching on. Well, like providing a super minimal kernel that would run inside  HVT and expose the SVT system calls to your SVT. So you have like Solar 5 and up to Solar  5. No, we haven't thought about that. Yeah, it's not recursive. Sorry.  Yeah. Any other questions? Yes. Yes.  That was the real code you will get. So there's some. There's no reason you should get into  because there aren't any signals being handled, especially not in the set comp case because  you don't even have SIG return in the set comp filter. So if you get a signal, then you  can't handle it. Sorry, your process just dies. I haven't actually seen, so I haven't  really seen E-inter happen in any other case in practice. I mean, if you're not handling  signals, then why would, why should you get that? But a good point. Yeah. Any other questions?  Kind of an out there question. I would love to use this for sandboxing desktop graphical  applications with super minimal interface for UI stuff. Have you put, has there any thought  or? You're not the first person that's asked about. So can this be used for sandboxing desktop  applications with actual UI? The question is what would that interface be because then  you run into the problem of, well, if you want performance, you want at least some kind  of, at least a frame buffer, but then that's very low level. So it's like, how do you decide  that abstraction? I mean, how would this run across this interface? I mean, you could,  of course, certainly run like an X11 client inside this thing, right, and have that talking  to an X server somewhere else. So that would be one way of accomplishing that, but yeah,  I mean, possibly not the, not the cleanest approach. I'm not sure. I know that there  are people that did things with, I mean, even the, even the genoed folks have been doing  things with virtualizing graphics hardware, but it's really quite a different area which  I don't know that much about. But yeah, you could do the X11 thing or, yeah.  Okay, thank you.  Thank you.
Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_0_167.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_168_875.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_876_5255.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_5256_6083.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_6084_7835.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_7836_8783.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_8784_12239.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_12240_13535.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_13536_16643.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_16644_19331.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_19332_21743.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_21744_26723.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_26724_30455.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_30456_35483.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_35484_35855.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_35856_35903.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_35904_35963.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_35964_36575.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_36576_36635.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_36636_36695.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_36696_36911.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_36912_36971.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_36972_37379.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_37380_40319.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_40320_40403.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_40404_40535.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_40536_40583.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_40584_40691.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_40692_41267.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_41268_41699.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_41700_41987.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_41988_42263.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_42264_42347.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_42348_43475.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_43476_43523.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_43524_43643.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_43644_43691.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_43692_43907.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_43908_44231.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_44232_44411.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_44412_44543.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_44544_44675.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_44676_45479.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_45480_45719.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_45720_48695.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_48696_49319.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_49320_49895.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_49896_51575.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_51576_55535.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_55536_56999.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_57000_58283.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_58284_58355.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_58356_60563.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_60564_61199.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_61200_61595.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_61596_62255.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_62256_62975.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_62976_64259.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_64260_65027.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_65028_65627.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_65628_66467.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_66468_73427.png Solo5__A_sandboxed__re-targetable_execution_environment_for_unikernels_73428_73546.png