HelenOS in the Year of the Pig
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Jermář, Jakub
This is going to be an all-encompassing update talk for HelenOS developments that happened in the Year of the Pig (since the last FOSDEM).
 Okay, the next speaker is Jakob, who's going to give a short update on Hellenos. So welcome. It's good to see so many familiar faces. In this short update talk, I will just go through the last year in Hellenos, basically from the last FOSDEM to this date. At first, a short recapitulation of what Hellenos is. So it's a portable microkernel-based multiserver operating system. I think it's fair to say that it's also a distribution of itself. So unlike some of the L4 projects that are more like unassembled LEGO bricks, this is also made of LEGO bricks but assembled already. It's designed and implemented from scratch. It's not deliberately trying to mimic some other operating system or API, even though we do provide limited compatibility for six layer. It's written in C and to a small degree also in C++ and it's almost 100% our own code. And it's composed of very fine-grained components. I will talk about that a little bit later. So this is the main timeline of events that took place last year. So one of those big events happened in May. It was the first and also the only release last year, 0.9.1, code name Armonia. I was once asked by Lazar, I guess, why Armonia. The reason is that in that port in that release, we merged the ARM64 port made by Petra somewhere there. The author is present, I guess, in this room, where are you? And also some other ARM work, so that's why it is called Armonia. I also participated in the L4-Ree retreat, where I worked on making Hellenos work on the L4-Ree hypervisor on X86. In July, there was this Susseha Creek where some of our developers worked on implementing the standard C++ library future header. And in August, there was the biggest Hellenos event of the year, the Hellenos camp. This time it was only three people, so no children and no spouses were involved, so we call it quasi-camp. But it produced most of the topic for this talk, so it was important, definitely. So what things made it to the Hellenos master and other branches? So we already mentioned C++ future header hacked on during the Susseha Creek week that's already merged. Then there was a really big feature merged into master, which is the switch of the build system from the old make into or to the meson build system that finally gave us the capability to do proper out of three builds. We are still recovering from the effects of this switch, especially in our infrastructure and testing setups. That's also the reason, by the way, why there was only one release last year, because this due to a showstopper, a blocker, it prevented a second release. Besides of that, I worked on the IPC version 2 branch. I will talk about that in just a couple of seconds. Martin Diedzky worked on turning Hellenos into a kernel. And Irka Soboda, who was not at the camp, but at the same time until today, has been working on a replacement for our compositor, the display server. So this is, let's start talking about the IPC version 2 stuff. This is a recapitulation from my talk from two years ago. On this picture, you can see how Hellenos IPC works. So there are some tasks that are communicating with each other. The orange one, that's a client. It has some capability. So it has a capability to a kernel object called a phone. Phone is connected to another task's answer box. And the answer box, that's a multi-Q object, which contains Qs for receiving requests from other tasks and also replies from other tasks. And as you can see, we have three tasks communicating, which means that the blue one, which is the recipient of the orange tasks, requests can either directly reply to a request or it can forward it to a third task, as is the case now. And then when the green tasks on this picture answers the request, the answer goes to the answer box of the orange task from where the orange tasks can fetch the answer. What I wanted to mention on this picture is the fact that in this setup, in this version of Hellenos IPC, the message, which is called the IPC call, is always allocated by the kernel, is fixed size. There is a limit on the size of the buffer that can be attached to it and is just one, it can be used only once. So the kernel always allocates it anew, then it sends to all the other tasks, answer it, and then it's deallocated. So you can think it's a little bit wasteful. Then I mentioned the phones, which if you have capabilities that kind of point to some destination endpoint for communication can be understood as a little bit unnecessary, especially if you can have more of these objects that represent endpoints, which is the answer box in this case. Also one important limitation of the Hellenos IPC is the fact that it was possible to have only one of these answer boxes per task, which is something that we were trying to fix or to limit. So two years ago I raised some questions like where would the answer go if you had more answer boxes and also how do you wait for multiple requests and also replies to an IPC. And this is something that I tried to solve in IPC version two. It's centered around some new kernel objects, specifically an IPC buffer, which is preallocated by the user. So you get a capability to an arbitrary really large piece of memory that is reusable, so it's not thrown away after it's first used in the IPC. And in IPC you basically only pass this capability to this buffer. You do not kind of allocate anything else or you just pass the capability basically. Then there is the IPC endpoint, which is a replacement for answer box. You can have as many as you like of these. And the third kind of kernel objects is like a kept place, which is a container for capabilities. It's something like in mach, there is this thing called port sets. It basically allows you to have a capability to a kernel object into which you put stuff and then you can refer to this set of objects by one capability. So how does this work in practice? So the client allocates a buffer, the server allocates an endpoint, and then the client basically sends the buffer to the endpoint. The server then waits on the endpoint and supplies a capability which is not yet associated with any object. Then when the buffer comes, the kernel makes the association, the server can put its answer into the buffer and can finish the buffer by basically replying to it. And then the client can wait on the capability of the buffer. This also answers the question of to which answer box does the reply go, it doesn't go to any answer box. You can just wait on the handle of the buffer itself. I would like to emphasize that in this blueish area, no allocations take place. So everything is allocated in the yellow area and when the communication takes place, everything is already allocated. And then after having waited for the buffer, the client can basically make another request with the same buffer using the same capability and the whole thing can repeat over and over again. So this is the simple scenario. There is also more complicated scenario which involves the capability lists. So here both the client and the server allocate a capability list, the kept list. The server immediately puts both of its endpoints. Now it's using two endpoints into the capability list. The client now allocates two buffers and upon sending those buffers, it specifies the kept list here which results in the kernel adding the buffer into the capability list. Then the server waits on the capability list so it doesn't know which endpoint will receive data first. So it doesn't matter here, any endpoint will be okay. Then it finishes the received buffer and then the client can also wait on a capability list. It also doesn't know which answer will come first. And then everything is allocated and destroyed. So this was in short how the new IPC stuff is supposed to work. If you are interested, talk to me afterwards or there will be a link to a GitHub ripple. So something completely different which is the any kernel stuff. So are you familiar with what any kernel is or what any kernels are? Yeah, I take it as yes. So for instance, an NBSD is a classic example of any kernel. It's a way of structuring operating systems in a way that they can provide components that can either form a monolithic kernel or components for micro kernels or unicarnels. So we also tried, Markin tried something like that with Hellenos. This time the benefit is that Hellenos is already kind of sliced into very fine grade components so you don't have to carve them out of a monolith which is the case with NBSD and you just make some adaptation to the linking process. Maybe it's slightly more complicated than that but you can imagine it has to do something with linking and you can then do things like transition from this thing which is the normal micro kernel stuff. You have both the client and the server as independent tasks. Here you have kernel and just by reorganizing stuff and making use of this component you can basically put it in the kernel and voila you have AVFS server which runs inside of the kernel. This is also a work in progress so if you're interested talk to Markin afterwards. And last but not least is the display server thingy. So right now we have a compositor in Hellenos which provides a GUI with windows, some nice visual features like transparency and rotation. But there are also some aspects of this compositor that are being now perceived as drawbacks or maybe not ideal things. So Yerka Sobora is working on a replacement that should eventually replace the compositor and the console server. Should also be possible with this new stuff to have something like a GUI in a text environment or in a pseudo graphic environment and it should be basically still the same thing. It shouldn't matter whether it's graphical or textual. I personally feel that this topic is too deep for me to talk about in this very short time so if you're interested I would like to refer you to this ticket 816 or Yerka's GitHub ticket which is the, well this is the ticket and this is his repository so if you're interested in that go there. And I think I'm just about to run out of time so thank you very much for your attention.
 Okay, the next speaker is Jakob, who's going to give a short update on Hellenos.  So welcome.  It's good to see so many familiar faces.  In this short update talk, I will just go through the last year in Hellenos, basically  from the last FOSDEM to this date.  At first, a short recapitulation of what Hellenos is.  So it's a portable microkernel-based multiserver operating system.  I think it's fair to say that it's also a distribution of itself.  So unlike some of the L4 projects that are more like unassembled LEGO bricks, this is  also made of LEGO bricks but assembled already.  It's designed and implemented from scratch.  It's not deliberately trying to mimic some other operating system or API, even though  we do provide limited compatibility for six layer.  It's written in C and to a small degree also in C++ and it's almost 100%  our own code.  And it's composed of very fine-grained components.  I will talk about that a little bit later.  So this is the main timeline of events that took place last year.  So one of those big events happened in May.  It was the first and also the only release last year, 0.9.1, code name Armonia.  I was once asked by Lazar, I guess, why Armonia.  The reason is that in that port in that release, we merged the ARM64 port made by Petra somewhere  there.  The author is present, I guess, in this room, where are you?  And also some other ARM work, so that's why it is called Armonia.  I also participated in the L4-Ree retreat, where I worked on making Hellenos work on  the L4-Ree hypervisor on X86.  In July, there was this Susseha Creek where some of our developers worked on implementing  the standard C++ library future header.  And in August, there was the biggest Hellenos event of the year, the Hellenos camp.  This time it was only three people, so no children and no spouses were involved, so  we call it quasi-camp.  But it produced most of the topic for this talk, so it was important, definitely.  So what things made it to the Hellenos master and other branches?  So we already mentioned C++ future header hacked on during the Susseha Creek week that's  already merged.  Then there was a really big feature merged into master, which is the switch of the build  system from the old make into or to the meson build system that finally gave us the capability  to do proper out of three builds.  We are still recovering from the effects of this switch, especially in our infrastructure  and testing setups.  That's also the reason, by the way, why there was only one release last year, because this  due to a showstopper, a blocker, it prevented a second release.  Besides of that, I worked on the IPC version 2 branch.  I will talk about that in just a couple of seconds.  Martin Diedzky worked on turning Hellenos into a kernel.  And Irka Soboda, who was not at the camp, but at the same time until today, has been  working on a replacement for our compositor, the display server.  So this is, let's start talking about the IPC version 2 stuff.  This is a recapitulation from my talk from two years ago.  On this picture, you can see how Hellenos IPC works.  So there are some tasks that are communicating with each other.  The orange one, that's a client.  It has some capability.  So it has a capability to a kernel object called a phone.  Phone is connected to another task's answer box.  And the answer box, that's a multi-Q object, which contains Qs for receiving requests from  other tasks and also replies from other tasks.  And as you can see, we have three tasks communicating, which means that the blue one, which is the  recipient of the orange tasks, requests can either directly reply to a request or it can  forward it to a third task, as is the case now.  And then when the green tasks on this picture answers the request, the answer goes to the  answer box of the orange task from where the orange tasks can fetch the answer.  What I wanted to mention on this picture is the fact that in this setup, in this version  of Hellenos IPC, the message, which is called the IPC call, is always allocated by the kernel,  is fixed size.  There is a limit on the size of the buffer that can be attached to it and is just one,  it can be used only once.  So the kernel always allocates it anew, then it sends to all the other tasks, answer it,  and then it's deallocated.  So you can think it's a little bit wasteful.  Then I mentioned the phones, which if you have capabilities that kind of point to some  destination endpoint for communication can be understood as a little bit unnecessary,  especially if you can have more of these objects that represent endpoints, which is the answer  box in this case.  Also one important limitation of the Hellenos IPC is the fact that it was possible to have  only one of these answer boxes per task, which is something that we were trying to fix or  to limit.  So two years ago I raised some questions like where would the answer go if you had more  answer boxes and also how do you wait for multiple requests and also replies to an IPC.  And this is something that I tried to solve in IPC version two.  It's centered around some new kernel objects, specifically an IPC buffer, which is preallocated  by the user.  So you get a capability to an arbitrary really large piece of memory that is reusable, so  it's not thrown away after it's first used in the IPC.  And in IPC you basically only pass this capability to this buffer.  You do not kind of allocate anything else or you just pass the capability basically.  Then there is the IPC endpoint, which is a replacement for answer box.  You can have as many as you like of these.  And the third kind of kernel objects is like a kept place, which is a container for capabilities.  It's something like in mach, there is this thing called port sets.  It basically allows you to have a capability to a kernel object into which you put stuff  and then you can refer to this set of objects by one capability.  So how does this work in practice?  So the client allocates a buffer, the server allocates an endpoint, and then the client  basically sends the buffer to the endpoint.  The server then waits on the endpoint and supplies a capability which is not yet associated  with any object.  Then when the buffer comes, the kernel makes the association, the server can put its answer  into the buffer and can finish the buffer by basically replying to it.  And then the client can wait on the capability of the buffer.  This also answers the question of to which answer box does the reply go, it doesn't go  to any answer box.  You can just wait on the handle of the buffer itself.  I would like to emphasize that in this blueish area, no allocations take place.  So everything is allocated in the yellow area and when the communication takes place, everything  is already allocated.  And then after having waited for the buffer, the client can basically make another request  with the same buffer using the same capability and the whole thing can repeat over and over  again.  So this is the simple scenario.  There is also more complicated scenario which involves the capability lists.  So here both the client and the server allocate a capability list, the kept list.  The server immediately puts both of its endpoints.  Now it's using two endpoints into the capability list.  The client now allocates two buffers and upon sending those buffers, it specifies the  kept list here which results in the kernel adding the buffer into the capability list.  Then the server waits on the capability list so it doesn't know which endpoint will receive  data first.  So it doesn't matter here, any endpoint will be okay.  Then it finishes the received buffer and then the client can also wait on a capability list.  It also doesn't know which answer will come first.  And then everything is allocated and destroyed.  So this was in short how the new IPC stuff is supposed to work.  If you are interested, talk to me afterwards or there will be a link to a GitHub ripple.  So something completely different which is the any kernel stuff.  So are you familiar with what any kernel is or what any kernels are?  Yeah, I take it as yes.  So for instance, an NBSD is a classic example of any kernel.  It's a way of structuring operating systems in a way that they can provide components  that can either form a monolithic kernel or components for micro kernels or unicarnels.  So we also tried, Markin tried something like that with Hellenos.  This time the benefit is that Hellenos is already kind of sliced into very fine grade  components so you don't have to carve them out of a monolith which is the case with NBSD  and you just make some adaptation to the linking process.  Maybe it's slightly more complicated than that but you can imagine it has to do something  with linking and you can then do things like transition from this thing which is the normal  micro kernel stuff.  You have both the client and the server as independent tasks.  Here you have kernel and just by reorganizing stuff and making use of this component you  can basically put it in the kernel and voila you have AVFS server which runs inside of  the kernel.  This is also a work in progress so if you're interested talk to Markin afterwards.  And last but not least is the display server thingy.  So right now we have a compositor in Hellenos which provides a GUI with windows, some nice  visual features like transparency and rotation.  But there are also some aspects of this compositor that are being now perceived as drawbacks  or maybe not ideal things.  So Yerka Sobora is working on a replacement that should eventually replace the compositor  and the console server.  Should also be possible with this new stuff to have something like a GUI in a text environment  or in a pseudo graphic environment and it should be basically still the same thing.  It shouldn't matter whether it's graphical or textual.  I personally feel that this topic is too deep for me to talk about in this very short time  so if you're interested I would like to refer you to this ticket 816 or Yerka's GitHub ticket  which is the, well this is the ticket and this is his repository so if you're interested  in that go there.  And I think I'm just about to run out of time so thank you very much for your attention.
uk_helenos_0_155.png uk_helenos_156_203.png uk_helenos_204_551.png uk_helenos_552_2303.png uk_helenos_2304_4691.png uk_helenos_4692_6851.png uk_helenos_6852_9071.png uk_helenos_9072_11003.png uk_helenos_11004_12779.png uk_helenos_12780_14939.png uk_helenos_14940_16823.png uk_helenos_16824_18407.png uk_helenos_18408_18875.png uk_helenos_18876_19235.png uk_helenos_19236_20999.png uk_helenos_21000_21347.png uk_helenos_21348_22151.png uk_helenos_22152_22269.png