Optimal buffer allocation on Wayland
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Ser, Simon
Wayland compositors try to make of most of the hardware by displaying directly without any copy pixel buffers coming from clients such as games, browsers and video players. This lowers battery usage, improves latency and leaves the 3D engine free for clients to use. The display engines found in modern GPUs often can support this zero-copy mechanism only if the buffers have been allocated in a special fashion. However, buffers allocated this way won't be optimal for rendering, and only a handful of buffers can be directly displayed. As a result, a trade-off between zero-copy display and optimal rendering needs to be made. The compositor is the natural place where a decision can be made, because it has a global view of all apps which need to be presented. Once the compositor has taken a decision, it needs to be communicated to the clients. The brand new linux-dmabuf feedback protocol enables this negotiation between the compositor and the clients.
 Hi, I'm Simon Serre, also known as Immersion. I'm working for SOSHOT, and today I'm going to talk about Optimal Buffer Allocation and Wayland. We've been working with other Wayland developers on a new extension code, Linux EMA Buff, and I'm going to explain why we need it, how it works, and our plans for the future. So let's get started. Why do we need this new shiny extension at all? So I'm going to explain a few details about the Linux and in particular Wayland graphics stacks. I'm going to explain also to present a bunch of examples, cases we need to handle, but quite a few. So let's get started with the basics about Wayland architecture. In Wayland, there are some clients which talk to compositors. So an example of a client can be a web browser, a text editor, a terminal emulator, a video player game. All of these are clients, and they are talking to compositors such as Matter for Oknoomshel, Queen for KDE, Sway, and Western and few others exist also. So what clients do is basically they allocate a buffer and the GPU, then they do their own rendering so functions again can render a scene with a character maybe. When the rendering is done, they hand off the buffer to the compositor and say, hey, please, can you display this buffer for me? So the compositor receives multiple buffers from each client, and then it decides which to show on screen. It needs to paint multiple buffers into one final buffer. This final buffer will be hand over to the kernel, and the kernel driver will program the hardware to display it on a screen. So compositors usually have an optimization called direct scan out. Maybe what happens without direct scan out is, as I said earlier, clients do their own rendering, then they pass the buffer to the compositor. The compositor does the composition step. It's a copy operation. Copy the client buffers into a final buffer, and then hand over the final buffer to the kernel to display it. But in case there's only one buffer to display on screen, for instance, if you're running with a full-screen video player, then the compositor can skip completely the composition step. So it skips a copy. So that's the red arrow here. So it's an optional optimization. It's pretty good because it allows the compositor to lower the battery usage. So if you're on a laptop, it's pretty important. It improves latency, and it leaves some parts of the GPU free for other clients to use. So if a game is keeping the GPU busy, the compositor won't disturb the game rendering, for instance. So apparently most compositors have this optimization implemented for full-screen windows. But a problem with this all looks great and all, but sometimes this direct scan out path doesn't work. So to explain why sometimes it works and sometimes not, I need to go into a bit of a bit to explain tile buffers. In general, when you need to store an image in memory, the straightforward thing to do is to take the first row of the image, put it in a buffer, then take the second row of pixels of the image, append it to the buffer, and so on and so forth till the last row. So that's what we call a linear layout. But there are other ways to represent an image in memory. For instance, it's possible to divide an image into multiple tiles. So here, for instance, the image is divided into nine tiles. And the idea is to, instead of storing the image row by row, it's stored tile by tile. So here the first tile is stored in memory first, and then the second tile is impended to the buffer, and so on and so forth till the last tile. So this is important because it allows the GPU to access the image in a manner which is more convenient and optimized in regards to the cache, for instance. So it's an important optimization for the GPU. So because this has been introduced by hardware designers, of course, each vendor, Intel, AMD, and NVIDIA, each of these vendors have come up with their own way to divide an image into tiles. Oftentimes, they are more complicated. This is the basic idea, but there are more complicated schemes which I use, more complicated tiling, compression, and so on, features like this. And each vendor has multiple ways to represent an image in memory. And of course, each way to describe an image in memory from a specific vendor is not compatible with other vendors. So we end up with a lot of different possible layouts. In DRM, we call these layouts modifiers. We describe the layout with something called a modifier. So we end up in a situation where some parts of the GPU are able to handle some layouts, some modifiers, but some other parts of the GPU are not necessarily able to support all of the possible layouts. So GPU has multiple parts in it. The most commonly understood part, possibly, is 3D Engines. It's usually responsible for running shaders, for instance. So 3D Engines, in general, can deal with a lot of tiling layouts, a lot of modifiers. It generally prefers quite advanced tiling layouts. The display engine and the other end usually doesn't support as many modifiers as the 3D Engines. And the video engine also has a different set of modifiers. So I gave a few examples here on the screen. We are only scratching the surface here, and the hardware constraints when it comes to modifiers are very complicated. For instance, if the screen is rotated, maybe the GPU will not be able to display linear images at all. It will only be able to handle tiled images. So we're starting to see that there needs to be some kind of negotiation. There needs to be someone who sits down and says, hey, I want to use to write an image with a 3D engine and display it with a display engine on the screen. So I need to pick a modifier which works with both of these parts. And I want to pick the best one. I don't really want to pick linear because it's not very optimized for the 3D engine to write into. With that in mind, let's move on to multi-GPU. So multi-GPU is basically when there are multiple GPUs connected to a single machine. So it can happen when, for instance, you have a box with two PC express ports and one AMD GPU and one NVIDIA GPU all connected to the same box. Because there are also some laptops which have an integrated GPU, an Intel integrated GPU, for instance, and a discrete GPU and the AMD, for instance. So here, for instance, let's suppose GPU 1 is an Intel integrated GPU with GPU 2, a discrete GPU on a laptop and battery. So when I'm battery, I don't really want to power on the second GPU, the discrete GPU, because that uses a lot of power. I want to save power. So the compositor decides to use the integrated GPU. And the compositor wants to tell clients to also use the same GPU to save power. So when it's cooked to be some way for the compositor to say to clients, hey, please use this GPU I'm already using so that we don't just randomly power on the second one. And clients need to know which GPU to pick. Now let's say that I plugged my laptop in power. So I don't want to save battery anymore. And I want instead to play game. So maybe the game is a bit demanding. And it would prefer to use the second GPU, the discrete GPU, because it's more powerful. So here the client locates a buffer on the discrete GPU, then does its rendering, then hands over the buffer to the compositor. On that done, the compositor needs to access, read the buffer on the discrete GPU, and render it with the composited with the first GPU. Because the screen is connected to the integrated GPU, the laptop screen. So there's already some kind of multi-GPU buffer sharing going on here. There are also some cases of external GPUs which can be plugged and unplugged, for instance, connected via USB-C. And that further complicates the matter, because the discrete GPU can come and go as a user plugs and plugs. So that's starting to be a bit fun. And there are even more complicated scenarios. For instance, if I have an external GPU and I start to plug a screen and the external GPU. And the compositor may want to also open both GPUs and take the client buffer composited with the second GPU and then display it and then that puts it connected to the second GPU. There are also other kinds of systems which makes things a bit hairy. For instance, arm boards for embedded use cases. I don't know if you've probably heard about Raspberry Pi, for instance. This is an example of an arm board. This is the arm board manufacturers want to have a bit of a fun evening. And they decide to pick parts from multiple manufacturers and glue them together. So for instance, one could have a display engine made by a whole chip or a winner or a M logic and glue it together with a render engine made by arm, a Mali engine or something else. So even though this is a single stock from user space, this looks like a multi GPU situation with two devices. The first device can only display images and cannot use OpenGL at all. The second device can only do OpenGL and has no outputs, has no notion of outputs. Screens connected. So yeah, this can be a bit of a problem. So with all of this in mind, our goal while designing the new Waylon protocol extension was to be able to tell the client which optimal device and format and modifier to use. But it's not that simple because clients may still have their own constraints and preferences. For instance, a video player may be wants to take advantage of hardware acceleration and wants to use video engine. And the video engine comes with some constraints and preferences when it comes to modifiers and devices and formats. So we need to reconcile both the compositor and the client and somehow pick the best combination which is still supported by everything both the compositor and the client wants to use. One thing which makes everything a bit more complicated is that things are dynamic. As I said, for instance, external GPU can come and go. And the number of that windows can move between screens. So a window could move from a screen which is plugged in GPU A into a screen which is plugged in GPU B. And windows can be on the screen like floating in which case direct scan out is not possible for most compositors today. And when the window becomes full screen, dynamic scan out becomes possible. And to take advantage of direct scan out, the client needs to maybe change its device format and modify it. Something to keep in mind is that some clients can't have all of the fancy stuff. Some clients are basic, have some other constraints. And maybe they pick the best device, format and modifier to use at start up and then everything is frozen. They can't change anymore. Finally, some compositors, the old compositors, always need a full back plan. So it's not possible for a compositor to say, okay, I'll always be able to use direct scan out. Sometimes it won't work and modifiers are not, can express some of the stuff which makes direct scan out possible, but it's not enough. Never guarantee that the optimized cut path will always work. So compositors need always to have a way to fall back to composition. So boring, copy and everything that always needs to be read through. Compositors can just take client buffers and do nothing with them. It results in a black screen for users and that's not acceptable. All right, so let's see how the protocol looks like. We've introduced a few concepts. So we've introduced a concept of main device. The main device is basically a good device, good GPU for clients to use if they don't have any other preferences. We've also introduced a concept of preference range. So a preference range is a set of device formats and modifiers which indicates some supported combinations and each set is ordered by preference. So the first set will be the one which will bring supposedly the most optimized way to present an image. The second one can be used as a fallback if the first one doesn't work and so on and so forth. And the last one, the compositor really needs to make sure that the last bench is always usable or else maybe the client won't be able to display an image at all. So the first thing a client will do is obtain a feedback object. There are two ways to obtain a feedback object. The first way is to get the default static feedback object. It's like a global feedback object and tied to anything. And the second way is to obtain a feedback object from a surface. A surface is like a window and way to simplify. So the feedback object will be tied to this surface and as the surface changes its position, for instance, the feedback object will receive some updates, some new events and the client will be able to react accordingly if it wants to. Once the client has obtained a feedback object, it will receive feedback events. So here's an example of typical feedback events. It's a compositor. The compositor first sends the main device event. Here it says if you don't have any other preference, it's a good idea to use GP1. And the compositor will send the trenches. So here we have two trenches. The first trench says this trench is about GPU2. If possible, use format ARGB8888 with the modifier X-style or linear. If you're using formats and modifiers from this trench, I'll be trying to use direct scanout. And that's it. If it's not possible, the client can use the other trench. The second trench says this trench is about the third GPU, GP1. If you're using this GPU, you'll be able to allocate a buffer with the format ARGB8888 with modifier Y-style or X-style or linear. And I won't be attempting to use direct scanout if you're using this trench. And that's pretty much it. So let's see how clients are able to handle these events. There are multiple types of clients. Let's start with the basic baseline support the clients could have, most minimal stuff. Such a client would use the get default feedback request to obtain a global static feedback object which won't really change. And this client will open the device indicated by the main device event. And for each trench which is about this main device, the client will try to allocate a buffer with format and modifier specified in the trench. So here, the client will receive main device GP1. He will see a trench about GPU2. It will ignore it completely and it will directly have a look at the second trench which is about the main device. And it will allocate a buffer with format specified here. Let's continue with a bit more sophisticated client. There are other types of clients, for instance, clients using Vulkan. These clients already have specific GPU selected, pre-selected. For instance, games often have a menu with checkbox to select which GPU to use. So these clients will call the get default device, the feedback to get a static global feedback and they will completely ignore main device because they already have selected one. And instead of, I'll go back to the example. Let's say the client has selected GPU2 at the start-up and we will only look at trenches about this pre-selected GPU. So here, the client will only take it into account the first trench and completely ignore the second trench and it will do the same as the first client by one of these formats. One thing to note is that maybe the buffer input to the Compostor will fail because in multi-GPU situations, the client selects device which is different from the GPUs used by the Compositor. Maybe the buffers won't be able to be accessed from the other GPU. So clients need to keep this in mind and handle this gracefully. Having a client even more complicated will be able to dynamically change its buffer format and modifier. So such a client will on the request get the feedback object tied to a specific surface. So as a client and start-up, it will look at all the events as usual. But the feedback object will send new events when the window is moved, for instance. So clients will listen for these events and we allocate its buffer as needed. So just works the same, but clients is able to dynamically change its... we allocate its buffer and there's the kind of mix and match where some of the even more complicated clients will be able to dynamically change the GPU they're using. So that user gets the best feedback, selects the main device and then listens for events and if the main device changes, they can close the old device, open the new device and then we allocate all of the buffers. And as before, buffer imports the Compositor can pay. So what's the current status of the protocol and what are the next steps? Implementation wise, we're pretty good, at least on the compositor side. All of Mutter, Wroots, Weston and Queen have the protocol implemented. So yeah, it's already possible to use it to the at least in the Git version of these compositors. In terms of clients, we only have a single implementation right now that I know of in Mesa, a GL subsystem, so this means Wayland native open GL clients are able to benefit from all of this stuff and they're able to dynamically change the modifier depending on whether the window is full screen or not and stuff like this. So that's pretty cool. There are still some things to be done, for instance, we're still missing Vulkan support. There's a mesh request open for this, they still need to review. We maybe something worth doing would be to add support for XYLand. There's some early patches for this, but yeah, we'll see what goes because we need to plan this down to the XYLand protocol also. So yeah, we see how it goes. And also other clients could benefit from it, like maybe VIP clients, G-streamer, Codie, stuff like this. Then also which could be improved is multi-GPU support in existing compositors. Right now, the compositors, all compositors that I know of have basic support for multi-GPU, but they always do the composition step with a single fixed GPU that cannot change at all. And they don't support direct scan out and secondary GPUs. So that would be something to improve. And they don't support either multi-GPU composition. So if a game is running in discrete GPU and then is displaying a window which is shown and a screen plugged into the discrete GPU, the same GPU, the compositor will still import the game buffer into the integrated GPU and then export it again to the discrete GPU. So just for the composition step, so that's not great. Also as I said earlier, it's not possible to express all those hardware constraints with modifiers. So we're still missing some work here that could be done to optimize even further buffer allocations. So we've talked about this with James Jones from NVIDIA and we have some ideas to improve this but see my existing 2020 talk about this for more details. So that's what's in my chat. Here are a few references you could look into if you're interested and feel free to discuss as question and weigh-in in an IRC. I'm going to do a small demo of all of this. So Western is shipping with a demo called Western Simple DMEber Feedback. It will basically act as the third kind of clients we've seen which dynamically changes its format and modifiers. So I'm going to run a window to open a window. Then full screen it. We're going to see what kind of events we see from the compositor. Window is opened, I full screen it and it's closed. So what happened? Here we have a log of all of the events that happened. So the client first received some events with the main device set to this device. So the first render. I only have a single GPU so things are a bit simple. And the compositor says I cannot do direct scan out. This is because at first the window was not full screen so that's expected. So with this first bunch of events, there's only a single branch. And that's it. And once a full screen the window, the compositor sends another burst of events. And you can see already that the first branch is different and has a scan out bit set to true true. That means the compositor will attempt to use direct scan out if the client uses when the format modifiers from this branch. Oh, I missed something here. Oh yeah, here. And we can see that in this burst of events there are two trenches and the second one, so the first one is to use direct scan out and the second one is to not use direct scan out. So in case one of the buffers above in the first range wasn't usable for the client, the client can always go back to the composition branch. And that's pretty much it. So if you have any questions. Now is a good time, I believe, to ask them. And thanks for listening. Hello, so we have Simon with us here live to answer all your questions. First off, thanks for your presentation and for all your work on the Weyland ecosystem. It's greatly appreciated. And let's start with a question for me. So do you have any updates on the presentation? Has anything changed or something else come to your mind that you haven't touched on in your presentation during the week because this was recorded over a week ago? You should probably unmute yourself. You should unmute yourself because you're still muted in Jitsi. Oh, sorry about that. Okay. All good now. Yeah, so yeah, thank you. For more context, yeah, I've been doing this presentation in a bit of a hurry. So I may have missed something and I wanted to do a bit more, but oh well. Maybe something I didn't mention quite explicitly is when I was talking about different parts of the GPU and I was planning that there is a 3D engine, display engine and video engine. So when I first got into graphics, I didn't really understand that the GPU was split like this. So the 3D engine is responsible for drawing the frames, for running the shaders. And the display engine is where you plug your connectors, HDMI and display port and all of this stuff. And the video engine is for video decoding, hardware accelerated video decoding. And yeah, it's important to understand that these parts of the GPU are not just one chip, they are multiple components driving these parts in the kernel as separate. So yeah, that's something I didn't quite mention explicitly. In terms of updates, Mesaveulcan is still installed because it needs a review time and review time is very valuable. X-YLAND has seen some activity in the last few days. Someone from NVDA has been working on it, so that's pretty cool. As I explained, we need to update the X11 protocol as well. We should be good, needs a bit more time. Yeah, I think that's pretty much it. I have some more material for the NVQ&A, but maybe we can start with questions now. Okay, yeah, sure. So we can wait with the material a little bit later. And the first question from the community is from Anunes. And the question is, what is the status of the clients using toolkits like GTK, QT, do they already benefit from this work through just Mesa EGL? Yeah, yeah, the good news is that as a client developer, if you're just using EGL, you don't need to do anything to benefit from all this work. The implementation in Mesa will allow most applications to just work out of the box. And I've been hearing reports that applications like MPV are just working fine and benefiting from this already. So you can check yourself by setting the Wayland display equals one environment variable before starting an EGL app, for instance, and check all the messages related to a DM above. And maybe you can have a look and check that it's used. But yeah, it should just work. There's an extension to the same question by the same author. So how about some more complicated applications like Firefox? Oh, so yeah, Firefox is a bit of a more involved case because I think it just uses low-level EGL. It doesn't really use the rule. So I think it manages the buffers itself with fine buffer objects. I don't remember exactly. Yeah, we need to investigate a bit more. I don't know, a friend. I know someone who maybe knows better. So I can poke him later. Another question is from the underscore ED. The clean needs to be aware of the GPU changes. How abstract that this is in Mesa? Yeah. So GPU changes are more involved because by default, most clients will just select a GPU and start up and then just use that. And so EGL and VLCAD clients won't work for dynamic GPU changes. So if a client wants to support dynamic GPU changes, it needs to listen to the Wayland events directly. And it can do that. So both Mesa, EGL, VLCAD and the client itself can listen to the changes in power advancement problem. So the client will need to tear down the old graphics context and recreate it with a new device manually to implement this thing. Okay. Thanks for the answer. And another question from Ismail. How would this play with multiple scannered buffers? Some cards display engines. In theory, can display multiple buffers on screen at once, effectively doing some composition themselves? Yep. So this should be, so most of the support for KMS planes, so deploying multiple buffers at once with direct scan out. Most compositor don't support it apart from full screen. So the only exception right now is Western. Western can do this. And with the extension, this new Wayland extension, it should just work. Western will send feedback to the surfaces, to individual surfaces accordingly. So extension doesn't need to be changed to accommodate for this new use case. But compositors need to be worked on so that this can be made to work more on motor, on KDE and WL routes. And this is something I plan to work on, at least for the WL route sites with things like LibLiftF that have been working for a few years now on. But we'll see. So so far that was the last question I have over here. And thank you for all your answers. And you've said that you have something extra for us. Yeah, yeah. So there are a lot of things I left out in the talk. Let's see. So one of the things I left out is that we designed a bunch of more technical stuff around the protocol. Up to now, the protocol sends the list of available formats and modifiers via directly on the wire and the unique sockets. It just sends the whole list of formats and modifiers. But it turned out with some AMD GPUs, for instance, there, or NVDA also, there are a lot, a lot of modifiers. So the list was getting pretty large to like multiple kilobytes, like, I don't know, 10 kilobytes or something. And for a single way on the message, this is too big. So we change that. And now the compositor will write to a shared memory file, the list of formats and modifiers, and we'll share this file in written remote to all clients, just like it does for things like the keyboard, the key map. So there should be also a performance improvement when you launch a client because the compositor won't be broadcasting all of the lists all the time. So that's when improvement. Maybe you've seen from the talk that we try to handle as many use cases as possible, and there are many interwind use cases. And it's really been a challenge to come up with a design which work well with all of these use cases. And something I haven't mentioned is to adjust another layer of use case is that some we also wanted to support drivers, which are legacy and don't support modifiers. And this was a bit of a pain because there are many problems with that. In particular, when it comes to multi GPU buffer sharing, if you don't have the format modifier, which indicates really how the pixels are laid out in memory, then the driver will just choose one implicit format modifier. And you don't know about this format modifier. And if you share this image between two GPUs, maybe the implicit format modifier used by one GPU will be different than the one used for the other GPU, and you will end up with a garbled image because both GPUs don't agree on the tiling to use. So yeah. We support these old drivers, mainly old AMD cards, but need to be a bit careful about multi GPU situations mainly. And yeah, to add on a bit on the next steps, also something to be worked on would be a better explicit synchronization. Right now there's a Waylon extension for explicit synchronization, but it's using sync files and it's exchanging file descriptors. Each time a frame is displayed and screened. So that's not really efficient. So we'd want to move on to something like DM sync ops, and that's more work. Yeah, a lot to be done yet. That's about it for me, I think. Yeah, so thanks for the extra content, and we have two more questions. So is this multi GPU configuration also applicable to virtual GPUs, like provided by SRIOV? I don't know a lot about virtual GPUs, so I'm not sure. I really need more context to be able to tell, so we can discuss afterwards if you're interested. Okay, and another question, not directly related to buffer feedback, but what happens when you run a full screen client with direct scan out and then notification from a different client pops up? So, yeah, planes. So if you're lucky or if you have the new Waylon extension, when there's a full screen client, it uses direct scan out. If a notification shows up and you're running not a subnome, KDE or sway, then the compositor will fall back to regular compositions, will fall back to open GL for compositing. And we'll stop doing direct scan out because it won't be able to display both buffers at once, but hopefully in the future, we'll be able to support direct scan out with multiple buffers at the same time with multiple KMS planes. But as I've said earlier, it's something that we need to implement in compositors and the extension already works for this use case. And if you're using Western, it should already work as is. And Western should be able to display both the Firefox and the notification on top with direct scan out. Okay, thanks for the answer and the last question seems to be, it's prefaced by clarification that it's a very new question. And you said Linux DMA buffers implementation of MSI EGL. And the question is, are MSI EGL and EGL streams the NVIDIA one? The same thing. So one thing to be noted here first is that EGL stream is an EGL standard. It's an EGL extension. And the NVIDIA implementation of EGL is not called EGL stream. It's just called the NVIDIA proprietary driver. So I guess the question is really, is this extension implemented in, is this new extension implemented in the NVIDIA proprietary driver? I don't think it is right now. But I don't know. Yeah, I don't know. You need to ask NVIDIA engineers about that one. Or maybe wait a few weeks and see if it comes up. OK, I didn't see any more questions for now about the two people typing. So yeah, wait for that. And yeah, thank you again for an awesome talk and awesome Q&A, extra content and all that you have done for the community. Because like your work on Wayland is just outstanding. Yeah, thanks. Glad you liked it. It's a lot of work. But also it's very time consuming and energy consuming to work on Windows system integration in particular, because you need to make people agree and there's a lot of politics in it. And not a lot of people have time to work on it. So with all of this, it's quite hard to work on this stuff. So we have a follow up for one of the previous questions. I assume the latency for the direct scan out is lower. So there is a glitch when switching back and forth. So does it happen? Is it visible or noticeable when you switch from direct scan out to not direct scan out when something pops up? So everything is implemented correctly. It shouldn't be visible. So there is a difference in latency in that with direct scan outs, when so the compositor basically needs to do a frame every 16 milliseconds or so. So with our direct scan outs, the application has less time to render because the compositor will need to also have a render step. But with direct scan outs, we skip this step and the application has more time to render. So it kind of depends on the situation. If the application already isn't taking a lot of time to render, it should be fine. If the application is like right under the limit is like taking maybe I don't know exactly 16 milliseconds to render, then you'll see the application will be fluid if it's using direct scan out and then it stops using direct scan out. Maybe it will become a bit choppy. But I don't think we can do a lot about this. It just if we can't use direct scan out, we just can't use the optimised file. And yeah, there's no saving grace here. So maybe in some situations, we are a bit of a glitch, but this extension should just try to make things as not choppy as possible most of the time. Okay, it looks like there is just one last question being typed out right now in the chat, I assume that's the question because someone is typing. Yeah, those were really good answers and good questions. And it feels like your presentation was definitely one of the most popular ones here. So that's cool. And now two people are typing. So that's going to be interesting. Yeah, while they are typing, maybe I can give a few words to thank Colaborah. Colaborah has been helping a lot on this. Because Leandro and Daniel Stone, they've both been providing some Western implementation, providing a MESA implementation. So they've been pivotal in making this happen. So really thank you for this. Okay, the last thing actually wasn't a question, but thank you for the answer because the person who asked about EGL just read, seen the word EGL used when talking about Nvidia Preparator driver and MESA is also using EGL. So it was confusing. Cool. Okay, we have still five minutes and there are still people typing. So any more shout outs? Oh, also we have Luke now here as well. Thank you, Eric, for organizing all this, therefore, I guess. Maybe now is a good time to say it. Well, thank you for speaking and thank you to all the speakers today as well. And most of the time, well, thank you for Eric and Martin for being the other different organizers, but most of the time should go to the FOSDM developers, the FOSDM organizers who spent the last months in a mad dash as they usually do. And they will probably go in hibernation for half a year again as they do every year. And then somewhere in August, they will reemerge from their sleep and they will start working on hopefully the next real life FOSDM that we don't have to do a virtual one again. So yeah, everybody get vaccinated. It's the best hope we have to have a proper real life FOSDM again next year. And thanks for watching from my side at least. Thank you. Okay, so that was the last question. The only other things that were typed in the chat since then were thank you for your presentation and that's it. So I guess we can wrap it here. Thanks to all the presenters. Thanks to all the attendees and hopefully see you next year in the Meet Space. Yeah, sure. All right, all right. See you. Okay.
 Hi, I'm Simon Serre, also known as Immersion.  I'm working for SOSHOT, and today I'm going to talk about Optimal Buffer Allocation and  Wayland.  We've been working with other Wayland developers on a new extension code, Linux EMA Buff,  and I'm going to explain why we need it, how it works, and our plans for the future.  So let's get started.  Why do we need this new shiny extension at all?  So I'm going to explain a few details about the Linux and in particular Wayland graphics  stacks.  I'm going to explain also to present a bunch of examples, cases we need to handle, but  quite a few.  So let's get started with the basics about Wayland architecture.  In Wayland, there are some clients which talk to compositors.  So an example of a client can be a web browser, a text editor, a terminal emulator, a video  player game.  All of these are clients, and they are talking to compositors such as Matter for Oknoomshel,  Queen for KDE, Sway, and Western and few others exist also.  So what clients do is basically they allocate a buffer and the GPU, then they do their own  rendering so functions again can render a scene with a character maybe.  When the rendering is done, they hand off the buffer to the compositor and say, hey,  please, can you display this buffer for me?  So the compositor receives multiple buffers from each client, and then it decides which  to show on screen.  It needs to paint multiple buffers into one final buffer.  This final buffer will be hand over to the kernel, and the kernel driver will program  the hardware to display it on a screen.  So compositors usually have an optimization called direct scan out.  Maybe what happens without direct scan out is, as I said earlier, clients do their own  rendering, then they pass the buffer to the compositor.  The compositor does the composition step.  It's a copy operation.  Copy the client buffers into a final buffer, and then hand over the final buffer to the  kernel to display it.  But in case there's only one buffer to display on screen, for instance, if you're running  with a full-screen video player, then the compositor can skip completely the composition  step.  So it skips a copy.  So that's the red arrow here.  So it's an optional optimization.  It's pretty good because it allows the compositor to lower the battery usage.  So if you're on a laptop, it's pretty important.  It improves latency, and it leaves some parts of the GPU free for other clients to use.  So if a game is keeping the GPU busy, the compositor won't disturb the game rendering,  for instance.  So apparently most compositors have this optimization implemented for full-screen  windows.  But a problem with this all looks great and all, but sometimes this direct scan out path  doesn't work.  So to explain why sometimes it works and sometimes not, I need to go into a bit of a  bit to explain tile buffers.  In general, when you need to store an image in memory, the straightforward thing to do  is to take the first row of the image, put it in a buffer, then take the second row of  pixels of the image, append it to the buffer, and so on and so forth till the last row.  So that's what we call a linear layout.  But there are other ways to represent an image in memory.  For instance, it's possible to divide an image into multiple tiles.  So here, for instance, the image is divided into nine tiles.  And the idea is to, instead of storing the image row by row, it's stored tile by tile.  So here the first tile is stored in memory first, and then the second tile is impended  to the buffer, and so on and so forth till the last tile.  So this is important because it allows the GPU to access the image in a manner which  is more convenient and optimized in regards to the cache, for instance.  So it's an important optimization for the GPU.  So because this has been introduced by hardware designers, of course, each vendor, Intel,  AMD, and NVIDIA, each of these vendors have come up with their own way to divide an image  into tiles.  Oftentimes, they are more complicated.  This is the basic idea, but there are more complicated schemes which I use, more complicated  tiling, compression, and so on, features like this.  And each vendor has multiple ways to represent an image in memory.  And of course, each way to describe an image in memory from a specific vendor is not compatible  with other vendors.  So we end up with a lot of different possible layouts.  In DRM, we call these layouts modifiers.  We describe the layout with something called a modifier.  So we end up in a situation where some parts of the GPU are able to handle some layouts,  some modifiers, but some other parts of the GPU are not necessarily able to support all  of the possible layouts.  So GPU has multiple parts in it.  The most commonly understood part, possibly, is 3D Engines.  It's usually responsible for running shaders, for instance.  So 3D Engines, in general, can deal with a lot of tiling layouts, a lot of modifiers.  It generally prefers quite advanced tiling layouts.  The display engine and the other end usually doesn't support as many modifiers as the 3D  Engines.  And the video engine also has a different set of modifiers.  So I gave a few examples here on the screen.  We are only scratching the surface here, and the hardware constraints when it comes to  modifiers are very complicated.  For instance, if the screen is rotated, maybe the GPU will not be able to display linear  images at all.  It will only be able to handle tiled images.  So we're starting to see that there needs to be some kind of negotiation.  There needs to be someone who sits down and says, hey, I want to use to write an image  with a 3D engine and display it with a display engine on the screen.  So I need to pick a modifier which works with both of these parts.  And I want to pick the best one.  I don't really want to pick linear because it's not very optimized for the 3D engine  to write into.  With that in mind, let's move on to multi-GPU.  So multi-GPU is basically when there are multiple GPUs connected to a single machine.  So it can happen when, for instance, you have a box with two PC express ports and one AMD  GPU and one NVIDIA GPU all connected to the same box.  Because there are also some laptops which have an integrated GPU, an Intel integrated  GPU, for instance, and a discrete GPU and the AMD, for instance.  So here, for instance, let's suppose GPU 1 is an Intel integrated GPU with GPU 2, a  discrete GPU on a laptop and battery.  So when I'm battery, I don't really want to power on the second GPU, the discrete GPU,  because that uses a lot of power.  I want to save power.  So the compositor decides to use the integrated GPU.  And the compositor wants to tell clients to also use the same GPU to save power.  So when it's cooked to be some way for the compositor to say to clients, hey, please use  this GPU I'm already using so that we don't just randomly power on the second one.  And clients need to know which GPU to pick.  Now let's say that I plugged my laptop in power.  So I don't want to save battery anymore.  And I want instead to play game.  So maybe the game is a bit demanding.  And it would prefer to use the second GPU, the discrete GPU, because it's more powerful.  So here the client locates a buffer on the discrete GPU, then does its rendering, then  hands over the buffer to the compositor.  On that done, the compositor needs to access, read the buffer on the discrete GPU, and render  it with the composited with the first GPU.  Because the screen is connected to the integrated GPU, the laptop screen.  So there's already some kind of multi-GPU buffer sharing going on here.  There are also some cases of external GPUs which can be plugged and unplugged, for instance,  connected via USB-C.  And that further complicates the matter, because the discrete GPU can come and go as a user  plugs and plugs.  So that's starting to be a bit fun.  And there are even more complicated scenarios.  For instance, if I have an external GPU and I start to plug a screen and the external  GPU.  And the compositor may want to also open both GPUs and take the client buffer composited  with the second GPU and then display it and then that puts it connected to the second GPU.  There are also other kinds of systems which makes things a bit hairy.  For instance, arm boards for embedded use cases.  I don't know if you've probably heard about Raspberry Pi, for instance.  This is an example of an arm board.  This is the arm board manufacturers want to have a bit of a fun evening.  And they decide to pick parts from multiple manufacturers and glue them together.  So for instance, one could have a display engine made by a whole chip or a winner or  a M logic and glue it together with a render engine made by arm, a Mali engine or something  else.  So even though this is a single stock from user space, this looks like a multi GPU situation  with two devices.  The first device can only display images and cannot use OpenGL at all.  The second device can only do OpenGL and has no outputs, has no notion of outputs.  Screens connected.  So yeah, this can be a bit of a problem.  So with all of this in mind, our goal while designing the new Waylon protocol extension  was to be able to tell the client which optimal device and format and modifier to use.  But it's not that simple because clients may still have their own constraints and preferences.  For instance, a video player may be wants to take advantage of hardware acceleration  and wants to use video engine.  And the video engine comes with some constraints and preferences when it comes to modifiers  and devices and formats.  So we need to reconcile both the compositor and the client and somehow pick the best combination  which is still supported by everything both the compositor and the client wants to use.  One thing which makes everything a bit more complicated is that things are dynamic.  As I said, for instance, external GPU can come and go.  And the number of that windows can move between screens.  So a window could move from a screen which is plugged in GPU A into a screen which is  plugged in GPU B. And windows can be on the screen like floating in which case direct  scan out is not possible for most compositors today.  And when the window becomes full screen, dynamic scan out becomes possible.  And to take advantage of direct scan out, the client needs to maybe change its device  format and modify it.  Something to keep in mind is that some clients can't have all of the fancy stuff.  Some clients are basic, have some other constraints.  And maybe they pick the best device, format and modifier to use at start up and then everything  is frozen.  They can't change anymore.  Finally, some compositors, the old compositors, always need a full back plan.  So it's not possible for a compositor to say, okay, I'll always be able to use direct scan  out.  Sometimes it won't work and modifiers are not, can express some of the stuff which makes  direct scan out possible, but it's not enough.  Never guarantee that the optimized cut path will always work.  So compositors need always to have a way to fall back to composition.  So boring, copy and everything that always needs to be read through.  Compositors can just take client buffers and do nothing with them.  It results in a black screen for users and that's not acceptable.  All right, so let's see how the protocol looks like.  We've introduced a few concepts.  So we've introduced a concept of main device.  The main device is basically a good device, good GPU for clients to use if they don't  have any other preferences.  We've also introduced a concept of preference range.  So a preference range is a set of device formats and modifiers which indicates some  supported combinations and each set is ordered by preference.  So the first set will be the one which will bring supposedly the most optimized way to  present an image.  The second one can be used as a fallback if the first one doesn't work and so on and  so forth.  And the last one, the compositor really needs to make sure that the last bench is always  usable or else maybe the client won't be able to display an image at all.  So the first thing a client will do is obtain a feedback object.  There are two ways to obtain a feedback object.  The first way is to get the default static feedback object.  It's like a global feedback object and tied to anything.  And the second way is to obtain a feedback object from a surface.  A surface is like a window and way to simplify.  So the feedback object will be tied to this surface and as the surface changes its position,  for instance, the feedback object will receive some updates, some new events and the client  will be able to react accordingly if it wants to.  Once the client has obtained a feedback object, it will receive feedback events.  So here's an example of typical feedback events.  It's a compositor.  The compositor first sends the main device event.  Here it says if you don't have any other preference, it's a good idea to use GP1.  And the compositor will send the trenches.  So here we have two trenches.  The first trench says this trench is about GPU2.  If possible, use format ARGB8888 with the modifier X-style or linear.  If you're using formats and modifiers from this trench, I'll be trying to use direct  scanout.  And that's it.  If it's not possible, the client can use the other trench.  The second trench says this trench is about the third GPU, GP1.  If you're using this GPU, you'll be able to allocate a buffer with the format ARGB8888  with modifier Y-style or X-style or linear.  And I won't be attempting to use direct scanout if you're using this trench.  And that's pretty much it.  So let's see how clients are able to handle these events.  There are multiple types of clients.  Let's start with the basic baseline support the clients could have, most minimal stuff.  Such a client would use the get default feedback request to obtain a global static feedback  object which won't really change.  And this client will open the device indicated by the main device event.  And for each trench which is about this main device, the client will try to allocate a  buffer with format and modifier specified in the trench.  So here, the client will receive main device GP1.  He will see a trench about GPU2.  It will ignore it completely and it will directly have a look at the second trench which is  about the main device.  And it will allocate a buffer with format specified here.  Let's continue with a bit more sophisticated client.  There are other types of clients, for instance, clients using Vulkan.  These clients already have specific GPU selected, pre-selected.  For instance, games often have a menu with checkbox to select which GPU to use.  So these clients will call the get default device, the feedback to get a static global  feedback and they will completely ignore main device because they already have selected  one.  And instead of, I'll go back to the example.  Let's say the client has selected GPU2 at the start-up and we will only look at trenches  about this pre-selected GPU.  So here, the client will only take it into account the first trench and completely ignore  the second trench and it will do the same as the first client by one of these formats.  One thing to note is that maybe the buffer input to the Compostor will fail because in  multi-GPU situations, the client selects device which is different from the GPUs used by the  Compositor.  Maybe the buffers won't be able to be accessed from the other GPU.  So clients need to keep this in mind and handle this gracefully.  Having a client even more complicated will be able to dynamically change its buffer format  and modifier.  So such a client will on the request get the feedback object tied to a specific surface.  So as a client and start-up, it will look at all the events as usual.  But the feedback object will send new events when the window is moved, for instance.  So clients will listen for these events and we allocate its buffer as needed.  So just works the same, but clients is able to dynamically change its... we allocate its  buffer and there's the kind of mix and match where some of the even more complicated clients  will be able to dynamically change the GPU they're using.  So that user gets the best feedback, selects the main device and then listens for events  and if the main device changes, they can close the old device, open the new device and then  we allocate all of the buffers.  And as before, buffer imports the Compositor can pay.  So what's the current status of the protocol and what are the next steps?  Implementation wise, we're pretty good, at least on the compositor side.  All of Mutter, Wroots, Weston and Queen have the protocol implemented.  So yeah, it's already possible to use it to the at least in the Git version of these  compositors.  In terms of clients, we only have a single implementation right now that I know of in  Mesa, a GL subsystem, so this means Wayland native open GL clients are able to benefit  from all of this stuff and they're able to dynamically change the modifier depending  on whether the window is full screen or not and stuff like this.  So that's pretty cool.  There are still some things to be done, for instance, we're still missing Vulkan support.  There's a mesh request open for this, they still need to review.  We maybe something worth doing would be to add support for XYLand.  There's some early patches for this, but yeah, we'll see what goes because we need to plan  this down to the XYLand protocol also.  So yeah, we see how it goes.  And also other clients could benefit from it, like maybe VIP clients, G-streamer, Codie,  stuff like this.  Then also which could be improved is multi-GPU support in existing compositors.  Right now, the compositors, all compositors that I know of have basic support for multi-GPU,  but they always do the composition step with a single fixed GPU that cannot change at all.  And they don't support direct scan out and secondary GPUs.  So that would be something to improve.  And they don't support either multi-GPU composition.  So if a game is running in discrete GPU and then is displaying a window which is shown  and a screen plugged into the discrete GPU, the same GPU, the compositor will still import  the game buffer into the integrated GPU and then export it again to the discrete GPU.  So just for the composition step, so that's not great.  Also as I said earlier, it's not possible to express all those hardware constraints  with modifiers.  So we're still missing some work here that could be done to optimize even further buffer  allocations.  So we've talked about this with James Jones from NVIDIA and we have some ideas to improve  this but see my existing 2020 talk about this for more details.  So that's what's in my chat.  Here are a few references you could look into if you're interested and feel free to discuss  as question and weigh-in in an IRC.  I'm going to do a small demo of all of this.  So Western is shipping with a demo called Western Simple DMEber Feedback.  It will basically act as the third kind of clients we've seen which dynamically changes  its format and modifiers.  So I'm going to run a window to open a window.  Then full screen it.  We're going to see what kind of events we see from the compositor.  Window is opened, I full screen it and it's closed.  So what happened?  Here we have a log of all of the events that happened.  So the client first received some events with the main device set to this device.  So the first render.  I only have a single GPU so things are a bit simple.  And the compositor says I cannot do direct scan out.  This is because at first the window was not full screen so that's expected.  So with this first bunch of events, there's only a single branch.  And that's it.  And once a full screen the window, the compositor sends another burst of events.  And you can see already that the first branch is different and has a scan out bit set to  true true.  That means the compositor will attempt to use direct scan out if the client uses when  the format modifiers from this branch.  Oh, I missed something here.  Oh yeah, here.  And we can see that in this burst of events there are two trenches and the second one,  so the first one is to use direct scan out and the second one is to not use direct scan  out.  So in case one of the buffers above in the first range wasn't usable for the client,  the client can always go back to the composition branch.  And that's pretty much it.  So if you have any questions.  Now is a good time, I believe, to ask them.  And thanks for listening.  Hello, so we have Simon with us here live to answer all your questions.  First off, thanks for your presentation and for all your work on the Weyland ecosystem.  It's greatly appreciated.  And let's start with a question for me.  So do you have any updates on the presentation?  Has anything changed or something else come to your mind that you haven't touched on in  your presentation during the week because this was recorded over a week ago?  You should probably unmute yourself.  You should unmute yourself because you're still muted in Jitsi.  Oh, sorry about that.  Okay.  All good now.  Yeah, so yeah, thank you.  For more context, yeah, I've been doing this presentation in a bit of a hurry.  So I may have missed something and I wanted to do a bit more, but oh well.  Maybe something I didn't mention quite explicitly is when I was talking about different parts  of the GPU and I was planning that there is a 3D engine, display engine and video engine.  So when I first got into graphics, I didn't really understand that the GPU was split like  this.  So the 3D engine is responsible for drawing the frames, for running the shaders.  And the display engine is where you plug your connectors, HDMI and display port and all of  this stuff.  And the video engine is for video decoding, hardware accelerated video decoding.  And yeah, it's important to understand that these parts of the GPU are not just one chip,  they are multiple components driving these parts in the kernel as separate.  So yeah, that's something I didn't quite mention explicitly.  In terms of updates, Mesaveulcan is still installed because it needs a review time and  review time is very valuable.  X-YLAND has seen some activity in the last few days.  Someone from NVDA has been working on it, so that's pretty cool.  As I explained, we need to update the X11 protocol as well.  We should be good, needs a bit more time.  Yeah, I think that's pretty much it.  I have some more material for the NVQ&A, but maybe we can start with questions now.  Okay, yeah, sure.  So we can wait with the material a little bit later.  And the first question from the community is from Anunes.  And the question is, what is the status of the clients using toolkits like GTK, QT, do  they already benefit from this work through just Mesa EGL?  Yeah, yeah, the good news is that as a client developer, if you're just using EGL, you don't  need to do anything to benefit from all this work.  The implementation in Mesa will allow most applications to just work out of the box.  And I've been hearing reports that applications like MPV are just working fine and benefiting  from this already.  So you can check yourself by setting the Wayland display equals one environment variable before  starting an EGL app, for instance, and check all the messages related to a DM above.  And maybe you can have a look and check that it's used.  But yeah, it should just work.  There's an extension to the same question by the same author.  So how about some more complicated applications like Firefox?  Oh, so yeah, Firefox is a bit of a more involved case because I think it just uses low-level  EGL.  It doesn't really use the rule.  So I think it manages the buffers itself with fine buffer objects.  I don't remember exactly.  Yeah, we need to investigate a bit more.  I don't know, a friend.  I know someone who maybe knows better.  So I can poke him later.  Another question is from the underscore ED.  The clean needs to be aware of the GPU changes.  How abstract that this is in Mesa?  Yeah.  So GPU changes are more involved because by default, most clients will just select a  GPU and start up and then just use that.  And so EGL and VLCAD clients won't work for dynamic GPU changes.  So if a client wants to support dynamic GPU changes, it needs to listen to the Wayland  events directly.  And it can do that.  So both Mesa, EGL, VLCAD and the client itself can listen to the changes in power advancement  problem.  So the client will need to tear down the old graphics context and recreate it with a new  device manually to implement this thing.  Okay.  Thanks for the answer.  And another question from Ismail.  How would this play with multiple scannered buffers?  Some cards display engines.  In theory, can display multiple buffers on screen at once, effectively doing some composition  themselves?  Yep.  So this should be, so most of the support for KMS planes, so deploying multiple buffers  at once with direct scan out.  Most compositor don't support it apart from full screen.  So the only exception right now is Western.  Western can do this.  And with the extension, this new Wayland extension, it should just work.  Western will send feedback to the surfaces, to individual surfaces accordingly.  So extension doesn't need to be changed to accommodate for this new use case.  But compositors need to be worked on so that this can be made to work more on motor, on  KDE and WL routes.  And this is something I plan to work on, at least for the WL route sites with things like  LibLiftF that have been working for a few years now on.  But we'll see.  So so far that was the last question I have over here.  And thank you for all your answers.  And you've said that you have something extra for us.  Yeah, yeah.  So there are a lot of things I left out in the talk.  Let's see.  So one of the things I left out is that we designed a bunch of more technical stuff around  the protocol.  Up to now, the protocol sends the list of available formats and modifiers via directly  on the wire and the unique sockets.  It just sends the whole list of formats and modifiers.  But it turned out with some AMD GPUs, for instance, there, or NVDA also, there are a lot, a lot  of modifiers.  So the list was getting pretty large to like multiple kilobytes, like, I don't know, 10  kilobytes or something.  And for a single way on the message, this is too big.  So we change that.  And now the compositor will write to a shared memory file, the list of formats and modifiers,  and we'll share this file in written remote to all clients, just like it does for things  like the keyboard, the key map.  So there should be also a performance improvement when you launch a client because the compositor  won't be broadcasting all of the lists all the time.  So that's when improvement.  Maybe you've seen from the talk that we try to handle as many use cases as possible, and  there are many interwind use cases.  And it's really been a challenge to come up with a design which work well with all of  these use cases.  And something I haven't mentioned is to adjust another layer of use case is that some we  also wanted to support drivers, which are legacy and don't support modifiers.  And this was a bit of a pain because there are many problems with that.  In particular, when it comes to multi GPU buffer sharing, if you don't have the format  modifier, which indicates really how the pixels are laid out in memory, then the driver will  just choose one implicit format modifier.  And you don't know about this format modifier.  And if you share this image between two GPUs, maybe the implicit format modifier used by  one GPU will be different than the one used for the other GPU, and you will end up with  a garbled image because both GPUs don't agree on the tiling to use.  So yeah.  We support these old drivers, mainly old AMD cards, but need to be a bit careful about  multi GPU situations mainly.  And yeah, to add on a bit on the next steps, also something to be worked on would be a  better explicit synchronization.  Right now there's a Waylon extension for explicit synchronization, but it's using sync  files and it's exchanging file descriptors.  Each time a frame is displayed and screened.  So that's not really efficient.  So we'd want to move on to something like DM sync ops, and that's more work.  Yeah, a lot to be done yet.  That's about it for me, I think.  Yeah, so thanks for the extra content, and we have two more questions.  So is this multi GPU configuration also applicable to virtual GPUs, like provided by SRIOV?  I don't know a lot about virtual GPUs, so I'm not sure.  I really need more context to be able to tell, so we can discuss afterwards if you're interested.  Okay, and another question, not directly related to buffer feedback, but what happens when  you run a full screen client with direct scan out and then notification from a different  client pops up?  So, yeah, planes.  So if you're lucky or if you have the new Waylon extension, when there's a full screen  client, it uses direct scan out.  If a notification shows up and you're running not a subnome, KDE or sway, then the compositor  will fall back to regular compositions, will fall back to open GL for compositing.  And we'll stop doing direct scan out because it won't be able to display both buffers at  once, but hopefully in the future, we'll be able to support direct scan out with multiple  buffers at the same time with multiple KMS planes.  But as I've said earlier, it's something that we need to implement in compositors and the  extension already works for this use case.  And if you're using Western, it should already work as is.  And Western should be able to display both the Firefox and the notification on top with  direct scan out.  Okay, thanks for the answer and the last question seems to be, it's prefaced by clarification  that it's a very new question.  And you said Linux DMA buffers implementation of MSI EGL.  And the question is, are MSI EGL and EGL streams the NVIDIA one?  The same thing.  So one thing to be noted here first is that EGL stream is an EGL standard.  It's an EGL extension.  And the NVIDIA implementation of EGL is not called EGL stream.  It's just called the NVIDIA proprietary driver.  So I guess the question is really, is this extension implemented in, is this new extension  implemented in the NVIDIA proprietary driver?  I don't think it is right now.  But I don't know.  Yeah, I don't know.  You need to ask NVIDIA engineers about that one.  Or maybe wait a few weeks and see if it comes up.  OK, I didn't see any more questions for now about the two people typing.  So yeah, wait for that.  And yeah, thank you again for an awesome talk and awesome Q&A, extra content and all that  you have done for the community.  Because like your work on Wayland is just outstanding.  Yeah, thanks.  Glad you liked it.  It's a lot of work.  But also it's very time consuming and energy consuming to work on Windows system integration  in particular, because you need to make people agree and there's a lot of politics in it.  And not a lot of people have time to work on it.  So with all of this, it's quite hard to work on this stuff.  So we have a follow up for one of the previous questions.  I assume the latency for the direct scan out is lower.  So there is a glitch when switching back and forth.  So does it happen?  Is it visible or noticeable when you switch from direct scan out to not direct scan out  when something pops up?  So everything is implemented correctly.  It shouldn't be visible.  So there is a difference in latency in that with direct scan outs, when so the compositor  basically needs to do a frame every 16 milliseconds or so.  So with our direct scan outs, the application has less time to render because the compositor  will need to also have a render step.  But with direct scan outs, we skip this step and the application has more time to render.  So it kind of depends on the situation.  If the application already isn't taking a lot of time to render, it should be fine.  If the application is like right under the limit is like taking maybe I don't know exactly  16 milliseconds to render, then you'll see the application will be fluid if it's using  direct scan out and then it stops using direct scan out.  Maybe it will become a bit choppy.  But I don't think we can do a lot about this.  It just if we can't use direct scan out, we just can't use the optimised file.  And yeah, there's no saving grace here.  So maybe in some situations, we are a bit of a glitch, but this extension should just  try to make things as not choppy as possible most of the time.  Okay, it looks like there is just one last question being typed out right now in the  chat, I assume that's the question because someone is typing.  Yeah, those were really good answers and good questions.  And it feels like your presentation was definitely one of the most popular ones here.  So that's cool.  And now two people are typing.  So that's going to be interesting.  Yeah, while they are typing, maybe I can give a few words to thank Colaborah.  Colaborah has been helping a lot on this.  Because Leandro and Daniel Stone, they've both been providing some Western implementation,  providing a MESA implementation.  So they've been pivotal in making this happen.  So really thank you for this.  Okay, the last thing actually wasn't a question, but thank you for the answer because the person  who asked about EGL just read, seen the word EGL used when talking about Nvidia Preparator  driver and MESA is also using EGL.  So it was confusing.  Cool.  Okay, we have still five minutes and there are still people typing.  So any more shout outs?  Oh, also we have Luke now here as well.  Thank you, Eric, for organizing all this, therefore, I guess.  Maybe now is a good time to say it.  Well, thank you for speaking and thank you to all the speakers today as well.  And most of the time, well, thank you for Eric and Martin for being the other different  organizers, but most of the time should go to the FOSDM developers, the FOSDM organizers  who spent the last months in a mad dash as they usually do.  And they will probably go in hibernation for half a year again as they do every year.  And then somewhere in August, they will reemerge from their sleep and they will start working  on hopefully the next real life FOSDM that we don't have to do a virtual one again.  So yeah, everybody get vaccinated.  It's the best hope we have to have a proper real life FOSDM again next year.  And thanks for watching from my side at least.  Thank you.  Okay, so that was the last question.  The only other things that were typed in the chat since then were thank you for your presentation  and that's it.  So I guess we can wrap it here.  Thanks to all the presenters.  Thanks to all the attendees and hopefully see you next year in the Meet Space.  Yeah, sure.  All right, all right.  See you.  Okay.
dmabuffeedback_0_155.png dmabuffeedback_156_1763.png dmabuffeedback_1764_4223.png dmabuffeedback_4224_21491.png dmabuffeedback_21492_27935.png dmabuffeedback_27936_28643.png dmabuffeedback_28644_29063.png dmabuffeedback_29064_29183.png dmabuffeedback_29184_41927.png dmabuffeedback_41928_42119.png dmabuffeedback_42120_47627.png dmabuffeedback_47628_48911.png dmabuffeedback_48912_48995.png dmabuffeedback_48996_50279.png dmabuffeedback_50280_50339.png dmabuffeedback_50340_51203.png dmabuffeedback_51204_51299.png dmabuffeedback_51300_51347.png dmabuffeedback_51348_52223.png dmabuffeedback_52224_52355.png dmabuffeedback_52356_52415.png dmabuffeedback_52416_53411.png dmabuffeedback_53412_54419.png dmabuffeedback_54420_56531.png dmabuffeedback_56532_56591.png dmabuffeedback_56592_57563.png dmabuffeedback_57564_58079.png dmabuffeedback_58080_59279.png dmabuffeedback_59280_59459.png dmabuffeedback_59460_61835.png dmabuffeedback_61836_62339.png dmabuffeedback_62340_63491.png dmabuffeedback_63492_64007.png dmabuffeedback_64008_64235.png dmabuffeedback_64236_64583.png dmabuffeedback_64584_65615.png dmabuffeedback_65616_66275.png dmabuffeedback_66276_66671.png dmabuffeedback_66672_67115.png dmabuffeedback_67116_69395.png dmabuffeedback_69396_69455.png dmabuffeedback_69456_69587.png dmabuffeedback_69588_70091.png dmabuffeedback_70092_70343.png dmabuffeedback_70344_70427.png dmabuffeedback_70428_70799.png dmabuffeedback_70800_71171.png dmabuffeedback_71172_73319.png dmabuffeedback_73320_74843.png dmabuffeedback_74844_75119.png dmabuffeedback_75120_75707.png dmabuffeedback_75708_75911.png dmabuffeedback_75912_76367.png dmabuffeedback_76368_76475.png dmabuffeedback_76476_76979.png dmabuffeedback_76980_79091.png dmabuffeedback_79092_79331.png dmabuffeedback_79332_79571.png dmabuffeedback_79572_80075.png dmabuffeedback_80076_80231.png dmabuffeedback_80232_80675.png dmabuffeedback_80676_80795.png dmabuffeedback_80796_82259.png dmabuffeedback_82260_82355.png dmabuffeedback_82356_82499.png dmabuffeedback_82500_82547.png dmabuffeedback_82548_83123.png dmabuffeedback_83124_83207.png dmabuffeedback_83208_83891.png dmabuffeedback_83892_84503.png dmabuffeedback_84504_84743.png dmabuffeedback_84744_84862.png