libcapsule
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Mohapatra, Vivek Das
libcapsule is a project that allows segregated dynamic linking: Access to the symbols of a library without being exposed to any of the dependencies of that library without requiring recompilation of the binary that pulls it in. libcapsule's goal is to improve portability of programs that are distributed with runtimes (cf flatpak) that still need access to [some] libraries from the host (eg libGL) while insulating said program from any system libraries outside the runtime other than those it directly requires.
 So, hi. My name is Vake. I work for Collabra. I've probably seen a couple of my colleagues talk here already before. And today I'm going to talk about something that may not mean very much to you, which I'm calling segregated dynamic linking, because naming things is hard and this will do until somebody comes up with something better. Okay, so let's jump into it. What is it? You may be wondering. So, let's start with a statement of the problem that we're trying to solve. These days a lot of applications are containerized. I'm using that term kind of loosely. They're in some kind of wrapper. They have libraries that come with them. The libraries come from the runtime rather than from the host. It's just a way of improving portability and stability because, you know, we all know, most people here know about the kind of portability problems we have. That's kind of mostly true. It's a little bit of a lie. Some of them still need to come from the host. Notably, your graphics stuff. It's quite closely tied to the hardware. There's no reasonable way of shipping it in a given runtime and still having it work with whatever hardware you end up having to work with, which is all fine for the most part. You know, you can sort of do that. But then the problem which we were trying to solve in the first place with our containerized applications, which is that the host and runtime libraries may be incompatible. This is kind of a really very primitive sketch of what the problem might look like. You can see that we've got a dependency down there and we've got something in the runtime that uses it. The main executable uses it. And unfortunately, we also need to have this library from the host, probably libgl or something similar. It's incompatible. It's not going to run. It's really unhappy because we haven't solved the problem that we set out to solve. So what could a solution look like? Well, we kind of want to have segregated linking. We want the host library that needs its compatible version or incompatible version of the library linked to it. But we want the rest of everything else not to see that. We want it to be generally unaware that there's another incompatible version of this dependency over here. And for those of you who are familiar with how linking works, how many people is that, by the way, just so I know how many people I'm lying to? It's a fair few. So if you spot me telling any lies, don't tell anyone else. Let's keep the illusion of competence alive. So you'll know that this isn't how linking on ELF works. There are other systems in which link is a tree and that has its own problems. But in general, we have one copy of the library and everything uses it, which is kind of the problem that shared libraries set out to solve in the first place. So what are our objectives here? We want to expose only the library that we are pulling in. So we want to expose libhost, but we don't want to expose this copy here, which is incompatible. As it's a dependency, it's the thing we're trying to protect ourselves from. We don't want to see it. We don't want to have to change the code because if we had to change the code of our application or our libraries, that would kind of defeat the point. We could say to people, hey, you're going to have to rewrite your programs to make them all portable. Well, that's been tried several times. We all know how that goes. It doesn't happen. We don't want to have to recompile the applications. Again, same thing. It's just going to be nonsensical if we say. And then you have to rebuild whatever the large application is that you want to run. And we would very much like for there to be no performance hit. We also want it to be transparent. It should happen when someone loads their containerized application or game or whatever. It should happen pretty much transparently. The user should not know anything is happening here. And just to make life easy for ourselves, we want this to be done with minimal manual intervention. We want this to be as frictionless as possible because any time you put a barrier in an application developer's way, they won't do it. With reason. We're asking them to do something and they should say, well, I shouldn't have to worry about this. It's not my problem. It's yours. So well, let's go through this and we'll see how many of these we can achieve. So what are the pieces of the puzzle that we're going to need for this? First of all, private dependencies. A library should have isolated dependencies. Normally, there's a single link list effectively of the dependencies in your program space. Everything will come from there. You'll get anything that you mentioned the name of as your dependency. You will see symbols from that. And it's just one flat list. There's no segregation going on there. So we need to kind of defeat that somehow. Worst case scenario, we're going to have to rewrite the linker. I really, really don't want to do that. So let's hope I don't have to. And the answer turns out to be, yeah, I'm open. So this is a new library call that was introduced in G-Lib C and around, I want to say 2.19. I think it was sort of existed before then, but that's sort of the first version that works without immediately exploding in most cases. It's not quite there yet. It's a work in progress. Patches are still being fed upstream. But from then on, it kind of works. And what does it do? It is like DL open. It adds a library after you've loaded up to your link map. But it can also create a new secondary or tertiary link map and add libraries to that instead. Sounds kind of promising. It's basically what we ordered for this. And just like DL open, it automatically loads dependencies. So we have a problem in that we want to pick the right libraries. Like if you remember, from back we had two copies of the dependency. We want to be able to pick the right one. Now by default, it's going to go to your standard loading path and it's going to load the one that's coming from your runtime with your containerized application. See, libraries are picked from the search path. We kind of need to defeat that because we will still get the wrong copy for at least one side of our link chain if we just let it do its thing. So we want isolated libraries to come from the non-standard search path. We must divert searches for dependencies of isolated libraries. Can we do that? Yeah, sort of. The linker loads all listed dependencies. But it won't reload items that are already in the link map it's targeting. So if a library is already there, it looks at your dependencies and said, you asked for libfoo. I have something called libfoo. My work here is done. Moving on to the next one. So if we load the libraries for the isolated library by their full path, not just by their SO name, not just by libfoo.so, we can say, you know, run host whatever, wherever we've exposed the host libraries. We do them in reverse dependency order. So we walk the dependencies backwards. We say, oh, libhost needs libdep. I will go and get that from wherever I've mapped in the host libraries. And by the time the linker gets to it, it won't go and search the standard search path. It will look in the link map it's loading to and say, I already have that. My work here is done. I will get my symbols from here. We can move on. So, yeah, we can do it. We can get what we wanted. Sort of. We have gotten part way there. We haven't had to rewrite the linker, which is always a plus in my book. But what we've achieved is we've isolated the DLM open symbols completely. They're not available by default to anything. So we still haven't met our goal of all of this being frictionless and seamless. So we need a caller of our library that's coming from the host to get those symbols automatically. And in order to do that, we need to understand how dynamic library calls work. So we're going to do a little bit of a dive into that. Now, don't worry about it because we're going to go into some detail. But at the end of it, what we will have figured out is that we don't need to know any of that detail because it sort of works the way we need it to, which is always nice. So this is what jumping to a foreign function call, a call from another elf object, a library in this case, looks like. We start by pushing our arguments for the call onto the stack. And then we look into something called the procedure linkage table. So each foreign function that you have has an entry in the procedure linkage table. And we look up its address. We read it back from something called a relocation record. And the things that are important to note here are the red and yellow sections are read-only. They can't be touched once they've been loaded. It's only the blue columns, the stack and the RR that are written to. Everything else is read-only, which sort of brings us some nice features of shared libraries. For example, they can be shared between processes without any problems. So this is the first step of calling a function. So we've got an address from the relocation record. And then what happens is that the first time you call a function, you don't actually get the address of the function. You get something called the fix-up code. So the fix-up code pointed by the relocation record asks the linker for the real address. The linker searches the dependencies of the calling object for where it should look for the symbol. And then it writes that address back into the relocation record. So this chunk here in green, so this is blue, this happens every time you call a foreign function. This happens only the first time you call it. And then this happens every other time. We've read the record back from the relocation record. We jump to the text. The function does whatever it needs to do, whatever it was written to do. The return value is pushed onto the stack, and then we jump back to the caller. So if we scribble on that relocation record before this first call ever happens, the PLT fix-up will never be invoked. So we bypass the normal symbol resolution entirely. It's up to us where this goes. The linker never resolves the symbol address. We have total control over where the function call goes. This is sort of the thing that a lot of people writing malware use to redirect your functions. Thing is, they have a rather nice feature in that they only need to work about maybe one time out of a million, and they've got a successful attack. We need this to work every time, which kind of makes things harder for us. They're amateurs basically. This is an important bit. At no point did we need to know the signature of the function we were calling. The caller puts the arguments onto the stack in the expected way. That was handled by the compiler because it had the signatures when the program was compiled. The callee pulled the arguments off the stack. It knew what it was expecting. But even though we're diverting the call to a completely different location, we didn't actually need to know the details. It was taken over just before and just after we did anything. So we didn't need to know that, which is good because it makes it much easier for us to automate this whole process. That's a huge amount less that we need to know. We don't need to know the arguments. We don't need to know how they are laid out on the stack. All we need to know, which is guaranteed for us, is that they are there on the stack when the call happens. So the key question is, can we find that relocation record and scribble on it reliably? The answer, luckily for me, again, no need to relink it, is yes, we can. The link map entry has a pointer to the ELF data for each library and libElf can interrogate this. We've added one dependency, libElf, but I'm pretty sure that's not going to change incompatibly any time soon. So great. We can now put these pieces together. We can make a shim library with the target library's SO name so that the linker, when the linker loads it, and we put it on the search path. So, for example, if I want to load libGL, there's no actual libGL inside my runtime with my containerized application, but I can put a shim libGL there, which will be loaded by the linker. And during the init phase of that shim library, I can deal them open the real library and all of its dependencies in reverse order, as I've said. I can search the alternate library path to do it, so I know I'm getting the host one because I want the one that didn't come with my runtime. And then I walk the link map, I'm making this sentence do a lot of work here. I walk the link map and scribble on all the relevant relocation records so that all of the functions in my binary, instead of going to the plt fixup and then finding the location of the stub function in my shim library, they'll actually go to the real one. They don't ever realize they've been diverted because I've bypassed the normal symbol lookup code. So, yeah, kind of we can basically do this. Now, obviously, I've simplified a lot. There are some details and some problems. The self-shaving yet has not yet been invented. So I'm just going to establish a couple of terms up front. I've called an isolated set of libraries a capsule. There may be a better name. Naming things is hard. If you can think of a better name, by all means, let me know. And then I've assumed that they come from file system mounted at slash host. So within my charoot or container or runtime or bubble wrapped environment or whatever, I have mapped in the actual host file system read only at slash host. For example, it could actually be anywhere. Other problems. DL open can't be called from inside a DL open namespace. It causes the C library to explode, which is not fun for anyone. So I need to use this relocation record scribbling mechanism, which I've already established, to replace the DL open inside a capsule with a wrapper that does the following. It calls DL open instead of DL open. It does the path remapping and simulink resolution relative to slash host or wherever I've told it the foreign libraries are located. It doesn't accept RTLD global because DL open doesn't yet have semantics for that. I'm discussing that with G-Lib C upstream. When we've hashed that out, it will, but for the time being, I need to mask that. The reason I need to mask that is because libgl does the... It DL opens itself, RTLD global, to expose symbols to its plug-ins, which was fun in games when I figured it out, but until I did was a puzzling crash as far as I was concerned. What else? OK. DL sim, which is how you get symbols from a DL open library, now has to have a split personality. It needs a little bit of extra work. Remember, we want this all to be seamless, but now there are two namespaces, not one. That means that when something in our application calls DL sim, I need to transparently go, ah, you may have meant this namespace over here and look in there, but I only wanted to find the symbols of the directly targeted library. I don't want to look at any dependencies, so I need to be quite careful. Again, similarly, I will scribble on the relocation record for DL sim and replace it with a wrapper that does these kind of safety checks for me. Another interesting thing, so the usage of libgl, which was the primary use case, varies quite a lot. Some things link statically against it, some things DL open libgl, there's a whole bunch of ways you can use it. So any time another library that's not necessarily one that I've exposed is DL opened, I need to scribble on its relocation record as well, which means that I similarly have to wrap DL open outside the capsule in the same way so that any time something is DL opened, I scribble on its relocation record. So if it needs, for example, a libgl function, it gets the right one. Extra problems. Currently each, this is kind of a lie in my demo because it is fixed, but the patches have not yet been reviewed and accepted upstream, so currently each namespace has its own glibc copy, which works a lot better than you might think in that it actually works at all. But it does make anything to do with threads quite, it's not actually a deadlock, but it looks like a deadlock prone. Alloc and free is interesting, of course. As long as all of your allocation of freeing of a particular pointer happens on the same side of the capsule boundary, you're okay, it works, it's fine. If you pass a pointer to across the boundary and expect it to be freed on the other side and you've got two glibcs, then, yeah, glibc immediately throws a hissy fit and says, what are you asking me to free? I've never seen that pointer in my life, corruption, pull the ripcord, bail out immediately, we have a disaster on our hands. So, yeah, there is going to be an extra flag called RTLD shared, which is currently being reviewed, and what that will do is allow me to put sort of the fundamental libraries like the glibc cluster have the same instance on both sides. It's building on work that's there in glibc anyway because, as you can imagine, there can only be one copy of the loader. It's sort of half already does this, but it needs to know to do it for glibc as well, and that's kind of one of the things that I'm feeding upstream. So, the workaround I had before was to replace the alloc and free clusters inside the capsule, which kind of in a hackish sort of way tried to dispatch the pointers to the right place to be freed. It doesn't really come up much when using glibgl because glibgl doesn't really allow allocation to pass across the boundary. It's API, but for other libraries, it might be relevant. Right, so, and finally, does it actually work? I'm going to postage the fortune here, I'm going to attempt a live demo. So, these things have all been tried and worked. I'm going to try glxgears and then if that works, I'm going to try open arena. So, where are we going? So, this is a cheroot which has had the libgl surgically removed from it, and I'm going to try and launch glxgears. You should see some spew from the capsule, which is where you can see it's mapped in the different things, and we've got a glx running, and you can see there it says, oh, gl over and wrapper cannot pass RTLD global, so I know I'm intercepting these calls. Where's my pointer gone? Someone see my pointer? Oh, there it is. So, if we page up a bit, you can see I've got, this is the default namespace, you can see I've got some shim libraries mapped in here, which are stubs, they don't contain any actual code, they're just a placeholder, so that the linker can map things in and then I can scribble on the relocation records later, and then if I go up a bit, there we go, we can see there's a capsule there which has host libraries mapped in, which is where the real libraries reside, and these are isolated from glxgears except where I've redirected them with relocation scribbling. So, that was one test. Now, that's a relatively simple test because it has libgl linked in, not static linked in, but linked in. Now, open arena uses SDL, which is going to DL open libgl, which is kind of a more complicated case because things are going to happen much later. If I'm really lucky, this will work, and I've got about 50% with this demo, so let's see if we can get that percentage up a little higher. Let's put the window here. Why is it the window here? Let me... I don't know if I can move the open arena window up. Damn it. Let's put it on the wrong screen. Maybe I can just turn this around. Demos, they never work, right? Okay. Tiny, but you should be able to see open arena actually running with a diverted libgl. Okay. I think that's kind of the end of the talk. And... So, the code is there. It's all kind of published already. This talk as well, if you're into that sort of thing, is available there. Any questions? I didn't understand then what makes the, like, your open arena demo only 50% reliable? Oh, that was kind of unrelated to this. It was a pulse audio not being happy running in a charoot problem, which I eventually tracked down. But, yeah. It's one of those things. It breaks, and quite often because there's a lot of moving parts. So, a colleague of mine, so this work was originally, well, and still is sponsored by Valve, because they want to improve game stability is one of the things. And so, a colleague of mine, Simon Vagitti, is working on the other side of this, which is assembling all of the right libraries within the runtime, sort of hybridizing your runtime libraries with the bits from the host that you need, you need to pick the newest libc of the ones that are available, and a couple of other things. So, he's working on that side of things, but those pieces don't exist yet. So, I kind of had to hack it together by hand, which is why it was unreliable. It wasn't anything intrinsic to the project. So, I would like to ask if you have support for RTDeepLink to go to the next symbol in the chain? Sorry, to the support for deep linking when you load the symbol from the next library in the chain. No, I don't think we do. This is pretty much just gotten to this stage, so not yet. Going in a different direction. Sure. Could you use this to make symbols like soft linking where you resolve a function if the library is installed, it uses the library, and if it's not installed, just returns an error. Yeah, what happens when this one is, actually, I didn't demo it because my auto-generated stubs direct everything, but, yeah, if you can have, what would happen there if a symbol wasn't available and you called it, you would immediately get a backtrace saying you called a symbol that wasn't available in the target library. Here's the backtrace, where you called it from here. So, yes. I'll answer your question. Yeah, but I don't want to get a backtrace. I want to, so for example, I have a program and the program links to five different compression libraries. Sure. And then if one of them is not installed, I would just want to say no, cannot decompress this, you know, like try a different one or something like that. Yeah, you could do that. Okay. Thanks. I was wondering if you can speak a little about what Valve is planning to do longer term with this, where they're going. Okay, sure. I mean, the idea is, I mean, pretty much as a laid out at the beginning, so, you know, you need libgl or in the future, you'll need your Wayland drivers to come, Vulkan drivers, sorry, to come from the actual system because they're the thing that's closely tied to your hardware and there's no real sane way to put those cram all the different ones into a runtime. You know, they would quite like a game that you bought 10 or whatever number of years ago to keep running even though you're on a completely different system. So the idea is that the game will come tied to a runtime, a version, like pretty much similar to Waythings like Flatpak work, you know, you've got your application in a runtime, you'll have a game, and it will specify, I want this flavor of runtime at this version. But then you need to parachute in the most recent libc out of the two that's available, and you need to parachute in the graphics drive, the stuff that's tied to the hardware needs to be brought in from the host. And so that's what it's going to be for. And the idea is that when my colleague's work is done, all of that will sort of be assembled into your sort of hybrid runtime. So hopefully, the idea is that in the glorious flying car future, we will improve game stability a lot, right? If you have an old console, you can plug a game in that you bought X number of years ago and it will work. Now, they have the advantage that it's all baked into the hardware, so they don't really have to deal with things changing. But they would really like that to be the case, because what their thing is that they want to sell you stuff electronically, and they want you to have the confidence that whatever you've paid for it goes on working. So that's the goal here. Sounds great. Is there an ETA? There sort of is, but it's tied to other things that I can't talk about. So what if there is a different system that uses the same Mac, for example, a heap profiler that wants to replace all mellocs in your entire program? What if there is? I don't know what would happen. It might work. It would depend on the order in which things happened. Was that what you were saying, would they clash? Yeah, I mean, can those interact? Is there a way to make it work? I don't know the answer to that. I'd have to try it. I wonder if you could talk a little about the surface area of GL that's relevant to change, because in my memory there's a couple of APIs where you put uniform in, and then you honk a giant string of program text, and that goes across a terrifying barrier into the drivers. But that is a function that says int and gives you a void star and stuff. To be honest, I don't actually know that much about how GL worked. There was a possibility that I'd need to learn it to do this, but I got to the point where it worked and then I stopped looking. As you may know, there's a whole number of different ways of using libgl. There's the simple GLXgears approach of just linking against it. There's the SDL approach of GL opening it and finding the functions. There's the one where you get a specific function from GL, and then you ask it for the function pointers that you want to call. But they all sort of work with this approach, because it's only the small set of well-known functions that people can call that need to be proxied in this way, and everything else just kind of works amazingly enough. Okay. For the dynamically linked GLXgears, can you just show what it looked like when you run through LDD? Yeah, sure. Let's see. Let's see. Nope. Where is it? It's right. It doesn't seem to be able to get focused back. Where's it gone? Oh, right. Sorry. Giant screen. Let me quit that. There we go. There we go. So you can see libgl is going to a shim. If we looked in that shim, we would just find stub functions with no actual code in them. Well, there's a little bit of code. There's code that generates a backtrace if you ever actually managed to call those functions. So the search path, the shim search path you added to the container is LDD conf. Okay. Oh, okay. Yes. Let me... Nope. It's going to work from here. Uh... Oh, isn't it? Oh, okay. I'll make it public. You're the first person to actually ask. So, yeah. Okay. Any other questions? I'm going to make it show the LDD output again. Okay. I saw that you had... So why then here is... So you see like the fit to last line there. Why is LDD saying that GLX gears, which you didn't have to recompile or anything, why is it dependent on lib capsule now? It's coming from the shim. The shim depends on lib capsule. So lib capsule is kind of rather than baking the code into every single shim, they link against lib capsule, which does that. Okay. That seems to be old questions. So there it is. Segregated dynamic linking.
 So, hi. My name is Vake. I work for Collabra. I've probably seen a couple of my colleagues  talk here already before. And today I'm going to talk about something that may not mean  very much to you, which I'm calling segregated dynamic linking, because naming things is  hard and this will do until somebody comes up with something better. Okay, so let's jump  into it. What is it? You may be wondering. So, let's start with a statement of the problem  that we're trying to solve. These days a lot of applications are containerized. I'm using  that term kind of loosely. They're in some kind of wrapper. They have libraries that  come with them. The libraries come from the runtime rather than from the host. It's just  a way of improving portability and stability because, you know, we all know, most people  here know about the kind of portability problems we have. That's kind of mostly true. It's  a little bit of a lie. Some of them still need to come from the host. Notably, your  graphics stuff. It's quite closely tied to the hardware. There's no reasonable way of  shipping it in a given runtime and still having it work with whatever hardware you end up  having to work with, which is all fine for the most part. You know, you can sort of do  that. But then the problem which we were trying to solve in the first place with our container ized applications, which is that the host and runtime libraries may be incompatible.  This is kind of a really very primitive sketch of what the problem might look like. You can  see that we've got a dependency down there and we've got something in the runtime that  uses it. The main executable uses it. And unfortunately, we also need to have this library from the  host, probably libgl or something similar. It's incompatible. It's not going to run.  It's really unhappy because we haven't solved the problem that we set out to solve. So what  could a solution look like? Well, we kind of want to have segregated linking. We want  the host library that needs its compatible version or incompatible version of the library  linked to it. But we want the rest of everything else not to see that. We want it to be generally  unaware that there's another incompatible version of this dependency over here. And  for those of you who are familiar with how linking works, how many people is that, by  the way, just so I know how many people I'm lying to? It's a fair few. So if you spot  me telling any lies, don't tell anyone else. Let's keep the illusion of competence alive.  So you'll know that this isn't how linking on ELF works. There are other systems in  which link is a tree and that has its own problems. But in general, we have one copy  of the library and everything uses it, which is kind of the problem that shared libraries  set out to solve in the first place. So what are our objectives here? We want to expose  only the library that we are pulling in. So we want to expose libhost, but we don't want  to expose this copy here, which is incompatible. As it's a dependency, it's the thing we're  trying to protect ourselves from. We don't want to see it. We don't want to have to change  the code because if we had to change the code of our application or our libraries, that  would kind of defeat the point. We could say to people, hey, you're going to have to rewrite  your programs to make them all portable. Well, that's been tried several times. We all know  how that goes. It doesn't happen. We don't want to have to recompile the applications.  Again, same thing. It's just going to be nonsensical if we say. And then you have to rebuild whatever  the large application is that you want to run. And we would very much like for there  to be no performance hit. We also want it to be transparent. It should happen when someone  loads their containerized application or game or whatever. It should happen pretty much  transparently. The user should not know anything is happening here. And just to make life easy  for ourselves, we want this to be done with minimal manual intervention. We want this to  be as frictionless as possible because any time you put a barrier in an application developer's  way, they won't do it. With reason. We're asking them to do something and they should  say, well, I shouldn't have to worry about this. It's not my problem. It's yours.  So well, let's go through this and we'll see how many of these we can achieve. So what  are the pieces of the puzzle that we're going to need for this? First of all, private dependencies.  A library should have isolated dependencies. Normally, there's a single link list effectively  of the dependencies in your program space. Everything will come from there. You'll get  anything that you mentioned the name of as your dependency. You will see symbols from  that. And it's just one flat list. There's no segregation going on there. So we need  to kind of defeat that somehow. Worst case scenario, we're going to have to rewrite  the linker. I really, really don't want to do that. So let's hope I don't have to. And  the answer turns out to be, yeah, I'm open. So this is a new library call that was introduced  in G-Lib C and around, I want to say 2.19. I think it was sort of existed before then,  but that's sort of the first version that works without immediately exploding in most  cases. It's not quite there yet. It's a work in progress. Patches are still being fed  upstream. But from then on, it kind of works. And what does it do? It is like DL open. It  adds a library after you've loaded up to your link map. But it can also create a new secondary  or tertiary link map and add libraries to that instead. Sounds kind of promising. It's basically  what we ordered for this. And just like DL open, it automatically loads dependencies.  So we have a problem in that we want to pick the right libraries. Like if you remember,  from back we had two copies of the dependency. We want to be able to pick the right one.  Now by default, it's going to go to your standard loading path and it's going to load the one  that's coming from your runtime with your containerized application. See, libraries are picked from  the search path. We kind of need to defeat that because we will still get the wrong copy  for at least one side of our link chain if we just let it do its thing. So we want isolated  libraries to come from the non-standard search path. We must divert searches for dependencies  of isolated libraries. Can we do that? Yeah, sort of. The linker loads all listed dependencies.  But it won't reload items that are already in the link map it's targeting. So if a library  is already there, it looks at your dependencies and said, you asked for libfoo. I have something  called libfoo. My work here is done. Moving on to the next one. So if we load the libraries  for the isolated library by their full path, not just by their SO name, not just by libfoo.so,  we can say, you know, run host whatever, wherever we've exposed the host libraries. We do them  in reverse dependency order. So we walk the dependencies backwards. We say, oh, libhost  needs libdep. I will go and get that from wherever I've mapped in the host libraries.  And by the time the linker gets to it, it won't go and search the standard search path. It  will look in the link map it's loading to and say, I already have that. My work here  is done. I will get my symbols from here. We can move on. So, yeah, we can do it. We  can get what we wanted. Sort of. We have gotten part way there. We haven't had to rewrite  the linker, which is always a plus in my book. But what we've achieved is we've isolated  the DLM open symbols completely. They're not available by default to anything. So we still  haven't met our goal of all of this being frictionless and seamless. So we need a caller  of our library that's coming from the host to get those symbols automatically. And in  order to do that, we need to understand how dynamic library calls work. So we're going  to do a little bit of a dive into that. Now, don't worry about it because we're going to  go into some detail. But at the end of it, what we will have figured out is that we don't  need to know any of that detail because it sort of works the way we need it to, which  is always nice. So this is what jumping to a foreign function call, a call from another  elf object, a library in this case, looks like. We start by pushing our arguments for  the call onto the stack. And then we look into something called the procedure linkage  table. So each foreign function that you have has an entry in the procedure linkage table.  And we look up its address. We read it back from something called a relocation record.  And the things that are important to note here are the red and yellow sections are read-only.  They can't be touched once they've been loaded. It's only the blue columns, the stack and  the RR that are written to. Everything else is read-only, which sort of brings us some  nice features of shared libraries. For example, they can be shared between processes without  any problems. So this is the first step of calling a function. So we've got an address  from the relocation record. And then what happens is that the first time you call a  function, you don't actually get the address of the function. You get something called  the fix-up code. So the fix-up code pointed by the relocation record asks the linker for  the real address. The linker searches the dependencies of the calling object for where  it should look for the symbol. And then it writes that address back into the relocation  record. So this chunk here in green, so this is blue, this happens every time you call  a foreign function. This happens only the first time you call it. And then this happens  every other time. We've read the record back from the relocation record. We jump to the  text. The function does whatever it needs to do, whatever it was written to do. The  return value is pushed onto the stack, and then we jump back to the caller. So if we  scribble on that relocation record before this first call ever happens, the PLT fix-up will  never be invoked. So we bypass the normal symbol resolution entirely. It's up to us  where this goes. The linker never resolves the symbol address. We have total control  over where the function call goes. This is sort of the thing that a lot of people writing  malware use to redirect your functions. Thing is, they have a rather nice feature in that  they only need to work about maybe one time out of a million, and they've got a successful  attack. We need this to work every time, which kind of makes things harder for us. They're  amateurs basically. This is an important bit. At no point did we need to know the signature  of the function we were calling. The caller puts the arguments onto the stack in the expected  way. That was handled by the compiler because it had the signatures when the program was  compiled. The callee pulled the arguments off the stack. It knew what it was expecting.  But even though we're diverting the call to a completely different location, we didn't  actually need to know the details. It was taken over just before and just after we did  anything. So we didn't need to know that, which is good because it makes it much easier  for us to automate this whole process. That's a huge amount less that we need to know. We  don't need to know the arguments. We don't need to know how they are laid out on the  stack. All we need to know, which is guaranteed for us, is that they are there on the stack  when the call happens. So the key question is, can we find that relocation record and  scribble on it reliably? The answer, luckily for me, again, no need to  relink it, is yes, we can. The link map entry has a pointer to the ELF data for each library  and libElf can interrogate this. We've added one dependency, libElf, but I'm pretty sure  that's not going to change incompatibly any time soon. So great. We can now put these  pieces together. We can make a shim library with the target library's SO name so that  the linker, when the linker loads it, and we put it on the search path. So, for example,  if I want to load libGL, there's no actual libGL inside my runtime with my containerized  application, but I can put a shim libGL there, which will be loaded by the linker. And during  the init phase of that shim library, I can deal them open the real library and all of  its dependencies in reverse order, as I've said. I can search the alternate library path  to do it, so I know I'm getting the host one because I want the one that didn't come with  my runtime. And then I walk the link map, I'm making this sentence do a lot of work  here. I walk the link map and scribble on all the relevant relocation records so that  all of the functions in my binary, instead of going to the plt fixup and then finding  the location of the stub function in my shim library, they'll actually go to the real one.  They don't ever realize they've been diverted because I've bypassed the normal symbol lookup  code. So, yeah, kind of we can basically do this. Now, obviously, I've simplified a lot.  There are some details and some problems. The self-shaving yet has not yet been invented.  So I'm just going to establish a couple of terms up front. I've called an isolated set  of libraries a capsule. There may be a better name. Naming things is hard. If you can think  of a better name, by all means, let me know. And then I've assumed that they come from  file system mounted at slash host. So within my charoot or container or runtime or bubble  wrapped environment or whatever, I have mapped in the actual host file system read only at  slash host. For example, it could actually be anywhere. Other problems. DL open can't  be called from inside a DL open namespace. It causes the C library to explode, which  is not fun for anyone. So I need to use this relocation record scribbling mechanism, which  I've already established, to replace the DL open inside a capsule with a wrapper that  does the following. It calls DL open instead of DL open. It does the path remapping and  simulink resolution relative to slash host or wherever I've told it the foreign libraries  are located. It doesn't accept RTLD global because DL open doesn't yet have semantics  for that. I'm discussing that with G-Lib C upstream. When we've hashed that out, it  will, but for the time being, I need to mask that. The reason I need to mask that is because  libgl does the... It DL opens itself, RTLD global, to expose symbols to its plug-ins,  which was fun in games when I figured it out, but until I did was a puzzling crash as far  as I was concerned. What else? OK. DL sim, which is how you get symbols from a DL open  library, now has to have a split personality. It needs a little bit of extra work. Remember,  we want this all to be seamless, but now there are two namespaces, not one. That means that  when something in our application calls DL sim, I need to transparently go, ah, you  may have meant this namespace over here and look in there, but I only wanted to find the  symbols of the directly targeted library. I don't want to look at any dependencies,  so I need to be quite careful. Again, similarly, I will scribble on the relocation record for  DL sim and replace it with a wrapper that does these kind of safety checks for me. Another  interesting thing, so the usage of libgl, which was the primary use case, varies quite a lot.  Some things link statically against it, some things DL open libgl, there's a whole bunch  of ways you can use it. So any time another library that's not necessarily one that I've  exposed is DL opened, I need to scribble on its relocation record as well, which means  that I similarly have to wrap DL open outside the capsule in the same way so that any time  something is DL opened, I scribble on its relocation record. So if it needs, for example,  a libgl function, it gets the right one. Extra problems. Currently each, this is kind of a  lie in my demo because it is fixed, but the patches have not yet been reviewed and accepted  upstream, so currently each namespace has its own glibc copy, which works a lot better  than you might think in that it actually works at all. But it does make anything to do with  threads quite, it's not actually a deadlock, but it looks like a deadlock prone. Alloc  and free is interesting, of course. As long as all of your allocation of freeing of a  particular pointer happens on the same side of the capsule boundary, you're okay, it works,  it's fine. If you pass a pointer to across the boundary and expect it to be freed on  the other side and you've got two glibcs, then, yeah, glibc immediately throws a hissy  fit and says, what are you asking me to free? I've never seen that pointer in my life, corruption,  pull the ripcord, bail out immediately, we have a disaster on our hands. So, yeah, there is going  to be an extra flag called RTLD shared, which is currently being reviewed, and what that will do  is allow me to put sort of the fundamental libraries like the glibc cluster have the same  instance on both sides. It's building on work that's there in glibc anyway because, as you can  imagine, there can only be one copy of the loader. It's sort of half already does this,  but it needs to know to do it for glibc as well, and that's kind of one of the things that I'm  feeding upstream. So, the workaround I had before was to replace the alloc and free clusters inside  the capsule, which kind of in a hackish sort of way tried to dispatch the pointers to the right  place to be freed. It doesn't really come up much when using glibgl because glibgl doesn't really  allow allocation to pass across the boundary. It's API, but for other libraries, it might be  relevant. Right, so, and finally, does it actually work? I'm going to postage the fortune here,  I'm going to attempt a live demo. So, these things have all been tried and worked. I'm going to try  glxgears and then if that works, I'm going to try open arena. So, where are we going? So, this is  a cheroot which has had the libgl surgically removed from it, and I'm going to try and launch  glxgears. You should see some spew from the capsule, which is where you can see it's mapped in the  different things, and we've got a glx running, and you can see there it says, oh, gl over and  wrapper cannot pass RTLD global, so I know I'm intercepting these calls.  Where's my pointer gone? Someone see my pointer? Oh, there it is. So, if we page up a bit,  you can see I've got, this is the default namespace, you can see I've got some shim libraries mapped  in here, which are stubs, they don't contain any actual code, they're just a placeholder,  so that the linker can map things in and then I can scribble on the relocation records later,  and then if I go up a bit, there we go, we can see there's a capsule there which has host libraries  mapped in, which is where the real libraries reside, and these are isolated from glxgears  except where I've redirected them with relocation scribbling. So, that was one test. Now, that's  a relatively simple test because it has libgl linked in, not static linked in, but linked in.  Now, open arena uses SDL, which is going to DL open libgl, which is kind of a more complicated  case because things are going to happen much later. If I'm really lucky, this will work,  and I've got about 50% with this demo, so let's see if we can get that percentage up a little higher.  Let's put the window here. Why is it the window here? Let me...  I don't know if I can move the open arena window up. Damn it. Let's put it on the wrong screen.  Maybe I can just turn this around. Demos, they never work, right? Okay.  Tiny, but you should be able to see open arena actually running with a diverted libgl.  Okay. I think that's kind of the end of the talk. And...  So, the code is there. It's all kind of published already. This talk as well,  if you're into that sort of thing, is available there. Any questions?  I didn't understand then what makes the, like, your open arena demo only 50% reliable?  Oh, that was kind of unrelated to this. It was a pulse audio not being happy running in a charoot  problem, which I eventually tracked down. But, yeah. It's one of those things. It breaks,  and quite often because there's a lot of moving parts. So, a colleague of mine,  so this work was originally, well, and still is sponsored by Valve, because they want to improve  game stability is one of the things. And so, a colleague of mine, Simon Vagitti, is working on  the other side of this, which is assembling all of the right libraries within the runtime,  sort of hybridizing your runtime libraries with the bits from the host that you need,  you need to pick the newest libc of the ones that are available, and a couple of other things.  So, he's working on that side of things, but those pieces don't exist yet. So, I kind of had to  hack it together by hand, which is why it was unreliable. It wasn't anything intrinsic to the  project. So, I would like to ask if you have support for RTDeepLink to go to the next symbol  in the chain? Sorry, to the support for deep linking when you load the symbol from the next  library in the chain. No, I don't think we do. This is pretty much just gotten to this stage,  so not yet. Going in a different direction. Sure. Could you use this to make symbols like  soft linking where you resolve a function if the library is installed, it uses the library,  and if it's not installed, just returns an error. Yeah, what happens when this one is,  actually, I didn't demo it because my auto-generated stubs direct everything, but,  yeah, if you can have, what would happen there if a symbol wasn't available  and you called it, you would immediately get a backtrace saying you called a symbol that  wasn't available in the target library. Here's the backtrace, where you called it from here.  So, yes. I'll answer your question. Yeah, but I don't want to get a backtrace. I want to,  so for example, I have a program and the program links to five different compression libraries.  Sure. And then if one of them is not installed, I would just want to say no, cannot decompress this,  you know, like try a different one or something like that. Yeah, you could do that. Okay.  Thanks. I was wondering if you can speak a little about what  Valve is planning to do longer term with this, where they're going.  Okay, sure. I mean, the idea is, I mean, pretty much as a laid out at the beginning, so, you know,  you need libgl or in the future, you'll need your Wayland drivers to come,  Vulkan drivers, sorry, to come from the actual system because they're the thing that's closely  tied to your hardware and there's no real sane way to put those cram all the different ones into  a runtime. You know, they would quite like a game that you bought 10 or whatever number of years  ago to keep running even though you're on a completely different system. So the idea is  that the game will come tied to a runtime, a version, like pretty much similar to Waythings  like Flatpak work, you know, you've got your application in a runtime, you'll have a game,  and it will specify, I want this flavor of runtime at this version.  But then you need to parachute in the most recent libc out of the two that's available,  and you need to parachute in the graphics drive, the stuff that's tied to the hardware needs to be  brought in from the host. And so that's what it's going to be for. And the idea is that  when my colleague's work is done, all of that will sort of be assembled into your sort of hybrid  runtime. So hopefully, the idea is that in the glorious flying car future, we will improve  game stability a lot, right? If you have an old console, you can plug a game in that you  bought X number of years ago and it will work. Now, they have the advantage that it's all baked  into the hardware, so they don't really have to deal with things changing. But they would really  like that to be the case, because what their thing is that they want to sell you stuff electronically,  and they want you to have the confidence that whatever you've paid for it goes on working.  So that's the goal here. Sounds great. Is there an ETA?  There sort of is, but it's tied to other things that I can't talk about. So what if there is a  different system that uses the same Mac, for example, a heap profiler that wants to replace all  mellocs in your entire program? What if there is? I don't know what would happen. It might work.  It would depend on the order in which things happened. Was that what you were saying, would  they clash? Yeah, I mean, can those interact? Is there a way to make it work? I don't know the  answer to that. I'd have to try it. I wonder if you could talk a little about the surface area  of GL that's relevant to change, because in my memory there's a couple of APIs where you put  uniform in, and then you honk a giant string of program text, and that goes across a terrifying  barrier into the drivers. But that is a function that says int and gives you a void star and stuff.  To be honest, I don't actually know that much about how GL worked. There was a possibility that  I'd need to learn it to do this, but I got to the point where it worked and then I stopped looking.  As you may know, there's a whole number of different ways of using libgl. There's the simple  GLXgears approach of just linking against it. There's the SDL approach of GL opening it and  finding the functions. There's the one where you get a specific function from GL, and then you ask  it for the function pointers that you want to call. But they all sort of work with this approach,  because it's only the small set of well-known functions that people can call that need to  be proxied in this way, and everything else just kind of works amazingly enough.  Okay. For the dynamically linked GLXgears, can you just show what it looked like when you run  through LDD? Yeah, sure. Let's see.  Let's see. Nope.  Where is it?  It's right. It doesn't seem to be able to get focused back. Where's it gone?  Oh, right. Sorry. Giant screen. Let me quit that. There we go.  There we go.  So you can see libgl is going to a shim. If we looked in that shim, we would just find stub  functions with no actual code in them. Well, there's a little bit of code. There's code that  generates a backtrace if you ever actually managed to call those functions. So the search path,  the shim search path you added to the container is LDD conf.  Okay.  Oh, okay. Yes.  Let me... Nope. It's going to work from here.  Uh...  Oh, isn't it? Oh, okay. I'll make it public. You're the first person to actually ask. So,  yeah. Okay. Any other questions?  I'm going to make it show the LDD output again. Okay.  I saw that you had... So why then here is... So you see like the fit to last line there.  Why is LDD saying that GLX gears, which you didn't have to recompile or anything,  why is it dependent on lib capsule now? It's coming from the shim. The shim depends on  lib capsule. So lib capsule is kind of rather than baking the code into every single shim,  they link against lib capsule, which does that.  Okay. That seems to be old questions. So there it is. Segregated dynamic linking.
libcapsule_Rb1wdDdOizc_0_155.png libcapsule_Rb1wdDdOizc_156_2963.png libcapsule_Rb1wdDdOizc_2964_3047.png libcapsule_Rb1wdDdOizc_3048_6587.png libcapsule_Rb1wdDdOizc_6588_6719.png libcapsule_Rb1wdDdOizc_6720_9947.png libcapsule_Rb1wdDdOizc_9948_12659.png libcapsule_Rb1wdDdOizc_12660_15155.png libcapsule_Rb1wdDdOizc_15156_15623.png libcapsule_Rb1wdDdOizc_15624_20363.png libcapsule_Rb1wdDdOizc_20364_28931.png libcapsule_Rb1wdDdOizc_28932_28991.png libcapsule_Rb1wdDdOizc_28992_29267.png libcapsule_Rb1wdDdOizc_29268_30155.png libcapsule_Rb1wdDdOizc_30156_30227.png libcapsule_Rb1wdDdOizc_30228_30383.png libcapsule_Rb1wdDdOizc_30384_30935.png libcapsule_Rb1wdDdOizc_30936_31259.png libcapsule_Rb1wdDdOizc_31260_31583.png libcapsule_Rb1wdDdOizc_31584_31727.png libcapsule_Rb1wdDdOizc_31728_31775.png libcapsule_Rb1wdDdOizc_31776_31871.png libcapsule_Rb1wdDdOizc_31872_31931.png libcapsule_Rb1wdDdOizc_31932_32027.png libcapsule_Rb1wdDdOizc_32028_36647.png libcapsule_Rb1wdDdOizc_36648_36851.png libcapsule_Rb1wdDdOizc_36852_38975.png libcapsule_Rb1wdDdOizc_38976_39047.png libcapsule_Rb1wdDdOizc_39048_39251.png libcapsule_Rb1wdDdOizc_39252_39551.png libcapsule_Rb1wdDdOizc_39552_39983.png libcapsule_Rb1wdDdOizc_39984_40931.png libcapsule_Rb1wdDdOizc_40932_40991.png libcapsule_Rb1wdDdOizc_40992_43067.png libcapsule_Rb1wdDdOizc_43068_43199.png libcapsule_Rb1wdDdOizc_43200_43535.png libcapsule_Rb1wdDdOizc_43536_43619.png libcapsule_Rb1wdDdOizc_43620_44243.png libcapsule_Rb1wdDdOizc_44244_44783.png libcapsule_Rb1wdDdOizc_44784_44879.png libcapsule_Rb1wdDdOizc_44880_44939.png libcapsule_Rb1wdDdOizc_44940_45767.png libcapsule_Rb1wdDdOizc_45768_45827.png libcapsule_Rb1wdDdOizc_45828_46945.png