Mach-O linker in Zig: linking in the era of Apple Silicon
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Konka, Jakub
One of Zig's main goals is seamless cross-compilation from any target to any target. This includes macOS and Apple's recent switch from x64 to ARM64 architecture with the introduction of the M1 SoC. This talk will delve deep into the internals of Mach-O linking in ZIg for both x64 and ARM64 targets, discussing new strict requirements the ARM64-version of the XNU kernel places on the binaries such as embedded code signature and inode caching, and how Zig mustered up to address and solve them for fantastic developer experience on macOS. I hope for this talk to deliver on two fronts: 1) to showcase and convince the audience that Zig takes cross-compilation very seriously and is a language worth learning, and 2) demistify Mach-O linking in general. Yet another great strength of Zig is that it is a very readable language, therefore, even though the code examples will be in Zig, it will not hinder anyone's ability to enjoy figuring out the finer points of Mach-O linking on ARM64 aka the Apple Silicon which are language-agnostic. Finally, there naturally will be coding examples interleaved here and there to demonstrate the different linking concepts that the talk will touch upon.
 Hi, everyone. Welcome to ZIG Dev Room at FOSDEM. My name is Jacob, and it's with great pleasure that today I will talk to you about linking in ZIG. And hopefully, I hope it inspires you to try ZIG on your own, have fun, and maybe contribute. So without further ado, let's get cracking. So as I mentioned, today we'll talk about Maho Linkarn ZIG, and specifically what change or what do we have to do to actually satisfy Apple Silicon for the Maho Textual Work, or handcrafted Maho binaries. Okay, so first things first, who is this guy? As I said, my name is Jacob. I guess most people usually associate me with my GitHub, Twitter, and all the other handles, KubeCon. I am a software engineer at Microsoft, and I am a ZIG core team member. So today, I'd like to propose the following agenda. Don't worry, there will be some actual coding involved as well. We love those, right? So I can skip on this. So first of all, we'll briefly discuss what linking is, actually. Then I would like to introduce my friend Maho. You might know him, you might not know him. It's kind of crucial that we discuss this format for going further, because this kind of has an impact on linking in general, how we do that for Mac OS and ZIG. Then we're going to move on to briefly discuss what changed in Apple Silicon, what new requirements Apple is putting on linking, on the binaries themselves, and even shared libraries, of course. So then finally discuss incremental linking in ZIG in the cell phone compiler. So this is our own in-house linker that satisfies all of the requirements of Apple Silicon, and you can actually use it to perform incremental linking target in Mac OS. Finally, if you have some time, and I'm pretty sure we will, I can't skip this bit, I'll briefly show you that we are able already to cross-compile C to Apple Silicon with ZIG. Okay, so that's good cracking. Right, so what is linking anyway? So most of you, I think, associate linking with this traditional approach. In this traditional approach, we have the compiler and linker separate, we have some source files in the source language, like Rust or C, and then we compile them into the things called intermediate object files. These are also for Mac OS, these are also Mac OS files, but they're not in their final form. So you still need the linker to actually read those, like multiple of those, put them together, link in any, like specifying any, if we want to link with the Shared Library, like LAPS system, we also have to specify the end, and then we end up with the final artifact, which is the executable Maho. Okay, so in this approach, compilation, as I said, is separate from linking. Source changes always lead, sorry, to full relinking, right? So if we change one source file, we will only have to update one intermediate object file, but the final artifact will have to be redone from scratch. And this entire chain is LVM favored. What do I mean by this? I mean that actually in LVM, this is how everything is laid out, if you want to pull an LVM. Okay, so how do we do this in ZIG? What is the, what is the ZIG's deal when it comes to linking? So actually, we take the concept of compiler and linker, and we tightly couple them together. So instead of having two separate drivers, we have one. While this might be not as generic as what you would have in some traditional tool trains, this gives us this possibility to actually do a lot of more targeted things that otherwise would be impossible. What do I mean by this? So for example, we can do incremental linking, and what does that mean? That means that we actually added the final binary in place. So instead of relying on intermediate object files, which we don't have in this case, we actually only overwrite bits in Maho that need that. So this leads to short, up-to-turn around time. We don't have to wait, I'm pretty sure most of you have experienced this once, or at least once in your life, that when you do, for example, compile a massive project, the very final stage, even if you don't change that many things, this linking bit is constantly the bottleneck. It actually takes a long time to link everything, because you have to link again. So if you remember, you basically toss the macro binary, and then you recreate it. Whereas with Zig's approach, you don't have to do this. You only have to override specific bits that changed, like specific symbols, I guess, but we'll get to this. Okay, so what is incremental linking? So I should first of all add and Zig. We only plan to have this on in the bug mode, and this will be on by default. While in the bug mode, this is for two main reasons, is that incremental linking inevitably generates larger artifacts, because we need all the padding we can get. We don't want to be shifting, like rewriting sections or even major chunks within the final artifact too many times, because that will guess what leads to actually, in the end, rewriting the entire artifact, which will put us in square one with traditional linkers. And we cannot support linker time optimization, because, well, we only have one fast. We do it, like, on the fly, like hot code swapping, if you will. And that's our goal, in fact. But on the positive side, incremental linking leads to shorter compilation times. It means quicker prototyping, because you don't have to wait so much. You just change the code, update it, and then it recompiles on the fly. It should be instantaneous, and that's what we are working towards. And so basically doing the typical prototyping route of write, compile, change, recompile, change, recompile, change, recompile should be very, very responsive, like super quick. You won't have to wait too long, and that's exactly the point of actually working on your app, your program, whatever, in the bag mode, right? You want to basically, you want to have a very quick turnaround time. Cool. Okay, so with this out, let's meet Maho. So Maho is my buddy. Maho stands for Mah object file format. Maho is a time for kernel. It's used by all Apple operating systems. The good news is, the file structure is actually straightforward. The bad news is there is no up to date docs. So if you want to get into this stuff, well, yeah, you will need to basically browse the web, see whether there are some maybe guidelines, maybe somebody wrote a blog post, but there isn't too many docs. So you will have to go through the source code eventually. Thankfully, Apple has released most of the stuff on their Apple open source website. And incremental linking actually is difficult, especially with the latest additions or requirements by Apple. So we'll get to it later. Okay, so before you is the overview of the Maho file format structure. It's actually, as I said, it's pretty straightforward. So we start with the header. This is where we insert magic number architecture. So it could be x8664, could be ARM64 for Apple Silicon. Also file type. Is it the intermediary object file? Or maybe this is the executable, or maybe it's a short library. Then directly after header, we've got load commands. So load commands, I guess you should think about them as your table of contents. So if you write your own parser for Maho, you will use those load commands to actually directly navigate through the entire file. Being able to locate things like the text section where the compiled machine code lies or the stops for the imports, things like this. So every segment load command describes the segment. It maps into virtual memory. So it's got the map into virtual memory. It's got the start address and the actual virtual memory size. And this is very important because this is what the loader uses to insert the binary into the address space. Okay, every segment command can have multiple sections from zero up to 255, I think. So in total, 256 is maximum. Oh yeah, zero is like the indexing. It doesn't matter. Right. So sections describe the actual data that is mapped into memory. For example, the text section will contain the compiled machine code. The steps section will contain the trampolines that the loader will use, or your program will use to actually work out what or where to call the import symbol, like write or exit from libc. And you can also have loader info. So for example, the opcodes that drive the dynamic loader and resolving those imported symbols. Okay, so so far so good. Let's quickly go through a couple of those. So we've got the map header. If you run auto, which is the fact of standard tool provided by Apple, they can use to have a look what the file structure actually is. We can have a look at the header. So that's dash H. The header structure is pretty simple. I have omitted a couple of fields because they're not pretty interesting. The main thing is that you've got magic for 64 bit maho binaries. This will always be feed facf or 32 bits. There used to be feed face, which I thought had a better ring to it. But hey, we've got file type, which in this case, well, in the cases that we're going to be describing here, it will always be executable. So that's mh underscore execute, number of commands is 15. In this particular example, and the size of commands in this example is just above 1200. We can have also additional flags. In this case, mhpy signifies that this binary is a position independent executable. And we'll get to all of this later on. Right, so now, navigating for the load commands, let's have a look at the segment command. So to print all the commands with auto, you would just append dash LV, where we have here. This is the second load command, actually, because we started from zero. Load command zero is always standard for executables. This is the reserve to virtual memory address space of four gigabytes that's reserved for the null symbol, or for null in general. And maho always, so basically every other segment has to start at four gig. It cannot be below this. So this in front of you is the text segment. So as I pointed out, we start at the virtual memory address of four gigabytes. Now the interesting things I should point out are crucial for the dynamic loader and the kernel not killing your program is that so command size always has to be eight bytes aligned. Okay, always, whatever that is, always has to be eight bytes aligned. Now, virtual memory size always has to be page aligned, always. File size also, and initial protection, well, that depends. Every segment will have a different set of flags for text segment that's read and execute. Now page aligned, that depends on the architecture for arm 64, that's 16K, for Intel, that's 4K. Okay, so far so good. So in front of you, this is an example of a text segment. As you can see that section text starts at address four gig plus 1000, and it extends all the way to 1100. Then, and this is very interesting, we have zero part of space, and only then we have, for example, stabs at address 56,000. Okay, now the important thing to note here, and this if you'll be working on an incremental linker or linker in general, you don't have to worry about zero padding or empty space between sections within a segment. That's fine. They don't have to be adjacent. Obviously, if you have a new space within a segment, you're going to be exploding your binary in size. But this is still fine. I mean, for incremental linking, that's inevitable. And we want to have as much padding as possible so that we don't have to shift everything at every second right or so. Okay, another thing to note, the segments themselves have to be laid out contiguously in memory. So we cannot have any empty space between them. No, there cannot be any space. So if text segments end at 58,000, then for example, data con segment has to start at 58,000 and then extend all the way to, for example, 5C, 1,000 for the next segment to start. Okay, moving on. Okay, now my favorite segment, the linker segment. Now, this is the odd one out, I would say. And why is that? Well, check it out. So first of all, the virtual memory size is page aligned, perfect. But file sizes are not. It is not page aligned. And there are no sections. Oh my God, link added segment never, well, I've never seen link added segment have any sections. So first things first, let's talk about the file size. It doesn't have to be page aligned because the last segment within the binary. And in this case, I assume it's link added. It doesn't have to be. But, well, I've never seen a different layout from my file format, I guess this is kind of standard that the link added comes last. And it will have to because of the code signature, but we'll get to it. So the file size can be exactly what the sizes of the link added segment. And then the loader will actually expand it to be page aligned. It's fine. And then no sections because the link added segment get the sections of link added segment get their own load commands. So, okay, so let's describe a couple of link added. I call them hidden sections, but actually, as I said, they get their own load commands. So for instance, what we have, we have a thing called dynamic loader info only. So this is the LC-DYLD info only. And this section contains all the op codes for the dynamic loader. So it's the rebase info, the binding info, lazy binding, and export. So everything to do with imports and exports. If you want to export a symbol, it has to be encoded for an export try within export info section. Or if you want to import something, how you resolve the addresses, where the DYLD stop binder should then populate them. So basically, you write the address into, which cell in the memory, that's also specified within the binding and lazy binding info. And rebase, of course. Okay, the next thing is that we have the symbol table. That's LC-SIMPAP. This one usually, so this is the symbol table that contains all of the locals, globals, and extras, all of it. We've got two bits that go within this load, the this load command describes two bit. This is the symbol table and string table. Important thing to notice. A thing that we discovered recently is that the symbol table has to always precede the string table. Meaning that if, for example, you end up expanding the symbol table so much that you have to move it somewhere else, you have to move string table as well. It doesn't have to, they don't have to be adjacent, but symbol table has to always precede the string table. Otherwise, the DYLD will refuse to load the binder. Okay, next thing is the LC-DISIMPAP. This one contains usually in the indirect symbol table. So this is the table of all the imports of all the external symbols. This is also non-optional. If you actually want to link with Lipsy properly, like actually import the symbols. And finally, a very important load command that is a new addition on macOS, starting with macOS 11 and Apple Silicon is the LC code signature that load command contains the code signature of the binary. And at the very minimum, the ad hoc code signature is basically a couple of hashes of the entire binary. Without this, the kernel just will refuse to run your binary. This is very important on newer Macs running on Apple Silicon. Okay, so let's move on to the changes with that Apple Silicon has brought to the table for Macs. So the main three ones are that the binary has to be ad hoc code sign. So that's what we're going to discuss next. It has to be also app positioned independent executable. So for Intel, it is still possible to create a static binary and it's going to be fine. Even though it's not recommended, but you can still do this, this is not possible or up on Apple Silicon, everything has to be position dependent, meaning that you cannot have any absolute addressing embedded within your binary. That's a no go. And finally, the binary has to be moved to a new I note, if it's modified in place. But this is a bit tricky. So I will let you think about this for a bit longer. But basically, what we'll describe it later in more detail about what happens is if you modify the binary that's already was run in place, the kernel will cut. Okay. So moving on, let's discuss briefly what ad hoc code signature actually is. So first things first, this is the last link at its section. It has to be last because it has to calculate the hashes of the entire binary excluding obviously itself. It is pointed to by LC code signature load command. And as I said, it contains hashes of the entire binary. Now the tricky bit is it is in big Indian order. Now, on the outside, I guess at the face of it, it kind of looks weird because the entire binary is little Indian. But if you think about this, this code signature structure or whatever goes in there is actually used for more than a couple of code signatures, it can be used for full blown code signatures that are required if you, for example, submit your app to Apple to App Store. And it's in big Indian because predominantly the network by toward there is actually big Indian order. So that's why so that you don't have to do the unnecessary conversed between little and big. Okay. So this is the basic structure of the code signature that goes within within it that's embedded inside the macro binary. So we have the header. We have this, I'm not ignored there. This is pretty standard for ad hoc code signatures as far as I know, it will usually be fade 0CC0. Then we have the code directory, which is another structure embedded within the code signature structure. This one has got the CS ad hoc flag specified that this is the ad hoc code signature important bits that are configurable if you so wish to our hash type and page size the hash type for like, for example, it's shot 256 back and pick. There is a list of different hashing algorithms you can pick from. And then the page size, the page size usually matches that of the architecture. So for ARM or Apple Silicon, that's 16k. And following this, we basically get like a long table, long list of of hashes of each page. So basically we divide a binary into pages, and then we calculate the hash of each page. And that's it. Obviously, every hash should be of the hash of the page should be, should span exactly one page except for the last page, which does not have to be the full. And page size because the last page. Okay, so onto the requirement or the change that the kernel is actually caching inodes. So this is true since my co is 11 and Apple Silicon. Now what is happening? So imagine the scenario that you have some Hello work program, and then you execute it, then the kernel is caching the inode, noting the actual inode and the code signature that this Hello has embedded within. Obviously Iran, Iran's it and then sends yeah, cool, no problem. Okay. Then imagine that we modify it in place, we actually calculate recalculate the code signature so that everything is valid. Then we try to launch it. And then the kernel goes into registry notices that okay, I already got I know like this, but hang on the code signatures are much. Oh, something is wrong. Somebody is being funky here. So it sends the sick co refusing to launch the modified binary. So in general, I understand the principle that's meant to be more secure, I guess, things like this, but for incremental linking, especially for sake, this is a bit of a problem because as I said, we want to modify it by this in place. And because of this, we actually have to go through those additional hoops of copying the entire contents of the updated macro binary into a new file and then swapping and then basically creating a new inode and then rewriting or deleting the old one leaving the new one in place so that the user doesn't see any difference. So this is kind of costly, but this is what it is. And this is how it's done. So we underneath the sea, like under the sea, we actually, sorry, behind the scenes, we actually move the file in place after we've finished writing to it and recalculating the code signature so that the kernel so basically that we change the inode. Okay, so let's move on to ZIG. So how is incremental linking handled in self hosted ZIG compiler? I should point out that the self hosted is still a work in progress. So you should expect breaking things, you should expect sex, sex faults, things not implemented by working on it. And actually we're making some pretty good progress. So it's pretty exciting times. Okay, so just to recap, this is the self hosted approach. If you remember, we have compilation coupled with linking. So we actually write to the macro binary the final artifact in place. So what does it mean actually? So for example, imagine we have a function foo. This function will get its own symbol within the text section within the text segment within the text section. So we get its own symbol with star address and then the size. Okay, when we write that into the text section, the compile machine code, we also write a new entry in the ZIG global author table. Now, why call is it that's because a global it's a God is kind of taken by by the traditional, I guess, traditionally God is part of data or data con segment. And actually it's for ZIG, it's kind of a misnomer because it's not really a God. It's actually more of a jump table, as you'll see in a minute. So when we write, as I was saying, when we write, we create a symbol for foo. At the same time, we add a new entry in the God that features basically a trampoline that any other symbol that references foo or is calling foo at the call side will reference the entries that got rather than the symbol itself. And then by referencing the entries that got so calling to branching to that little snippet here, he will be able to automatically compute where to jump to in order to call foo. Okay, so just to do a bit of math here to show you how it works. First thing as far as if you notice, we've got the other instruction here on arm 64, which will get the address of the label foo displaced by that value here and 349480 is equivalent to actually 55528 in hex. Now, if we take the address of that instruction, which is 56560 and we subtract that value in hex, we'll get, well, surprise, surprise, the address of symbol foo. And this is exactly what we are after. Okay, cool. Now, I'm going to leave it at this and we'll come back to the one after the demo to show you what this implies and why it's really useful. Okay, so now to show you the principle of how we actually use that as I described briefly, if we have, for example, start and we call foo, as I mentioned, we actually don't call the actual, we don't compute the displacement to the actual symbol foo, we actually compute the displacement to the entry, its entry within the zig global offset table, which is 56560. And then by doing this, we then, and we basically invoke these two op codes to work out where foo actually is. Then we move the contents of x zero, which we populated with the actual address of foo into the link register, and then we brand from that link register to actually call foo. Okay, so this is just a slight indirection versus what you would actually do if you have it, if you had it properly optimized and you are actually branching directly to the symbol. Okay, some demo time. Cool, so first things first, let's create some source. Okay, so we're going to force them here. What we'll actually do, we will, just to show off that we've got some things already working pretty well, we will import two symbols. So I should point out here that you could write raw assembly targeting, basically write raw assembly to invoke the syscals on macOS, but this is not recommended because Apple does not guarantee that, for example, the syscal numbering will not change from version to version. So we don't do that. Instead, you're always recommended to use libc to actually invoke the syscals. So we've got, we've got write and that one accepts three u sizes. Actually, they should be different types, but I think we're missing the stuff from the code gen still when it comes to expanding or like growing or shortening end resolutions. So we'll just stick to u size and returns u size. I will also use exit for specifying the return code that we can then inspect when the program returns. Then we will export start. Now, we should really have weak linkage to main, but we don't actually do that. We do the custom entry point of start for the time being. This will change. We will actually use main as recommended, but hey, we didn't get to it yet. Let's print some message. So the message will be starting like hello. Oh, why not? Hello, Fosdm. Oops, sorry. So pointer to end means that we will actually convert the pointer to that C string or string, an array of bytes to an end immediately. That's because write requires that as the second argument here. So actually, let me write that out. So this is the file descriptor. This is the message. Actually, it's pointer to message, I guess. And this is length. So this is the length of the message. And that's the return code. Let's make it message. Actually, it's going to be easier to refer to it. So this is an int as well. And it will go into the second argument and write. Now, as far as length is concerned, here we have, I don't have to calculate that by hand now, three, six, seven, ten, thirteen, fourteen, fifteen bytes to write. Okay. Now, we don't care about the turn value here. Now, FD is going to be one for stood out. We've got the message and we also have the return code. Sorry, the length, which is length. Okay. Actually, let me move that immediately to another function just to demonstrate stuff later on. We're just going to put it in print. Okay. And then here we'll call print. I'm also called xin with zero. Everything works fine. Cool. So we've got our simple zig source. So now let's try and compile it. So let's use the stage two to do this. And build x, the fosdm zig. So this is the standard way you would build it doing a one off. And this is the standard way you would actually work with different languages, like Rust or C. You just rerun the entire thing and then it generates the artifact here, which is the fosdm executable. And then if we run it, we get hello fosdm. So the basics work. But actually, what I want to show you is this magical flag called watch. What happens here is that we entered an incremental linking mode, meaning that so we already have compiled it. You can rerun it to see that it works and it works. But now we can do modifications on the fly, which should make it a recompilation or relinking. Actually, in this case, recompilation on relinking super fast. So for instance, I can do print a couple of times. And what will happen here before we get to this, I'll tell you upfront, this will cause, actually, let's do it a couple of times. Okay, this will cause the start symbol to expand in size. We will actually need more size than we have allocated it initially when we compiled it initially. So what it will mean in order to grow, we'll have to first of all lift the entire symbol and then move it somewhere else. And normally, what that would lead to, this would lead to rewriting the entire x section, probably stabs as well. Maybe not, maybe stabs would stay. But if everything was so adjacent and squeezed in, like you would have in different tool chains like Rast C or C++ or whatnot, then you would need to actually indeed perform the full relinking. But because we leave padding, this is not needed. So before I show it to you, let me quickly show you what the actual, my mouse is gone. Oh, there it is. Cool. What the actual generated Mahobailin looks like. So as I mentioned to you, we've got all the segments and everything laid out like I described it. We have code signature, everything, you know, so far so good. Then within the text segment, we've got the start symbol. So this is the content of the old start that we have just modified, but we haven't recompiled yet. So we actually, we have this call here, which is the branch unconditionally to this address 560, which is an entry in the Zik code. And if you notice 560 will calculate the displacement to a symbol, which is 349,476 bytes below that address. And that points to print, because this is where we want to basically get the address of print and call it. And this is where we actually call the results address of print. And then we call exit symbol from let's see. So now the point to note here is that, so we even started a couple more, as you see here, we even said a couple more calls to print within start. So start probably doesn't have enough space here. And it doesn't, because if you notice we have, we only had one and now we have, I think at least 10 more, and we only have space for four opcodes more here. That's not, so it's the 102C all the way to 1038. That's not enough. So what will happen? In order to grow start, we will move the section below print somewhere, starting a this address or below. And it will be much larger in size. But actually what will happen, is that we will have to re-write everything. We will actually have to re-calculate the addresses again and things like this, because obviously we would actually, start will be now below print, and the displacement will be negative as a positive things like this. And the same thing happens with the other ones. So we will have to re-calculate the address again and things like this. And the displacement will be negative as a positive things like this. And the same will apply when we modify print in a minute. Okay, so let's have a look. So we did this. Now let's do update. Okay, as you can notice actually, this is where the update was. We have much less output than we had before. Look, here was the first setup. We were actually populating the metadata, like load commands, wherever it goes and things like this. Whereas when we update, we actually only care about updating automatically. So we actually care about updating as only those bits that need to be updated or that are required to be updated by Apple, by the dynamic loader. So as I was saying, we grow and relocate start from the old address to the new address. So this is the 10-8-C. We write the new offset table entry for that symbol at its address. So instead of modifying every other symbol within the text section, we basically only update the corresponding entry when the Zig got, and that's it. Okay, so far so good. So let's, before I show you the change structure, let's try and run it. Cool, we have HelloFuzz a couple of times. And now let's reopen that binary. And what we have here, so the old start actually stayed because this is now part of what we call a free list of text blocks that we can reuse if a symbol of a size that's not bigger than the old start fits. And then we'll just stick it here instead of writing it at the end so that we can conserve the space. Now print state, and as you can notice, in the new start, we still branch to the same symbol, which is the second entry within the Zig got, which is here. Okay, so cool, so far so good. So what is happening now? Let's now try and do a slight twist. So let's try, we've got the start, we have print. Why won't we write it a couple more times so that we will have to grow and relocate print as well? Okay, I think that should be plenty to do this. Let's do update. Okay, cool. Yeah, look, growing and relocating print. So again, we moved it from the old symbol address to the new one. And this one is again, this one now is following the new start. And same, sorry, the same thing applies now. If I reopen it, the binary here, what we get within the text section is that the new print, or first of all, look, start is calling print at the same address. Why? Because the entry in the Zig got, so the key in the in the dump table has not changed, the index is the same, but the entry, so where we point to, has been updated to point to the new address of print, which is this one. Okay, so far so good. And this, this is the whole principle. This is the entire idea behind incremental linking on, on, in Zig effectively speaking, this is how we want to manage everything. Okay, cool. So that's, I think, all about incremental linking. I'd like to show you guys. We've got still a couple of minutes. Yay. So let's keep going. So now the bonus. So cross compiling C to Apple Silicon with Zig. So Zig comes with its own C and C++ compilers. They're actually under, they are like, like, thin wrappers for clan, because this is in, in, in this is not self hosted, self hosted, we'll also have it. We'll also have LVM as an optional backend, but in this case, I'm talking about stage one. So this is the compiler that's written in mix of Zig and C++. So this one is based on LVM. It uses LVM for code, code gen and it uses LLD for linking. So with stage one compiler, we actually did traditional approach, which is not, you know, obviously it's devoid of incremental linking. We create intermediate files. And then we actually perform the full linkage at the end using LLD. Now the problem with this is that when Apple Silicon came out, LLD was not equipped to actually handle code signatures, not to measure trust compilation, but it still doesn't know how to handle. There is some, there is actually a lot of work being put into LLD for macOS in LLDM, which is great. But it's, you know, it's basically it's, it's, it's a, you know, it's a big project, right, right, the linker or rewrite it, I guess from scratch, because that's what's happening. So anyway, so our idea was, because this is going to take a while, and we obviously we're working on our own in-house linker, but still it's also not, not, not ready to handle everything. And in what I mean by this, it cannot handle the intermediate object files. So instead, what we noticed was that we can modify the output of LLD, kind of post mortem. So we can actually reparse the binary, the Maho binary, look for certain bits, the arrow of interest, fix them, and then at the end, append the code signature, and what was your uncle, and this should work. And guess what, it actually does. So yes, let me recap this. In stage one, on, for Apple Silicon, we actually, you actually don't need to, for example, fall back to system linker. So the linker that's supplied by Apple, you can use LLD in 99% of the cases to generate a valid Maho binary that's been code signed, ad hoc code signed properly, using RZ toolchain. It's just, just that it's not inside LVM, it just happens post mortem. So let's have a look. Actually, everything I've been showing you here, all the tooling, I'm actually talking to you from, um, um, sorry, from, uh, M1 MacBook Air. And all the tooling that I used here to actually compile stage two was done using our own post mortem, uh, embedding of code signatures within LLD's output. So to prove my point, um, that first of all, let me show you how you compile, for example, simple C with ZIG. And this is it. It's basically ZIGCC simple C. Now this generates A out like, well, you know what, the standard way. And then if we run it, we're going to get Hayder. And let's compare that with, um, simple C. That's indeed what it was. Okay. So now let's get rid of this A out. Now I am going to quickly associate into my local, um, Linux box. So we have here, 5.9 kernel. And this is the Intel, um, Intel box. Okay. Let's go to dev. Let's go to ZIG. Examples. Let me show this. Now, uh, let's have a look at simple C. We have pretty much the same, oh, we have hello, instead of hello world or Hayder. Okay. Now let's do the same thing. So let's first of all compile this C for that, um, target architecture and that operating system. So there we go. Now A out is actually an else. Perfect. So we should be able to run it. We did. Everything works. Awesome. Now let's do the same. But this time we are going to cross compile to macOS arm. Okay. So let's try it. Okay. Something happened. We still have A out. Let's see what that is. Okay. Cool. So now this is arm 64 executable. It's a pie and all the other flags. Awesome. Let's exit from that machine and let's actually copy. Okay. We don't have A out here. So no, no, no monkey business. Let's actually copy it from the server. So that's dev, ZIG examples, A out to here. Now if everything went fine, let's double check. Yep. If everything went fine, we should be able to run it. So let's have a look. Ta-da. It runs. So actually with ZIG, no kidding, you can do cross compilation of simple C or simple ZIG from anywhere to macOS, Apple Silicon and Intel. For Intel, actually, I can show you that we can do this here on this machine. Bear with me. So build bin ZIG CC. And then we have simple C. And then let's do target x8664 macOS. Okay. File A out. Co is executable for x8664. And it works. Awesome. So yeah, I think that's all I have for you guys today. I hope you enjoyed it. I think we've got more than, I think, 15 minutes. So, or maybe even close to like 20 or maybe even half an hour depends how everything is going to get organized for the actual FOS then. Obviously, I'm recording it a couple of weeks in advance. But I hope it's been interesting. Thanks for watching. And I hope you have lots of questions. I'd love to have loads of questions, especially for this session right now. Obviously, I'm available. You're convenience to answer anything you have. If anything wasn't clear, please let me know. At the same time, I'd like to strongly encourage you to try out ZIG. If you like it, contribute. ZIG's community is really, really welcoming. We welcome any and all contributions. We're just a really friendly bunch of people. We come from the first background. So, and there's plenty of work. So, if you have anything that is of particular interest to you, the sky's the limit here. Oh, and if you want to get in touch and be it directly right now, after the presentation at FOS, or even afterwards, feel free to ping me a message on Twitter or ping me an email. That's my email address. Or find me on GitHub. Cool. So, thank you again, and have a good day. It's funny how every single talk has the part where we are shutting down the recording that ends up being part of the recording itself. Okay, I think we are live, and we were probably already for a while. There's a bit of delay. Yes. I also recommended you mute yourself. Okay. Hello, everyone. Welcome to the Q&A with Jacob. Here's what we should do, I think. Let me do a super short introduction. Then I leave Jacob to talk about other things that he's working on, because his work on linkers is never done. Link outers can never rest. But in the meantime, if you have questions, write them down. I'll take care to stop Jacob at some point to answer them. So, I think that we should start by letting everybody know that Jacob has been doing an amazing job with linking. In the presentation, he talked about the incremental linking process and stuff specific to Apple Silicon. But that's not a full story. He's also now working on another linker, which he will talk about now. But just so you know, in the community, since he's been such an expert about macro format, people have started calling him the Macho Man. He doesn't like the name. I can understand why, but we like it too much. If you come to the community and ask Macho Man, that's Jacob. That's the thank you. That's how the community thanks Jacob for his efforts with Apple Silicon. Anyway, so as I was saying, just a super quick recap and then Jacob, I'll leave it to you to explain all the details. Incremental linking is a tool mostly, I would say even exclusively, geared towards development. Jacob explained this in the presentation. So the incremental linker is not a good linker by standard terms of efficiency. It's supposed to just be fast. And it does a ton of weird things to just be linking as fast as possible and implementing incremental linking. So this is a thing that is useful just for the development process, like you compile, make it run, make a change, recompile again, et cetera. This is one project which is separate from the main actual linking for releases. And so this is, I would say, more traditional linking, I guess. And this traditional linking has the issue that, for example, right now, Jacob, you can correct me if I'm wrong, but I believe that LLD, so the linker that is bundled with LLVM is not able to support everything that needs to be able to allow us to do code signing when cross-compiling, right? Yeah, correct. Yeah. So right now, we have this implemented as a hack where you wrote the code that basically reopens the executable after the linking stage and tries to patch things around. But this is not working 100% of the time based on the space available, et cetera. So this is what Jacob is working on, an actual linker for non-incremental linking, for normal linking. So Jacob, please let us know how are things going? They're going pretty well. I can actually do a demo with you guys. Sure. Yeah, so it's called ZLD for now, but people were saying that maybe we should rename it to something else. So it might be Mahuman, who knows? No spoilers. I need to get rid of the name, but anyway. So yeah, so the problem is that LLD is not tailored towards Apple Silicon at all. So code signing is missing. Actually, cross-compilation doesn't really work for either the Intel target, so X8664, nor Apple Silicon. So cross-compilation is just not possible with current LD. We do some, as I was showing in the recording, we do some like hot patching of the binary after it's being sped up by LD to actually support some cross-compilation, but it's not ideal. Things are missing. If I wanted to make it ideal, I would basically have to disassemble the entire thing and put it back together, which is just silly. We might as well write a linker, right? There's one caveat that, so the ZLD started only two weeks ago. So obviously things will change. I'll try to make it more generic, you know, the factor here and there, but my main idea, my main hope is just to have something that works for like, I don't know, 90% of cases. And we can use it as an experimental linking back-end for existing ZIC and anybody can turn it on with a flag or something to see how it behaves. Yeah, it's actually kind of promising. Let me try and share something. I'm going to share the screen. Okay, Lawrence, you'll have to tell me if you can see that. Yes, it's working. Oh, for the size, let me full screen. I can make it a bit bigger. Yeah, okay, this is reasonable. Yes. Okay, cool. So yeah, so the ZLD, that's a separate project to ZIC currently because I mean, I'm not sure we can actually give it a separate project because hopefully if it matures enough, it can be used by anybody. So you could actually use this as a linker for Rust, for C, C++, you know, instead of like a system linker or LD. It's a traditional in the sense that it basically reads static archives, shared libraries and intermediate object files and it should, you know, like create the final artifact. It doesn't do all those things yet, but we're getting there, I guess. Right, so let me remove all this stuff just to start fresh. I got TLV, like I got threat locals started working today this morning. So this is kind of, they're still up to clean up, but it's getting there. But anyway, so here we have the hello ZIG and believe it or not, creating a hello ZIG is actually pretty heavy. The amount of information that you get passed in from the compiler is quite a lot. We need two things. We need the main object file with like, you know, all the symbols for allocators, things like this. And we also need compiler runtime, like compiler RT. That's also, that bit is also passed in from ZIG. So we need to do those two object files and we can actually create a final artifact. So with this stuff, we can let me quickly just build the object file. So there we have it. Yeah, it's a, it's, oh yeah, you can't read from here, but it's a standard intermediate object file. And where was the LD? So this fancy like path is the path of the cache to the compiler RT. When I actually plug in ZLD directly into, into ZIG, you know, this will be passed automatically, but now I just have to like dig it out. So we actually, we specify two object files like you would normally do with LD. We can create a name for it, like whatever we want. It's some warning is because I don't handle dwarf yet. I've got some bits, like written down, but I need to do more parsing. Actually, I know how it works now. So that shouldn't be too, it shouldn't take too long. One of the load commands I'm not handling, I think it's the build version, but I can always get back to it. It's not really important. Actually, that was already linked. Right. So, oh, sorry. And that's a R executable. With everything, the cooler thing that I want to show. So this bit doesn't, it doesn't, it doesn't link properly with LVM linker. We actually used thread local vars for, inside of our panic handler, the default one in ZIG. And if anybody was following closely, that was the reason why you would get no backtrace, no backtrace, sorry, on, you know, when, when you would trap on Apple Silicon, it's because it has thread local and it would just not be handled properly. So instead you would get a sex halt. So we patched it by basically saying if it's Apple Silicon, then we just don't use thread local at all. With that LV actually, we will be able to actually handle this like properly. So if I followed the same thing, just build the object file, right? And then the, not still V compiler RT. Well, guess what? Oh, sorry, I didn't, I didn't rename it. Let me do that. Just a by default. I wanted to stay compatible with, with other linkers as well. But you'll be there you go. So it actually works. And this is because it's Apple Silicon. Everything is subjected to address space layout randomization. So you can be sure that we actually do the rebasing properly. Actually, the working out the rebase was the last bit that I couldn't, I couldn't figure out because, for example, when I was building, when I was linking the hello, hello world, it turned out that when I was plugging, so I was getting a sex fault, but then I would stick it in the debugger and it would work fine. And that's why I wrote this blog post about disabling, sorry, enabling a SLR in LDB, because actually I wanted it to be like, you know, the addressing being randomized, because I was getting sex faults when I would run it normally, and not when it would be run at the same address as it was specified in the, in the binders for the start of memory. And it was super frustrating. And then I actually discovered that it was basically me forgetting to the rebases on, on certain pointers that were stored in the data segment. So, so yeah, that's done. And the cool thing is if anybody was following the TLV stuff for Apple Silicon, it turns out it got changed a little bit. So I wrote a blog post about like, kind of, it was a, it was a fix or experiment to see what's needed to fix it. And there are two things I can actually quickly show this stuff. I'm just going to do it here. A port wait, that one. Uh, the LD. No, not fingers. Right. So, by the way, that actually the linker itself is very simple. Okay. J2, your blog posts are on are both on your blog, right? Jakub.com. Yeah. Okay. So I'll link it in, in the main chat show so people can follow along. Yeah. Thank you. Yeah. Can go there if they want to. Right. Oh, right. We're like, relax here. Okay. So here you can see that we handle a couple of different instructions that are passed from the compiler about relications. So we've got the branch. Uh, so the branching basically, we've got, um, the page is basically an instruction how to, you know, how many pages you have to advance. So this is for the ADRP for the address of the page. So this is like basically going, you know, sifting through pages and increments of, um, four kilobytes. And then you can narrow down with, um, the actual paid offset. So the page is actually PC relative. That one is not. And previously, so I can only assume that what was happening in LD, LD is that here they would have, there we go. Uh, so TLVP, that's for thread local variables. They would also have the pages here and it would handle them normally. So the way it works here, you can either have a load or store, or you can have an ad instruction. So you actually have to, you kind of have to, um, parse what instruction is, is, is, is for you to fix up. And then depending on what it is, you should either do an ad or you should do load and store scaling. That's, that's how the ARM isa works. Um, and that's why I had, but it turns out that TLV needs a separate thing. Um, which is basically that, so we have to, so I think this is what Apple, Apple linker is doing as well. Uh, we need to parse the instruction to get out the registers that we need to fix up together with the size, whether it's 64 bit or 32. But at the end of the day, TLV is always handled by an ad. It's never a load or a store. So that's the first thing that's different. And this is the stuff that was discussing in the blog post. And then the second one, and this is the most interesting one. So, um, for all this, uh, the cl- actually LVM is specifying pointers within the data segment. And we have to do rebases for this. And my current, the way I'm doing it now is basically when I'm parsing the locations for the pointers, I'm then saving where this pointer was. So then I can write that in the, in the rebase information for dyld, what to be what needs to be rebased because otherwise with ASLR, it would just get garbage because it would be pointing to addressing that doesn't make sense. And I was doing the same for TLVP. And I'm pretty sure this is exactly what, um, what, uh, LAD is doing because I think that's how it was done for x86, 64, but surprise, surprise, you can't do that with ARM like Apple Silicon anymore. So you don't do any rebasing actually. What is happening, you have to zero out the memory as well for the thread variable. So thread bars would be equivalent of data section within the data segment. So this is where you would have your, um, read write data for the, you know, for your binary to basically specify addressing and things like this, but you have to zero it out. And you still need to rebase, uh, do you? Yeah, I think so. You still need to rebase, uh, maybe not. Wait, I don't think you have to wait a sec. I'm still getting confused by all this. No. Okay. Yeah. So there's no rebasing and stuff for the thread local variables. Weird stuff. Um, so, right, we can close that now. And we can, I guess I can show where I'm going to hide this and this on. Okay, cool. I think I got it open. So let's have a look. So this is the intermediate object file. And what we have here, right, is that thread var is empty, but when you do the fix ups, you won't be able to see them because there's too many. When you do relocations, it tells you that, you know, it's going to, you have to specify addresses for that, for the, um, thread local variables here for the binary to do, actually later use it, but actually that's not true. You shouldn't do that. And if we look for main, uh, we have yet here. So global is actually the thread local variable. And you can see that actually what we are meant to fix up here is LDR instruction, but that's also not correct. So actually what is happening at, um, in fact here is that we do an add. And this is what the, I was basically reversing, doing what Apple linker is doing. We do an add. Furthermore, we have, uh, wait here, we have everything zero out. So there's no data here. And I think, uh, we won't be able to get it's only just addressing. There's no, um, no naming and, and rebase and for table. And that's it pretty much. I think. On yet here in the symbols, if you look for global, you're going to see that. Yeah, it's, it's correctly in the third bars and thread VSS for the, and it. So all things are good. Oh, and there's one more thing. It turns out that, um, one more thing, this, this, this, this magical TLV bootstrap, that was, um, that was meant to be an explicit, um, we, we, we needed to create a bind like explicitly binding in the past. So you would either put in non lazy symbol pointers or lazy symbol pointers, like lazy symbol pointers. That would be any lips functions, like read, write things like this and don't lazy you to usually get, um, uh, like actually a variables, I guess, like, or, or reference to the, why, why, why, why, where, where was actually working out the lazy symbol pointers later on. Nowadays, everything should go into lazy symbol pointers. Um, I did stick in the TLV bootstrap, but I actually did. It turns out you don't even have to have to be bootstrap anywhere. All you need to do, you just need to have the binding information populated properly here. And that's it. Okay. My optimization I'll need to do. So if I understand correctly, there is no initial setup done then for trial local storage. I'm understanding this correctly. Like it's just memory that you have. Yeah. Yeah. It's handled for you. It's, it turns out that actually I think it's handled for you. Like in the past, you would need to guide you by LD. And I think nowadays it just, um, I think it just knows it goes through the simple table and it can work it out. So you know, main differences. Okay. So more in general, what make you want to learn all this stuff about my quest? Why, why didn't you just throw your hands up in the air and say, well, whatever, I'm just going to switch to Linux. Oh, that's because we didn't have anything for, um, for Mac back in the Bay. So we had, uh, so I think Andrew wrote the first draft for incremental linker, uh, target in Linux, I think. So that's else. Right. And we had, I think Alex wrote some, uh, like the first pass, he had a first pass at PE and costs. So that's mainly for Windows, but there was nothing for, uh, for macOS. We had nothing. So I was like, well, might as well. And I thought, yeah, I'm going to learn something. And I'm, this is about like five months. And I use, you know, every single time like, oh my God, what now? So yeah, it's good fun, I guess. Okay. So there was a comment from, uh, from before, um, by June who, um, commented that, uh, I think it's a reasonable take that, uh, having complete parallel code pass, um, is, is problematic, right? Because you have different code paths that, uh, are, are you expecting to like all do the right thing, but, uh, they're different. So they might diverge. And, um, so, and you're applied about the testing routine. Um, how does that work? The testing routine for linker stuff. Um, how is it structured? Um, so we have, so basically, obviously we have some unit testing for some bits, um, within the linker itself. Uh, so for example, I, uh, when I was working on the export try, if you remember, that was also a lot of reverse engineering plus looking up the, you know, the source code of the, um, the existing system linker and LLD trying to figure out how to do, how to properly export information for the divi LLD to understand. I added some tests there. And obviously we have those for a lot more things, just to be sure that, you know, the tiny bits are properly encoded as bytes. But at the same time, we have a very powerful end to end tooling for, uh, for testing. So we actually, you know, you, you basically write source for the test, um, like zig, uh, zig, zig file and it's being compiled and then it's being linked and then it's being run. So, so you can actually be sure that, um, and you can obviously, because they get so powerful with comp time, you can actually do, um, a lot of very interesting branching on different architectures and operating systems. And like I said, this manner, so that's, that's perfect. So we will be actually able to test quirks on different, for different targets as well. The only thing that we're missing, and this is kind of sad at the moment is, for example, we don't have a CI for Apple Silicon. Uh, it's kind of difficult, you know, emulation. I don't think we'll cut it because it's a, while the basic isa is, I think it's like bespoke. So this is the standard, the standard isa that ARM has. There are some extensions that Apple has as well, which make it basically specific to Apple. So I guess we'll need to actually test it. Like if we want to run it, we'll need to test it on Apple Silicon unless we write an emulator, I guess, but that's another project. Yeah, that's, that's one more yag to shave. Okay. So, um, uh, let me ask you a quick question then. So since you like, um, Max so much, uh, are you going to get the new 16 inch MacBook Pro if it has a better M, I don't know, M1X processor? Yeah, probably, but not immediately, I don't think. Right. Because, um, so I've got this MacBook Air with M1 and it's perfect. It's actually faster than i9. So that's really awesome, right? It doesn't have fans, so it doesn't really get that hot, which is another plus. It's just, it's a really cool, it's a really cool piece of software hardware, sorry. So I'm really impressed with the efficiency of the M1. It's, um, but I guess everybody is right. Everybody who tried it, they were like, wow. Yeah. So I'm not sure. We'll see how much it is, you know, when, how much. I would love to see benchmarks comparing M1 with M1X and then to what it's called. Okay. I think we have two to three minutes more. Another question. Where would you like the first Z conference, like ERL conference to be? Where? Oh, that's it. Are we talking like after the pandemic or now? Yeah, well, after the pandemic, let's assume that you have 20 seconds to answer. Let's assume that the pandemic is over and you can go everywhere, but also, I know. Yeah, I don't have, I don't have Andrew's address, but it's going to be there.
 Hi, everyone. Welcome to ZIG Dev Room at FOSDEM. My name is Jacob, and it's with great pleasure  that today I will talk to you about linking in ZIG. And hopefully, I hope it inspires  you to try ZIG on your own, have fun, and maybe contribute. So without further ado,  let's get cracking. So as I mentioned, today we'll talk about Maho Linkarn ZIG, and specifically  what change or what do we have to do to actually satisfy Apple Silicon for the Maho Textual Work,  or handcrafted Maho binaries. Okay, so first things first, who is this guy? As I said,  my name is Jacob. I guess most people usually associate me with my GitHub, Twitter, and all  the other handles, KubeCon. I am a software engineer at Microsoft, and I am a ZIG core team member.  So today, I'd like to propose the following agenda. Don't worry, there will be some actual  coding involved as well. We love those, right? So I can skip on this. So first of all, we'll briefly  discuss what linking is, actually. Then I would like to introduce my friend Maho. You might know  him, you might not know him. It's kind of crucial that we discuss this format for going further,  because this kind of has an impact on linking in general, how we do that for Mac OS and ZIG.  Then we're going to move on to briefly discuss what changed in Apple Silicon, what new requirements  Apple is putting on linking, on the binaries themselves, and even shared libraries, of course.  So then finally discuss incremental linking in ZIG in the cell phone compiler. So this is our own  in-house linker that satisfies all of the requirements of Apple Silicon, and you can actually use it to  perform incremental linking target in Mac OS. Finally, if you have some time, and I'm pretty  sure we will, I can't skip this bit, I'll briefly show you that we are able already to cross-compile  C to Apple Silicon with ZIG. Okay, so that's good cracking. Right, so what is linking anyway?  So most of you, I think, associate linking with this traditional approach. In this traditional  approach, we have the compiler and linker separate, we have some source files in the source language,  like Rust or C, and then we compile them into the things called intermediate object files.  These are also for Mac OS, these are also Mac OS files, but they're not in their final form.  So you still need the linker to actually read those, like multiple of those, put them together,  link in any, like specifying any, if we want to link with the Shared Library, like LAPS system,  we also have to specify the end, and then we end up with the final artifact, which is the executable  Maho. Okay, so in this approach, compilation, as I said, is separate from linking. Source changes  always lead, sorry, to full relinking, right? So if we change one source file, we will only  have to update one intermediate object file, but the final artifact will have to be redone from scratch.  And this entire chain is LVM favored. What do I mean by this? I mean that actually in LVM,  this is how everything is laid out, if you want to pull an LVM. Okay, so how do we do this in ZIG?  What is the, what is the ZIG's deal when it comes to linking? So actually, we take the concept of  compiler and linker, and we tightly couple them together. So instead of having two separate  drivers, we have one. While this might be not as generic as what you would have in some traditional  tool trains, this gives us this possibility to actually do a lot of more targeted things that  otherwise would be impossible. What do I mean by this? So for example, we can do incremental linking,  and what does that mean? That means that we actually added the final binary in place. So instead of  relying on intermediate object files, which we don't have in this case, we actually only overwrite  bits in Maho that need that. So this leads to short, up-to-turn around time. We don't have to  wait, I'm pretty sure most of you have experienced this once, or at least once in your life, that  when you do, for example, compile a massive project, the very final stage, even if you don't change  that many things, this linking bit is constantly the bottleneck. It actually takes a long time  to link everything, because you have to link again. So if you remember, you basically toss  the macro binary, and then you recreate it. Whereas with Zig's approach, you don't have to do this.  You only have to override specific bits that changed, like specific symbols, I guess, but we'll get to this.  Okay, so what is incremental linking? So I should first of all add and Zig. We only plan to have  this on in the bug mode, and this will be on by default. While in the bug mode, this is for two  main reasons, is that incremental linking inevitably generates larger artifacts, because we need all  the padding we can get. We don't want to be shifting, like rewriting sections or even major  chunks within the final artifact too many times, because that will guess what leads to actually,  in the end, rewriting the entire artifact, which will put us in square one with traditional linkers.  And we cannot support linker time optimization, because, well, we only have one fast. We do it,  like, on the fly, like hot code swapping, if you will. And that's our goal, in fact. But on the  positive side, incremental linking leads to shorter compilation times. It means quicker prototyping,  because you don't have to wait so much. You just change the code, update it, and then it recompiles  on the fly. It should be instantaneous, and that's what we are working towards. And so basically  doing the typical prototyping route of write, compile, change, recompile, change, recompile,  change, recompile should be very, very responsive, like super quick. You won't have to wait too long,  and that's exactly the point of actually working on your app, your program, whatever, in the bag  mode, right? You want to basically, you want to have a very quick turnaround time. Cool.  Okay, so with this out, let's meet Maho. So Maho is my buddy. Maho stands for Mah object file  format. Maho is a time for kernel. It's used by all Apple operating systems. The good news is,  the file structure is actually straightforward. The bad news is there is no up to date docs. So  if you want to get into this stuff, well, yeah, you will need to basically browse the web, see  whether there are some maybe guidelines, maybe somebody wrote a blog post, but there isn't too  many docs. So you will have to go through the source code eventually. Thankfully, Apple has released  most of the stuff on their Apple open source website. And incremental linking actually is  difficult, especially with the latest additions or requirements by Apple. So we'll get to it later.  Okay, so before you is the overview of the Maho file format structure. It's actually,  as I said, it's pretty straightforward. So we start with the header. This is where we  insert magic number architecture. So it could be x8664, could be ARM64 for Apple Silicon.  Also file type. Is it the intermediary object file? Or maybe this is the executable, or maybe it's a  short library. Then directly after header, we've got load commands. So load commands,  I guess you should think about them as your table of contents. So if you write your own parser for  Maho, you will use those load commands to actually directly navigate through the entire file.  Being able to locate things like the text section where the compiled machine code  lies or the stops for the imports, things like this. So every segment load command  describes the segment. It maps into virtual memory. So it's got the map into virtual memory. It's  got the start address and the actual virtual memory size. And this is very important because this is  what the loader uses to insert the binary into the address space. Okay, every segment command  can have multiple sections from zero up to 255, I think. So in total, 256 is maximum.  Oh yeah, zero is like the indexing. It doesn't matter. Right. So sections describe the actual  data that is mapped into memory. For example, the text section will contain the compiled machine  code. The steps section will contain the trampolines that the loader will use, or your program will  use to actually work out what or where to call the import symbol, like write or exit from libc.  And you can also have loader info. So for example, the opcodes that drive the dynamic loader and  resolving those imported symbols. Okay, so so far so good. Let's quickly go through a couple of those.  So we've got the map header. If you run auto, which is the fact of standard tool provided by Apple,  they can use to have a look what the file structure actually is. We can have a look at the header. So  that's dash H. The header structure is pretty simple. I have omitted a couple of fields because  they're not pretty interesting. The main thing is that you've got magic for 64 bit  maho binaries. This will always be feed facf or 32 bits. There used to be feed face, which I thought  had a better ring to it. But hey, we've got file type, which in this case, well, in the cases that  we're going to be describing here, it will always be executable. So that's mh underscore execute,  number of commands is 15. In this particular example, and the size of commands in this example  is just above 1200. We can have also additional flags. In this case, mhpy signifies that this  binary is a position independent executable. And we'll get to all of this later on.  Right, so now, navigating for the load commands, let's have a look at the segment command.  So to print all the commands with auto, you would just append dash LV,  where we have here. This is the second load command, actually, because we started from zero.  Load command zero is always standard for executables. This is the reserve to virtual memory address space  of four gigabytes that's reserved for the null symbol, or for null in general. And  maho always, so basically every other segment has to start at four gig. It cannot be below this.  So this in front of you is the text segment. So as I pointed out, we start at the virtual memory  address of four gigabytes. Now the interesting things I should point out are crucial for the  dynamic loader and the kernel not killing your program is that so command size always has to be  eight bytes aligned. Okay, always, whatever that is, always has to be eight bytes aligned.  Now, virtual memory size always has to be page aligned, always. File size also, and initial  protection, well, that depends. Every segment will have a different set of flags for text  segment that's read and execute. Now page aligned, that depends on the architecture for arm 64,  that's 16K, for Intel, that's 4K. Okay, so far so good. So in front of you, this is an example  of a text segment. As you can see that section text starts at address four gig plus 1000,  and it extends all the way to 1100. Then, and this is very interesting, we have zero part of space,  and only then we have, for example, stabs at address 56,000. Okay, now the important thing to  note here, and this if you'll be working on an incremental linker or linker in general, you  don't have to worry about zero padding or empty space between sections within a segment. That's  fine. They don't have to be adjacent. Obviously, if you have a new space within a segment, you're  going to be exploding your binary in size. But this is still fine. I mean, for incremental linking,  that's inevitable. And we want to have as much padding as possible so that we don't have to  shift everything at every second right or so. Okay, another thing to note, the segments themselves  have to be laid out contiguously in memory. So we cannot have any empty space between them. No,  there cannot be any space. So if text segments end at 58,000, then for example, data con segment  has to start at 58,000 and then extend all the way to, for example, 5C,  1,000 for the next segment to start. Okay, moving on. Okay, now my favorite segment,  the linker segment. Now, this is the odd one out, I would say. And why is that? Well, check it out.  So first of all, the virtual memory size is page aligned, perfect. But file sizes are not. It is  not page aligned. And there are no sections. Oh my God, link added segment never, well, I've never  seen link added segment have any sections. So first things first, let's talk about the file size.  It doesn't have to be page aligned because the last segment within the binary. And in this case,  I assume it's link added. It doesn't have to be. But, well, I've never seen a different layout from  my file format, I guess this is kind of standard that the link added comes last. And it will have  to because of the code signature, but we'll get to it. So the file size can be exactly what  the sizes of the link added segment. And then the loader will actually expand it to be page  aligned. It's fine. And then no sections because the link added segment get the sections of link  added segment get their own load commands. So, okay, so let's describe a couple of  link added. I call them hidden sections, but actually, as I said, they get their own load  commands. So for instance, what we have, we have a thing called dynamic loader info only. So this  is the LC-DYLD info only. And this section contains all the op codes for the dynamic loader. So it's  the rebase info, the binding info, lazy binding, and export. So everything to do with imports and  exports. If you want to export a symbol, it has to be encoded for an export try within export info  section. Or if you want to import something, how you resolve the addresses, where the  DYLD stop binder should then populate them. So basically, you write the address into,  which cell in the memory, that's also specified within the binding and lazy binding info. And  rebase, of course. Okay, the next thing is that we have the symbol table. That's LC-SIMPAP. This  one usually, so this is the symbol table that contains all of the locals, globals, and extras,  all of it. We've got two bits that go within this load, the this load command describes two  bit. This is the symbol table and string table. Important thing to notice. A thing that we discovered  recently is that the symbol table has to always precede the string table. Meaning that if, for  example, you end up expanding the symbol table so much that you have to move it somewhere else,  you have to move string table as well. It doesn't have to, they don't have to be adjacent,  but symbol table has to always precede the string table. Otherwise, the DYLD will refuse to load  the binder. Okay, next thing is the LC-DISIMPAP. This one contains usually in the indirect symbol  table. So this is the table of all the imports of all the external symbols. This is also non-optional.  If you actually want to link with Lipsy properly, like actually import the symbols.  And finally, a very important load command that is a new addition on macOS, starting with  macOS 11 and Apple Silicon is the LC code signature that load command contains the code  signature of the binary. And at the very minimum, the ad hoc code signature is basically a couple  of hashes of the entire binary. Without this, the kernel just will refuse to run your binary.  This is very important on newer Macs running on Apple Silicon. Okay, so let's move on to the changes  with that Apple Silicon has brought to the table for Macs. So the main three ones are that the  binary has to be ad hoc code sign. So that's what we're going to discuss next. It has to be also  app positioned independent executable. So for Intel, it is still possible to create a static  binary and it's going to be fine. Even though it's not recommended, but you can still do this,  this is not possible or up on Apple Silicon, everything has to be position dependent,  meaning that you cannot have any absolute addressing embedded within your binary. That's a no go.  And finally, the binary has to be moved to a new I note, if it's modified in place.  But this is a bit tricky. So I will let you think about this for a bit longer. But basically,  what we'll describe it later in more detail about what happens is if you modify the binary that's  already was run in place, the kernel will cut. Okay. So moving on, let's discuss briefly what  ad hoc code signature actually is. So first things first, this is the last link at its section.  It has to be last because it has to calculate the hashes of the entire binary excluding obviously  itself. It is pointed to by LC code signature load command. And as I said, it contains hashes of the  entire binary. Now the tricky bit is it is in big Indian order. Now, on the outside, I guess  at the face of it, it kind of looks weird because the entire binary is little Indian.  But if you think about this, this code signature structure or whatever goes in there is actually  used for more than a couple of code signatures, it can be used for full blown code signatures  that are required if you, for example, submit your app to Apple to App Store. And it's in big Indian  because predominantly the network by toward there is actually big Indian order. So that's why so that  you don't have to do the unnecessary conversed between little and big. Okay. So this is the basic  structure of the code signature that goes within within it that's embedded inside the macro binary.  So we have the header. We have this, I'm not ignored there. This is pretty standard for  ad hoc code signatures as far as I know, it will usually be fade 0CC0. Then we have the code directory,  which is another structure embedded within the code signature structure. This one has got the CS  ad hoc flag specified that this is the ad hoc code signature important bits that are configurable  if you so wish to our hash type and page size the hash type for like, for example, it's shot 256  back and pick. There is a list of different hashing algorithms you can pick from. And then the page  size, the page size usually matches that of the architecture. So for ARM or Apple Silicon, that's  16k. And following this, we basically get like a long table, long list of of hashes of each page.  So basically we divide a binary into pages, and then we calculate the hash of each page. And that's  it. Obviously, every hash should be of the hash of the page should be, should span exactly one  page except for the last page, which does not have to be the full. And page size because the last page.  Okay, so onto the requirement or the change that the kernel is actually caching inodes.  So this is true since my co is 11 and Apple Silicon. Now what is happening? So imagine the  scenario that you have some Hello work program, and then you execute it, then the kernel is caching  the inode, noting the actual inode and the code signature that this Hello has embedded within.  Obviously Iran, Iran's it and then sends yeah, cool, no problem. Okay. Then imagine that we  modify it in place, we actually calculate recalculate the code signature so that everything  is valid. Then we try to launch it. And then the kernel goes into registry notices that okay,  I already got I know like this, but hang on the code signatures are much. Oh, something is wrong.  Somebody is being funky here. So it sends the sick co refusing to launch the modified binary.  So in general, I understand the principle that's meant to be more secure, I guess,  things like this, but for incremental linking, especially for sake, this is a bit of a problem  because as I said, we want to modify it by this in place. And because of this, we actually have  to go through those additional hoops of copying the entire contents of the updated macro binary  into a new file and then swapping and then basically creating a new inode and then rewriting  or deleting the old one leaving the new one in place so that the user doesn't see any difference.  So this is kind of costly, but this is what it is. And this is how it's done. So we underneath the  sea, like under the sea, we actually, sorry, behind the scenes, we actually move the file in place  after we've finished writing to it and recalculating the code signature so that the kernel  so basically that we change the inode. Okay, so let's move on to ZIG. So how is  incremental linking handled in self hosted ZIG compiler? I should point out that the  self hosted is still a work in progress. So you should expect breaking things, you should expect  sex, sex faults, things not implemented by working on it. And actually we're making  some pretty good progress. So it's pretty exciting times. Okay, so just to recap, this is the self  hosted approach. If you remember, we have compilation coupled with linking. So we actually  write to the macro binary the final artifact in place. So what does it mean actually? So for  example, imagine we have a function foo. This function will get its own symbol within the text  section within the text segment within the text section. So we get its own symbol with  star address and then the size. Okay, when we write that into the text section, the compile  machine code, we also write a new entry in the ZIG global author table. Now,  why call is it that's because a global it's a God is kind of taken by  by the traditional, I guess, traditionally God is part of data or data con segment.  And actually it's for ZIG, it's kind of a misnomer because it's not really a God.  It's actually more of a jump table, as you'll see in a minute. So when we write, as I was saying,  when we write, we create a symbol for foo. At the same time, we add a new entry in the God  that features basically a trampoline that any other symbol that references foo or is calling foo  at the call side will reference the entries that got rather than the symbol itself.  And then by referencing the entries that got so calling to branching to that little snippet here,  he will be able to automatically compute where to jump to in order to call foo. Okay, so just to  do a bit of math here to show you how it works. First thing as far as if you notice, we've got  the other instruction here on arm 64, which will get the address of the label  foo displaced by that value here and 349480 is equivalent to actually 55528 in hex.  Now, if we take the address of that instruction, which is 56560 and we subtract that value  in hex, we'll get, well, surprise, surprise, the address of symbol foo. And this is exactly  what we are after. Okay, cool. Now, I'm going to leave it at this and we'll come back to the  one after the demo to show you what this implies and why it's really useful. Okay, so now to show  you the principle of how we actually use that as I described briefly, if we have, for example, start  and we call foo, as I mentioned, we actually don't call the actual, we don't compute the  displacement to the actual symbol foo, we actually compute the displacement to the entry,  its entry within the zig global offset table, which is 56560. And then by doing this, we then,  and we basically invoke these two op codes to work out where foo actually is. Then we move the  contents of x zero, which we populated with the actual address of foo into the link register,  and then we brand from that link register to actually call foo. Okay, so this is just a  slight indirection versus what you would actually do if you have it, if you had it properly optimized  and you are actually branching directly to the symbol. Okay, some demo time. Cool, so first  things first, let's create some source. Okay, so we're going to force them here.  What we'll actually do, we will, just to show off that we've got some things already working  pretty well, we will import two symbols. So I should point out here that you could write  raw assembly targeting, basically write raw assembly to invoke the syscals on macOS, but this  is not recommended because Apple does not guarantee that, for example, the syscal numbering will not  change from version to version. So we don't do that. Instead, you're always recommended to use  libc to actually invoke the syscals. So we've got, we've got write and that one accepts  three u sizes. Actually, they should be different types, but I think we're missing the stuff from  the code gen still when it comes to expanding or like growing or shortening end resolutions. So  we'll just stick to u size and returns u size. I will also use exit for specifying the return  code that we can then inspect when the program returns. Then we will export start. Now,  we should really have weak linkage to main, but we don't actually do that. We do the custom  entry point of start for the time being. This will change. We will actually use main as recommended,  but hey, we didn't get to it yet. Let's print some message. So the message will be starting  like hello. Oh, why not? Hello, Fosdm. Oops, sorry. So pointer to end means that we will  actually convert the pointer to that C string or string, an array of bytes to an end immediately.  That's because write requires that as the second argument here. So actually, let me write that out.  So this is the file descriptor. This is the message. Actually, it's pointer to message, I guess.  And this is length. So this is the length of the message. And that's the return code.  Let's make it message. Actually, it's going to be easier to refer to it. So this is an int as  well. And it will go into the second argument and write. Now, as far as length is concerned,  here we have, I don't have to calculate that by hand now, three, six, seven, ten,  thirteen, fourteen, fifteen bytes to write. Okay. Now, we don't care about the turn value here.  Now, FD is going to be one for stood out. We've got the message and we also have  the return code. Sorry, the length, which is length. Okay. Actually, let me move that immediately to  another function just to demonstrate stuff later on. We're just going to put it in print.  Okay. And then here we'll call print. I'm also called xin with zero. Everything works fine.  Cool. So we've got our simple zig source. So now let's try and compile it. So let's use the  stage two to do this. And build x, the fosdm zig. So this is the standard way you would build it  doing a one off. And this is the standard way you would actually work with different languages,  like Rust or C. You just rerun the entire thing and then it generates the artifact here, which is  the fosdm executable. And then if we run it, we get hello fosdm. So the basics work. But actually,  what I want to show you is this magical flag called watch. What happens here is that we entered  an incremental linking mode, meaning that so we already have compiled it. You can rerun it to see  that it works and it works. But now we can do modifications on the fly, which should make it  a recompilation or relinking. Actually, in this case, recompilation on relinking super fast. So  for instance, I can do print a couple of times. And what will happen here before we get to this,  I'll tell you upfront, this will cause, actually, let's do it a couple of times. Okay, this will  cause the start symbol to expand in size. We will actually need more size than we have allocated  it initially when we compiled it initially. So what it will mean in order to grow, we'll have to  first of all lift the entire symbol and then move it somewhere else. And normally, what that would  lead to, this would lead to rewriting the entire x section, probably stabs as well. Maybe not,  maybe stabs would stay. But if everything was so adjacent and squeezed in, like you would have in  different tool chains like Rast C or C++ or whatnot, then you would need to actually indeed  perform the full relinking. But because we leave padding, this is not needed. So before I show it  to you, let me quickly show you what the actual, my mouse is gone. Oh, there it is. Cool. What the  actual generated Mahobailin looks like. So as I mentioned to you, we've got all the segments  and everything laid out like I described it. We have code signature, everything, you know, so far  so good. Then within the text segment, we've got the start symbol. So this is the content of the  old start that we have just modified, but we haven't recompiled yet. So we actually, we have this  call here, which is the branch unconditionally to this address 560, which is an entry in the Zik  code. And if you notice 560 will calculate the displacement to a symbol, which is 349,476 bytes  below that address. And that points to print, because this is where we want to basically get  the address of print and call it. And this is where we actually call the results address of print.  And then we call exit symbol from let's see. So now the point to note here is  that, so we even started a couple more, as you see here, we even said a couple more calls to print  within start. So start probably doesn't have enough space here. And it doesn't, because if you notice  we have, we only had one and now we have, I think at least 10 more, and we only have space for four  opcodes more here. That's not, so it's the 102C all the way to 1038. That's not enough. So what will  happen? In order to grow start, we will move the section below print somewhere, starting a  this address or below. And it will be much larger in size. But actually what will happen,  is that we will have to re-write everything. We will actually have to re-calculate the addresses  again and things like this, because obviously we would actually, start will be now below print,  and the displacement will be negative as a positive things like this. And the same  thing happens with the other ones. So we will have to re-calculate the address again and things  like this. And the displacement will be negative as a positive things like this. And the same  will apply when we modify print in a minute. Okay, so let's have a look. So we did this. Now let's  do update. Okay, as you can notice actually, this is where the update was. We have much less output  than we had before. Look, here was the first setup. We were actually populating the metadata,  like load commands, wherever it goes and things like this. Whereas when we update,  we actually only care about updating automatically. So we actually care about updating as only those  bits that need to be updated or that are required to be updated by Apple, by the dynamic loader.  So as I was saying, we grow and relocate start from the old address to the new address. So this is  the 10-8-C. We write the new offset table entry for that symbol at its address. So instead of  modifying every other symbol within the text section, we basically only update the corresponding  entry when the Zig got, and that's it. Okay, so far so good. So let's, before I show you the  change structure, let's try and run it. Cool, we have HelloFuzz a couple of times. And now let's  reopen that binary. And what we have here, so the old start actually stayed because this is now part  of what we call a free list of text blocks that we can reuse if a symbol of a size that's not  bigger than the old start fits. And then we'll just stick it here instead of writing it at the end  so that we can conserve the space. Now print state, and as you can notice, in the new start,  we still branch to the same symbol, which is the second entry within the Zig got, which is here.  Okay, so cool, so far so good. So what is happening now?  Let's now try and do  a slight twist. So let's try, we've got the start, we have print. Why won't we write it a couple  more times so that we will have to grow and relocate print as well? Okay, I think that should  be plenty to do this. Let's do update. Okay, cool. Yeah, look, growing and relocating print. So again,  we moved it from the old symbol address to the new one. And this one is again, this one now is  following the new start. And same, sorry, the same thing applies now. If I reopen it, the binary here,  what we get within the text section is that the new print, or first of all, look, start is calling  print at the same address. Why? Because the entry in the Zig got, so the key in the  in the dump table has not changed, the index is the same, but the entry, so where we point to,  has been updated to point to the new address of print, which is this one. Okay, so far so good.  And this, this is the whole principle. This is the entire idea behind incremental linking on,  on, in Zig effectively speaking, this is how we want to manage everything. Okay, cool. So that's,  I think, all about incremental linking. I'd like to show you guys. We've got still a couple of  minutes. Yay. So let's keep going. So now the bonus. So cross compiling C to Apple Silicon with Zig.  So Zig comes with its own C and C++ compilers. They're actually under, they are like,  like, thin wrappers for clan, because this is in, in, in this is not self hosted, self hosted,  we'll also have it. We'll also have LVM as an optional backend, but in this case, I'm talking  about stage one. So this is the compiler that's written in mix of Zig and C++. So this one is  based on LVM. It uses LVM for code, code gen and it uses LLD for linking. So with stage one compiler,  we actually did traditional approach, which is not, you know, obviously it's devoid of incremental  linking. We create intermediate files. And then we actually perform the full linkage at the end  using LLD. Now the problem with this is that when Apple Silicon came out, LLD was not equipped to  actually handle code signatures, not to measure trust compilation, but it still doesn't know how  to handle. There is some, there is actually a lot of work being put into LLD for macOS  in LLDM, which is great. But it's, you know, it's basically it's, it's, it's a, you know, it's a big  project, right, right, the linker or rewrite it, I guess from scratch, because that's what's happening.  So anyway, so our idea was, because this is going to take a while,  and we obviously we're working on our own in-house linker, but still it's also not, not, not ready  to handle everything. And in what I mean by this, it cannot handle the intermediate object files.  So instead, what we noticed was that we can modify the output of LLD, kind of post mortem.  So we can actually reparse the binary, the Maho binary, look for certain bits, the arrow of interest,  fix them, and then at the end, append the code signature, and what was your uncle, and this  should work. And guess what, it actually does. So yes, let me recap this. In stage one, on, for  Apple Silicon, we actually, you actually don't need to, for example, fall back to system linker.  So the linker that's supplied by Apple, you can use LLD in 99% of the cases to generate a valid  Maho binary that's been code signed, ad hoc code signed properly, using RZ toolchain. It's just,  just that it's not inside LVM, it just happens post mortem. So let's have a look. Actually,  everything I've been showing you here, all the tooling, I'm actually talking to you from,  um, um, sorry, from, uh, M1 MacBook Air. And all the tooling that I used here to actually compile  stage two was done using our own post mortem, uh, embedding of code signatures within LLD's output.  So to prove my point, um, that first of all, let me show you how you compile, for example,  simple C with ZIG. And this is it. It's basically ZIGCC simple C. Now this generates A out like,  well, you know what, the standard way. And then if we run it, we're going to get Hayder.  And let's compare that with, um, simple C. That's indeed what it was. Okay. So now let's get rid  of this A out. Now I am going to quickly associate into my local, um, Linux box. So we have here,  5.9 kernel. And this is the Intel, um, Intel box. Okay. Let's go to dev. Let's go to ZIG.  Examples. Let me show this. Now, uh, let's have a look at simple C. We have pretty much the same,  oh, we have hello, instead of hello world or Hayder. Okay. Now let's do the same thing. So let's  first of all compile this C for that, um, target architecture and that operating system.  So there we go. Now A out is actually an else. Perfect. So we should be able to run it. We did.  Everything works. Awesome. Now let's do the same. But this time we are going to cross compile  to macOS arm. Okay. So let's try it. Okay. Something happened. We still have A out. Let's see what  that is. Okay. Cool. So now this is arm 64 executable. It's a pie and all the other flags. Awesome.  Let's exit from that machine and let's actually copy.  Okay. We don't have A out here. So no, no, no monkey business. Let's actually copy it from the  server. So that's dev, ZIG examples, A out to here. Now if everything went fine,  let's double check. Yep. If everything went fine, we should be able to run it. So let's have a look.  Ta-da. It runs. So actually with ZIG, no kidding, you can do cross compilation of simple C or simple  ZIG from anywhere to macOS, Apple Silicon and Intel. For Intel, actually, I can show you that we  can do this here on this machine. Bear with me. So build bin ZIG CC. And then we have simple C.  And then let's do target x8664 macOS. Okay. File A out. Co is executable for x8664.  And it works. Awesome. So yeah, I think that's all I have for you guys today.  I hope you enjoyed it. I think we've got more than, I think, 15 minutes. So,  or maybe even close to like 20 or maybe even half an hour depends how everything is going  to get organized for the actual FOS then. Obviously, I'm recording it a couple of weeks in advance.  But I hope it's been interesting. Thanks for watching. And I hope you have lots of questions.  I'd love to have loads of questions, especially for this session right now. Obviously, I'm available.  You're convenience to answer anything you have. If anything wasn't clear, please let me know.  At the same time, I'd like to strongly encourage you to try out ZIG. If you like it, contribute.  ZIG's community is really, really welcoming. We welcome any and all contributions.  We're just a really friendly bunch of people. We come from the first background. So, and there's  plenty of work. So, if you have anything that is of particular interest to you,  the sky's the limit here. Oh, and if you want to get in touch and be it  directly right now, after the presentation at FOS, or even afterwards, feel free to  ping me a message on Twitter or ping me an email. That's my email address. Or find me on GitHub.  Cool. So, thank you again, and have a good day.  It's funny how every single talk has the part where we are shutting down the recording that  ends up being part of the recording itself. Okay, I think we are live, and we were probably  already for a while. There's a bit of delay. Yes. I also recommended you mute yourself.  Okay. Hello, everyone. Welcome to the Q&A with Jacob. Here's what we should do, I think.  Let me do a super short introduction. Then I leave Jacob to talk about other things that he's  working on, because his work on linkers is never done. Link outers can never rest.  But in the meantime, if you have questions, write them down. I'll take care to stop Jacob at  some point to answer them. So, I think that we should start by letting everybody know that  Jacob has been doing an amazing job with linking. In the presentation, he talked about the  incremental linking process and stuff specific to Apple Silicon. But that's not a full story.  He's also now working on another linker, which he will talk about now. But just so you know,  in the community, since he's been such an expert about macro format, people have started calling  him the Macho Man. He doesn't like the name. I can understand why, but we like it too much.  If you come to the community and ask Macho Man, that's Jacob.  That's the thank you. That's how the community thanks Jacob for his efforts with Apple Silicon.  Anyway, so as I was saying, just a super quick recap and then Jacob, I'll leave it to you to  explain all the details. Incremental linking is a tool mostly, I would say even exclusively,  geared towards development. Jacob explained this in the presentation. So the incremental linker  is not a good linker by standard terms of efficiency. It's supposed to just be fast. And  it does a ton of weird things to just be linking as fast as possible and implementing incremental  linking. So this is a thing that is useful just for the development process, like you compile,  make it run, make a change, recompile again, et cetera. This is one project which is separate  from the main actual linking for releases. And so this is, I would say, more traditional linking,  I guess. And this traditional linking has the issue that, for example, right now,  Jacob, you can correct me if I'm wrong, but I believe that LLD, so the linker that is bundled  with LLVM is not able to support everything that needs to be able to allow us to do code signing  when cross-compiling, right? Yeah, correct. Yeah. So right now, we have this implemented as a hack  where you wrote the code that basically reopens the executable after the linking stage and tries  to patch things around. But this is not working 100% of the time based on the space available,  et cetera. So this is what Jacob is working on, an actual linker for non-incremental linking,  for normal linking. So Jacob, please let us know how are things going?  They're going pretty well. I can actually do a demo with you guys. Sure. Yeah, so it's called  ZLD for now, but people were saying that maybe we should rename it to something else. So it might  be Mahuman, who knows? No spoilers. I need to get rid of the name, but anyway. So yeah, so the problem  is that LLD is not tailored towards Apple Silicon at all. So code signing is missing. Actually,  cross-compilation doesn't really work for either the Intel target, so X8664, nor Apple Silicon. So  cross-compilation is just not possible with current LD. We do some, as I was showing in the  recording, we do some like hot patching of the binary after it's being sped up by LD to actually  support some cross-compilation, but it's not ideal. Things are missing. If I wanted to make it ideal,  I would basically have to disassemble the entire thing and put it back together, which is just  silly. We might as well write a linker, right? There's one caveat that, so the ZLD started only  two weeks ago. So obviously things will change. I'll try to make it more generic, you know,  the factor here and there, but my main idea, my main hope is just to have something that works  for like, I don't know, 90% of cases. And we can use it as an experimental linking back-end for  existing ZIC and anybody can turn it on with a flag or something to see how it behaves.  Yeah, it's actually kind of promising. Let me try and share something.  I'm going to share the screen. Okay, Lawrence, you'll have to tell me if you can see that.  Yes, it's working. Oh, for the size, let me full screen. I can make it a bit bigger.  Yeah, okay, this is reasonable. Yes. Okay, cool. So yeah, so the ZLD, that's a separate project to  ZIC currently because I mean, I'm not sure we can actually give it a separate project because  hopefully if it matures enough, it can be used by anybody. So you could actually use this as a  linker for Rust, for C, C++, you know, instead of like a system linker or LD. It's a traditional  in the sense that it basically reads static archives, shared libraries and intermediate  object files and it should, you know, like create the final artifact. It doesn't do all those things  yet, but we're getting there, I guess. Right, so let me remove all this stuff just to start fresh.  I got TLV, like I got threat locals started working today this morning.  So this is kind of, they're still up to clean up, but it's getting there. But anyway,  so here we have the hello ZIG and believe it or not, creating a hello ZIG is actually pretty heavy.  The amount of information that you get passed in from the compiler is quite a lot. We need  two things. We need the main object file with like, you know, all the symbols for allocators,  things like this. And we also need compiler runtime, like compiler RT.  That's also, that bit is also passed in from ZIG. So we need to do those two object files and we  can actually create a final artifact. So with this stuff, we can let me quickly just build the  object file. So there we have it. Yeah, it's a, it's, oh yeah, you can't read from here, but it's a standard  intermediate object file. And where was the LD? So this fancy like path is the path of the cache  to the compiler RT. When I actually plug in ZLD directly into, into ZIG, you know, this will  be passed automatically, but now I just have to like dig it out. So we actually, we specify  two object files like you would normally do with LD. We can create a name for it,  like whatever we want. It's some warning is because I don't handle dwarf yet. I've got some bits,  like written down, but I need to do more parsing. Actually, I know how it works now. So that shouldn't  be too, it shouldn't take too long. One of the load commands I'm not handling, I think it's the build  version, but I can always get back to it. It's not really important. Actually, that was already linked.  Right. So, oh, sorry. And that's a R executable. With everything, the cooler thing that I want to show.  So this bit doesn't, it doesn't, it doesn't link properly with LVM linker. We actually used  thread local vars for, inside of our panic handler, the default one in ZIG. And if anybody was  following closely, that was the reason why you would get no backtrace, no backtrace, sorry, on,  you know, when, when you would trap on Apple Silicon, it's because it has thread local and it  would just not be handled properly. So instead you would get a sex halt. So we patched it by  basically saying if it's Apple Silicon, then we just don't use thread local at all. With that LV  actually, we will be able to actually handle this like properly. So if I followed the same thing,  just build the object file, right? And then the, not still V compiler RT. Well, guess what?  Oh, sorry, I didn't, I didn't rename it. Let me do that. Just a by default. I wanted to stay  compatible with, with other linkers as well. But you'll be there you go. So it actually works. And  this is because it's Apple Silicon. Everything is subjected to address space layout randomization.  So you can be sure that we actually do the rebasing properly. Actually, the working out the rebase  was the last bit that I couldn't, I couldn't figure out because, for example, when I was building,  when I was linking the hello, hello world, it turned out that when I was plugging, so I was  getting a sex fault, but then I would stick it in the debugger and it would work fine. And that's  why I wrote this blog post about disabling, sorry, enabling a SLR in LDB, because actually I wanted  it to be like, you know, the addressing being randomized, because I was getting sex faults  when I would run it normally, and not when it would be run at the same address as it was specified  in the, in the binders for the start of memory. And it was super frustrating. And then I actually  discovered that it was basically me forgetting to the rebases on, on certain pointers that were  stored in the data segment. So, so yeah, that's done. And the cool thing is if anybody was following  the TLV stuff for Apple Silicon, it turns out it got changed a little bit. So I wrote a blog post  about like, kind of, it was a, it was a fix or experiment to see what's needed to fix it. And  there are two things I can actually quickly show this stuff. I'm just going to do it here.  A port wait, that one.  Uh, the LD. No, not fingers. Right. So, by the way, that actually the linker itself is very simple.  Okay. J2, your blog posts are on are both on your blog, right? Jakub.com.  Yeah.  Okay. So I'll link it in, in the main chat show so people can follow along. Yeah. Thank you.  Yeah. Can go there if they want to.  Right. Oh, right. We're like, relax here. Okay. So here you can see that we handle a couple of  different instructions that are passed from the compiler about relications. So we've got the branch.  Uh, so the branching basically, we've got, um, the page is basically an instruction how to, you  know, how many pages you have to advance. So this is for the ADRP for the address of the page.  So this is like basically going, you know, sifting through pages and increments of, um,  four kilobytes. And then you can narrow down with, um, the actual paid offset. So the page is  actually PC relative. That one is not. And previously, so I can only assume that what was  happening in LD, LD is that here they would have, there we go. Uh, so TLVP, that's for thread local  variables. They would also have the pages here and it would handle them normally. So the way it  works here, you can either have a load or store, or you can have an ad instruction. So you actually  have to, you kind of have to, um, parse what instruction is, is, is, is for you to fix up.  And then depending on what it is, you should either do an ad or you should do load and store  scaling. That's, that's how the ARM isa works. Um, and that's why I had, but it turns out that  TLV needs a separate thing. Um, which is basically that, so we have to, so I think this is what Apple,  Apple linker is doing as well. Uh, we need to parse the instruction to get out the registers that we  need to fix up together with the size, whether it's 64 bit or 32. But at the end of the day, TLV is  always handled by an ad. It's never a load or a store. So that's the first thing that's different.  And this is the stuff that was discussing in the blog post. And then the second one, and this is  the most interesting one. So, um, for all this, uh, the cl- actually LVM is specifying pointers  within the data segment. And we have to do rebases for this. And my current, the way I'm doing it  now is basically when I'm parsing the locations for the pointers, I'm then saving where this  pointer was. So then I can write that in the, in the rebase information for dyld, what to be  what needs to be rebased because otherwise with ASLR, it would just get garbage because it would  be pointing to addressing that doesn't make sense. And I was doing the same for TLVP. And I'm pretty  sure this is exactly what, um, what, uh, LAD is doing because I think that's how it was done for  x86, 64, but surprise, surprise, you can't do that with ARM like Apple Silicon anymore. So you  don't do any rebasing actually. What is happening, you have to zero out the memory as well for the  thread variable. So thread bars would be equivalent of data section within the data segment. So this  is where you would have your, um, read write data for the, you know, for your binary to basically  specify addressing and things like this, but you have to zero it out. And you still need to rebase,  uh, do you? Yeah, I think so. You still need to rebase, uh, maybe not. Wait, I don't think you  have to wait a sec. I'm still getting confused by all this. No. Okay. Yeah. So there's no rebasing  and stuff for the thread local variables. Weird stuff. Um, so, right, we can close that now. And we can,  I guess I can show where I'm going to hide this and this on. Okay, cool. I think I got it open. So  let's have a look. So this is the intermediate object file. And what we have here, right,  is that thread var is empty, but when you do the fix ups, you won't be able to see them because  there's too many. When you do relocations, it tells you that, you know, it's going to, you have to  specify addresses for that, for the, um, thread local variables here for the binary to do,  actually later use it, but actually that's not true. You shouldn't do that. And if we look for  main, uh, we have yet here. So global is actually the thread local variable. And you can see that  actually what we are meant to fix up here is LDR instruction, but that's also not correct. So  actually what is happening at, um, in fact here is that we do an add. And this is what the,  I was basically reversing, doing what Apple linker is doing. We do an add.  Furthermore, we have, uh, wait here, we have everything zero out. So there's no data here.  And I think, uh, we won't be able to get it's only just addressing. There's no, um,  no naming and, and rebase and for table. And that's it pretty much.  I think. On yet here in the symbols, if you look for global, you're going to see that. Yeah,  it's, it's correctly in the third bars and thread VSS for the, and it. So all things are good. Oh,  and there's one more thing. It turns out that, um, one more thing, this, this, this, this  magical TLV bootstrap, that was, um, that was meant to be an explicit, um, we, we, we needed to  create a bind like explicitly binding in the past. So you would either put in non lazy symbol  pointers or lazy symbol pointers, like lazy symbol pointers. That would be any lips functions,  like read, write things like this and don't lazy you to usually get, um,  uh, like actually a variables, I guess, like, or, or reference to the, why,  why, why, why, where, where was actually working out the lazy symbol pointers later on.  Nowadays, everything should go into lazy symbol pointers. Um, I did stick in the TLV bootstrap,  but I actually did. It turns out you don't even have to have to be bootstrap anywhere.  All you need to do, you just need to have the binding information populated properly here.  And that's it. Okay. My optimization I'll need to do. So if I understand correctly, there is no  initial setup done then for trial local storage. I'm understanding this correctly. Like it's just  memory that you have. Yeah. Yeah. It's handled for you. It's, it turns out that actually I think  it's handled for you. Like in the past, you would need to guide you by LD. And I think nowadays  it just, um, I think it just knows it goes through the simple table and it can work it out. So  you know, main differences. Okay. So more in general, what make you want to learn  all this stuff about my quest? Why, why didn't you just throw your hands up in the air and say,  well, whatever, I'm just going to switch to Linux. Oh, that's because we didn't have anything for,  um, for Mac back in the Bay. So we had, uh, so I think Andrew wrote the first draft for  incremental linker, uh, target in Linux, I think. So that's else. Right. And we had, I think Alex  wrote some, uh, like the first pass, he had a first pass at PE and costs. So that's mainly for  Windows, but there was nothing for, uh, for macOS. We had nothing. So I was like, well, might as well.  And I thought, yeah, I'm going to learn something. And I'm, this is about like five months.  And I use, you know, every single time like, oh my God, what now? So yeah, it's good fun, I guess.  Okay. So there was a comment from, uh, from before, um, by June who, um,  commented that, uh, I think it's a reasonable take that, uh, having complete parallel code pass,  um, is, is problematic, right? Because you have different code paths that, uh, are,  are you expecting to like all do the right thing, but, uh, they're different. So they might diverge.  And, um, so, and you're applied about the testing routine. Um, how does that work? The testing  routine for linker stuff. Um, how is it structured? Um, so we have, so basically, obviously we have  some unit testing for some bits, um, within the linker itself. Uh, so for example, I, uh, when I  was working on the export try, if you remember, that was also a lot of reverse engineering plus  looking up the, you know, the source code of the, um, the existing system linker and LLD trying to  figure out how to do, how to properly export information for the divi LLD to understand.  I added some tests there. And obviously we have those for a lot more things, just to be sure that,  you know, the tiny bits are properly encoded as bytes. But at the same time, we have a very  powerful end to end tooling for, uh, for testing. So we actually, you know, you, you basically write  source for the test, um, like zig, uh, zig, zig file and it's being compiled and then it's being  linked and then it's being run. So, so you can actually be sure that, um, and you can obviously,  because they get so powerful with comp time, you can actually do, um, a lot of very interesting  branching on different architectures and operating systems. And like I said, this manner, so that's,  that's perfect. So we will be actually able to test quirks on different, for different targets as well.  The only thing that we're missing, and this is kind of sad at the moment is, for example,  we don't have a CI for Apple Silicon. Uh, it's kind of difficult, you know, emulation. I don't  think we'll cut it because it's a, while the basic isa is, I think it's like bespoke. So this is the  standard, the standard isa that ARM has. There are some extensions that Apple has as well, which  make it basically specific to Apple. So I guess we'll need to actually test it. Like if we want  to run it, we'll need to test it on Apple Silicon unless we write an emulator, I guess, but that's  another project. Yeah, that's, that's one more yag to shave. Okay. So, um, uh, let me ask you a  quick question then. So since you like, um, Max so much, uh, are you going to get the new 16 inch  MacBook Pro if it has a better M, I don't know, M1X processor? Yeah, probably, but not immediately,  I don't think. Right. Because, um, so I've got this MacBook Air with M1 and it's perfect. It's  actually faster than i9. So that's really awesome, right? It doesn't have fans, so it doesn't really  get that hot, which is another plus. It's just, it's a really cool, it's a really cool piece of  software hardware, sorry. So I'm really impressed with the efficiency of the M1. It's, um,  but I guess everybody is right. Everybody who tried it, they were like, wow. Yeah.  So I'm not sure. We'll see how much it is, you know, when, how much.  I would love to see benchmarks comparing M1 with M1X and then to what it's called.  Okay. I think we have two to three minutes more. Another question. Where would you like the first  Z conference, like ERL conference to be? Where? Oh, that's it. Are we talking like after the pandemic  or now? Yeah, well, after the pandemic, let's assume that you have 20 seconds to answer. Let's  assume that the pandemic is over and you can go everywhere, but also, I know. Yeah, I don't have,  I don't have Andrew's address, but it's going to be there.
zig_macho_0_179.png zig_macho_180_15551.png zig_macho_15552_33671.png zig_macho_33672_36899.png zig_macho_36900_42047.png zig_macho_42048_45611.png zig_macho_45612_45719.png zig_macho_45720_45791.png zig_macho_45792_45875.png zig_macho_45876_45923.png zig_macho_45924_45995.png zig_macho_45996_46055.png zig_macho_46056_46187.png zig_macho_46188_48059.png zig_macho_48060_48371.png zig_macho_48372_49811.png zig_macho_49812_49859.png zig_macho_49860_51083.png zig_macho_51084_51383.png zig_macho_51384_53603.png zig_macho_53604_55511.png zig_macho_55512_56903.png zig_macho_56904_57275.png zig_macho_57276_57323.png zig_macho_57324_58847.png zig_macho_58848_59063.png zig_macho_59064_59411.png zig_macho_59412_59663.png zig_macho_59664_59891.png zig_macho_59892_60851.png zig_macho_60852_66419.png zig_macho_66420_68207.png zig_macho_68208_68459.png zig_macho_68460_69551.png zig_macho_69552_69923.png zig_macho_69924_70979.png zig_macho_70980_71147.png zig_macho_71148_82655.png zig_macho_82656_83951.png zig_macho_83952_84479.png zig_macho_84480_84551.png zig_macho_84552_85079.png zig_macho_85080_86315.png zig_macho_86316_86363.png zig_macho_86364_88535.png zig_macho_88536_88859.png zig_macho_88860_88967.png zig_macho_88968_91271.png zig_macho_91272_91487.png zig_macho_91488_91571.png zig_macho_91572_97799.png zig_macho_97800_97955.png zig_macho_97956_98003.png zig_macho_98004_98627.png zig_macho_98628_99251.png zig_macho_99252_99803.png zig_macho_99804_99995.png zig_macho_99996_104627.png zig_macho_104628_108287.png zig_macho_108288_109019.png zig_macho_109020_109127.png zig_macho_109128_111251.png