Analyzing ABI changes with Abigail
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Seketeli, Dodji
Libabigail is an infrastructure for semantic analysis of ELF binaries containing C or C++ programs. It powers command line tools like ‘abidiff’, which let users compare ABI changes between two different versions of a given ELF binary by analyzing just the binary and its ancillary debug information. The result of the binary comparison is a kind of hierarchical ‘diff’ which shows changes up to the types of the interfaces that constitute the ABI of an ELF program. This infrastructure allows software distributors (among other actors) to build specific tooling to review and analyze ABI changes that might occur whenever a shared library package is updated. That tooling might even be tailored to automatically prevent packages with unwanted incompatible ABI changes to reach users. This talk intends to present Libabigail, its architecture, its capabilities, its current limits, its associated tools and how it might be used to further build highly tailored ABI verification tooling. The talk will also explore the potential improvement paths that are currently identified, and from the feedback of the audience, explore improvement paths that are not yet identified.
 So, hello everybody. Thank you for attending this talk. I know it's right after lunch, so you might be... Well, thank you for being here. So yeah, my name is Dujie. I work for Red Hat in the tools team. And my background is, well, a little bit in compilers and debuggers. And since a little while, I've been working on static analysis of ELF binaries to infer interesting things about ABI and stuff. So as a result, we came up with a framework named ABGAL, which is an acronym for ABI generic analysis and instrumentation library. Well, it's also the name of the wife of my... Well, the guy we started the project with. So, anyway. So we use ABGAL to basically compare binaries and come up with reports about the differences they have as far as ABI is concerned. So this can be really interesting to folks ranging from developers up to people pushing bits coming out of the developer's hand, namely these two people. And I thought it would be interesting to come and talk to you guys about what we've been doing in this area. And maybe we can find areas where we can work together and improve what we have or change directions, well, and discuss. So in today's talk, I'm going to first try and define what we mean by ABI. This is a quite fluid concept. It's not well-defined. And so it means different things for everyone. So I thought it would be an interesting take to try to define what we actually mean by that in this project. And then we'll talk about what we mean by ABI compatibility once we have a definition for the ABI. And then I'll jump straight to some examples of change reports we can have with the framework we have today. Once we have that, we'll dive into what ABGAL is, what it does, how we perform those static analysis tasks, and see how it is used in real life today. And then we'll talk about possible improvement. And well, I say we'll talk because I'll present something. But if you have ideas, please feel free to come up with those. And so here we are. So to define what an ABI is, well, let's set a context first. OK, suppose we have a binary, which we'll name E. And that binary uses a code from another binary that we call L. So basically, E can be an executable, well, that's where the E comes from, or a shared library. If there is something that is not clear, please just stop me right ahead rather than waiting for the end. So that's what E is. And L can be, well, a shared library. It's a library. That's where the L comes from. And it can also be a dynamically loaded module. So this is the basic context. So at execution time, what happens is that E expects properties from L. In this context, I wanted to say that we're talking mostly about L, the L Linux format right now. This is what we support. We could support other things in the future. But right now, we're talking about L. So E expects properties from L. At execution time. So those properties can be things like the format of the binary, the architecture, well, an executable, an x86 executable will expect an x86 library, for instance, right? x86 dwarf, well, elf will expect an x86 elf library, for instance. It will also expect the presence of certain elf symbols. Those coming from either functions or global variables or other things. Specific layout of data. When I talk about this, I mean types. Specific types of data and specific size, alignment, offset, et cetera, et cetera. There are also other stuff like calling convention, et cetera, et cetera. But the first three are the things we're going to focus on more here. So I wanted to stress also that those properties are structural. They talk about the structure of the program, not its behavior. We're not talking about bugs where we were supposed to add two numbers and we are now dividing them, things like that. We're not talking about dynamic things. We're talking about structure. You know, types, layout, things like that. So those properties that are loosely defined somehow are what we call the ABI. Those properties that E expect from L. Those are the ABI of L. So more specifically, when we talk about the ABI of a library or a binary in general, we'll be talking about the set of symbols that it defines and exports. And also the layout of the data expected by those symbols. So from now on, I will stop talking about symbols. I'll talk about functions and variables because, well, that is what programmers care about, even though a function in source code end up being, okay, usually a symbol in L. But we want to talk about things that are meaningful to developers. So I'm going to talk about now functions and variables now. So one thing to keep in mind is that ABI changes are there to stay. They're inevitable in our free software world. I mean, we want our software to evolve. You know, we've been talking about this all these three, during all these three days. So we don't want to cast things in stone and say, oh, no, you can't change anything anymore because you're going to break ABI, whatever that means. Things are going to change and the ABI is going to evolve because we are fixing bugs, we're adding features, we want to kick us, basically. So we're going to add new functions, we're going to add new global variables, how we are going to change the signatures, types, and whatever of the existing functions, all those things are going to happen. And we want these things to keep happening. And then what we want is to be able to detect the subset of those changes that are harmful. So can you read this? Yeah. So in this context to us, only what we call ABI incompatible changes are harmful. For instance, if you remove a function that was there and that was there in L in the library and that was expected by the executable, that is a harmful ABI change. It's an ABI incompatible change because existing Banner is executable Banner is that are in nature. Well, we'll expect that function that you removed from your newer version of the library. The same thing goes for incompatible data layout changes. Basically you have a type in which you removed a structure in which you removed a data member, things like that. Or you removed or added a function parameter, things like that. So you see I'm not talking about symbols anymore at this point, even though we're looking at Banner is. And it follows that ABI compatible changes are fine. They're fine, but we might need to see those, to still see them and review them and be sure that this change, even though it's not bad, is really a kind of change I expected, just from looking at the binary. So one thing to keep in mind is that we want to be able to detect all those things by looking at the binaries only, not by looking at the source code. This is really important to keep in mind because there are many reasons why we want this, but I think most of the distro people get this. When you have, well, compilation is a kind of destructive process. There's some information that are not there anymore. But there are also some information that are there that you can't see when you look at the source code. For instance, when you look at, well, we'll see that later in some of these examples that I have, but when you look at a data structure, a class or a struct, for instance, you don't know what the offsets of the data members are just by looking at the source code. You have to think, right? But when you look at the binary, boom, you have those information. So looking at the binary is really interesting because it reveals things that are hard to see when you just look at the source code. And we want to be able to detect those things as soon as possible. It's not when you are building the final stage of the distro that you want to detect those things as soon as possible in the dev process is what we want. And we need to keep in mind that there is no magic here. Most of the interesting changes need a human review to know if they are harmful or not. So you need to make those changes stick out from the noise and then have some folks to review them just like you review patches. So for that, what we want to use is to use the diff paradigm we have already. To let people review those ABI changes just as we review patches today, which is something we don't do yet, right? But well, this is the philosophy we're following while developing those tools and framework and so on and so forth. So yeah, I danced ahead the music here. So let's look at some examples. Okay, so this is the place where the time where I switched it. So in this case, I just wanted to show you something very quick. I have a friend here, Frédéric, who told me once, it was probably 15 years ago or something, showing code about what you're demonstrating is a really bad thing to do, right? But here, I'm not showing the code of Libabigel. I'm showing the code, well, Libabigel works on code, so it's different what I'm doing. So here, I wanted to show you, I hit this. So here you have a small function, F1, that has one parameter. And here you have in a new version, F1, another version, in which I added a parameter, right? And I probably, yeah, that's the only thing I did, I guess. So I wanted to show you what a Libabigel-based tool sees. So I've compiled these things. Well, I can run the compiler again, just to show that I'm not... I'm not... Ah, yeah. This is an artist setup, and it's like I'm on a piano or something. Yeah, okay. So I compiled this, and I'm just calling a tool named ABIDIF, and I give ABIDIF the first version of the binary, LibExample1. Yeah, I wrote this this morning while having a chat with a fellow here, so I couldn't find a better name. I was more involved in our discussion. So LibExample1.so.zero compared to LibExample1.so.one. Oh, boom, oh, sorry, I should put that in a red-rex that into a file named ABI. So have you seen what I did? So I called a program named ABIDIF, right? It's a program part of the Libabigel toolset that compares the ABI of two binaries, ABIDIF. And so here is the results of what it says. So can you read that? Yeah? Okay, it just says that... Here, so it says the function f1, like, you know, with its signature, has some changes in one of its subtypes, right? And the change is that the parameter 2 was added. That's all. And this is actually what we did if I do a diff of the source code. Sorry. V1. So you see here the sort... Do you see that? Can you read that? So this is the diff. And so this is... And so this is the source code. And this is an example of the kind of report you get, you know, from looking at the binary. So you see that even though you're just looking at the binary and its debug information, because I've compiled this with debug information enabled, you can have quite some detailed information, right? Like, like, light numbers, etc., etc. So we can move to something more involved with, you know, C++. So here we have... In blue here we have a class. Don't really need to read the details. We have a class here. And that class is used by another class, a structure. It is used here because we use C. You know, class name is C and we use C in another type underneath here. And you see, I still have a function named F1 that uses C. And I have a function named F2 that uses C as well here, right? And of course, on the second side of the screen there, the thing that is highlighted in blue is... Well, I created a base class. So I changed C and made C inherit a new class like this. You know, I was listening to the fellow over there. So this was my... It's because of him, you know, that inspiration came from you. So I just made this change. And well, I was like, okay. How about adding a virtual here, for instance, in the class? The idea is just to make some changes and see what ABI div says. And I kept F1 as is. I didn't change the signature. And I kept F2 as is too. Oh, well, there is also an F3. But anyway, let's go. Did I try it? Oh, yes. So I compiled it as previously. And I guess... Oh, yeah. I'm putting the result into a file. So I'm running ABI div on the two binaries, putting a result into a file, text file, and then I'm opening the text file in Emacs. So the changes are interesting here. So first, let's look at the summary of the changes that is shown by... Let's all say that there is one function that was changed. Change means that one of the types used by the function, or subtypes, you know, types of those types, changed. Right? And it also says that one function was added. Adding a function is not a problem. As I said earlier, it's an ABI change, but it's not necessarily a problem. But we still want to see it and review it and be sure that this change is really what we need. So we see that... And it says that the function that I added was C, the getM0. Did I add a function? I don't think so. I mean, I don't remember. So let's look at... So this was the first class I had. So you see getM0 was already there in the first version. And in the second one, what did I do? In the second one, I still have getM0. And you see I had a little change here in F1. In F1 here, I called... You know, I invoked the getM0. You see here? In blue. Whereas in the first case, there was no code. Okay. So this is C++. GetM0 here is inlined. You know what inline means? So it's not generated per se. Oh, sorry. Yeah. GetM0 is inline. So it's not generated, right? By default. But in the second case, it is generated. GCC generates it because it is called. Because someone uses it, then it is generated. You know? This is also because I compiled without optimization. If I compile with optimization, we can try that later because time is flowing. You'll see that this added function will disappear because of optimization. The thing, the function is just inline and not generated as a symbol, right? So these are the interesting things that you see when you just look at the binaries rather than looking at the source code. The same source code can produce different binaries depending on how you compiled it. How you change the code, et cetera, et cetera. This is what I meant by when I was saying we want to look at the binaries only. Well, being able to look at those. So added function. You see? And in here, you see the symbol of the function, right? And here, you see its signature, right? And so let's look at the changes, what it says. So, first, it says that the return type of F1 changed. Of course, that's true. I changed it from void to int. So we catch that. So you see that just by looking, okay, there are some tools around that would say, that we'll just look at the symbols, and because in C++, the symbols are mangled, you know, you could derive the types used by the function from the mangling. Do you follow me? Yeah, people say that. People think that. Well, they're wrong. Because, for instance, the return types of functions are not part of the mangling. Only the parameters are part of the mangling. And here, I made that change on purpose. You see that the return type of the function changed. And we want to catch that, too. So we really need to go look at the debug info and analyze all that, you know. So you see here that the next change it shows us is that the first parameter, which is a pointer to C, pointer to the class C, changed. Well, and how did it change? It changed because the thing that is pointed to by that pointer, which is C, changed. How did C changed? It had one base class insertion. You remember I said that I added a new base class. I made C inherit a new class. So we can catch that, too, by looking at the parameter. This is what we do. And as a result, well, as a result of all those changes, the size of C changes, too. This is something you cannot see by just looking at the source code, right? Well, you can see that it might change, but you're not sure. Actually, the reason why the size change here is because of the virtual. You remember I added a virtual keyword to the destructor. So because of that, virtual that I added, a V table got added to the class. And the V table is added at the beginning. And it's the V table. The V table is the reason why the size changed here, actually. So we see here that there is a V table insertion. And we can say at which line it was inserted and the V table offset, et cetera, et cetera. And as a result, because something got inserted at the beginning of the class, the offset of all the data members of the class changed, too. So we have these changes, too. Just by adding a virtual, right? So yeah, we can learn a lot by just looking at the binaries. So now I guess I'm done with showing off what we can do with this kind of tooling. So what do we do behind the scenes? Basically, Abigail and, well, what I call it, Libabigal, because it's the name of the project. Libabigal being the library that implements all the stuff and also a library of tools that come on top of that library. So basically, we represent ABI artifact. What does that mean? We have an in-memory models for types and declarations, just like a compiler does, just like LLVM does. So Abigail represents color types, integers, characters, and so on and so forth, aggregate types that are arrays, structures, classes, unions, blah, blah, blah. And declarations, a declaration being something that has a name and a type. So all those things are represented. You can write code saying, okay, this is a function. The function has parameters, and I'm adding a new parameter. You can manipulate those things. On top of that, Abigail has a model of what I call bundles of ABI artifacts. The first bundle is obviously a translation unit. So whenever you look at a binary, the binary has debug information, when it has, and those debug information describe things, translation unit per translation unit. And it is inside those translation units that you have declarations and their types. So we represent those translation units in the Abigail model, and we represent what we call ABI corpus, and ABI corpus is the representation of a set of translation units. So basically a shared library, the ABI things, artifacts of a shared library, is represented in Abigail's pick as an ABI corpus. That's how we call it. And once we have ABI corpora, yeah, because the plural of corpus to Latin word is corpora, I was told anyway. So when you have those corpora, you can, well, compare them. And so as usual, we build a model of the diff. If you make a parallel with the GNU-Diff tool, when the GNU-Diff tool computes the difference of two files, what does it do with the difference? It just emits it. If you look at the source code, it's what it does. It sees the different boom, it emits it as a text. It doesn't do anything with that difference. In our case, we build a model of the difference, and we actually build a graph of those differences. So if there is a difference between two structures, for instance, that difference might come from the fact that one of the data members of those two structures changed. So you will have one tree representing the two structures, difference between the two structures, and a child tree note representing the detail of that change, the detail being the change of the data members. Do you understand what I mean? And like that, you have a full graph of the changes in memory. Once we have that, we can categorize the changes. I mean, we can walk the changes and mark them saying, okay, this change is that kind of change because, I don't know, for instance, in C++, you can have a change about, I don't know, private data member becoming public. This is a change. But it might not be the same kind of change as removing a data member. So we will put the first change in a category named private change category, and the second change will be put in another category. So categorizing changes. One of the good things about categorizing changes is that we can see changes that come over and over. In the previous example I showed you, you've seen that there were two functions using the class C, right, function F1 and F2. If you don't remember, I'm telling you again. But then the reports, in the reports you saw the change about F1 only, saying F1 change, blah, blah, blah, and then because of C. And you didn't see the change about F2. Well, F2 changed to because of C. You didn't see it because the tool categorized the changes on F2 as being redundant. And why? Because it's the same C that changed that we've seen before. If we have time, we can go back and I'll show you how to, you know, ABI Dev has an option to show you the redundant changes too if you want to see everything. And we can do this kind of analysis because we keep a model of the changes in memory. We don't just emit those changes as soon as we see them. One thing that is interesting too that I wanted to show but time is moving and then, well, we can come back to that later if we're earlier is what we call suppression specifications. You guys are, you know about Valgrind, right? Yeah. In Valgrind you have suppressions. You know that. It's the central, it's, to me, it's the most, it's the best thing ever in Valgrind, right? You can, you have an error that keeps popping. You know about it but you don't want to see that error again because for you it's noise, right? Suppression specifications. We have the same thing in Babigel actually. So and it's a bit more complicated, well, let's say, you know, evolved than in Valgrind because you can say things that like, you can say things like, don't show me changes about a type which name is Foo. But you can also say, okay, don't show me changes about that type if the change is an addition, a new member, add it at the end of C. Or you can say things like, don't show me changes about function F if F is part of, if F is part of a file named blah, blah, blah. You know, there are many, many different things you can express and of course those things come from users. We're saying, I have this case where, you know, I would like to, you know, okay, can you do something? You know, and we discuss the price and, you know, we end up with something, a new category. So we can do, we do this kind of things. So afterwards we report about the changes and reporting means again, working the graph of diffs and depending on the category of a diff node, report about it or not. If the category says, oh, this node has been suppressed by the user, then you won't report it. Right? If the category says, oh, this is a type diff name change. Sometimes you have changes like that. You have an, I don't know, you have, somewhere you have a type that is an integer and someone renamed that, well, someone defines a type diff saying, you know, that integer's name is now fancy type, for instance. So this is not a real, you know, it's a change, but it's not necessarily harmful. Right? Some folks want to be able to, you know, dismiss, well, suppress those things. So we have a category for those kind of changes too. So the reporting engine knows about categorization and decides to show things depending on that. And of course, for now we only have one kind of text reports. But we could have more, but well, that's what we have. So we've just seen what we can build as an in-memory model. Right? So where do we build that model from? We build it from reading ELF and DWARF. DWARF is the format for debug info in ELF, basically. Those guys like Tolkien's books and well, I guess, but anyway. So in Libabigel, there is a reader component that knows how to read ELF, so it's written in C++. And from that, it builds the Abigail model I was talking to you about earlier in memory. So it does that for shared libraries, object files, and also actually also executable if only if, even if I don't mention that here. Once we have that, once we can read from ELF, build a model, well, we can also write it down in files, obviously, so that people can define snapshots of the ABI of a binary or a package. You have a binary, you don't want to put that into Git. Well, you can extract its ABI and stash that ABI into Git. The format is an XML ad hoc format we came up with and we call that ABI XML. And of course, we can read it back. So this is interesting because once you read it back, well, you build an Abigail model and so it lets you be able to compare, well, I'll talk about that later, you can compare a binary against an ABI XML file because you can read both. We come up with the in-memory model and then we can work on the in-memory model. So which brings me to the tooling around the Abigail. So I'll just skim quickly through some of the tools we have today. So first we have ABI DW which emits, you know, it serializes the ABI representation of a binary to a text file, that's a tool. We have ABI diff that you have seen. So ABI diff can compare two binaries, two ABI XML file or one ABI XML file and one binary, you know, things like that. So you can compare an ABI, a binary against a baseline. There is ABI package diff that compares the ABI of binaries that are inside RPMs or depth files. So that was, yeah, written for someone else and now I maintain it. And the last one that came to the family is Fed ABI package diff. These guys keep coming with longer names. We need to do something about this. I didn't write this one. It's someone else, a colleague who wrote it on his own time, by the way. So this one is interesting. It talks, okay, ABI package diff just so that you have things in mind. It works on RPMs. But when you work on RPMs, the debug info is split. In another, it's in a separate package. So you've seen with ABI diff, when we want to compare two binaries, we just need the two binaries. If you want to compare two packages, you need actually four. Yeah, because you need the two debug info packages. So the command line starts being a bit longer. You can actually need even more in certain cases because ABI, ABIgal sees everything, basically. I mean, there are some changes. I should show you some examples. But there are some changes to some types that some developers don't want to see. Because they say that, oh, this type is private, whatever that means. In EF, there is no such concept as a private type. But what the developer means by that, it means that type was not defined in the public header shipped by the package or whatever. That's what it means by private. But in the binary, we see all the types. We don't care about, so we tell you about all the changes. So people came to me saying, oh, I don't want to see the changes about those types. So there is an option to ABI package diff in which you give it the develop packages in which there are the header files. And so it won't show you changes about types that are not defined in the header files. So six packages. You need six packages just to run ABI package diff. That's a bit too much when you type command lines. So Fed ABI package diff is quite handy because you type it. The name is long, but then afterwards it becomes shorter. You say Fed ABI package diff dash dash from, you say Fedora 10, I don't know, dash dash to Fedora 20, HTTP, the HTTP package. So it will go fetch all the packages, the HTTP packages from Fedora 10 and 20, you know, the debugging for packages, develop packages, whatever, compare them and show you the difference. So that's what it does. So maybe we would need something like this for OBS or, wow, just saying. Anyway, so this is the part of the talk I'm going to change if I go talk to the Debian guys. But anyway, so I'm talking out loud here. It's a chance that it's not recorded, right? So doing this, we faced and we're still facing some interesting challenges. We, well, speed and space matters a lot, a lot more than what people think. When you're dealing with something like Alibaba Gale, you are looking at the entire binary. And usually your compilation tool chain, even your compilation tool chain usually doesn't do that unless you do something called LTO, but we're not, we'll put that aside. Usually your compiler just compiles things file by file, translation unit by translation unit, right? And the linker that sees everything doesn't analyze types, right? So it doesn't see types either. We see all the types of all the translation units of the shell library so that it can be a lot, a lot, especially when you have like big C++, you know, binaries. And this is also due to the fact that the same types get included over and over and over again. Did I forget an over? Okay, over again. This is thanks to the famous hashing clue thing that I love, right? So think about it. We build representation, in memory representation for those types, for all of them. If we have hundreds of thousands of types in memory, that can do a lot. You can end up with gigabytes and gigabytes of memory used just to represent those types. And you need two copies because of course you're comparing two libraries, right? So we need to do something about that. We need to de-doplicate those types. So basically what it means, compiler people have fancy names for easy concept. When you see a type foo, first time, okay, type foo, and then you go to the next translation you need and you see type foo again, well, you shouldn't rebuild an in-memory representation for this foo. You should reuse the first one you saw. De-doplication. But to do that, so it's an easy concept. But then to do that, you need to be fast because how do you know that the second foo is the same one as the first one? It is not because it is named foo that it is necessarily the same as the first one. In C++ normally it should be the same because there is something named the one definition rule, which is a C++ rule saying that if two entities have the same name, then they ought to be the same entity. But then in C, you don't have such thing. Yeah, C, anyway. So in C you can have foo in a file. The same, okay, a different foo in a other file. The two foos are exported, well, used by external functions and they're not the same. So to be able to know that we want, well, we want to de-doplicate things. So we need to know if the second foo is the same as the first one. So we need to compare them. And comparison by construction is an exponential problem. You compare things member-wise and those members have types themselves that are structures that have members that are structures that blah, blah, blah, and then you end up, you know, it's exponential. And seriously, I mean, if you do these things naively, it can take hours to complete just to compare two packages. So we need to use some heuristics to transform the problem into a more linear one so that things can complete quickly. So it's kind of, there are some interesting, you know, graph algorithms underneath. We also need to control, well, to avoid seeing things we don't want to see. And when I say this, when I say we, the we depends on the project. A change, an ABI change for your project might be harmless and that very same change might be harmful for his project. He might want to see those. So we need to give you ways to, you know, yeah, to avoid seeing what you don't want to see. So these are, this is why we came up with suppression specifications, but it goes further. For instance, if you have a file named dot, well, that ends up with dot ABI ignore, ABI ignore in your package, in your RPM file. ABI package will detect that and use it, for instance, as a suppression specification. Right. And think about it, if you build, for instance, an ABI verifier at the distro level, you will need to provide users with this kind of capabilities, provide them with the ability to say, okay, this change is okay. I don't want to see it again in my next runs. And of course, this is a library that we're talking about, not only tools, so it needs more documentation. So basically, all the entry points need to be documented, and we're doing that already, but we need more documentation about the inside and things like that. It needs, and we're doing, I mean, the things that are in this section challenges are things we're doing already, but I wanted to, you know, stress on them so that you guys are aware of that. We need a huge set of banners for regression testing. Elf and dwarf is a very loose format. You can put everything, anything in elf or in dwarf. So unless you test stuff, you don't know what you are expecting or what you're really doing. So we need a lot of tests. So the LeBabieGalTarble is more than 100 megabytes of size. The reason is that we have lots of banners just for testing when you make check. So yeah, and we need even more than that. And now what we're planning to do is to actually use Fed ABI package diff or something else to go actually use the packages in the distros and distro histories to use that as regression testing. For instance, if you compare a package against itself, it should yield the empty set. For instance, just doing that on thousands of packages will be good. So this is really important, just like we test comparlers. So in Fedora, we're using this thing to compare the ABI of new packages that are pushed. So it's been in production since a couple of months. So yeah, basically, we perform an ABI diff of the new package against the previous stable one, and it sends a friendly message to the package maintainer. So the changes are categorized. Now you know what category is, basically, but roughly in this case, it's less detailed than what we can do, basically. But this is what they wanted. If, well, there are certain kinds of changes for which we are sure that they're going to cause a problem. Incompatible ABI changed. For instance, removed symbols, for those, those are flagged as failed, for the build, basically. And the most interesting changes, for instance, the changes I show you about the subtypes of functions, those are flagged as need inspection. If your package, well, so the package maintainer needs to inspect the package in those cases, and otherwise the package is flagged as passed as far as this ABI test is concerned. So it is based on Taskotron. I think you guys know about that. The task name, the task that runs on each package is named ABI check. You can Google that. And it uses ABI package diff to perform those things. It's written in Python. So that's it. At the same time, package maintainers can use Fed ABI package diff, so they don't have to wait until they're submitted their package to try to see ABI changes. And so it is in production. The limitations, you know it. It only works on C and C++ shared libraries today. But, well, tomorrow, if folks doing Rust want support, I mean, if it is Elf, we can support it. We can do some work. And it just runs on a small set, well, small set, several hundred packages of, you know, packages that are in the critical path. They call that. And improvements that we need. So I've just ticked this here. So I'm currently working on supporting, you know, the comparison of Linux kernels. This is, well, it's well underway. The Linux kernel is huge because we're not just comparing VM Linux. We're comparing the union of VM Linux and all the modules. There's a couple of those. And the Linux kernel has its own way of handling Elf. So it's quite interesting. So we need more, you know, better support of C and C++ language construct, mostly C++, you know, the new stuff, lambdas, blah, blah, blah, you know, the new stuff. It keeps coming. It just needs work to keep. Because we basically have, it's like a compiler where we have, you know, you need to parse. There is a front end, which is the dwarf thing, which builds a middle-end representation, which is, you know, the Abigail model, and then, well, the back end is emitting reports. So people will come with better API change categorization requests, saying, ah, this change, you categorize it as harmful and harmless. Please, can you put that in harmful place? And we'll discuss and, you know, we'll see, you know, how we can do that. Some people want additional API change reports. You see here in the reports I show you first, we show the changes, but we also show the impact, you know, what are the functions that are impacted or global variables. Some people just want to see the types that changed and don't care about the functions that were impacted because there are so many types, you know. So it's a valid request. So we would need, well, we can do that. We have all the information to do that. We just need to, you know, emit other kinds of reports for that. Some people want more friendly web reporting. We need work from that. And someone even came with a crazy idea of something web, blah, anyway, you know, you submit two packages and the thing does its thing and shows you web reports, you know. Anyway, so if someone wants to do that, it will be great. But I don't know if, well, maybe I can try and learn some JavaScript. Anyway, so yeah, that's it for me, I guess. Now is the time for question. I'm sorry, but I finished on time. Now you should do the same. You have two minutes. Anyone? What architecture does it support? Only x8664 or so? Well, right now, well, it is known to work on x86, ARM, PowerPC, you know, and even stuff. The right answer is if we have DWARF, if we on ELF, it should work. PowerPC showed that there are some corner cases sometimes, but in theory it should work. Yeah. Another question, does it also catch calling convention changes? This is an interesting question. Right now, no, short answer. But there is a, I have a bugzilla open to support that. Not necessarily calling convention changes, but not necessarily only that, but things that are more subtle than that. For instance, a parameter, for instance, the second parameter of a function, the way that parameter is passed can change from a compiler to another. For instance, you have an LLVM generated binary that passes that parameter on the stack, and GCC will pass that or the other way around in registers, things like that. So the idea is to have, is to be sure that we have that information in DWARF. And if we have that, and I think we have part of it, then we can implement it on our site. Well, if the structure of, in C, in the size of the structure changes, can you somehow detect from the library code whether it is IBA compatible or not? This is a problem when the function fills the whole structure, including the last member and the size is in the public interface, then it could override a data. Okay. To me, so if I understand the question correctly is if the size changes, can we say if that change is compatible or not? Right. That's, well, to answer that question, well, I would say that if that size change impacts another type that uses that type which size changed, then it's ABI incompatible. But then there is not only an impact, the impact is not necessarily only on other types. It can be on code too. You can have code that says size of that structure. So my short answer will be if the size changes, it's bad, period. If it's not bad, then you haven't looked close enough. So yeah, because the question came out over and over. And for instance, if a code says size of, that size of is lost because it's a constant and we don't see that in the generated code anymore. So I don't think we can really be sure that we're doing the complete impact analysis of these kind of things. And I think we would err on the safe side by saying that this is bad period. If you do it, you know that it's potentially bad. Yeah. It's bad. You mentioned that you were looking for ways to identify whether two certain structs are the same and C, which doesn't have to you ODR. Have you thought about building like an MD5 sum of a struct and then comparing that? Oh, well, to us, the problem is solved. I mean, well, it's solved. The question is interesting. If to build an MD5 sum or a sum, you need to walk the structure. Right? What I want to avoid is to walk the structure. Exactly. And that is what we do. That's what I meant by saying that trying to transform the problem from an exponential one to a linear one, this is basically the kind of technique we use. We use some kind of hashing to avoid walking over and over. We just walk once per type, name and kind. Yes. It's just that the hash that we end up using is not, we don't save it. It's just in memory. So I guess, thank you very much.
 So, hello everybody.  Thank you for attending this talk.  I know it's right after lunch, so you might be...  Well, thank you for being here.  So yeah, my name is Dujie.  I work for Red Hat in the tools team.  And my background is, well, a little bit in compilers and debuggers.  And since a little while, I've been working on static analysis of ELF binaries to infer  interesting things about ABI and stuff.  So as a result, we came up with a framework named ABGAL, which is an acronym for ABI generic  analysis and instrumentation library.  Well, it's also the name of the wife of my...  Well, the guy we started the project with.  So, anyway.  So we use ABGAL to basically compare binaries and come up with reports about the differences  they have as far as ABI is concerned.  So this can be really interesting to folks ranging from developers up to people pushing  bits coming out of the developer's hand, namely these two people.  And I thought it would be interesting to come and talk to you guys about what we've been  doing in this area.  And maybe we can find areas where we can work together and improve what we have or change  directions, well, and discuss.  So in today's talk, I'm going to first try and define what we mean by ABI.  This is a quite fluid concept.  It's not well-defined.  And so it means different things for everyone.  So I thought it would be an interesting take to try to define what we actually mean by  that in this project.  And then we'll talk about what we mean by ABI compatibility once we have a definition  for the ABI.  And then I'll jump straight to some examples of change reports we can have with the framework  we have today.  Once we have that, we'll dive into what ABGAL is, what it does, how we perform those static  analysis tasks, and see how it is used in real life today.  And then we'll talk about possible improvement.  And well, I say we'll talk because I'll present something.  But if you have ideas, please feel free to come up with those.  And so here we are.  So to define what an ABI is, well, let's set a context first.  OK, suppose we have a binary, which we'll name E. And that binary uses a code from another  binary that we call L. So basically, E can be an executable, well, that's where the E  comes from, or a shared library.  If there is something that is not clear, please just stop me right ahead rather than waiting  for the end.  So that's what E is.  And L can be, well, a shared library.  It's a library.  That's where the L comes from.  And it can also be a dynamically loaded module.  So this is the basic context.  So at execution time, what happens is that E expects properties from L. In this context,  I wanted to say that we're talking mostly about L, the L Linux format right now.  This is what we support.  We could support other things in the future.  But right now, we're talking about L. So E expects properties from L.  At execution time.  So those properties can be things like the format of the binary, the architecture, well,  an executable, an x86 executable will expect an x86 library, for instance, right?  x86 dwarf, well, elf will expect an x86 elf library, for instance.  It will also expect the presence of certain elf symbols.  Those coming from either functions or global variables or other things.  Specific layout of data.  When I talk about this, I mean types.  Specific types of data and specific size, alignment, offset, et cetera, et cetera.  There are also other stuff like calling convention, et cetera, et cetera.  But the first three are the things we're going to focus on more here.  So I wanted to stress also that those properties are structural.  They talk about the structure of the program, not its behavior.  We're not talking about bugs where we were supposed to add two numbers and we are now  dividing them, things like that.  We're not talking about dynamic things.  We're talking about structure.  You know, types, layout, things like that.  So those properties that are loosely defined somehow are what we call the ABI.  Those properties that E expect from L. Those are the ABI of L.  So more specifically, when we talk about the ABI of a library or a binary in general, we'll  be talking about the set of symbols that it defines and exports.  And also the layout of the data expected by those symbols.  So from now on, I will stop talking about symbols.  I'll talk about functions and variables because, well, that is what programmers care about,  even though a function in source code end up being, okay, usually a symbol in L.  But we want to talk about things that are meaningful to developers.  So I'm going to talk about now functions and variables now.  So one thing to keep in mind is that ABI changes are there to stay.  They're inevitable in our free software world.  I mean, we want our software to evolve.  You know, we've been talking about this all these three, during all these three days.  So we don't want to cast things in stone and say, oh, no, you can't change anything anymore  because you're going to break ABI, whatever that means.  Things are going to change and the ABI is going to evolve because we are fixing bugs,  we're adding features, we want to kick us, basically.  So we're going to add new functions, we're going to add new global variables, how we  are going to change the signatures, types, and whatever of the existing functions, all  those things are going to happen.  And we want these things to keep happening.  And then what we want is to be able to detect the subset of those changes that are harmful.  So can you read this?  Yeah.  So in this context to us, only what we call ABI incompatible changes are harmful.  For instance, if you remove a function that was there and that was there in L in the library  and that was expected by the executable, that is a harmful ABI change.  It's an ABI incompatible change because existing Banner is executable Banner is that are in  nature.  Well, we'll expect that function that you removed from your newer version of the library.  The same thing goes for incompatible data layout changes.  Basically you have a type in which you removed a structure in which you removed a data member,  things like that.  Or you removed or added a function parameter, things like that.  So you see I'm not talking about symbols anymore at this point, even though we're looking  at Banner is.  And it follows that ABI compatible changes are fine.  They're fine, but we might need to see those, to still see them and review them and be sure  that this change, even though it's not bad, is really a kind of change I expected, just  from looking at the binary.  So one thing to keep in mind is that we want to be able to detect all those things by looking  at the binaries only, not by looking at the source code.  This is really important to keep in mind because there are many reasons why we want this, but  I think most of the distro people get this.  When you have, well, compilation is a kind of destructive process.  There's some information that are not there anymore.  But there are also some information that are there that you can't see when you look at  the source code.  For instance, when you look at, well, we'll see that later in some of these examples that  I have, but when you look at a data structure, a class or a struct, for instance, you don't  know what the offsets of the data members are just by looking at the source code.  You have to think, right?  But when you look at the binary, boom, you have those information.  So looking at the binary is really interesting because it reveals things that are hard to  see when you just look at the source code.  And we want to be able to detect those things as soon as possible.  It's not when you are building the final stage of the distro that you want to detect those  things as soon as possible in the dev process is what we want.  And we need to keep in mind that there is no magic here.  Most of the interesting changes need a human review to know if they are harmful or not.  So you need to make those changes stick out from the noise and then have some folks to  review them just like you review patches.  So for that, what we want to use is to use the diff paradigm we have already.  To let people review those ABI changes just as we review patches today, which is something  we don't do yet, right?  But well, this is the philosophy we're following while developing those tools and framework  and so on and so forth.  So yeah, I danced ahead the music here.  So let's look at some examples.  Okay, so this is the place where the time where I switched it.  So in this case, I just wanted to show you something very quick.  I have a friend here, Frédéric, who told me once, it was probably 15 years ago or something,  showing code about what you're demonstrating is a really bad thing to do, right?  But here, I'm not showing the code of Libabigel.  I'm showing the code, well, Libabigel works on code, so it's different what I'm doing.  So here, I wanted to show you, I hit this.  So here you have a small function, F1, that has one parameter.  And here you have in a new version, F1, another version, in which I added a parameter, right?  And I probably, yeah, that's the only thing I did, I guess.  So I wanted to show you what a Libabigel-based tool sees.  So I've compiled these things.  Well, I can run the compiler again, just to show that I'm not...  I'm not...  Ah, yeah.  This is an artist setup, and it's like I'm on a piano or something.  Yeah, okay.  So I compiled this, and I'm just calling a tool named ABIDIF, and I give ABIDIF the first  version of the binary, LibExample1.  Yeah, I wrote this this morning while having a chat with a fellow here, so I couldn't find  a better name.  I was more involved in our discussion.  So LibExample1.so.zero compared to LibExample1.so.one.  Oh, boom, oh, sorry, I should put that in a red-rex that into a file named ABI.  So have you seen what I did?  So I called a program named ABIDIF, right?  It's a program part of the Libabigel toolset that compares the ABI of two binaries, ABIDIF.  And so here is the results of what it says.  So can you read that?  Yeah?  Okay, it just says that...  Here, so it says the function f1, like, you know, with its signature, has some changes  in one of its subtypes, right?  And the change is that the parameter 2 was added.  That's all.  And this is actually what we did if I do a diff of the source code.  Sorry.  V1.  So you see here the sort...  Do you see that?  Can you read that?  So this is the diff.  And so this is...  And so this is the source code.  And this is an example of the kind of report you get, you know, from looking at the binary.  So you see that even though you're just looking at the binary and its debug information, because  I've compiled this with debug information enabled, you can have quite some detailed information,  right?  Like, like, light numbers, etc., etc.  So we can move to something more involved with, you know, C++.  So here we have...  In blue here we have a class.  Don't really need to read the details.  We have a class here.  And that class is used by another class, a structure.  It is used here because we use C.  You know, class name is C and we use C in another type underneath here.  And you see, I still have a function named F1 that uses C.  And I have a function named F2 that uses C as well here, right?  And of course, on the second side of the screen there, the thing that is highlighted in blue  is...  Well, I created a base class.  So I changed C and made C inherit a new class like this.  You know, I was listening to the fellow over there.  So this was my...  It's because of him, you know, that inspiration came from you.  So I just made this change.  And well, I was like, okay.  How about adding a virtual here, for instance, in the class?  The idea is just to make some changes and see what ABI div says.  And I kept F1 as is.  I didn't change the signature.  And I kept F2 as is too.  Oh, well, there is also an F3.  But anyway, let's go.  Did I try it?  Oh, yes.  So I compiled it as previously.  And I guess...  Oh, yeah.  I'm putting the result into a file.  So I'm running ABI div on the two binaries, putting a result into a file, text file, and  then I'm opening the text file in Emacs.  So the changes are interesting here.  So first, let's look at the summary of the changes that is shown by...  Let's all say that there is one function that was changed.  Change means that one of the types used by the function, or subtypes, you know, types  of those types, changed.  Right?  And it also says that one function was added.  Adding a function is not a problem.  As I said earlier, it's an ABI change, but it's not necessarily a problem.  But we still want to see it and review it and be sure that this change is really what  we need.  So we see that...  And it says that the function that I added was C, the getM0.  Did I add a function?  I don't think so.  I mean, I don't remember.  So let's look at...  So this was the first class I had.  So you see getM0 was already there in the first version.  And in the second one, what did I do?  In the second one, I still have getM0.  And you see I had a little change here in F1.  In F1 here, I called...  You know, I invoked the getM0.  You see here?  In blue.  Whereas in the first case, there was no code.  Okay.  So this is C++.  GetM0 here is inlined.  You know what inline means?  So it's not generated per se.  Oh, sorry.  Yeah.  GetM0 is inline.  So it's not generated, right?  By default.  But in the second case, it is generated.  GCC generates it because it is called.  Because someone uses it, then it is generated.  You know?  This is also because I compiled without optimization.  If I compile with optimization, we can try that later because time is flowing.  You'll see that this added function will disappear because of optimization.  The thing, the function is just inline and not generated as a symbol, right?  So these are the interesting things that you see when you just look at the binaries rather  than looking at the source code.  The same source code can produce different binaries depending on how you compiled it.  How you change the code, et cetera, et cetera.  This is what I meant by when I was saying we want to look at the binaries only.  Well, being able to look at those.  So added function.  You see?  And in here, you see the symbol of the function, right?  And here, you see its signature, right?  And so let's look at the changes, what it says.  So, first, it says that the return type of F1 changed.  Of course, that's true.  I changed it from void to int.  So we catch that.  So you see that just by looking, okay, there are some tools around that would say, that  we'll just look at the symbols, and because in C++, the symbols are mangled, you know,  you could derive the types used by the function from the mangling.  Do you follow me?  Yeah, people say that.  People think that.  Well, they're wrong.  Because, for instance, the return types of functions are not part of the mangling.  Only the parameters are part of the mangling.  And here, I made that change on purpose.  You see that the return type of the function changed.  And we want to catch that, too.  So we really need to go look at the debug info and analyze all that, you know.  So you see here that the next change it shows us is that the first parameter, which is a  pointer to C, pointer to the class C, changed.  Well, and how did it change?  It changed because the thing that is pointed to by that pointer, which is C, changed.  How did C changed?  It had one base class insertion.  You remember I said that I added a new base class.  I made C inherit a new class.  So we can catch that, too, by looking at the parameter.  This is what we do.  And as a result, well, as a result of all those changes, the size of C changes, too.  This is something you cannot see by just looking at the source code, right?  Well, you can see that it might change, but you're not sure.  Actually, the reason why the size change here is because of the virtual.  You remember I added a virtual keyword to the destructor.  So because of that, virtual that I added, a V table got added to the class.  And the V table is added at the beginning.  And it's the V table.  The V table is the reason why the size changed here, actually.  So we see here that there is a V table insertion.  And we can say at which line it was inserted and the V table offset, et cetera, et cetera.  And as a result, because something got inserted at the beginning of the class, the offset  of all the data members of the class changed, too.  So we have these changes, too.  Just by adding a virtual, right?  So yeah, we can learn a lot by just looking at the binaries.  So now I guess I'm done with showing off what we can do with this kind of tooling.  So what do we do behind the scenes?  Basically, Abigail and, well, what I call it, Libabigal, because it's the name of the  project.  Libabigal being the library that implements all the stuff and also a library of tools  that come on top of that library.  So basically, we represent ABI artifact.  What does that mean?  We have an in-memory models for types and declarations, just like a compiler does, just  like LLVM does.  So Abigail represents color types, integers, characters, and so on and so forth, aggregate  types that are arrays, structures, classes, unions, blah, blah, blah.  And declarations, a declaration being something that has a name and a type.  So all those things are represented.  You can write code saying, okay, this is a function.  The function has parameters, and I'm adding a new parameter.  You can manipulate those things.  On top of that, Abigail has a model of what I call bundles of ABI artifacts.  The first bundle is obviously a translation unit.  So whenever you look at a binary, the binary has debug information, when it has, and those  debug information describe things, translation unit per translation unit.  And it is inside those translation units that you have declarations and their types.  So we represent those translation units in the Abigail model, and we represent what we  call ABI corpus, and ABI corpus is the representation of a set of translation units.  So basically a shared library, the ABI things, artifacts of a shared library, is represented  in Abigail's pick as an ABI corpus.  That's how we call it.  And once we have ABI corpora, yeah, because the plural of corpus to Latin word is corpora,  I was told anyway.  So when you have those corpora, you can, well, compare them.  And so as usual, we build a model of the diff.  If you make a parallel with the GNU-Diff tool, when the GNU-Diff tool computes the difference  of two files, what does it do with the difference?  It just emits it.  If you look at the source code, it's what it does.  It sees the different boom, it emits it as a text.  It doesn't do anything with that difference.  In our case, we build a model of the difference, and we actually build a graph of those differences.  So if there is a difference between two structures, for instance, that difference might come from  the fact that one of the data members of those two structures changed.  So you will have one tree representing the two structures, difference between the two  structures, and a child tree note representing the detail of that change, the detail being  the change of the data members.  Do you understand what I mean?  And like that, you have a full graph of the changes in memory.  Once we have that, we can categorize the changes.  I mean, we can walk the changes and mark them saying, okay, this change is that kind of  change because, I don't know, for instance, in C++, you can have a change about, I don't  know, private data member becoming public.  This is a change.  But it might not be the same kind of change as removing a data member.  So we will put the first change in a category named private change category, and the second  change will be put in another category.  So categorizing changes.  One of the good things about categorizing changes is that we can see changes that come  over and over.  In the previous example I showed you, you've seen that there were two functions using the  class C, right, function F1 and F2.  If you don't remember, I'm telling you again.  But then the reports, in the reports you saw the change about F1 only, saying F1 change,  blah, blah, blah, and then because of C. And you didn't see the change about F2.  Well, F2 changed to because of C. You didn't see it because the tool categorized the changes  on F2 as being redundant.  And why?  Because it's the same C that changed that we've seen before.  If we have time, we can go back and I'll show you how to, you know, ABI Dev has an option  to show you the redundant changes too if you want to see everything.  And we can do this kind of analysis because we keep a model of the changes in memory.  We don't just emit those changes as soon as we see them.  One thing that is interesting too that I wanted to show but time is moving and then, well,  we can come back to that later if we're earlier is what we call suppression specifications.  You guys are, you know about Valgrind, right?  Yeah.  In Valgrind you have suppressions.  You know that.  It's the central, it's, to me, it's the most, it's the best thing ever in Valgrind, right?  You can, you have an error that keeps popping.  You know about it but you don't want to see that error again because for you it's noise,  right?  Suppression specifications.  We have the same thing in Babigel actually.  So and it's a bit more complicated, well, let's say, you know, evolved than in Valgrind  because you can say things that like, you can say things like, don't show me changes  about a type which name is Foo.  But you can also say, okay, don't show me changes about that type if the change is an  addition, a new member, add it at the end of C. Or you can say things like, don't show  me changes about function F if F is part of, if F is part of a file named blah, blah,  blah.  You know, there are many, many different things you can express and of course those  things come from users.  We're saying, I have this case where, you know, I would like to, you know, okay, can  you do something?  You know, and we discuss the price and, you know, we end up with something, a new category.  So we can do, we do this kind of things.  So afterwards we report about the changes and reporting means again, working the graph of  diffs and depending on the category of a diff node, report about it or not.  If the category says, oh, this node has been suppressed by the user, then you won't report  it.  Right?  If the category says, oh, this is a type diff name change.  Sometimes you have changes like that.  You have an, I don't know, you have, somewhere you have a type that is an integer and someone  renamed that, well, someone defines a type diff saying, you know, that integer's name  is now fancy type, for instance.  So this is not a real, you know, it's a change, but it's not necessarily harmful.  Right?  Some folks want to be able to, you know, dismiss, well, suppress those things.  So we have a category for those kind of changes too.  So the reporting engine knows about categorization and decides to show things depending on that.  And of course, for now we only have one kind of text reports.  But we could have more, but well, that's what we have.  So we've just seen what we can build as an in-memory model.  Right?  So where do we build that model from?  We build it from reading ELF and DWARF.  DWARF is the format for debug info in ELF, basically.  Those guys like Tolkien's books and well, I guess, but anyway.  So in Libabigel, there is a reader component that knows how to read ELF, so it's written  in C++.  And from that, it builds the Abigail model I was talking to you about earlier in memory.  So it does that for shared libraries, object files, and also actually also executable if  only if, even if I don't mention that here.  Once we have that, once we can read from ELF, build a model, well, we can also write it  down in files, obviously, so that people can define snapshots of the ABI of a binary or  a package.  You have a binary, you don't want to put that into Git.  Well, you can extract its ABI and stash that ABI into Git.  The format is an XML ad hoc format we came up with and we call that ABI XML.  And of course, we can read it back.  So this is interesting because once you read it back, well, you build an Abigail model and  so it lets you be able to compare, well, I'll talk about that later, you can compare a binary  against an ABI XML file because you can read both.  We come up with the in-memory model and then we can work on the in-memory model.  So which brings me to the tooling around the Abigail.  So I'll just skim quickly through some of the tools we have today.  So first we have ABI DW which emits, you know, it serializes the ABI representation of a  binary to a text file, that's a tool.  We have ABI diff that you have seen.  So ABI diff can compare two binaries, two ABI XML file or one ABI XML file and one binary,  you know, things like that.  So you can compare an ABI, a binary against a baseline.  There is ABI package diff that compares the ABI of binaries that are inside RPMs or depth  files.  So that was, yeah, written for someone else and now I maintain it.  And the last one that came to the family is Fed ABI package diff.  These guys keep coming with longer names.  We need to do something about this.  I didn't write this one.  It's someone else, a colleague who wrote it on his own time, by the way.  So this one is interesting.  It talks, okay, ABI package diff just so that you have things in mind.  It works on RPMs.  But when you work on RPMs, the debug info is split.  In another, it's in a separate package.  So you've seen with ABI diff, when we want to compare two binaries, we just need the  two binaries.  If you want to compare two packages, you need actually four.  Yeah, because you need the two debug info packages.  So the command line starts being a bit longer.  You can actually need even more in certain cases because ABI, ABIgal sees everything,  basically.  I mean, there are some changes.  I should show you some examples.  But there are some changes to some types that some developers don't want to see.  Because they say that, oh, this type is private, whatever that means.  In EF, there is no such concept as a private type.  But what the developer means by that, it means that type was not defined in the public header  shipped by the package or whatever.  That's what it means by private.  But in the binary, we see all the types.  We don't care about, so we tell you about all the changes.  So people came to me saying, oh, I don't want to see the changes about those types.  So there is an option to ABI package diff in which you give it the develop packages in  which there are the header files.  And so it won't show you changes about types that are not defined in the header files.  So six packages.  You need six packages just to run ABI package diff.  That's a bit too much when you type command lines.  So Fed ABI package diff is quite handy because you type it.  The name is long, but then afterwards it becomes shorter.  You say Fed ABI package diff dash dash from, you say Fedora 10, I don't know, dash dash  to Fedora 20, HTTP, the HTTP package.  So it will go fetch all the packages, the HTTP packages from Fedora 10 and 20, you know,  the debugging for packages, develop packages, whatever, compare them and show you the difference.  So that's what it does.  So maybe we would need something like this for OBS or, wow, just saying.  Anyway, so this is the part of the talk I'm going to change if I go talk to the Debian  guys.  But anyway, so I'm talking out loud here.  It's a chance that it's not recorded, right?  So doing this, we faced and we're still facing some interesting challenges.  We, well, speed and space matters a lot, a lot more than what people think.  When you're dealing with something like Alibaba Gale, you are looking at the entire binary.  And usually your compilation tool chain, even your compilation tool chain usually doesn't  do that unless you do something called LTO, but we're not, we'll put that aside.  Usually your compiler just compiles things file by file, translation unit by translation  unit, right?  And the linker that sees everything doesn't analyze types, right?  So it doesn't see types either.  We see all the types of all the translation units of the shell library so that it can  be a lot, a lot, especially when you have like big C++, you know, binaries.  And this is also due to the fact that the same types get included over and over and  over again.  Did I forget an over?  Okay, over again.  This is thanks to the famous hashing clue thing that I love, right?  So think about it.  We build representation, in memory representation for those types, for all of them.  If we have hundreds of thousands of types in memory, that can do a lot.  You can end up with gigabytes and gigabytes of memory used just to represent those types.  And you need two copies because of course you're comparing two libraries, right?  So we need to do something about that.  We need to de-doplicate those types.  So basically what it means, compiler people have fancy names for easy concept.  When you see a type foo, first time, okay, type foo, and then you go to the next translation  you need and you see type foo again, well, you shouldn't rebuild an in-memory representation  for this foo.  You should reuse the first one you saw.  De-doplication.  But to do that, so it's an easy concept.  But then to do that, you need to be fast because how do you know that the second foo is the  same one as the first one?  It is not because it is named foo that it is necessarily the same as the first one.  In C++ normally it should be the same because there is something named the one definition  rule, which is a C++ rule saying that if two entities have the same name, then they ought  to be the same entity.  But then in C, you don't have such thing.  Yeah, C, anyway.  So in C you can have foo in a file.  The same, okay, a different foo in a other file.  The two foos are exported, well, used by external functions and they're not the same.  So to be able to know that we want, well, we want to de-doplicate things.  So we need to know if the second foo is the same as the first one.  So we need to compare them.  And comparison by construction is an exponential problem.  You compare things member-wise and those members have types themselves that are structures  that have members that are structures that blah, blah, blah, and then you end up, you  know, it's exponential.  And seriously, I mean, if you do these things naively, it can take hours to complete just  to compare two packages.  So we need to use some heuristics to transform the problem into a more linear one so that  things can complete quickly.  So it's kind of, there are some interesting, you know, graph algorithms underneath.  We also need to control, well, to avoid seeing things we don't want to see.  And when I say this, when I say we, the we depends on the project.  A change, an ABI change for your project might be harmless and that very same change might  be harmful for his project.  He might want to see those.  So we need to give you ways to, you know, yeah, to avoid seeing what you don't want  to see.  So these are, this is why we came up with suppression specifications, but it goes further.  For instance, if you have a file named dot, well, that ends up with dot ABI ignore, ABI  ignore in your package, in your RPM file.  ABI package will detect that and use it, for instance, as a suppression specification.  Right.  And think about it, if you build, for instance, an ABI verifier at the distro level, you will  need to provide users with this kind of capabilities, provide them with the ability to say, okay,  this change is okay.  I don't want to see it again in my next runs.  And of course, this is a library that we're talking about, not only tools, so it needs  more documentation.  So basically, all the entry points need to be documented, and we're doing that already,  but we need more documentation about the inside and things like that.  It needs, and we're doing, I mean, the things that are in this section challenges are things  we're doing already, but I wanted to, you know, stress on them so that you guys are  aware of that.  We need a huge set of banners for regression testing.  Elf and dwarf is a very loose format.  You can put everything, anything in elf or in dwarf.  So unless you test stuff, you don't know what you are expecting or what you're really doing.  So we need a lot of tests.  So the LeBabieGalTarble is more than 100 megabytes of size.  The reason is that we have lots of banners just for testing when you make check.  So yeah, and we need even more than that.  And now what we're planning to do is to actually use Fed ABI package diff or something else  to go actually use the packages in the distros and distro histories to use that as regression  testing.  For instance, if you compare a package against itself, it should yield the empty set.  For instance, just doing that on thousands of packages will be good.  So this is really important, just like we test comparlers.  So in Fedora, we're using this thing to compare the ABI of new packages that are pushed.  So it's been in production since a couple of months.  So yeah, basically, we perform an ABI diff of the new package against the previous stable  one, and it sends a friendly message to the package maintainer.  So the changes are categorized.  Now you know what category is, basically, but roughly in this case, it's less detailed  than what we can do, basically.  But this is what they wanted.  If, well, there are certain kinds of changes for which we are sure that they're going to  cause a problem.  Incompatible ABI changed.  For instance, removed symbols, for those, those are flagged as failed, for the build,  basically.  And the most interesting changes, for instance, the changes I show you about the subtypes  of functions, those are flagged as need inspection.  If your package, well, so the package maintainer needs to inspect the package in those cases,  and otherwise the package is flagged as passed as far as this ABI test is concerned.  So it is based on Taskotron.  I think you guys know about that.  The task name, the task that runs on each package is named ABI check.  You can Google that.  And it uses ABI package diff to perform those things.  It's written in Python.  So that's it.  At the same time, package maintainers can use Fed ABI package diff, so they don't have  to wait until they're submitted their package to try to see ABI changes.  And so it is in production.  The limitations, you know it.  It only works on C and C++ shared libraries today.  But, well, tomorrow, if folks doing Rust want support, I mean, if it is Elf, we can support  it.  We can do some work.  And it just runs on a small set, well, small set, several hundred packages of, you know,  packages that are in the critical path.  They call that.  And improvements that we need.  So I've just ticked this here.  So I'm currently working on supporting, you know, the comparison of Linux kernels.  This is, well, it's well underway.  The Linux kernel is huge because we're not just comparing VM Linux.  We're comparing the union of VM Linux and all the modules.  There's a couple of those.  And the Linux kernel has its own way of handling Elf.  So it's quite interesting.  So we need more, you know, better support of C and C++ language construct, mostly C++,  you know, the new stuff, lambdas, blah, blah, blah, you know, the new stuff.  It keeps coming.  It just needs work to keep.  Because we basically have, it's like a compiler where we have, you know, you need to parse.  There is a front end, which is the dwarf thing, which builds a middle-end representation,  which is, you know, the Abigail model, and then, well, the back end is emitting reports.  So people will come with better API change categorization requests, saying, ah, this  change, you categorize it as harmful and harmless.  Please, can you put that in harmful place?  And we'll discuss and, you know, we'll see, you know, how we can do that.  Some people want additional API change reports.  You see here in the reports I show you first, we show the changes, but we also show the  impact, you know, what are the functions that are impacted or global variables.  Some people just want to see the types that changed and don't care about the functions  that were impacted because there are so many types, you know.  So it's a valid request.  So we would need, well, we can do that.  We have all the information to do that.  We just need to, you know, emit other kinds of reports for that.  Some people want more friendly web reporting.  We need work from that.  And someone even came with a crazy idea of something web, blah, anyway, you know, you  submit two packages and the thing does its thing and shows you web reports, you know.  Anyway, so if someone wants to do that, it will be great.  But I don't know if, well, maybe I can try and learn some JavaScript.  Anyway, so yeah, that's it for me, I guess.  Now is the time for question.  I'm sorry, but I finished on time.  Now you should do the same.  You have two minutes.  Anyone?  What architecture does it support?  Only x8664 or so?  Well, right now, well, it is known to work on x86, ARM, PowerPC, you know, and even  stuff.  The right answer is if we have DWARF, if we on ELF, it should work.  PowerPC showed that there are some corner cases sometimes, but in theory it should work.  Yeah.  Another question, does it also catch calling convention changes?  This is an interesting question.  Right now, no, short answer.  But there is a, I have a bugzilla open to support that.  Not necessarily calling convention changes, but not necessarily only that, but things  that are more subtle than that.  For instance, a parameter, for instance, the second parameter of a function, the way that  parameter is passed can change from a compiler to another.  For instance, you have an LLVM generated binary that passes that parameter on the stack,  and GCC will pass that or the other way around in registers, things like that.  So the idea is to have, is to be sure that we have that information in DWARF.  And if we have that, and I think we have part of it, then we can implement it on our site.  Well, if the structure of, in C, in the size of the structure changes, can you somehow  detect from the library code whether it is IBA compatible or not?  This is a problem when the function fills the whole structure, including the last member  and the size is in the public interface, then it could override a data.  Okay.  To me, so if I understand the question correctly is if the size changes, can we say if that  change is compatible or not?  Right.  That's, well, to answer that question, well, I would say that if that size change impacts  another type that uses that type which size changed, then it's ABI incompatible.  But then there is not only an impact, the impact is not necessarily only on other types.  It can be on code too.  You can have code that says size of that structure.  So my short answer will be if the size changes, it's bad, period.  If it's not bad, then you haven't looked close enough.  So yeah, because the question came out over and over.  And for instance, if a code says size of, that size of is lost because it's a constant  and we don't see that in the generated code anymore.  So I don't think we can really be sure that we're doing the complete impact analysis of  these kind of things.  And I think we would err on the safe side by saying that this is bad period.  If you do it, you know that it's potentially bad.  Yeah.  It's bad.  You mentioned that you were looking for ways to identify whether two certain structs are  the same and C, which doesn't have to you ODR.  Have you thought about building like an MD5 sum of a struct and then comparing that?  Oh, well, to us, the problem is solved.  I mean, well, it's solved.  The question is interesting.  If to build an MD5 sum or a sum, you need to walk the structure.  Right?  What I want to avoid is to walk the structure.  Exactly.  And that is what we do.  That's what I meant by saying that trying to transform the problem from an exponential  one to a linear one, this is basically the kind of technique we use.  We use some kind of hashing to avoid walking over and over.  We just walk once per type, name and kind.  Yes.  It's just that the hash that we end up using is not, we don't save it.  It's just in memory.  So I guess, thank you very much.
openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_0_71.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_72_203.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_204_287.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_288_611.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_612_659.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_660_707.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_708_755.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_756_803.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_804_875.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_876_947.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_948_995.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_996_1043.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1044_1091.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1092_1139.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1140_1187.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1188_1235.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1236_1295.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1296_1355.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1356_1415.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1416_1463.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1464_1523.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1524_1691.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1692_1739.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1740_1787.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1788_1835.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1836_1883.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1884_1943.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1944_1991.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_1992_2051.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2052_2099.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2100_2147.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2148_2195.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2196_2255.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2256_2303.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2304_2495.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2496_2543.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2544_2627.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2628_2675.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2676_2735.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2736_2783.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2784_2831.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2832_2987.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_2988_3059.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_3060_3143.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_3144_3203.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_3204_3251.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_3252_3311.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_3312_19391.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_19392_19439.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_19440_19499.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_19500_21455.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_21456_21899.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_21900_22595.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_22596_22727.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_22728_22895.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_22896_25187.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_25188_26039.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_26040_27095.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_27096_28427.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_28428_28523.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_28524_29399.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_29400_30779.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_30780_30827.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_30828_30935.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_30936_31007.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_31008_31271.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_31272_39083.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_39084_39227.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_39228_39479.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_39480_39647.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_39648_39971.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_39972_40367.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_40368_77795.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_77796_81635.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_81636_90023.png openSUSE_Conference_2017_Libabigail__How_semantic__wxVBuZK8Dl0_90024_90142.png