Postmodern strace
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Levin, Dmitry
strace is a diagnostic, debugging and instructional utility for Linux. It is used to monitor and tamper with interactions between processes and the Linux kernel, which include system calls, signal deliveries, and changes of process state. In this talk the maintainer of strace will describe new features implemented since FOSDEM 2018. Several interesting features were implemented within strace project since FOSDEM 2018, including: seccomp-assisted system call filtering system call return status filtering PTRACE_GET_SYSCALL_INFO API support new options: -DD, -DDD, -X, -z, -Z In this talk the maintainer of strace will describe these new features and demonstrate what kinds of problems they help to solve.
 Hello, everybody. Thank you for coming. My name is Dmitry Levin. I am the Chief Software Architect at Vezalt, where we do GNULI, GNULI-NUC Separating System. But I am also the maintainer of S-Trace for the last, slightly more than last 10 years. So today I'll be talking about post-modern S-Trace. What is post-modern S-Trace? I used to talk about modern S-Trace last year, so I understood that I can name it modern S-Trace any longer if I'm talking about very recent features. So where traditional S-Trace ends and modern S-Trace begins, and when modern S-Trace ends, well, modern S-Trace never ends. So when it turns into post-modern, well, it's kind of subjective. So, well, my definition is very simple, that the S-Trace that was before I started maintaining it is traditional, and all the rest is modern. And, yeah, so here it goes. And, well, post-modern is all new features since the last talk at FOSDOM. So I'll be covering mostly what have changed for the last two years. But I'll remind you briefly about traditional features just to refresh these features in your memory. So S-Trace is a mostly Linux system called Tracer. It also can not just trace but tamper the system calls since, like, several years ago. But it has a lot of options to control its behavior in different ways, like whether it prints instruction pointers, whether it prints timestamps or not, how it prints strings, what system calls are printed and which way they are printed, what's abbreviated and what's not. There are also options to control what signals are printed. It can also dump the data that goes through descriptors. It can print its output in different ways. So you can, for example, redirect it into a pipe or collect output for each process separately. Yeah. A lot of features that control what these calls would be printed. It can also print statistics on system calling vacations. It can attach to already existing processes. It can follow forks and it can don't follow forks, depending on whether you specify the option. Well, that was traditional. There were also quite a few options added for the last 10 years. Like, you can print a lot of details about descriptors, like what paths are associated with them or what socket information is behind sockets when these descriptors are sockets. We can bring stack of user function calls. Yeah. You can filter system calls by path names. We finally got support for regular expressions for filtering system calls. So you can specify which these calls are printed using regular expressions and so on. Yeah. More ways to control how statistics is printed, what is being, how it's traced. So you can, for example, attach to many processes. You can run the traces and attach it, process and so on and so on. Well. And there's also this big feature which changes the trace. I mean, it changes not just the trace, but the way how people look at it. It's system call tampering. So you cannot just trace system calls but also inject various things, like starting with return code. Also, you can inject signals and delays. But this all was more or less covered in the previous talk. So in the last two years, we got Petris Getzis calling for support. It went both into the kernel and in the interest trace. We got system call return status filtering. We have SICOMP assisted SICOMP filter nowadays. There are also a lot of new system calls in the kernel that are supported. And we have more and more elaborated system call parses. We also finally have long options. Yeah. We had no choice. We will soon see why. And finally, a bit more than a year ago, we changed our B-Z style lines to a co-pileft license. So let's start with the first feature. Well, the story itself started very long. I think it was 2001. Then this new architecture, X8664, appeared. So the way it was added in Linux kernel, obviously, was to support both 64 and 32-bit processes for various reasons because it was the main like feature of this architecture compared to its competitor that it could run legacy code. In early years of this, there were a lot of legacy code and very little native 64-bit code. But the way it was implemented in the kernel, it allowed not just to mix instructions, but also mix system call notifications. So you could actually invoke from a native code both native 64-bit system calls, but also legacy 32-bit system calls. And it was very poorly documented, if at all. It was very surprising to many people. And it wasn't really exposed in the kernel API. So, yeah, what user space, JSON, debuggers could do, they could fetch the system call number, they could like, fetch this register that describes the bitness of process, and then they would just guess, do the wild guess and say, well, if the process is 64-bit, then probably the syscall is also 64-bit, right? It's mostly the case. And if it's a 32-bit process, then syscall is definitely fit to bit. And all the logic depended on this wild guess. And it mostly works because in most cases, it's exactly what happens. But sometimes it's not the case. And back in 2008, there was a bug report against a trace in debunked back-tacture. There is a very simple example that looks, you can see are very similar to it. It's somewhat simplified compared to the one reported in that bug report. So, the program does a very simple thing. It just brings a line of output. And then it invokes a 32-bit system call. And then it prints another line of output. But this 32-bit syscall is actually a fork. So, what happens is that there are two processes and each of them prints the line. So, if you compile a link around this program, you'll see an output similar to this. Maybe the numbers will change, but all the rest will be just very simple. But if you run this very simple program under stress, you will see something very strange. So, you will see this line is being printed. And then suddenly the process attaches. And then you see this ridiculous open system call with very odd, very impossible, I would say, arguments. But all you can say about this is what. And all the rest looks very usual and regular, making the whole picture completely ridiculous. Like, this ridiculous open among all nice expected system calls. So, yeah, if you run this program several times, you will see that all these odd open flags are different. You will never see the same combination or probably never see the same combination of flags. Because nowadays, thanks to kernel address indemnization, all these register contain garbage that changes. And this reminds me of a toy I had in my childhood, a kaleidoscope. You turn it slightly and you see a different nice picture. So, you can use this simple program as a kaleidoscope, if you like. Yeah, so, this problem was approached several times. But until 2018, there was no progress. And finally, well, thanks to two people who contributed this API in the kernel. And two, yeah, two authors and 20 more people who reviewed this. And I could buy this at the silent buys. It took us almost nine months to get this into the kernel. And like, don't remember how many iterations, but it was two digit number of iterations. So, finally, we have it in the kernel. And for all architectures that support trace troops, which are like, all supported architectures, or almost all, I would say, and some that are not supported but get it for free. We have this and, yeah, the API looks this way. There is a structure you can request from the kernel. It contains this crucial architecture field. And in other ways, it looks similar to secumb data. So, you can obtain in one go both the architecture, the syscall number, syscall arguments, also instruction pointer, step pointer. And this makes traces that use this API reliable in this respect, in respect to the original problem. So the same program now looks if Linux is fresh enough and it's traces fresh enough, you see this is as expected. So, process attaches, you see the proper for call and not this ridiculous open. And all looks good. So, I think other traces and debuggers that have something to do with the system calls should switch to this API. By the way, it also allows to find out what kind of trace stop is the current stop. Otherwise, up to this time, kernel provided no way to find out. So, they used to think that they alternate. So, first, you enter the syscall and exit the syscall, but it's not always the case. So, you actually can use this nice API to find out and what is the actually the trace stop you're dealing with. Okay. So, it was a very major feature for this trace. And, yeah, as I said, some other traces are welcome to use this, of course. Let's speak about system call filtering. There is a new option to filter system calls by written status. It had a very unusual history, first trace. So, it was actually introduced in 2002, but it was broken from the beginning. It was never announced. You couldn't find it exists unless you accidentally type it in or look into the source code because it was broken. So, what it did, it printed the beginning of system call and when it failed, it just didn't print the ending. It wasn't useful. But now you can filter system calls by written status. So, you can print only those system calls that succeeded or those that are failed. So, in this very simple example, you can see the difference. Well, if you run a very simple program like cat with a modified LD library path, it makes a dynamic linker to look into different places. I wonder whether you expect dynamic linker to look into so many different places. But, well, you can see the difference. As a very useful side effect of this option, you can have an aggregation for free. So, for example, if you trace several processes that are running asynchronously, then you will see a lot of this unfinished and resumed stuff. And sometimes it's not very convenient. We used to implement special aggregators to collect this data so it would look like this. But now, thanks, you can use this option also to aggregate. The only need, I would say, is that it might change the order of invocations. So, in this example, it looks like if nano-slipses calls were invoked sequentially, which is definitely not the case, they were invoked simultaneously. But because they were printed at the moment, they finished this system calls. It looks not the way you are used to. But then, when you're aggregating, it doesn't really matter in which order they are printed. So, there is also another option that, there is a funny story connected with it. So, when I tried to come up with something useful as an example, I started invoking all programs I had in my small cheroot. And I found out a few programs that were not printing their arguments correctly, then they couldn't find them. I just invoked programs with non-existing files. And I found two of these programs in Elfutils and I fixed them. But you can get an idea when this could be useful. For example, a command program doesn't print what's going on, you can like trace and have a look. When you are filtering system calls, you probably don't want to, if you don't want to print all the rest, you probably want to make both system calls you are not printing execute faster. And now, we have a very nice feature, which we planned for several years but couldn't get until, well, we had two GSOC, Google Summer Code students. In the year before last, the student made a prototype and, yeah, last year, we had a student who is going to talk about this feature very soon, I hope. So, he will describe how it works. But from user perspective, it looks like trace is no longer, no longer delays everything by two orders of magnitude on those system calls that are not traced. So, in this, yeah, it's a famous example because this is a modification of example, BPF people use to describe how slow a trace is. And now, we use the BPF stuff to show how fast a trace is. So, you can see that SikonBPF itself slows down things about 10%, which is nothing compared to what all these PTC stops do with the speed of running programs. Yeah, you can see this is a long option and it was actually the first option that we couldn't find a good short analog. So, we had quite a few, not as many as a LSE program has, but quite a few options and some of them are not obvious. And we had, I think, what is dash n in our prototype, but we couldn't find an explanation why it should be called dash n. So, we decided that it's time to introduce long options. And now we are starting adding our analysis for not so obvious names. Yeah, so SikonBPF was the first one. Another option which should probably have a long option analog and is the option that has named dash k. I don't know why it's called dash k. It prints a stack of user calls at the time of system calling vacation. Yeah, it's a very useful thing because you can see the logic that behind the program, if you don't know what's going on, you can just apply this. It will produce a lot of output, but it makes a trace somewhat kind of debugger more than a tracer. So, in this example, you can see why, for example, cat closes is the doubt. From names of these functions, you can see that it does some kind of exit handling. And it closes doubt to ensure that everything is written, otherwise it should return non-zero exit code. Another option, you can attach, you can use a trace as an attached process. So, because a trace affects traces in different ways, it's not always desirable. For example, if these processes interact with their parents and they want to know their PID numbers, so you can run a trace this way and be more transparent. There is also a relatively new option that says how all these symbolic constants should be printed. So, you can print, as usual, like translate these numbers into symbolic names. So, you can print both symbolic numbers and row numbers or just row numbers. It has various useful implications. You can debug programs that you suspect pass arguments to system calls in a wrong way, which is not very surprising because on different architectures, there are different system call, different APIs, different number and order of passing system call arguments. And this also can be used, I think it's used in CIScaller project. So, yeah, we added support for all new system calls that were added into the Linux kernel. And nowadays, they started adding system calls again. So, yeah, like a bunch of new system calls that work with mount points. Well, I can't describe them all. There are too many who should never look into man pages, probably. But we have support for them. We also have a lot of very sophisticated system call parses. And I'll show you an example which looks very monstrous, but you will get the idea how sophisticated system call parses could be. So we support the coding of netlink protocol. You can see this output. This is the output of a very simple routing table. And here you can see what's going to be hired. So you see this netlink protocol is very structured. It has some structures, substructures, subsubstructures, and everything is printed. Coloring is mine. All the rest is made by this trace. Yeah. And the last but not least is that in December of 2018, we changed the license. So a trace used to be released since the very beginning under backlist style license. So it was a request of Paul Crennbuck. I don't know this man. It was too early. So when we added support for Petrezges calling for API, it was kind of crucial point. Most contributors to a trace didn't want to contribute on the permissive license any longer. So we decided we would never change to a copy left. So test suite is released under GNU DPL V2 plus. And all the rest is a license that allow us to release this as a library someday in the future. You will manage to make a library out of this trace. Yeah. So this is more or less what I wanted to talk about. And if you have some questions or ideas or something related to this trace to discuss, you have some time. Yeah. Should I repeat the question? So please be concise. Should I pick, yeah, from the back to the front? Yeah. So you mentioned this before, which is that speeds up the rendering of this is called so much. So why is it actually on default? And we have to actually put it to the long argument to use it. So the question was why this Sikonbisystem gold filtering is not by default and why you have to type this very long option to use this feature. First of all, you can abbreviate long options. So you don't have to type that much. I think two letters usually enough. If not, then type three letters. Some shells allow expansions of program arguments. So I don't think this is a problem. Well, there are two important points about this way of filtering. It's first, it generates and attaches a BPF program. I'm not going to dive into details, but it makes this program and you can attach it, but you can't get rid of it unless you're privileged. So this implies that you have to follow forks. You have to follow all processes that are forked by the process you are tracing. And this kind of change behavior. And one of important points of a trace is that a trace is backwards compatible. So we can't enable follow forks by default because people are not used to this. Yeah. And if you specify this option and do not specify follow forks, it says that I am enabling follow forks. Hey. So this is one point. Another point is that unless you are privileged and a trace is used as a privileged program, you can attach a BPF program to another process. So you can attach to a process using Ptrace C or Ptrace Attach, but you can attach a BPF program to another process. You can only attach a BPF program to yourself. So one of important features of a trace is to trace already existing parts wouldn't work with this unless you're privileged. But if you're privileged, you can use a lot of kernel tracing nowadays. It's not really a big deal. Although they don't have so elaborate traces. Yeah. Yeah. Please, another question. Yes. So you mentioned that on the last slide that the color was your own. Would you consider adding color output to a trace? So the question was that on this slide, the color was my own. And would I consider coloring by a trace? It's kind of, this is a difficult question because we had actually a plan to generate a structured output from a trace. And if you generate, for example, some JSON output, you would apply already existing software to do all this fancy stuff like coloring. So we decided we will make structured output first, and then other people will do whatever coloring they like. But as you can see, there is no structured output yet, and I have to do all the coloring. Yes, please. Sorry? Me? I think I can. So the question was whether I can print this. I think this is pretty enough. So what was your question? So whether I can bring this in blocks, so it will be easy to read. Yeah. It's getting closer and closer to our idea of structured output. So, yeah, you can see why we decided to go the simple way, but it was not so simple. Is it over? Yes. Okay. Thank you for coming.
 Hello, everybody. Thank you for coming. My name is Dmitry Levin. I am the Chief Software  Architect at Vezalt, where we do GNULI, GNULI-NUC Separating System. But I am also the maintainer  of S-Trace for the last, slightly more than last 10 years. So today I'll be talking about  post-modern S-Trace. What is post-modern S-Trace? I used to talk about modern S-Trace last year,  so I understood that I can name it modern S-Trace any longer if I'm talking about very recent  features. So where traditional S-Trace ends and modern S-Trace begins, and when modern  S-Trace ends, well, modern S-Trace never ends. So when it turns into post-modern, well, it's  kind of subjective. So, well, my definition is very simple, that the S-Trace that was  before I started maintaining it is traditional, and all the rest is modern. And, yeah, so  here it goes. And, well, post-modern is all new features since the last talk at FOSDOM.  So I'll be covering mostly what have changed for the last two years. But I'll remind you  briefly about traditional features just to refresh these features in your memory. So  S-Trace is a mostly Linux system called Tracer. It also can not just trace but tamper the  system calls since, like, several years ago. But it has a lot of options to control its  behavior in different ways, like whether it prints instruction pointers, whether it prints  timestamps or not, how it prints strings, what system calls are printed and which way  they are printed, what's abbreviated and what's not. There are also options to control  what signals are printed. It can also dump the data that goes through descriptors. It  can print its output in different ways. So you can, for example, redirect it into a pipe  or collect output for each process separately. Yeah. A lot of features that control what  these calls would be printed. It can also print statistics on system calling vacations.  It can attach to already existing processes. It can follow forks and it can don't follow  forks, depending on whether you specify the option. Well, that was traditional. There  were also quite a few options added for the last 10 years. Like, you can print a lot of  details about descriptors, like what paths are associated with them or what socket information  is behind sockets when these descriptors are sockets. We can bring stack of user function  calls. Yeah. You can filter system calls by path names. We finally got support for regular  expressions for filtering system calls. So you can specify which these calls are printed  using regular expressions and so on. Yeah. More ways to control how statistics is printed,  what is being, how it's traced. So you can, for example, attach to many processes. You  can run the traces and attach it, process and so on and so on. Well. And there's also  this big feature which changes the trace. I mean, it changes not just the trace, but  the way how people look at it. It's system call tampering. So you cannot just trace system  calls but also inject various things, like starting with return code. Also, you can inject  signals and delays. But this all was more or less covered in the previous talk. So in  the last two years, we got Petris Getzis calling for support. It went both into the kernel  and in the interest trace. We got system call return status filtering. We have SICOMP  assisted SICOMP filter nowadays. There are also a lot of new system calls in the kernel  that are supported. And we have more and more elaborated system call parses. We also finally  have long options. Yeah. We had no choice. We will soon see why. And finally, a bit more  than a year ago, we changed our B-Z style lines to a co-pileft license. So let's start  with the first feature. Well, the story itself started very long. I think it was 2001. Then  this new architecture, X8664, appeared. So the way it was added in Linux kernel, obviously,  was to support both 64 and 32-bit processes for various reasons because it was the main  like feature of this architecture compared to its competitor that it could run legacy  code. In early years of this, there were a lot of legacy code and very little native  64-bit code. But the way it was implemented in the kernel, it allowed not just to mix  instructions, but also mix system call notifications. So you could actually invoke from a native  code both native 64-bit system calls, but also legacy 32-bit system calls. And it was  very poorly documented, if at all. It was very surprising to many people. And it wasn't  really exposed in the kernel API. So, yeah, what user space, JSON, debuggers could do,  they could fetch the system call number, they could like, fetch this register that describes  the bitness of process, and then they would just guess, do the wild guess and say, well,  if the process is 64-bit, then probably the syscall is also 64-bit, right? It's mostly  the case. And if it's a 32-bit process, then syscall is definitely fit to bit. And all  the logic depended on this wild guess. And it mostly works because in most cases, it's  exactly what happens. But sometimes it's not the case. And back in 2008, there was a bug  report against a trace in debunked back-tacture. There is a very simple example that looks,  you can see are very similar to it. It's somewhat simplified compared to the one reported in  that bug report. So, the program does a very simple thing. It just brings a line of output.  And then it invokes a 32-bit system call. And then it prints another line of output.  But this 32-bit syscall is actually a fork. So, what happens is that there are two processes  and each of them prints the line. So, if you compile a link around this program, you'll  see an output similar to this. Maybe the numbers will change, but all the rest will be just  very simple. But if you run this very simple program under stress, you will see something  very strange. So, you will see this line is being printed. And then suddenly the process  attaches. And then you see this ridiculous open system call with very odd, very impossible,  I would say, arguments. But all you can say about this is what. And all the rest looks  very usual and regular, making the whole picture completely ridiculous. Like, this ridiculous  open among all nice expected system calls. So, yeah, if you run this program several  times, you will see that all these odd open flags are different. You will never see the  same combination or probably never see the same combination of flags. Because nowadays,  thanks to kernel address indemnization, all these register contain garbage that changes.  And this reminds me of a toy I had in my childhood, a kaleidoscope. You turn it slightly and you  see a different nice picture. So, you can use this simple program as a kaleidoscope,  if you like. Yeah, so, this problem was approached several times. But until 2018, there was no  progress. And finally, well, thanks to two people who contributed this API in the kernel.  And two, yeah, two authors and 20 more people who reviewed this. And I could buy this at  the silent buys. It took us almost nine months to get this into the kernel. And like, don't  remember how many iterations, but it was two digit number of iterations. So, finally, we  have it in the kernel. And for all architectures that support trace troops, which are like,  all supported architectures, or almost all, I would say, and some that are not supported  but get it for free. We have this and, yeah, the API looks this way. There is a structure  you can request from the kernel. It contains this crucial architecture field. And in other  ways, it looks similar to secumb data. So, you can obtain in one go both the architecture,  the syscall number, syscall arguments, also instruction pointer, step pointer. And this  makes traces that use this API reliable in this respect, in respect to the original problem.  So the same program now looks if Linux is fresh enough and it's traces fresh enough,  you see this is as expected. So, process attaches, you see the proper for call and not this  ridiculous open. And all looks good. So, I think other traces and debuggers that have  something to do with the system calls should switch to this API. By the way, it also allows  to find out what kind of trace stop is the current stop. Otherwise, up to this time,  kernel provided no way to find out. So, they used to think that they alternate. So, first,  you enter the syscall and exit the syscall, but it's not always the case. So, you actually  can use this nice API to find out and what is the actually the trace stop you're dealing  with. Okay. So, it was a very major feature for this trace. And, yeah, as I said, some  other traces are welcome to use this, of course. Let's speak about system call filtering.  There is a new option to filter system calls by written status. It had a very unusual history,  first trace. So, it was actually introduced in 2002, but it was broken from the beginning.  It was never announced. You couldn't find it exists unless you accidentally type it  in or look into the source code because it was broken. So, what it did, it printed the  beginning of system call and when it failed, it just didn't print the ending. It wasn't  useful. But now you can filter system calls by written status. So, you can print only  those system calls that succeeded or those that are failed. So, in this very simple example,  you can see the difference. Well, if you run a very simple program like cat with a modified  LD library path, it makes a dynamic linker to look into different places. I wonder whether  you expect dynamic linker to look into so many different places. But, well, you can  see the difference. As a very useful side effect of this option, you can have an aggregation  for free. So, for example, if you trace several processes that are running asynchronously,  then you will see a lot of this unfinished and resumed stuff. And sometimes it's not  very convenient. We used to implement special aggregators to collect this data so it would  look like this. But now, thanks, you can use this option also to aggregate. The only need,  I would say, is that it might change the order of invocations. So, in this example, it looks  like if nano-slipses calls were invoked sequentially, which is definitely not the case, they were  invoked simultaneously. But because they were printed at the moment, they finished this  system calls. It looks not the way you are used to. But then, when you're aggregating,  it doesn't really matter in which order they are printed. So, there is also another option  that, there is a funny story connected with it. So, when I tried to come up with something  useful as an example, I started invoking all programs I had in my small cheroot. And I  found out a few programs that were not printing their arguments correctly, then they couldn't  find them. I just invoked programs with non-existing files. And I found two of these programs in  Elfutils and I fixed them. But you can get an idea when this could be useful. For example,  a command program doesn't print what's going on, you can like trace and have a look. When  you are filtering system calls, you probably don't want to, if you don't want to print  all the rest, you probably want to make both system calls you are not printing execute  faster. And now, we have a very nice feature, which we planned for several years but couldn't  get until, well, we had two GSOC, Google Summer Code students. In the year before last, the  student made a prototype and, yeah, last year, we had a student who is going to talk about  this feature very soon, I hope. So, he will describe how it works. But from user perspective,  it looks like trace is no longer, no longer delays everything by two orders of magnitude  on those system calls that are not traced. So, in this, yeah, it's a famous example because  this is a modification of example, BPF people use to describe how slow a trace is. And now,  we use the BPF stuff to show how fast a trace is. So, you can see that SikonBPF itself  slows down things about 10%, which is nothing compared to what all these PTC stops do with  the speed of running programs. Yeah, you can see this is a long option and it was actually  the first option that we couldn't find a good short analog. So, we had quite a few, not  as many as a LSE program has, but quite a few options and some of them are not obvious.  And we had, I think, what is dash n in our prototype, but we couldn't find an explanation  why it should be called dash n. So, we decided that it's time to introduce long options.  And now we are starting adding our analysis for not so obvious names. Yeah, so SikonBPF  was the first one. Another option which should probably have a long option analog and is  the option that has named dash k. I don't know why it's called dash k. It prints a stack  of user calls at the time of system calling vacation. Yeah, it's a very useful thing because  you can see the logic that behind the program, if you don't know what's going on, you can  just apply this. It will produce a lot of output, but it makes a trace somewhat kind  of debugger more than a tracer. So, in this example, you can see why, for example, cat  closes is the doubt. From names of these functions, you can see that it does some kind of exit  handling. And it closes doubt to ensure that everything is written, otherwise it should  return non-zero exit code. Another option, you can attach, you can use a trace as an  attached process. So, because a trace affects traces in different ways, it's not always  desirable. For example, if these processes interact with their parents and they want  to know their PID numbers, so you can run a trace this way and be more transparent. There  is also a relatively new option that says how all these symbolic constants should be  printed. So, you can print, as usual, like translate these numbers into symbolic names.  So, you can print both symbolic numbers and row numbers or just row numbers. It has various  useful implications. You can debug programs that you suspect pass arguments to system  calls in a wrong way, which is not very surprising because on different architectures, there  are different system call, different APIs, different number and order of passing system  call arguments. And this also can be used, I think it's used in CIScaller project. So,  yeah, we added support for all new system calls that were added into the Linux kernel.  And nowadays, they started adding system calls again. So, yeah, like a bunch of new system  calls that work with mount points. Well, I can't describe them all. There are too many  who should never look into man pages, probably. But we have support for them. We also have  a lot of very sophisticated system call parses. And I'll show you an example which looks very  monstrous, but you will get the idea how sophisticated system call parses could be.  So we support the coding of netlink protocol. You can see this output. This is the output  of a very simple routing table. And here you can see what's going to be hired. So you see  this netlink protocol is very structured. It has some structures, substructures, subsubstructures,  and everything is printed. Coloring is mine. All the rest is made by this trace. Yeah.  And the last but not least is that in December of 2018, we changed the license. So a trace  used to be released since the very beginning under backlist style license. So it was a  request of Paul Crennbuck. I don't know this man. It was too early. So when we added support  for Petrezges calling for API, it was kind of crucial point. Most contributors to a trace  didn't want to contribute on the permissive license any longer. So we decided we would  never change to a copy left. So test suite is released under GNU DPL V2 plus. And all  the rest is a license that allow us to release this as a library someday in the future. You  will manage to make a library out of this trace. Yeah. So this is more or less what  I wanted to talk about. And if you have some questions or ideas or something related to  this trace to discuss, you have some time. Yeah. Should I repeat the question? So please  be concise. Should I pick, yeah, from the back to the front? Yeah.  So you mentioned this before, which is that speeds up the rendering of this is called  so much. So why is it actually on default? And we have to actually put it to the long  argument to use it. So the question was why this Sikonbisystem gold filtering is not by  default and why you have to type this very long option to use this feature. First of  all, you can abbreviate long options. So you don't have to type that much. I think two  letters usually enough. If not, then type three letters. Some shells allow expansions  of program arguments. So I don't think this is a problem. Well, there are two important  points about this way of filtering. It's first, it generates and attaches a BPF program. I'm  not going to dive into details, but it makes this program and you can attach it, but you  can't get rid of it unless you're privileged. So this implies that you have to follow forks.  You have to follow all processes that are forked by the process you are tracing. And  this kind of change behavior. And one of important points of a trace is that a trace is backwards  compatible. So we can't enable follow forks by default because people are not used to  this. Yeah. And if you specify this option and do not specify follow forks, it says that  I am enabling follow forks. Hey. So this is one point. Another point is that unless you  are privileged and a trace is used as a privileged program, you can attach a BPF program to another  process. So you can attach to a process using Ptrace C or Ptrace Attach, but you can attach  a BPF program to another process. You can only attach a BPF program to yourself. So one  of important features of a trace is to trace already existing parts wouldn't work with  this unless you're privileged. But if you're privileged, you can use a lot of kernel tracing  nowadays. It's not really a big deal. Although they don't have so elaborate traces. Yeah.  Yeah. Please, another question. Yes. So you mentioned that on the last slide that the  color was your own. Would you consider adding color output to a trace? So the question was  that on this slide, the color was my own. And would I consider coloring by a trace?  It's kind of, this is a difficult question because we had actually a plan to generate  a structured output from a trace. And if you generate, for example, some JSON output, you  would apply already existing software to do all this fancy stuff like coloring. So we  decided we will make structured output first, and then other people will do whatever coloring  they like. But as you can see, there is no structured output yet, and I have to do all  the coloring. Yes, please. Sorry? Me? I think I can. So the question was whether I can  print this. I think this is pretty enough. So what was your question? So whether I can  bring this in blocks, so it will be easy to read. Yeah. It's getting closer and closer  to our idea of structured output. So, yeah, you can see why we decided to go the simple  way, but it was not so simple. Is it over? Yes. Okay. Thank you for coming.
debugging_strace_modern_0_155.png debugging_strace_modern_156_227.png debugging_strace_modern_228_335.png debugging_strace_modern_336_383.png debugging_strace_modern_384_827.png debugging_strace_modern_828_899.png debugging_strace_modern_900_2339.png debugging_strace_modern_2340_3419.png debugging_strace_modern_3420_4079.png debugging_strace_modern_4080_4571.png debugging_strace_modern_4572_5747.png debugging_strace_modern_5748_6179.png debugging_strace_modern_6180_6815.png debugging_strace_modern_6816_8207.png debugging_strace_modern_8208_10391.png debugging_strace_modern_10392_11351.png debugging_strace_modern_11352_13043.png debugging_strace_modern_13044_13607.png debugging_strace_modern_13608_13871.png debugging_strace_modern_13872_14291.png debugging_strace_modern_14292_15419.png debugging_strace_modern_15420_17015.png debugging_strace_modern_17016_17819.png debugging_strace_modern_17820_19247.png debugging_strace_modern_19248_19463.png debugging_strace_modern_19464_19547.png debugging_strace_modern_19548_21347.png debugging_strace_modern_21348_22067.png debugging_strace_modern_22068_22847.png debugging_strace_modern_22848_24227.png debugging_strace_modern_24228_25715.png debugging_strace_modern_25716_28379.png debugging_strace_modern_28380_29411.png debugging_strace_modern_29412_30911.png debugging_strace_modern_30912_31667.png debugging_strace_modern_31668_33131.png debugging_strace_modern_33132_33803.png debugging_strace_modern_33804_34535.png debugging_strace_modern_34536_35123.png debugging_strace_modern_35124_37055.png debugging_strace_modern_37056_37931.png debugging_strace_modern_37932_37979.png debugging_strace_modern_37980_38027.png debugging_strace_modern_38028_38075.png debugging_strace_modern_38076_38123.png debugging_strace_modern_38124_38171.png debugging_strace_modern_38172_41783.png debugging_strace_modern_41784_41831.png debugging_strace_modern_41832_44531.png debugging_strace_modern_44532_44651.png