HawkTracer profiler
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Kolny, Marcin
HawkTracer is low-overhead instrumentation-based profiler built at Amazon Video for platforms with limited capabilities. It's written in C but can be used almost with any other language (we've successfully used it with JavaScript, LUA, Python and Rust). It's highly extensible (at compile time) and portable so it can be run on almost any embedded device. In this talk I'll introduce the architecture of the profiler, present it's advantages and limitations, show how can you instrument the code and demonstrate the profiler in action by running it with an example cross-language (Rust, C++ and Python) project.
 Hi everybody, thanks for coming. I know for some of you it's quite early yet. My name is Marcin, I work for Prime Video, and this talk will be about the profiler that we did to fix performance issues in Prime Video app that's running on living room devices. And I'm not going to talk a lot about the internals of the profiler, I'd rather focus on how to use it, how you can use it and adapt to your project. But if you want to know more about internals, I'll be here around till the end of the day. So you can just speak to me anytime you want. So I'll start with explaining why we actually started a new profiler because as you might know, there's quite a lot of them on the market already as LTNG, Ftrace, UFtrace, Strace, Perf, and a lot of others. The thing is that the development environment that we had to work in was quite different than what you might get used to. So we've been doing applications that's running on living room devices such as game consoles, smart TVs, streaming sticks, and some of the platforms are very friendly for developers, like game consoles are amazing. They provide lots of tools, debugging, utilities and so on, but smart TVs are not that friendly for developers. So very often all we have is compiler, and we can't really deploy any tool to that device. It's very close. So all we can do is basically deploy the single app. That's our Prime Video app, and that's it. So we can, for example, deploy Strace or LTNG. We can't deploy some kernel modules and so on. So it's very, very limited environment. And also our application, the Prime Video application, is built using three different languages. So we have a native layer, which is a C++, and then that native layer exposes two scripted engines, one for Lua and one for JavaScript, and we run them at the same time. So there's basically three different languages. And because those platforms, some of those platforms are very low in terms of the CPU and memory, we had quite a few performance issues, especially on the cheaper platforms. And we wanted to debug it, but as I said, there was not much tools for debugging. There were some tools, but they didn't really meet our requirements. So we said, OK, maybe we can just quickly prototype something, and we see if that works. As you see, this presentation is about the profiler, so it actually worked. So we come up with a list of features that we want to have from our profiler. This is just a short list, but there's quite a few more features I just listed that are more important. So it's user space and instrumentation base. We basically don't have access to kernel by any means, so user space for us was the only option. Instrumentation base, we can't really deploy a second demon that, I don't know, for example, checks traces every second or so. So everything had to be built into the application since our application is written in C, and JavaScript, and Lua, we decided that we're going to write it in C++ or C, but it should be available for those languages as well, in somehow by providing a bindings layer, and so on. Also, because we port to different platforms, those platforms are very different to each other. Some of them have different operating systems, different CPUs, even different end-endness. So it should be easy to port this profiler to all the platforms that we support by Prime Video. Low overhead, I think this is something that all the profilers are trying to achieve. We wanted to measure timings because that's what usually people think of when they do profiling, how long does it take to execute a function. But apart from that, we also wanted to measure other metrics like memory usage, CPU usage, how many HTTP calls we've done, and so on. And the other thing that we wanted to achieve, we had a group of people who were very into performance and were responsible for fixing performance issues, but we wanted also other developers to use profiling tools on a daily basis. So while they're writing the code, they can focus on performance as well, so we don't have to do the job afterwards. So we wanted to provide consistent user experience across different platforms. So no matter what application, what device you're developing on, whether this is a streaming stick, web, game console, smart TV, you always have the same tool. So they just need to learn one tool and they can use it for all the different projects. So those are the features. And we come up with a very simple high level design. We have a profile device that's running the application. We have a Huxracer library attached to that application. That generates a binary stream of events. We call it HTDumpStream. And then we have a client, which is on the developer desktop that's converting this byte stream to some human readable form, like, for example, Chrome Trace Format, Flame Graphs, or you can, as I show you later in the demo, you can build your own client very easily to do some other visualizations that our client doesn't provide. This is a bit more detailed diagram. So there's an app. This app has the Huxracer library linked. And Huxracer has a timeline. A timeline is essentially a buffer. And application pushes events to this buffer. These events are serialized. And then once the buffer is full, we push it to a listener. And the listener is basically a function callback. So then the listener decides what to do. And we, Huxracer, by default, provides two listeners. It's either it stores the stream to a file, or it can send it over TCP IP. But you can extend it. I don't know if you want to send the stream over a serial port. For example, you can do that. And then the stream goes to the developer desktop. And again, we provide the library as written in C++ and has Python binding that deserializes the stream of events and provides you nice data structures in your language. So then you can build your client. And we already provide the Huxracer converter client that converts this binary stream to some common trace formats. We also have the library that's written in Rust. It's not really experimental, because I know people already use it. And I'm going to use it today during the demo. So if that's going to work, then it won't be experimental anymore. Hopefully it will work. So I mentioned here you see that there is a timeline. And the timeline is conceptually a buffer. But it actually is a little bit more. And you need to provide some configuration for that timeline. And so to simplify this process of creating the timeline, Huxracer provides a global timeline, which is quite efficient. It's multiple timelines, actually. It's timeline per thread. So if you push events to the timeline, we don't require any logs, because you have an instance per thread. So there is no problem with data races and so on. And you can easily access it by just calling htglobaltimelineget function. I recommend just using that. To be honest, I don't think in my real projects, I ever use different approach. So it's probably good enough. So I mentioned that the basic data unit in Huxracer is an event. So we have quite a few events defined in Huxracer, event types, but you can define your own. It's basically a C structure. You define it using a C macro. And you provide the event class name, which could be my event, for example. It supports inheritance. So all the events must eventually inherit from ht event, which is a very base class. And then you provide a set of fields that you want to have in this event. And you define it by three properties. As a type, it's either integer, string, struct, flow, double, or pointer, I think. Then you provide the actual C type of that. And the name. And that converts to the C structure. And additionally, it generates, automatically, a few helper methods, like, for example, a method for serializing the event. So when you have this event and you want to push it to the byte stream, this macro already will generate a function that does it for you. And then when you already have the event, if you want to push the instance of this event type to the timeline, you just call ht time line push event macro. You pass the timeline as a first parameter. The second parameter is the name of your event type, and then all the values, all the fields. And that's pretty much it. So I said that before that we provide the client that parses the binary stream. But then I said that you can write your own event type. So you might be wondering how the client knows how to deserialize those new event types. And for that purpose, we divided the event stream into two sub-streams. There's a metadata stream that describes all the types, and there's the actual event stream with all the values. And since everything in Hocktracer is an event, even the definition of the event is event as well. So we have special events that describe your type. And the first event is just describing the name. And the number of fields that you have. And then you have each new event for each field of this class. So in that case, we'll expect one event for class info event and three events for class field info event. And in class field info event, we provide information like the field type, a field name, the size of that field, and the data type, whether it is a string, integer, a struct, and so on. And both streams are serialized as a byte stream. So you can see that basically it's just 30 or 40 bytes for those events. And then eventually, you have the actual event. And this event has a type. So it's number nine. As you can see, the class ID is nine here. And the info class ID nine again. So all of them are connected to each other by this identifier. And it's important that you first need to send the definition of the event type before you actually send the first event of this type. Because otherwise, the parser doesn't know how to parse this event. But the hokter does everything for you. So it's just. Can I ask a question? Yeah, go ahead. If the parser loses sync with the byte stream, is there a way for it to regain sync later? So no. At the moment, it is not possible. I'm working on improvement that it will be actually possible. But at the moment, if you lose the metadata stream, yeah, it is a problem. So yeah. But I'm aware of this problem. And we'll be working on that. OK. So I mentioned that we also want to measure time. And that was actually quite important, requirement for us. So to make it even easier, we have predefined event types that are already for measuring time. They take a label. And they take the duration and the timestamp. And we have macros that automatically generate those events and push them to the timeline. And so for C++, depending on what you want to do, if you just trace the whole function, you just put this macro, httrace function at the beginning. And if you just want to trace the random scope, there is httrace. There is a few more macros that are more optimized for that. For example, they have some hash maps. So you don't have to send the string every time for every event. It's even more optimized version that uses some tricks with static thread local objects. So even hash map is not necessary for some of them. It's documented, so I don't want to spend much time on that. And we did bindings for a few languages. For now, the public one is Python and Rust. We have one for JavaScript as well. It's not ready yet to be published, but we'll do that in the future. And for Lua as well. So this is how you do that in Python. You import the trace decorator, and you just put it for the function that you want to trace. And the same for Rust, there are some macros. I'm not an author of the Rust bindings, so probably if you want to know more about this, there's Alexandru, he's in Fosdum. Not sure if he's today in this room, but he's an author of that. And basically, that's how you do that. There are some kind of Rust macros that you can trace the scope. OK, so that was it. And I just want to show you some demos. So you can see how you can use it. The first one is we have a C++ application that allocates a memory, and then we want to see on the graph how the memory grows. We also want to know how many allocations we've done in this program. So we'll write a simple Python client that receives the Hoke Tracer event stream and does the visualization. And I start with the program itself. So at the very beginning, we need to initialize the Hoke Tracer library. So this is the event type that we define. As I said, there's a name, the base type, and we have two fields, memory usage and allocation count. They both are integers size t type. And the event, before you want to use it, you need to register it. This method is auto-generated by the macro that I showed before by this macro, ht-declare event type. You just need to call it before you use it. It registers some information in the Hoke Tracer system. And then you create a listener to the timeline. I said the global timeline is the best option. So we're using global timeline here. There are some parameters like port. This is the buffer size. We set it to zero. So we'll get we don't actually buffer anything. We stream it directly to the client. And we say that we want to use TCP client. So we can stream it directly to the client. And we can get the real data. Hoke Tracer already provides the functions to get the memory usage. It also provides some functions to trace the allocation. So you just need to provide a callback. This is for malloc premalloc. There's post malloc, pre-realloc, post-realloc, and so on and so on. We are only interested in premalloc hooks. So we register that. And this is basically our function. We get the memory usage. We get the allocation count, which is read in the other callback. So here, this is our malloc callback. So we increase the allocation count every time somebody calls malloc. So we get that all in the context object. And that's basically what we do. We push the event. So this is our type. Those are our values. And this is the memory usage, virtual memory usage. And this is the allocation count. And this is the client. It's written in Python. All we need to do, we need to start the client, the Hoke Tracer client. We say, OK, listen to 8.76.5 port on this IP address. This is the animate method. That's basically the anim that's drawing the graph. But the most interesting bit is this one. So this is basically how we read events from Hoke Tracer. This is our client. So we're waiting for end of stream. If it's not, then we pull an event. If there is an event in the queue, we check the first. This is a tuple. And the first element of the tuple is the event name. So we check if this is the event that we want. And if so, we get the timestamp. We get the allocation count. And we get the memory usage. And then we put the memory usage on the graph. And we print the allocation count. So it's very, very simple. Let's run it. So we run the client. And now we run the program that we want to trace. So you can see that the memory usage is growing. And you can also see the allocation count is going up. And so you might be wondering why the memory usage is growing. If we go back to our application, we see that we have a loop. And we do malloc. And it's called 100,000 times. So we can see how many mallocs we actually got. It's more than 100,000. There's probably some other allocations going on. Yeah, and this is the graph. So that's how you can get the real data from the Hoke Tracer by defining your own event. It doesn't have to be memory usage. You, for example, can trace the number of HTTP calls, for example, and so on. Yeah, so that was the first demo. The second one is going to be more complex a little. I want to show you how you can trace multiple languages at the same time. Because that was actually our real use case. We had application written in C++. And then we run Lua and JavaScript on top of that. So I do a slightly different example. I have a main application written in Rust. This application downloads the image using a downloader library that's written in C. It uses curl. And then I rotate this image using some image. I think it's open CV. Maybe not open CV. I actually changed it. But yeah, it rotates the image in Python. So I run Python interpreter in Rust. And then at the same time, while I'm rotating the image, I also upload the image that I downloaded in the first step to the S3 backend. And I want to know how much time I spend in each particular operation. So I, again, start with the code. So this is our Rust client. Even though you might not be familiar with Rust, I think it's pretty simple. So we start with download file. This is download file RS. It's basically just the wrapper for the download file library, function from the library that's written in C. Then once we have the file downloaded, we spawn two threads. One is doing the rotation in Python. It starts the Python interpreter. And that's the rotation. And the other one, at the same time, uploads the file to S3. You might see that all of the functions here are decorated with those macros. And we also have some other trace points in here. We can look at the downloader file. It's pretty simple one. It's just the curl wrapped around. We trace the function here at the beginning. And we also see how long does it take to call curl easy perform. So we trace the whole scope here. And just last bit is the Python one. We import the decorator. And we decorate all the functions. So we basically rotate the image and save the image. We load the image. We rotate the image. And then we save the image to the file. So not really a rocket science here. It's important that if you want to run it with Python, this environment of variable needs to be set. Otherwise, the Python bypasses all the trace decorators. So it doesn't have much impact on the performance if you disable that. So you can even have it in the production code. OK, so let's run this. So this is our program. It downloaded the file successfully. It's doing the rotation. And it's doing the upload. So you can see that it actually did rotate some image. Let's start from zero. Yeah, so it's a hook. And it's basically all the angles are here. If we delete that one, we have also rotate the HTDAM, which is the binary file with all the events, all the traces. This is not going to work because I forgot this environment variable here, so we wouldn't have the Python traces. So I run it again. I'm going to delete the images. So we have the HTDAM file, which is the binary stream. We can now use a Hock Tracer converter. This is the one that's experimental. So it's working. And we generate the Chrome trace format. So we say it would be output.json. It converted. So now you can use the tool that's deployed with Chromium. And you see that first you see that we have three threads here, one, two, three. That's exactly what we had because we had the main thread. And then we spawned two extra threads for rotate and for upload. This is the first bit that's in C++ or C. We download the file, there's HTTP request. And then just after that, we start rotating Python. And those trace points here, those are from Python. Save image, rotate, main. Those trace points are from Python. And this one, upload to S3, is in Rust. So as you can see, you can trace. You can have everything in one trace file. And you can easily analyze what's going on in the program. You can also convert it to flame graphs. So if you want the flame graph view, you can just do that. This is a very simple program. So the flame graph also is very simple. But yeah. So that's it. We have some plans for improving this. There are some of the items, but not all of them. As you mentioned, we want to add some extra protection layer in case we lose some events on the way. And yeah, that's it. Thank you. There's a bunch of links. Feel free to contact me. There's a documentation on the website. There are tutorials. They say how to integrate it with your project. Basically, Hocktracer can be combined into one single C++ and header file. So it's easy to link it to your project if you want. There are other links for bindings and for converter. So thank you. And I think we'll have one minute for questions. But if that's not enough, then I'll be around. OK. Which methods like you showed TCP now, right? So in addition to TCP, what do you currently use? So this is the TCP that we use in the first demo. And the file, which was used as a second demo. Yeah, so those are natively supported at the moment. What's something like serial or can you have to use for all your tests? Sorry? So if you want something like serial? Yeah, you need to write the neural. Sorry, the question was, what listener types we support? And my answer is currently we support file and TCP listeners. Yeah. Thank you.
 Hi everybody, thanks for coming.  I know for some of you it's quite early yet.  My name is Marcin, I work for Prime Video,  and this talk will be about the profiler that we did  to fix performance issues in Prime Video app  that's running on living room devices.  And I'm not going to talk a lot about  the internals of the profiler,  I'd rather focus on how to use it,  how you can use it and adapt to your project.  But if you want to know more about internals,  I'll be here around till the end of the day.  So you can just speak to me anytime you want.  So I'll start with explaining why we actually started  a new profiler because as you might know,  there's quite a lot of them on the market already as  LTNG, Ftrace, UFtrace, Strace, Perf, and a lot of others.  The thing is that the development environment that we had  to work in was quite different than what you might get used to.  So we've been doing applications that's running on  living room devices such as game consoles,  smart TVs, streaming sticks, and some of the platforms  are very friendly for developers,  like game consoles are amazing.  They provide lots of tools, debugging,  utilities and so on, but smart TVs are not  that friendly for developers.  So very often all we have is compiler,  and we can't really deploy any tool to that device.  It's very close.  So all we can do is basically deploy the single app.  That's our Prime Video app, and that's it.  So we can, for example, deploy Strace or LTNG.  We can't deploy some kernel modules and so on.  So it's very, very limited environment.  And also our application, the Prime Video application,  is built using three different languages.  So we have a native layer, which is a C++,  and then that native layer exposes two scripted engines,  one for Lua and one for JavaScript,  and we run them at the same time.  So there's basically three different languages.  And because those platforms, some of those platforms  are very low in terms of the CPU and memory,  we had quite a few performance issues,  especially on the cheaper platforms.  And we wanted to debug it, but as I said,  there was not much tools for debugging.  There were some tools, but they didn't really  meet our requirements.  So we said, OK, maybe we can just quickly prototype something,  and we see if that works.  As you see, this presentation is about the profiler,  so it actually worked.  So we come up with a list of features  that we want to have from our profiler.  This is just a short list, but there's quite a few more  features I just listed that are more important.  So it's user space and instrumentation base.  We basically don't have access to kernel by any means,  so user space for us was the only option.  Instrumentation base, we can't really  deploy a second demon that, I don't know, for example,  checks traces every second or so.  So everything had to be built into the application  since our application is written in C,  and JavaScript, and Lua, we decided  that we're going to write it in C++ or C,  but it should be available for those languages as well,  in somehow by providing a bindings layer, and so on.  Also, because we port to different platforms,  those platforms are very different to each other.  Some of them have different operating systems, different CPUs,  even different end-endness.  So it should be easy to port this profiler to all the platforms  that we support by Prime Video.  Low overhead, I think this is something  that all the profilers are trying to achieve.  We wanted to measure timings because that's what usually  people think of when they do profiling,  how long does it take to execute a function.  But apart from that, we also wanted  to measure other metrics like memory usage, CPU usage,  how many HTTP calls we've done, and so on.  And the other thing that we wanted to achieve,  we had a group of people who were very into performance  and were responsible for fixing performance issues,  but we wanted also other developers  to use profiling tools on a daily basis.  So while they're writing the code,  they can focus on performance as well,  so we don't have to do the job afterwards.  So we wanted to provide consistent user experience  across different platforms.  So no matter what application, what device you're developing on,  whether this is a streaming stick, web, game console,  smart TV, you always have the same tool.  So they just need to learn one tool  and they can use it for all the different projects.  So those are the features.  And we come up with a very simple high level design.  We have a profile device that's running the application.  We have a Huxracer library attached to that application.  That generates a binary stream of events.  We call it HTDumpStream.  And then we have a client, which is on the developer desktop  that's converting this byte stream to some human readable  form, like, for example, Chrome Trace Format, Flame Graphs,  or you can, as I show you later in the demo,  you can build your own client very easily  to do some other visualizations that our client doesn't provide.  This is a bit more detailed diagram.  So there's an app.  This app has the Huxracer library linked.  And Huxracer has a timeline.  A timeline is essentially a buffer.  And application pushes events to this buffer.  These events are serialized.  And then once the buffer is full, we push it to a listener.  And the listener is basically a function callback.  So then the listener decides what to do.  And we, Huxracer, by default, provides two listeners.  It's either it stores the stream to a file,  or it can send it over TCP IP.  But you can extend it.  I don't know if you want to send the stream over a serial port.  For example, you can do that.  And then the stream goes to the developer desktop.  And again, we provide the library as written in C++  and has Python binding that deserializes the stream of events  and provides you nice data structures in your language.  So then you can build your client.  And we already provide the Huxracer converter client  that converts this binary stream to some common trace formats.  We also have the library that's written in Rust.  It's not really experimental, because I know people already  use it.  And I'm going to use it today during the demo.  So if that's going to work, then it won't be experimental  anymore.  Hopefully it will work.  So I mentioned here you see that there is a timeline.  And the timeline is conceptually a buffer.  But it actually is a little bit more.  And you need to provide some configuration for that timeline.  And so to simplify this process of creating the timeline,  Huxracer provides a global timeline, which is quite efficient.  It's multiple timelines, actually.  It's timeline per thread.  So if you push events to the timeline,  we don't require any logs, because you  have an instance per thread.  So there is no problem with data races and so on.  And you can easily access it by just calling  htglobaltimelineget function.  I recommend just using that.  To be honest, I don't think in my real projects,  I ever use different approach.  So it's probably good enough.  So I mentioned that the basic data unit in Huxracer  is an event.  So we have quite a few events defined in Huxracer,  event types, but you can define your own.  It's basically a C structure.  You define it using a C macro.  And you provide the event class name, which could be my event,  for example.  It supports inheritance.  So all the events must eventually inherit from ht event,  which is a very base class.  And then you provide a set of fields  that you want to have in this event.  And you define it by three properties.  As a type, it's either integer, string, struct, flow,  double, or pointer, I think.  Then you provide the actual C type of that.  And the name.  And that converts to the C structure.  And additionally, it generates, automatically,  a few helper methods, like, for example, a method  for serializing the event.  So when you have this event and you  want to push it to the byte stream,  this macro already will generate a function that  does it for you.  And then when you already have the event,  if you want to push the instance of this event  type to the timeline, you just call  ht time line push event macro.  You pass the timeline as a first parameter.  The second parameter is the name of your event type,  and then all the values, all the fields.  And that's pretty much it.  So I said that before that we provide the client that  parses the binary stream.  But then I said that you can write your own event type.  So you might be wondering how the client knows how to deserialize  those new event types.  And for that purpose, we divided the event stream  into two sub-streams.  There's a metadata stream that describes all the types,  and there's the actual event stream with all the values.  And since everything in Hocktracer is an event,  even the definition of the event is event as well.  So we have special events that describe your type.  And the first event is just describing the name.  And the number of fields that you have.  And then you have each new event for each field of this class.  So in that case, we'll expect one event for class info  event and three events for class field info event.  And in class field info event, we provide information  like the field type, a field name, the size of that field,  and the data type, whether it is a string, integer,  a struct, and so on.  And both streams are serialized as a byte stream.  So you can see that basically it's just 30 or 40 bytes  for those events.  And then eventually, you have the actual event.  And this event has a type.  So it's number nine.  As you can see, the class ID is nine here.  And the info class ID nine again.  So all of them are connected to each other by this identifier.  And it's important that you first  need to send the definition of the event type  before you actually send the first event of this type.  Because otherwise, the parser doesn't know how to parse  this event.  But the hokter does everything for you.  So it's just.  Can I ask a question?  Yeah, go ahead.  If the parser loses sync with the byte stream,  is there a way for it to regain sync later?  So no.  At the moment, it is not possible.  I'm working on improvement that it will be actually possible.  But at the moment, if you lose the metadata stream,  yeah, it is a problem.  So yeah.  But I'm aware of this problem.  And we'll be working on that.  OK.  So I mentioned that we also want to measure time.  And that was actually quite important, requirement for us.  So to make it even easier, we have  predefined event types that are already for measuring time.  They take a label.  And they take the duration and the timestamp.  And we have macros that automatically generate those events  and push them to the timeline.  And so for C++, depending on what you want to do,  if you just trace the whole function,  you just put this macro, httrace function at the beginning.  And if you just want to trace the random scope,  there is httrace.  There is a few more macros that are more optimized for that.  For example, they have some hash maps.  So you don't have to send the string every time for every event.  It's even more optimized version that  uses some tricks with static thread local objects.  So even hash map is not necessary for some of them.  It's documented, so I don't want to spend much time on that.  And we did bindings for a few languages.  For now, the public one is Python and Rust.  We have one for JavaScript as well.  It's not ready yet to be published,  but we'll do that in the future.  And for Lua as well.  So this is how you do that in Python.  You import the trace decorator, and you just  put it for the function that you want to trace.  And the same for Rust, there are some macros.  I'm not an author of the Rust bindings,  so probably if you want to know more about this,  there's Alexandru, he's in Fosdum.  Not sure if he's today in this room,  but he's an author of that.  And basically, that's how you do that.  There are some kind of Rust macros  that you can trace the scope.  OK, so that was it.  And I just want to show you some demos.  So you can see how you can use it.  The first one is we have a C++ application that allocates  a memory, and then we want to see on the graph  how the memory grows.  We also want to know how many allocations  we've done in this program.  So we'll write a simple Python client that  receives the Hoke Tracer event stream  and does the visualization.  And I start with the program itself.  So at the very beginning, we need  to initialize the Hoke Tracer library.  So this is the event type that we define.  As I said, there's a name, the base type,  and we have two fields, memory usage and allocation count.  They both are integers size t type.  And the event, before you want to use it,  you need to register it.  This method is auto-generated by the macro  that I showed before by this macro, ht-declare event type.  You just need to call it before you use it.  It registers some information in the Hoke Tracer system.  And then you create a listener to the timeline.  I said the global timeline is the best option.  So we're using global timeline here.  There are some parameters like port.  This is the buffer size.  We set it to zero.  So we'll get we don't actually buffer anything.  We stream it directly to the client.  And we say that we want to use TCP client.  So we can stream it directly to the client.  And we can get the real data.  Hoke Tracer already provides the functions  to get the memory usage.  It also provides some functions to trace the allocation.  So you just need to provide a callback.  This is for malloc premalloc.  There's post malloc, pre-realloc, post-realloc,  and so on and so on.  We are only interested in premalloc hooks.  So we register that.  And this is basically our function.  We get the memory usage.  We get the allocation count, which  is read in the other callback.  So here, this is our malloc callback.  So we increase the allocation count  every time somebody calls malloc.  So we get that all in the context object.  And that's basically what we do.  We push the event.  So this is our type.  Those are our values.  And this is the memory usage, virtual memory usage.  And this is the allocation count.  And this is the client.  It's written in Python.  All we need to do, we need to start the client, the Hoke Tracer  client.  We say, OK, listen to 8.76.5 port on this IP address.  This is the animate method.  That's basically the anim that's drawing the graph.  But the most interesting bit is this one.  So this is basically how we read events from Hoke Tracer.  This is our client.  So we're waiting for end of stream.  If it's not, then we pull an event.  If there is an event in the queue,  we check the first.  This is a tuple.  And the first element of the tuple is the event name.  So we check if this is the event that we want.  And if so, we get the timestamp.  We get the allocation count.  And we get the memory usage.  And then we put the memory usage on the graph.  And we print the allocation count.  So it's very, very simple.  Let's run it.  So we run the client.  And now we run the program that we want to trace.  So you can see that the memory usage is growing.  And you can also see the allocation count is going up.  And so you might be wondering why the memory usage is growing.  If we go back to our application,  we see that we have a loop.  And we do malloc.  And it's called 100,000 times.  So we can see how many mallocs we actually got.  It's more than 100,000.  There's probably some other allocations going on.  Yeah, and this is the graph.  So that's how you can get the real data from the Hoke Tracer  by defining your own event.  It doesn't have to be memory usage.  You, for example, can trace the number of HTTP calls,  for example, and so on.  Yeah, so that was the first demo.  The second one is going to be more complex a little.  I want to show you how you can trace multiple languages  at the same time.  Because that was actually our real use case.  We had application written in C++.  And then we run Lua and JavaScript on top of that.  So I do a slightly different example.  I have a main application written in Rust.  This application downloads the image using a downloader library  that's written in C. It uses curl.  And then I rotate this image using some image.  I think it's open CV.  Maybe not open CV.  I actually changed it.  But yeah, it rotates the image in Python.  So I run Python interpreter in Rust.  And then at the same time, while I'm rotating the image,  I also upload the image that I downloaded in the first step  to the S3 backend.  And I want to know how much time I  spend in each particular operation.  So I, again, start with the code.  So this is our Rust client.  Even though you might not be familiar with Rust,  I think it's pretty simple.  So we start with download file.  This is download file RS.  It's basically just the wrapper for the download file  library, function from the library that's written in C.  Then once we have the file downloaded, we spawn two threads.  One is doing the rotation in Python.  It starts the Python interpreter.  And that's the rotation.  And the other one, at the same time,  uploads the file to S3.  You might see that all of the functions here  are decorated with those macros.  And we also have some other trace points in here.  We can look at the downloader file.  It's pretty simple one.  It's just the curl wrapped around.  We trace the function here at the beginning.  And we also see how long does it take to call curl easy perform.  So we trace the whole scope here.  And just last bit is the Python one.  We import the decorator.  And we decorate all the functions.  So we basically rotate the image and save the image.  We load the image.  We rotate the image.  And then we save the image to the file.  So not really a rocket science here.  It's important that if you want to run it with Python,  this environment of variable needs to be set.  Otherwise, the Python bypasses all the trace decorators.  So it doesn't have much impact on the performance  if you disable that.  So you can even have it in the production code.  OK, so let's run this.  So this is our program.  It downloaded the file successfully.  It's doing the rotation.  And it's doing the upload.  So you can see that it actually did rotate some image.  Let's start from zero.  Yeah, so it's a hook.  And it's basically all the angles are here.  If we delete that one, we have also  rotate the HTDAM, which is the binary file  with all the events, all the traces.  This is not going to work because I forgot this environment  variable here, so we wouldn't have the Python traces.  So I run it again.  I'm going to delete the images.  So we have the HTDAM file, which is the binary stream.  We can now use a Hock Tracer converter.  This is the one that's experimental.  So it's working.  And we generate the Chrome trace format.  So we say it would be output.json.  It converted.  So now you can use the tool that's deployed with Chromium.  And you see that first you see that we have three threads here,  one, two, three.  That's exactly what we had because we had the main thread.  And then we spawned two extra threads for rotate and for upload.  This is the first bit that's in C++ or C.  We download the file, there's HTTP request.  And then just after that, we start rotating Python.  And those trace points here, those are from Python.  Save image, rotate, main.  Those trace points are from Python.  And this one, upload to S3, is in Rust.  So as you can see, you can trace.  You can have everything in one trace file.  And you can easily analyze what's going on in the program.  You can also convert it to flame graphs.  So if you want the flame graph view, you can just do that.  This is a very simple program.  So the flame graph also is very simple.  But yeah.  So that's it.  We have some plans for improving this.  There are some of the items, but not all of them.  As you mentioned, we want to add some extra protection layer  in case we lose some events on the way.  And yeah, that's it.  Thank you.  There's a bunch of links.  Feel free to contact me.  There's a documentation on the website.  There are tutorials.  They say how to integrate it with your project.  Basically, Hocktracer can be combined into one single C++  and header file.  So it's easy to link it to your project  if you want. There are other links for bindings  and for converter.  So thank you.  And I think we'll have one minute for questions.  But if that's not enough, then I'll be around.  OK.  Which methods like you showed TCP now, right?  So in addition to TCP, what do you currently  use?  So this is the TCP that we use in the first demo.  And the file, which was used as a second demo.  Yeah, so those are natively supported at the moment.  What's something like serial or can you  have to use for all your tests?  Sorry?  So if you want something like serial?  Yeah, you need to write the neural.  Sorry, the question was, what listener types we support?  And my answer is currently we support file and TCP listeners.  Yeah.  Thank you.
debugging_hawktrace_0_155.png debugging_hawktrace_156_1019.png debugging_hawktrace_1020_7427.png debugging_hawktrace_7428_8399.png debugging_hawktrace_8400_20651.png debugging_hawktrace_20652_21191.png debugging_hawktrace_21192_23231.png debugging_hawktrace_23232_23963.png debugging_hawktrace_23964_26027.png debugging_hawktrace_26028_26975.png debugging_hawktrace_26976_27371.png debugging_hawktrace_27372_28835.png debugging_hawktrace_28836_31499.png debugging_hawktrace_31500_31955.png debugging_hawktrace_31956_32075.png debugging_hawktrace_32076_32639.png debugging_hawktrace_32640_32879.png debugging_hawktrace_32880_33443.png debugging_hawktrace_33444_33491.png debugging_hawktrace_33492_33563.png debugging_hawktrace_33564_34667.png debugging_hawktrace_34668_34847.png debugging_hawktrace_34848_34955.png debugging_hawktrace_34956_35027.png debugging_hawktrace_35028_35219.png debugging_hawktrace_35220_37103.png debugging_hawktrace_37104_37283.png debugging_hawktrace_37284_37511.png debugging_hawktrace_37512_37627.png