Harnessing Intel Processor Trace on Windows for fuzzing and dynamic analysis
CC Attribution 4.0 International:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Allievi, Andrea
Johnson, Richard
This talk will explore Intel Processor Trace, the new hardware branch tracing feature included in Intel Skylake processors. We will explain the design of Intel Processor trace and detail how the current generation implementation works, including the various filtering modes and output configurations. This year we designed and developed the first open-source Intel PT driver for the Microsoft Windows operating system. We will discuss the architecture of the driver and the large number of low level programming hurdles we had to overcome throughout the development of the driver to program the PMU, including registering Performance Montering Interrupts (PMI), locating the Local Vector Table (LVT), managing physical memory. We will introduce even the new features of the latest version, like the IP filtering, and multi-processor support. We will demonstrate the usage of Intel PT in Windows environments for diagnostic and debugging purposes, showing a “tracing” demo and our new IDA Plugin, able to decode and apply the trace data directly to the visual assembly graph. Finally we discuss how we’ve harnessed this branch tracing engine for guided fuzzing. We have added the Intel PT tracing mode as an engine for targeting Windows binaries in the widely used evolutionary fuzzer, American Fuzzy Lop. This fuzzer is capable of using random mutation fuzzing with a code coverage feedback loop to explore new areas. Using our new Intel PT driver for Windows, we provide the fastest hardware supported engine for targeting binaries with evolutionary fuzzing. In addition we have added new functionality to AFL for guided fuzzing, which allows users to specify targeted areas on a program control flow graph that are of interest. This can be combined with static analysis results or known-vulnerable locations to help automate the creation of trigger inputs to reproduce a vulnerability without the limits of symbolic execution. To keep performance as the highest priority, we have also created new methods for efficiently encoding weighted graphs into an efficiently comparable bytemap.
 Okay, before start a brief introduction about us, I'm Andrea Llevi, I'm a security research engineer of Microsoft. Now I have worked for three years for the TALOS team and I'm specialized in Windows internal and low level reverse engineering. I have a previous worker for Prevex, Webroot and Saferbeds that are companies and located even at one easy in Italy. I'm the original designer of the first UFI boot kit in the year 2012 and the first patch got the 80.1 bypass presented in the year 2014. I'm one of the designer of the Windows PT driver that we are going to present. And I'm Richard Johnson, I'm the research technical lead for Cisco TALOS. We have a team that does vulnerability research and I help guide those efforts. You may see some of our vulnerabilities in the past year and we focus on technologies mostly for finding bugs. And so this was, as he mentioned, some of my talks the last couple years at recon have been in the focus of high performance applied to fuzzing and engineering technologies that give us feedback driven fuzzing and componentizes the different parts of the process to try to make it as fast as possible. I came across this new architectural feature in Intel CPUs called Intel processor trace about two years ago and basically it's a hardware supported mechanism for doing code coverage and I made a prototype for this in 2015 that worked for Linux because there was experimental support via some open source drivers. After evaluating it I realized that this would be a great thing to bring to the Windows operating system but the Intel support for it was not going to be suitable for what we were looking for. So last year at recon we had our very first version of the Windows driver working like an hour before we got on stage and it was doing a raw decode and so we're going to talk about the last six months of developments which have brought all kinds of support to the driver and this picks up where the last one went off. So Andrea is going to give you the introduction of the technical details of the driver and the low level implementation and some demos and then we'll address that as applied to fuzzing and finding bugs. Okay, speaking about processor trace, the processor trace is a new feature of the latest Intel Skylake CPU. It's very useful because it can trace whatever your CPU is going to execute like in hardware and it has some particular benefit especially for dynamic code coverage. For example if you would like to understand what a particular piece of software would like to do for malware analysis or for whatever. I mean the usage are various. I would like because we have not a lot of time I would like to be quite fast in describing how the Intel PT is executed in a CPU and I would like to concentrate on the new feature of our driver. Basically being fast to discover the Intel processor trace in a CPU you can do even in user mode only emitting to CPU ID instruction with two different leaves. One is used to detect the support for Intel processor trace. The second one is used to detect the feature of processor trace because different CPU can implement different features. Processor trace has been implemented in the first time in the first in broadware architecture but it was limited. Now in Skylake there is the full support and you can trace whatever you would like. On CPU there will be a release of Intel in the second quarter of 2017. They are quite run. Okay, here in this slide I would like to show the code for detecting the processor trace. As you can see it is quite easy and there is nothing special only a pair of CPU ID instructions. Okay, let's speak about why it is so interesting. It is because it is implemented in T-Railin hardware. One of the basic things that we can say about this is that it is not detectable by software, I mean in user mode software. One important thing to say is that you can trace whatever you want. Even SMI, SMM, handler, even an hypervisor code or whatever. The only thing you can trace is like the SCX secure container but it is a good thing because the SCX by design should work in an inosolate environment. Okay, even here quite fast how it works. The processor trace works in three modes. It can trace using three different kinds of filtering. The first is by current privilege level. I mean you can differentiate between kernel mode software and user mode software. The second filtering mode is by PML four page table. In that way you can trace only a single process because you can track the processor trace to trace only a specific PML physical address page table. In that way you can trace only a single process. Otherwise the last filtering mode is by instruction pointer. You can set a start point and the end point and ask the processor trace to trace only that window of code. And this is very cool. The output logging is done directly in memory, in physical memory. That's why we need a driver to manage that. And the logging could be implemented in two things, in two ways. The first one is single range. There is a circular buffer and the trace is written always in the same place in memory. The second type is the table of physical address, also known as TOPA. Okay, even here like quite fast. To implement the single range you should allocate a continuous physical memory buffer. And then you should set two proper mode specific registers. One is the RT high T output base and the output mask. And then you have to start the trace setting the trace PML flag in the counter register. The buffer is automatically filled in a circular manner by the CPU. The table of physical address. The table of physical addresses is a better implementation of the output because you can set various physical memory address and you can create a like table in which you instruct the CPU where to write exactly memory. It's very smart because you can even set up PMI interrupt that is raised by the CPU if a certain part of the buffer is filled by the log. And then you can stop, you can resume, you can do whatever you want. Okay, different kind of packets. To log the software execution, processor trace uses a different kind of trace packet. There are a lot of timing packets that we are not interested in. The packet that we are interested in are the branch packet. They are taken, not taken. They target the PM flow update packets. Those packets are the most interested because with those you can trace the execution of the software and follow its execution even maybe checking the assembly code. Here is a big diagram. I will refer you to the internal manual if you would like to understand the nitty-gritty details of each packet. The one that we are interested in are the branch packets. They are not taken, target the PM flow update. Okay, let's speak about the Intel PT driver implementation. Okay, we have decided to write this driver to be able to perform the trace directly from a Windows operating system. At this time of this presentation, the driver is quite stable and version 0.5. It supports all the filtering mode combination and output modes. Some new features of this release is that it supports even multi-processors and this supports even kernel mode code tracing. As I have told you, you can use the processor trace to trace even kernel software without any problem. In the developing of this driver, we had overcome a lot of problems. One of the most big problems are the mapping of the PMI interrupt because there was no documentation at all for how to do that. Even the multi-processors because you have to manage and you have to enable the processor trace to once in each processor. Okay, let's speak first about the PMI interrupt. The PMI interrupt has been raised by the processor trace when our buffer is full. To do that, we have programmed the table of physical addresses in inserting this PMI interrupt request at the end of the buffer. How we can be able to manage that? When the PMI interrupt raise, we suspend the target process, dump the physical memory and then resume. We had some problems implementing this because as you probably know, all the interrupts inside the X86 architecture run at a very high, higher quell. This is quite a problem because from that code, you can do quite anything. Even the user mode buffer, very fast even here, we have found a way to directly map the physical memory in a user mode buffer. We do this in a smart manner. I think it's not we respect the security boundaries and we map it only the log buffer and that's all in user mode. Nothing kernel mode is important because if you use a very big buffer, you have the problem that for the virtual address space, that's now it's not a problem in 64-bit systems, what could be? Okay. Here is where the things get interesting because in version 0.5, we have been able to support the multiprocessor and multi-thread application. Each CPU, as we have implemented in that way in which each CPU has its buffer associated with it and it's mapped in user mode. We signal a PMI interrupt where the buffer will be filled. But here, there is a problem because if we fired the PMI interrupt in the same way, the user mode application is not fast and is not able to detect which CPU is as the buffer full in a real-time way, in a fast enough way. Then, we have switched the implementation and implementing the user mode callbacks. I mean, our user application has spawned one thread for each CPU and each thread registered a PMI callback function in that way when a CPU has the buffer at this full called the exactly right user mode callback. Here, we have tested that even in a real big multiprocessor environment, if we don't decode in real-time the binary log to transform it in a human-readable text, the performance is really good. I mean, we can't see some slowdowns or something in the Traced application. Speaking in summary, we have overcome this problem, but the only problem that we had is managing a multi-threaded application because as you already know, for a CPU point of view, a thread doesn't exist. I mean, the CPU is executing some codes. That's not the CPU doesn't know if it belongs to one thread or another one. Okay. If you try to launch CALC, the standard calculator in Windows 10, you will find that it's not a standard process. It's a new upcontainer process that spawns another process. This is an example of the increasing complex of the even the standard process that spawns another process or multi-threaded. Here, it could be a problem for our tracing purposes because of that. To overcome that, we can identify the paging information packets of each process and use the processor tracing only an instructing it to filtering by CR3. But we have a big drawback because the size of the log is huge. I mean, in that way, you trace all the process of the user that runs in user mode, all the loader code, whatever. This is a problem. The second way to overcome this could be to register a process through the creation CALC in kernel mode and then trace only one process at a time. This solution is simple and it works. But sometimes it's not acceptable because, for example, some hardware or some complex component like, for example, Microsoft Word, require the interaction from different process. This is a problem. But we are researching a new way to do this because originally we would like to enable processor trace by each thread, by using a thread construction. The thread is known only in software. Only Windows kernel know how to manage the threads. Our original idea was to intercept the thread context switcher code and save manually all the mode specific registers used in processor trace to an area and then restore back when the context switcher restore the original thread. When I was doing this, I was manually saving all the mode specific registers on an external buffer. But someone has been pointing to me on the existence of another very cool instruction that is not known by the research community but is very useful. Do you remember the old push-ad instruction in X86 architecture? Basically what it does is it pushes all the general purpose registers in user mode stack directly only using one instruction. Now in 64-bit environment, something like that doesn't exist anymore. But Intel has made this new cool instruction called XSave. The XSave is a new opcode in the MD64 instruction set that basically saves all the some extended registers, some registers that belong to the Intel architecture on a specific area. I have found that the XSave instruction can save MMX, SSE, AVX, AVX registers. I have written here that what is AVX 512 because I didn't know the existence. The cool feature about the XSave is that it can save even the register that belongs to Intel processor trace and the new Intel memory protection extension. And it is very cool because using only one instruction we can save all the registers that belong to the processor trace directly in a very, very fast manner without any problem. If you open the Intel manual you will find that you use this instruction, it's a bit complex because there is an XSave instruction that belongs only to user mode. And to set what to save you have to set an extended control register in user mode using a new instruction that's called XSBV. But to be able to save the mod specific register directly you have to use another instruction. You have to use the XSave S that it means XSave supervisor and it's good to use it in kernel mode. Our driver supports completely the XSave. And it was very funny because when I implemented this I have found that the new Windows 10 context switch already implemented the support of this XSave but only for user mode. Our original intention was to find a way to intercept or divert the key swap context routine that is the kernel mode routine that Windows uses to perform the context switcher from one thread to another thread. If we are able to intercept this we can save all the mod specific register belongs to processor trace directly in an area and then restore later. In that way we can implement the tracing by thread that is a completely software, it leaves in a completely software point of view. But we found some problems because as you probably know you can't touch any kernel mode in an official way using for example an hook, I don't know a code diversion or whatever because otherwise Pashkar will blue screen off that your system. And we found that this way is not feasible in a public system. We can use our debug system. In a debug environment you can do that and Pashkar doesn't run and it's not a problem but it's not a viable way in a production system. The second solution that we found is the usage of ETW. If you check the documentation about ETW there is a way to intercept the context switcher but it's we are still doing some research because the API are very complex and we are trying to get if we are able to use in a legally way ETW to implement the trace by thread. Another cool feature of the new relays is that the driver now fully supports the kernel mode tracing. We have implemented 11 new kernel APIs that you can attach to your driver and you can use it manually to decide what to trace, how to trace and to do whatever from a kernel driver. But we were not happy with this because we would like even to implement the tracing from a user mode application. Then we have created some high-octls that you can communicate with our driver and it's able to do kernel tracing directly from user mode. And to do this we have overcame a lot of security problems but now you can do that. From our user mode application you can even trace the kernel code. I said I wrote here that in this way we are able to for example trace the loading or unloading of a kernel module or even if maybe you are studying some high-octl in a root kit or whatever you can even trace only the high-octl code because as you know the high-octl in kernel environment are processed as synchronously when a user mode application asks. Okay, some quick words about how to use the driver. As you can see the code is quite simple. First of all you have to grab an handle of our device. It's quite easy. The device name is named Windows Intel PT Dev. After that you have to fill a data structure named PT user rig that is in our request and then you have to ask the high-octl manager to send the high-octl to our device. As specified the user request. After all, after this the trace starts. You can decide where to stop the trace using another high-octl. This is very important because if you close the application without doing that it means that your processor is still tracing something and this could lead to a problem if you try to upload our driver because the processor is not cleared. But the driver is able to detect this and to overcome this. But it's a good practice to do that. For multi-processor code you can spawn a user mode thread directly without any problem and then from the user mode thread you have to call the IO manager to send the register PMI routine high-octl to our device. That's all. Then you wait in an infinite loop. The only thing to take care is that there is a parameter in the sleephacks function that is named a table or not. You have to insert through. That way every time the CPU buffer gets filled the callback will be called without any problem. Now it's time for a demo. I don't know how much. We have to be very fast. I have prepared a demo for you. As you can see there is the code of a very simple application. It asks only some questions to the user. Let's try to run it. No kernel trees. Target process is our simple application. Andrea, you can see here. Just a moment. How many CPU? At the beginning let's do one. Only one CPU. He is asking you to increase the font size. You can mirror it. While he is setting this up I will recap what he went over. When we presented it last year in June all we had shown is the raw code. He is going to show you a tracing in different modes and visualizing it in IDA. How many conferences have you attended? Let's say three. The application has ended saying you are on a good track. Let's exit and try to open the executable using IDA. Here is the code. Our goal is to trace the software. We have developed IDA plugin that does this for us. Let's feed it with the text log. Wait a moment. An exception. As you can see, this is exactly the code that it has run. The exception was not voluntary. We use a library from Intel called lib.ipt. It provides the decoding of the binary trace to a text file that we are parsing in the plugin for right now. Later we will integrate the decoding within the plugin. I would like to show you the same with the multi-processor environment. Let's see where is the part of my process. How many processors? Four. Now let's say eight conferences. The results are different. You can see in the summary each processor has its own buffer with different number of packets. It is this dump. As you can see, there are different binary files and text files for each processor. Now we will increase the font size. Can you see guys? As you can see, this is CPU number one. It has all the packets, the taken not taken and the paging, paging generation, enable and disable. Because the context switcher of Windows switch to different threads. This is one CPU. But for example, if you are lucky, we are not lucky. There is even some logs that are quite clear because it means that the Windows has executed these executables only for a small period of time. This is not the case because luckily the context switcher runs for each different CPU. But sometimes it happens that even the log for one processor is empty. And this is our implementation. Can you pull up one of those text files again? Okay. I just wanted to point out. So when you read these text files, you can see that for indirect branches and return addresses, we actually get the full 64-bit target address. But if you look at the TNT packets, those are indicating whether or not a conditional branch was taken. So they only store a single bit that determines whether or not you took the true or false branch. So you have to recover that later on and disassemble in real time to recover what the target addresses were for those conditional branches. We have time. Yeah. I would like to show you an experimental demo that uses kernel tracing directly for user mode. For this demo, I have chosen ACP driver, not for a specific reason, because it was randomly chosen. And I found that the system calls this interface a lot of times. Let's try to do even this, to take even those dumps. The explorer has been blocked. Amazing. Okay. Let's say yes, ACPI.cs. Let's say go, number of process of CPU, just one from this time. Okay. It asks, I'm tracing, do something and then stop me. I do, for example, some movement on my PC. And then at a point of time, I will say stop. In this time, we were quite lucky because, as you can see, the process on number one has taken some packets. Let's see what are those packets. Okay. As you can see, there is not so a lot of time, a lot of things registered here, but something has been taken. Let's try to use our plugin to be able to trace what you can see. This is the code of ACPI. Unfortunately, I have no symbol about this because it's an in-flight release of Windows 10. But let's try. Okay. The plugin has worked. You can see now because the driver entry has never been called, has been already called when you have switched on your system. But if you see the log, there is a lot of time called this special function. That could be an IOCTL or whatever. Let's try to go. Yes, this is IOCTL. You can see. Just a moment. This should be blindly without knowing anything about the interface of the driver. We can probably say that this is an IOCTL because the code is executed a lot of times because for the color, the color is darker. It means that the CPU has executed this function a lot of times. As you can see, there are traces of all the branches. This is... Ah, come on. It's not taken. And you can see there is the branch. And all the branches are traced. And that is the demo about the kernel tracing from user mode. From kernel mode, if you develop your driver, you are even being able to trace the driver entry or driver unload routine. I can show that right now because first we don't have time and second, my computer is switched on in a relay's environment. Because for doing that, of course, we can't use a signet driver. But it's feasible. If you do that, if you write your kernel driver and you sign your kernel driver, you can do whatever you would like. Okay, let's write it back today. So now we'll switch. Oh, okay, yes. So yeah, to recap, the driver now supports kernel tracing and user mode tracing. You can filter based upon the CR3, so a single whole process. You can trace the entire kernel space or you can isolate ranges of contiguous IP. And you get up to four different ranges. So now I'm going to demonstrate this in a practical real world scenario inside of AFL. So we have this fast tracing engine and you'll see some performance numbers that are real world here. But in the manuals, they are targeting a 5 to 15% trace overhead for the entire system. So per core, you should be able to trace both kernel mode operations and user mode operations for only 5 to 15%. And I'll actually be able to show you that and how that works. I'm sorry. Yes, yes. Oh, I'm sorry. That will help. That's one. Yes. Okay. So how do we use this and apply it for vulnerability discovery? So who here is familiar with American Fuzzy Lop? Has used it? We have people who do fuzzing in the crowd. Okay, a good portion here. So in the last few years, we've seen, well, an evolutionary jump in fuzzing technology. Basically, we've gone from using dumb fuzzing or grammar-based fuzzing that wasn't able to determine whether or not the samples that were being generated were useful to applying a new technology or engineering an older technology into something that's performant enough to be used. And we call that evolutionary fuzzing. So we take the idea of dumb fuzzing mutation and we combine it with the ability to collect a feedback signal using code coverage. And then we assess the fitness of that new randomly generated input against the entire lifetime of your fuzzing cycle. And so basically what we can do is we can look at this code coverage information, determine if this newly generated input actually gets us to a different part of the code. If it does, then we'll introduce that into our entire pool of samples and continue to mutate and fuzz those as we go. So over time, we're refining our set of inputs and getting, we're building a corpus and each one of those inputs exercises a slightly different part of the code. And in effect, what we've seen is the last three or four years since this has been available, this has highly optimized your compute time when it comes to doing dumb fuzzing. And so the last couple of talks I've given have focused on this technology. And so I encourage you to go look at previous slide decks which are on the recon website or my website at moflo.org. But basically through researching this, I've resolved that the main things that we need to effectively deploy this technology is we need a fast tracing engine. And of course, that was the inspiration to look into Intel processor trace because the promise of 15% overhead against closed source binary software is pretty incredible compared to the technologies that we've had available before. Previously, hardware tracing is not new to Intel. Since the P4, there's been the ability to do hardware tracing. There's mechanisms called branch trace store, BTS, which is works in a similar fashion, but was not designed in a way that was optimized. It didn't write to physical RAM. It polluted your cache and things like that. So we saw a massive slowdown. And then you have another option which is called last branch record, which is only 32 registers and modern processors. And those only give you the last 32 branches. So you'd have to interrupt every time every 32 branches to parse that and use that. And or you'd have to write a driver that flushes that out to a different cache and do other things. So while this isn't new, this is designed for the first time to be highly performance up until Intel processor trace. It was actually faster to use software based tracers, things that would do dynamic binary instrumentation using Dynamo Rio or PIN or something like that. So now we have this fast tracing engine. That's great. We need fast logging, which is something that we get out of the design of AFL. AFL uses a bloom filter that allows you to quickly look up whether or not you've done the same code coverage. So instead of parsing a text file or binary file, that's just a list of addresses of basic blocks consecutively. We actually parse that real time and fill a bloom filter. So you can just check to see if this 64k of RAM is identical to another 64k of RAM instead of doing a comparison of each address. And then through some of my other research, there's been other attempts at evolutionary fuzzing starting about 2004 or 2005. Jared DeMott did some of his PhD research on this, and he had something called the evolutionary fuzzing system. But it was based upon basically the BTS record or debugger breakpoint. So it was quite slow in its tracing. And then it was also over engineered and trying to incorporate too much of the research that's in evolutionary, like, biology side of things. So the key is to have this fast tracing engine efficient logging and to keep the analysis to the minimum. So in 2013, Michael Zolesky, who's contributed a lot of great stuff to our industry, produced the first performant open source evolutionary fuzzer called American Fuzzy Lot. It uses a pretty comprehensive list of types of fuzzing strategies, whether it's bit flipping, byte flipping, d-word flipping, and so on, crossovers, and various types of mutation. It uses, originally it used block coverage via a plugin, a post process of the GCC compilation. It would compile your code to assembler and then annotate the assembler to add callback hooks at every basic block entry point. And then that's, through that modified source code, that's how you got your code coverage. And then, as I mentioned, he takes those edge transitions, basically shifts and zoars them together, and then increments offsets into this bloom filter or byte map that is basically able to track, you know, whether or not you've seen this edge before. Now, you can't get out what those addresses were originally because it's simply an offset into this mapping, but you can very quickly look up, have I been here before? And that's all we care about as far as whether or not to keep that sample. And then, of course, it was written on top of the POSIX API, so it wasn't Windows-compatible out the gate. The benefits that it has, it tracks edge transitions and not just block entry. It uses the bloom filter. It had a fork server built into it. And then, basically, after your process has initialized, it waited until all your libraries are loaded and all the linking and everything is done. And then once you get to the parser code, then it would fork. And so, you skip all that initialization time, which is an optimization. And then, very importantly, he introduced persistent mode fuzzing, which is an in-memory type of fuzzing where you're not exiting and recreating the process every time. So, you're giving it a pointer to a function and the number of arguments to that function and saying, okay, once you exit this section of code, start over again and take our new inputs as inputs to this function. And so, that also reduces the amount of code that you're tracing and executing down to the minimal points as an optimization. And then, importantly, you can use this to build a corpus on open-source software. It's very fast, and so you can use those as inputs into your pipeline on maybe slower or more heavyweight analysis on other types of fuzzing. So, the way they do their tracing is every block gets a unique ID. The edges are indexed in that map. It creates that hash using the shift and zore, and then we increment the map. So, this was great. I was looking into this and how to optimize this and bring this to the Windows platform. Obviously, we can't use source code instrumentation for the majority of the software that we're targeting, so we needed something that could do binary targeting. And so, this seemed ideal versus the other options of using PIN or Dynamo Rio and so on. And so, last summer, we decided to start looking into this. Well, around that same time of my talk at Recon last year, Ivan Fratric from Google, I think he's Project Zero or Google Security, released Win AFL, which was a port of Michael Zeweski's AFL to Windows using Dynamo Rio as a backend. Are you guys familiar with PIN or Dynamo Rio? They're basically loaders for your program, and as you visit each new basic block of code, it caches that and allows you to modify it in real time. So, it's a kind of, Valgrind works the same way if you're familiar with that. So, it was using that as a backend. It's really cool. It works. It was like the first thing that you could just go download right now and start fuzzing Windows GDI in five minutes. Beautiful. The biggest thing that's allowed it to be a performance is that it uses this persistent mode where it doesn't exit the process. So, these loaders like PIN or Dynamo Rio, they have to disassemble your program to instrument them. So, he was able to, and I did some experimentation on trying to do forking in Windows previous talks, and it turns out it's just real pain. So, using persistent mode, you get things like, you don't have to worry about ASLR because you're not exiting the process, and you don't have to re-disassemble the process every time because you're using that code cache. So, Win AFL turned out to be pretty well engineered. And basically, you can tell it how many iterations to persist, like, you know, maybe do a thousand iterations, and then go ahead and exit and restart the process so that we can, if we have any memory leaks or we're not quite cleaning up properly, we can just handle that through delaying the restart. Now, persistence is key because every time you load this in the DBI, it's going to disassemble it. So, if we were to just do it every time, we'd get two executions a second on this GDI plus demo I'm going to show you. If we persisted 100 times before we restarted 72 executions a second and so on, it reaches its ceiling somewhere around a thousand or so iterations. So, we have now integrated our Intel PT driver into Win AFL as an alternative tracer engine. And now, this brings some problems because the reason I had him show you the text version of that dump was that we don't have all the addresses in the log file. So, we have to recover some of those along the way. We don't have persistence mode working quite yet, unfortunately. I have done some experimentation here. It's around the corner. By the next time we present it to Hack in the Box, this will be available. I'm building my tooling on top of Alex Inescu's great work from last year at Recon on the application verifier hooking system. And so, basically, as it is, I'm using the IP filtering mode so you can specify up to four DLLs that you want to trace or, you know, four modules or any address ranges in your process that you want to trace and so on. The current status now is that we do accurately decode the full trace, so doing disassembly online and using cache for the control filigree that you recover. So, I first looked to see if we've already resolved what this upcoming conditional branch is. And if so, then it's a quick index. If not, I have to disassemble forward to determine the targets for the conditional branch and then store those in a structure. The edge and source destination are recorded as expected, so we're not reduced to basic blocks. We actually do get the edges. And I'm currently just using create process, so we're doing this iteratively rather than forking or persistence mode yet. So, in order to determine the performance of this tracing mechanism, I first made a dummy looping benchmark. So, basically, it was just creating the process and waiting for it to exit. So, we get our kind of maximum bounds on how many iterations we'll be able to execute with a sample. In this case, I found that we could get 85 executions a second without doing any tracing. We're just generating the fuzzer input and running it without anything like that. We're not parsing the log file. We're not doing anything. So, once we enable the tracing, that was reduced to 72 executions a second, which is right in that sweet zone of 15% overhead that Intel was promising for this particular sample. So, parsing the log file was an additional 22% overhead. So, now we're down to 55 executions a second. And I'll demo what this all means for you here. So, we'll compare it to the... I'm going to show you... So, we're going to fuzz GDI plus. This is an experiment that comes with the Win AFL out the gate. You just pass it some image files and it uses Windows to render them without rendering to the screen. And this is a live demo of that working. So, currently this is using Dynamo Rio in persistence mode with the maximum number of iterations possible. And we see that it's getting 127 executions a second. The lighting is not good here, unfortunately, but hopefully you guys can see that a little bit. I can increase the font just quickly here. So, what we're looking at here is this number specifically. So, we get 126 executions a second using Dynamo Rio versus GDI. So, let's see how our Windows PC driver performs in comparison. Sorry, that's me decoding the log here. So, let's redirect that out to Noel. Okay, so we're seeing with that overhead of the 15% for tracing and then the additional overhead of 22% for decoding. This will creep up a little bit, but we're only getting about 40 something executions a second. That's a little bit disappointing. We're kind of hoping to see a little bit faster. Now, you have to keep in mind again that this is just iterative tracing. We're not doing in-memory fuzzing. So, once we get to doing in-memory fuzzing, this number will increase significantly. However, this is not the end of the story. I originally was doing all my testing against their setup with the GDI plus wrapper. And as you can see in my command line here, this is tracing only the Windows Codex DLL and the GDI plus DLL in the process. So, I made another demo in order to compare the performance. And this time we'll trace libpng using WinAFL. And so, this is just what libpng statically compiled into a small harness that will just load a png file. It will fuzz it and then it will fix up check sums and then it will parse it through libpng. And we see that the performance becomes quite abysmal, actually. So, this is the Dynamo Rio. That's expected. But we're seeing only 0.5 executions per second. So, this is quite slow, obviously. This is not where we want to be. And this is because since this statically compiled all of the code involved in libpng, including the encryption or compression and things like that, are included here, which causes some issues in the Dynamo Rio backend. Now, here's the fun part. We have a constant overhead when we use Intel PT. So, using Intel PT, we can see that we're back up in that. This will creep up to about 55-60 executions a second. So, instead of being only a half an execution a second, this is 100 times faster than the Dynamo Rio backend. And this is doing the full code coverage tracing, finding new paths, and so on. So, with this engine, we've done 100 times performance increase depending on your target application. And this is not specifically chosen. This was just randomly I had this fuzzer laying around and uses the demo. So, yeah. So, that's my demos for this plugged into 1AFL. Thank you. And so, just closing remarks now. Number one, all the code that we've written is already open source. They're going to be open source. We're on GitHub slash Intel PT. Real simple. The driver is already there. I actually have a pull request from Andrea from last night. So, the latest version is on GitHub. It'll be merged tonight. The 1AFL needs a little cleanup. I was up till 3.30 last night making my final preparation. So, that code will be up next week. And then we just have a few more things that we need to address. But obviously, you know, we see that code coverage is being finally harnessed to make our fuzzing better. We can load this information into IDA to do our analysis of crash or malware or whatever it might be. And using the hardware supported tracing engine, we don't have any issues like you would have with other software-based instrumentation and hooking engines. You know, once, you know, one of our future plans is to get this into a hypervisor so that we can trace the guests inside of a hypervisor. And then your, you know, your malware tracing will be basically, you know, unobservable and you won't be able to disable it in that method. There are capabilities of deploying Intel PT to trace things like SGX mode and SMM and BMM. And those are kind of future areas of work. And then also, my end goal is, of course, to get this fully supported with persistent mode and everything as well. One thing that we need to do as part of that is to finish the ETW-based thread context switch awareness, because we need to separate these logs out into a per-thread instance. Otherwise, you have to use the timing information to determine where the synchronization is between the threads, and that slows down the parsing a lot. So our goal is to get the logs individualized before you do your parsing, and so you only have to parse the thread that you care about that's doing your, you know, your file or network I.O. And did you have any further comments, solutions? Just wanted to say that the new version that we are going to release already support the XAVE feature, and if you download it, you can test it directly, and it's quite a cool feature in my opinion. Okay, so you can get this code and you can reach us on Twitter, and thank you very much. Thank you. Hopefully, I think we might have a minute for questions. We're just on time. Yes. Okay, Trace, inside the filter machine. So currently, there are, we, there are any hypervisors that currently expose this and virtualize this. For example, in the other hardware tracing modes, the hypervisor has to virtualize the support for writing those MSRs. This does exist, like, for example, in VMware using the BTS mechanism, but Intel PT is rather new, so there aren't any hypervisors that are available yet. So we are either going to have to modify Zen or KBM, and we've been actually even just today talking about perhaps being able to trace the entirety of the hypervisor and then later on pull out only the user mode processor kernel threads that you're interested in. So currently, no, but absolutely, we'll continue working on this until we get there. Thanks. Yeah, and I know that that will hopefully be applicable for cuckoo sandbox, so. Were there any other questions? Okay, feel free to grab us. Thank you very much for your attention.
 Okay, before start a brief introduction about us, I'm Andrea Llevi, I'm a security research  engineer of Microsoft. Now I have worked for three years for the TALOS team and I'm specialized  in Windows internal and low level reverse engineering. I have a previous worker for  Prevex, Webroot and Saferbeds that are companies and located even at one easy in Italy. I'm  the original designer of the first UFI boot kit in the year 2012 and the first patch got  the 80.1 bypass presented in the year 2014. I'm one of the designer of the Windows PT  driver that we are going to present. And I'm Richard Johnson, I'm the research technical  lead for Cisco TALOS. We have a team that does vulnerability research and I help guide those  efforts. You may see some of our vulnerabilities in the past year and we focus on technologies  mostly for finding bugs. And so this was, as he mentioned, some of my talks the last couple  years at recon have been in the focus of high performance applied to fuzzing and engineering  technologies that give us feedback driven fuzzing and componentizes the different parts of the  process to try to make it as fast as possible. I came across this new architectural feature  in Intel CPUs called Intel processor trace about two years ago and basically it's a hardware  supported mechanism for doing code coverage and I made a prototype for this in 2015 that worked  for Linux because there was experimental support via some open source drivers. After evaluating  it I realized that this would be a great thing to bring to the Windows operating system but  the Intel support for it was not going to be suitable for what we were looking for. So last  year at recon we had our very first version of the Windows driver working like an hour  before we got on stage and it was doing a raw decode and so we're going to talk about the  last six months of developments which have brought all kinds of support to the driver and this  picks up where the last one went off. So Andrea is going to give you the introduction of the  technical details of the driver and the low level implementation and some demos and then  we'll address that as applied to fuzzing and finding bugs.  Okay, speaking about processor trace, the processor trace is a new feature of the  latest Intel Skylake CPU. It's very useful because it can trace whatever your CPU is going to  execute like in hardware and it has some particular benefit especially for dynamic code  coverage. For example if you would like to understand what a particular piece of software  would like to do for malware analysis or for whatever. I mean the usage are various.  I would like because we have not a lot of time I would like to be quite fast in describing  how the Intel PT is executed in a CPU and I would like to concentrate on the new feature  of our driver. Basically being fast to discover the Intel processor trace in a CPU you can  do even in user mode only emitting to CPU ID instruction with two different leaves.  One is used to detect the support for Intel processor trace. The second one is used to  detect the feature of processor trace because different CPU can implement different features.  Processor trace has been implemented in the first time in the first in broadware architecture  but it was limited. Now in Skylake there is the full support and you can trace whatever you  would like. On CPU there will be a release of Intel in the second quarter of 2017.  They are quite run. Okay, here in this slide I would like to show the code for detecting  the processor trace. As you can see it is quite easy and there is nothing special only a pair  of CPU ID instructions. Okay, let's speak about why it is so interesting. It is because  it is implemented in T-Railin hardware. One of the basic things that we can say about  this is that it is not detectable by software, I mean in user mode software. One important  thing to say is that you can trace whatever you want. Even SMI, SMM, handler, even an  hypervisor code or whatever. The only thing you can trace is like the SCX secure container  but it is a good thing because the SCX by design should work in an inosolate environment.  Okay, even here quite fast how it works. The processor trace works in three modes. It can trace  using three different kinds of filtering. The first is by current privilege level. I mean you can  differentiate between kernel mode software and user mode software. The second filtering mode  is by PML four page table. In that way you can trace only a single process because you can  track the processor trace to trace only a specific PML physical address page table. In that way  you can trace only a single process. Otherwise the last filtering mode is by instruction  pointer. You can set a start point and the end point and ask the processor trace to trace  only that window of code. And this is very cool. The output logging is done directly in  memory, in physical memory. That's why we need a driver to manage that. And the logging could be  implemented in two things, in two ways. The first one is single range. There is a circular buffer  and the trace is written always in the same place in memory. The second type is the table of  physical address, also known as TOPA. Okay, even here like quite fast. To implement the single  range you should allocate a continuous physical memory buffer. And then you should set two  proper mode specific registers. One is the RT high T output base and the output mask. And then  you have to start the trace setting the trace PML flag in the counter register. The buffer is  automatically filled in a circular manner by the CPU. The table of physical address. The table of  physical addresses is a better implementation of the output because you can set various physical  memory address and you can create a like table in which you instruct the CPU where to write  exactly memory. It's very smart because you can even set up PMI interrupt that is raised by the  CPU if a certain part of the buffer is filled by the log. And then you can stop, you can  resume, you can do whatever you want. Okay, different kind of packets. To log the software  execution, processor trace uses a different kind of trace packet. There are a lot of timing  packets that we are not interested in. The packet that we are interested in are the branch packet.  They are taken, not taken. They target the PM flow update packets. Those packets are the most  interested because with those you can trace the execution of the software and follow its  execution even maybe checking the assembly code. Here is a big diagram. I will refer you to the  internal manual if you would like to understand the nitty-gritty details of each packet. The one  that we are interested in are the branch packets. They are not taken, target the PM flow  update. Okay, let's speak about the Intel PT driver implementation. Okay, we have decided to  write this driver to be able to perform the trace directly from a Windows operating system. At this  time of this presentation, the driver is quite stable and version 0.5. It supports all the  filtering mode combination and output modes. Some new features of this release is that it  supports even multi-processors and this supports even kernel mode code tracing. As I have told  you, you can use the processor trace to trace even kernel software without any problem. In the  developing of this driver, we had overcome a lot of problems. One of the most big problems are the  mapping of the PMI interrupt because there was no documentation at all for how to do that. Even  the multi-processors because you have to manage and you have to enable the processor trace to  once in each processor. Okay, let's speak first about the PMI interrupt. The PMI interrupt has  been raised by the processor trace when our buffer is full. To do that, we have programmed the  table of physical addresses in inserting this PMI interrupt request at the end of the buffer. How  we can be able to manage that? When the PMI interrupt raise, we suspend the target process, dump  the physical memory and then resume. We had some problems implementing this because as you  probably know, all the interrupts inside the X86 architecture run at a very high, higher  quell. This is quite a problem because from that code, you can do quite anything. Even the  user mode buffer, very fast even here, we have found a way to directly map the physical memory  in a user mode buffer. We do this in a smart manner. I think it's not we respect the security  boundaries and we map it only the log buffer and that's all in user mode. Nothing kernel  mode is important because if you use a very big buffer, you have the problem that for the virtual  address space, that's now it's not a problem in 64-bit systems, what could be?  Okay. Here is where the things get interesting because in version 0.5, we have been able to  support the multiprocessor and multi-thread application. Each CPU, as we have implemented  in that way in which each CPU has its buffer associated with it and it's mapped in user  mode. We signal a PMI interrupt where the buffer will be filled. But here, there is a  problem because if we fired the PMI interrupt in the same way, the user mode application  is not fast and is not able to detect which CPU is as the buffer full in a real-time way,  in a fast enough way. Then, we have switched the implementation and implementing the user  mode callbacks. I mean, our user application has spawned one thread for each CPU and each  thread registered a PMI callback function in that way when a CPU has the buffer at this  full called the exactly right user mode callback. Here, we have tested that even in a real big  multiprocessor environment, if we don't decode in real-time the binary log to transform it  in a human-readable text, the performance is really good. I mean, we can't see some slowdowns  or something in the Traced application. Speaking in summary, we have overcome this problem,  but the only problem that we had is managing a multi-threaded application because as you  already know, for a CPU point of view, a thread doesn't exist. I mean, the CPU is executing  some codes. That's not the CPU doesn't know if it belongs to one thread or another one.  Okay. If you try to launch CALC, the standard calculator in Windows 10, you will find that  it's not a standard process. It's a new upcontainer process that spawns another process. This is  an example of the increasing complex of the even the standard process that spawns another  process or multi-threaded. Here, it could be a problem for our tracing purposes because  of that. To overcome that, we can identify the paging information packets of each process  and use the processor tracing only an instructing it to filtering by CR3. But we have a big  drawback because the size of the log is huge. I mean, in that way, you trace all the process  of the user that runs in user mode, all the loader code, whatever. This is a problem.  The second way to overcome this could be to register a process through the creation CALC  in kernel mode and then trace only one process at a time. This solution is simple and it  works. But sometimes it's not acceptable because, for example, some hardware or some complex  component like, for example, Microsoft Word, require the interaction from different process.  This is a problem. But we are researching a new way to do this because originally we would  like to enable processor trace by each thread, by using a thread construction. The thread  is known only in software. Only Windows kernel know how to manage the threads. Our original  idea was to intercept the thread context switcher code and save manually all the mode specific  registers used in processor trace to an area and then restore back when the context switcher  restore the original thread. When I was doing this, I was manually saving all the mode specific  registers on an external buffer. But someone has been pointing to me on the existence of  another very cool instruction that is not known by the research community but is very useful.  Do you remember the old push-ad instruction in X86 architecture? Basically what it does  is it pushes all the general purpose registers in user mode stack directly only using one  instruction. Now in 64-bit environment, something like that doesn't exist anymore. But Intel has  made this new cool instruction called XSave. The XSave is a new opcode in the MD64 instruction  set that basically saves all the some extended registers, some registers that belong to the  Intel architecture on a specific area. I have found that the XSave instruction can save MMX,  SSE, AVX, AVX registers. I have written here that what is AVX 512 because I didn't know the  existence. The cool feature about the XSave is that it can save even the register that belongs  to Intel processor trace and the new Intel memory protection extension. And it is very cool  because using only one instruction we can save all the registers that belong to the processor  trace directly in a very, very fast manner without any problem. If you open the Intel  manual you will find that you use this instruction, it's a bit complex because there is an XSave  instruction that belongs only to user mode. And to set what to save you have to set an  extended control register in user mode using a new instruction that's called XSBV. But to be able  to save the mod specific register directly you have to use another instruction. You have to use  the XSave S that it means XSave supervisor and it's good to use it in kernel mode. Our driver  supports completely the XSave. And it was very funny because when I implemented this I have  found that the new Windows 10 context switch already implemented the support of this XSave but  only for user mode. Our original intention was to find a way to intercept or divert the key swap  context routine that is the kernel mode routine that Windows uses to perform the context switcher  from one thread to another thread. If we are able to intercept this we can save all the mod  specific register belongs to processor trace directly in an area and then restore later. In  that way we can implement the tracing by thread that is a completely software, it leaves in a  completely software point of view. But we found some problems because as you probably know you  can't touch any kernel mode in an official way using for example an hook, I don't know a code  diversion or whatever because otherwise Pashkar will blue screen off that your system. And we  found that this way is not feasible in a public system. We can use our debug system. In a debug  environment you can do that and Pashkar doesn't run and it's not a problem but it's not a  viable way in a production system. The second solution that we found is the usage of ETW.  If you check the documentation about ETW there is a way to intercept the context switcher but  it's we are still doing some research because the API are very complex and we are trying to  get if we are able to use in a legally way ETW to implement the trace by thread.  Another cool feature of the new relays is that the driver now fully supports the kernel  mode tracing. We have implemented 11 new kernel APIs that you can attach to your driver and you  can use it manually to decide what to trace, how to trace and to do whatever from a kernel  driver. But we were not happy with this because we would like even to implement the tracing from a  user mode application. Then we have created some high-octls that you can communicate with our  driver and it's able to do kernel tracing directly from user mode. And to do this we have  overcame a lot of security problems but now you can do that. From our user mode application you  can even trace the kernel code. I said I wrote here that in this way we are able to for example trace  the loading or unloading of a kernel module or even if maybe you are studying some high-octl in a root  kit or whatever you can even trace only the high-octl code because as you know the high-octl in kernel  environment are processed as synchronously when a user mode application asks.  Okay, some quick words about how to use the driver. As you can see the code is quite simple.  First of all you have to grab an handle of our device. It's quite easy. The device name is  named Windows Intel PT Dev. After that you have to fill a data structure named PT user rig that is  in our request and then you have to ask the high-octl manager to send the high-octl to our device.  As specified the user request. After all, after this the trace starts. You can decide where to stop the  trace using another high-octl. This is very important because if you close the application without doing  that it means that your processor is still tracing something and this could lead to a problem if you  try to upload our driver because the processor is not cleared. But the driver is able to detect this  and to overcome this. But it's a good practice to do that. For multi-processor code you can  spawn a user mode thread directly without any problem and then from the user mode thread you  have to call the IO manager to send the register PMI routine high-octl to our device. That's all.  Then you wait in an infinite loop. The only thing to take care is that there is a parameter in the  sleephacks function that is named a table or not. You have to insert through. That way every time the  CPU buffer gets filled the callback will be called without any problem.  Now it's time for a demo. I don't know how much. We have to be very fast. I have prepared a demo for you.  As you can see there is the code of a very simple application. It asks only some questions to the user.  Let's try to run it. No kernel trees. Target process is our simple application.  Andrea, you can see here.  Just a moment. How many CPU? At the beginning let's do one. Only one CPU.  He is asking you to increase the font size. You can mirror it.  While he is setting this up I will recap what he went over. When we presented it last year in June all we had shown is the raw  code. He is going to show you a tracing in different modes and visualizing it in IDA.  How many conferences have you attended? Let's say three. The application has ended saying you are on a good track.  Let's exit and try to open the executable using IDA. Here is the code.  Our goal is to trace the software. We have developed IDA plugin that does this for us.  Let's feed it with the text log. Wait a moment. An exception.  As you can see, this is exactly the code that it has run.  The exception was not voluntary.  We use a library from Intel called lib.ipt. It provides the decoding of the binary trace to a text file that we are parsing in the plugin for right now.  Later we will integrate the decoding within the plugin.  I would like to show you the same with the multi-processor environment. Let's see where is the part of my process.  How many processors? Four.  Now let's say eight conferences. The results are different. You can see in the summary each processor has its own buffer with different number of packets.  It is this dump. As you can see, there are different binary files and text files for each processor. Now we will increase the font size.  Can you see guys?  As you can see, this is CPU number one. It has all the packets, the taken not taken and the paging, paging generation, enable and disable.  Because the context switcher of Windows switch to different threads.  This is one CPU. But for example, if you are lucky, we are not lucky.  There is even some logs that are quite clear because it means that the Windows has executed these executables only for a small period of time.  This is not the case because luckily the context switcher runs for each different CPU.  But sometimes it happens that even the log for one processor is empty.  And this is our implementation.  Can you pull up one of those text files again?  Okay.  I just wanted to point out. So when you read these text files, you can see that for indirect branches and return addresses, we actually get the full 64-bit target address.  But if you look at the TNT packets, those are indicating whether or not a conditional branch was taken.  So they only store a single bit that determines whether or not you took the true or false branch.  So you have to recover that later on and disassemble in real time to recover what the target addresses were for those conditional branches.  We have time. Yeah.  I would like to show you an experimental demo that uses kernel tracing directly for user mode.  For this demo, I have chosen ACP driver, not for a specific reason, because it was randomly chosen.  And I found that the system calls this interface a lot of times.  Let's try to do even this, to take even those dumps.  The explorer has been blocked. Amazing.  Okay.  Let's say yes, ACPI.cs.  Let's say go, number of process of CPU, just one from this time.  Okay. It asks, I'm tracing, do something and then stop me.  I do, for example, some movement on my PC.  And then at a point of time, I will say stop.  In this time, we were quite lucky because, as you can see, the process on number one has taken some packets.  Let's see what are those packets.  Okay. As you can see, there is not so a lot of time, a lot of things registered here, but something has been taken.  Let's try to use our plugin to be able to trace what you can see.  This is the code of ACPI. Unfortunately, I have no symbol about this because it's an in-flight release of Windows 10.  But let's try.  Okay. The plugin has worked. You can see now because the driver entry has never been called, has been already called when you have switched on your system.  But if you see the log, there is a lot of time called this special function.  That could be an IOCTL or whatever.  Let's try to go.  Yes, this is IOCTL. You can see. Just a moment.  This should be blindly without knowing anything about the interface of the driver.  We can probably say that this is an IOCTL because the code is executed a lot of times because for the color, the color is darker.  It means that the CPU has executed this function a lot of times.  As you can see, there are traces of all the branches.  This is... Ah, come on. It's not taken.  And you can see there is the branch. And all the branches are traced.  And that is the demo about the kernel tracing from user mode.  From kernel mode, if you develop your driver, you are even being able to trace the driver entry or driver unload routine.  I can show that right now because first we don't have time and second, my computer is switched on in a relay's environment.  Because for doing that, of course, we can't use a signet driver.  But it's feasible. If you do that, if you write your kernel driver and you sign your kernel driver, you can do whatever you would like.  Okay, let's write it back today.  So now we'll switch.  Oh, okay, yes.  So yeah, to recap, the driver now supports kernel tracing and user mode tracing.  You can filter based upon the CR3, so a single whole process.  You can trace the entire kernel space or you can isolate ranges of contiguous IP.  And you get up to four different ranges.  So now I'm going to demonstrate this in a practical real world scenario inside of AFL.  So we have this fast tracing engine and you'll see some performance numbers that are real world here.  But in the manuals, they are targeting a 5 to 15% trace overhead for the entire system.  So per core, you should be able to trace both kernel mode operations and user mode operations for only 5 to 15%.  And I'll actually be able to show you that and how that works.  I'm sorry.  Yes, yes.  Oh, I'm sorry.  That will help.  That's one.  Yes.  Okay.  So how do we use this and apply it for vulnerability discovery?  So who here is familiar with American Fuzzy Lop?  Has used it?  We have people who do fuzzing in the crowd.  Okay, a good portion here.  So in the last few years, we've seen, well, an evolutionary jump in fuzzing technology.  Basically, we've gone from using dumb fuzzing or grammar-based fuzzing that wasn't able to determine whether or not the samples that were being generated were useful to applying a new technology or engineering an older technology into something that's performant enough to be used.  And we call that evolutionary fuzzing.  So we take the idea of dumb fuzzing mutation and we combine it with the ability to collect a feedback signal using code coverage.  And then we assess the fitness of that new randomly generated input against the entire lifetime of your fuzzing cycle.  And so basically what we can do is we can look at this code coverage information, determine if this newly generated input actually gets us to a different part of the code.  If it does, then we'll introduce that into our entire pool of samples and continue to mutate and fuzz those as we go.  So over time, we're refining our set of inputs and getting, we're building a corpus and each one of those inputs exercises a slightly different part of the code.  And in effect, what we've seen is the last three or four years since this has been available, this has highly optimized your compute time when it comes to doing dumb fuzzing.  And so the last couple of talks I've given have focused on this technology.  And so I encourage you to go look at previous slide decks which are on the recon website or my website at moflo.org.  But basically through researching this, I've resolved that the main things that we need to effectively deploy this technology is we need a fast tracing engine.  And of course, that was the inspiration to look into Intel processor trace because the promise of 15% overhead against closed source binary software  is pretty incredible compared to the technologies that we've had available before.  Previously, hardware tracing is not new to Intel.  Since the P4, there's been the ability to do hardware tracing.  There's mechanisms called branch trace store, BTS, which is works in a similar fashion, but was not designed in a way that was optimized.  It didn't write to physical RAM.  It polluted your cache and things like that.  So we saw a massive slowdown.  And then you have another option which is called last branch record, which is only 32 registers and modern processors.  And those only give you the last 32 branches.  So you'd have to interrupt every time every 32 branches to parse that and use that.  And or you'd have to write a driver that flushes that out to a different cache and do other things.  So while this isn't new, this is designed for the first time to be highly performance up until Intel processor trace.  It was actually faster to use software based tracers, things that would do dynamic binary instrumentation using Dynamo Rio or PIN or something like that.  So now we have this fast tracing engine.  That's great.  We need fast logging, which is something that we get out of the design of AFL.  AFL uses a bloom filter that allows you to quickly look up whether or not you've done the same code coverage.  So instead of parsing a text file or binary file, that's just a list of addresses of basic blocks consecutively.  We actually parse that real time and fill a bloom filter.  So you can just check to see if this 64k of RAM is identical to another 64k of RAM instead of doing a comparison of each address.  And then through some of my other research, there's been other attempts at evolutionary fuzzing starting about 2004 or 2005.  Jared DeMott did some of his PhD research on this, and he had something called the evolutionary fuzzing system.  But it was based upon basically the BTS record or debugger breakpoint.  So it was quite slow in its tracing.  And then it was also over engineered and trying to incorporate too much of the research that's in evolutionary, like, biology side of things.  So the key is to have this fast tracing engine efficient logging and to keep the analysis to the minimum.  So in 2013, Michael Zolesky, who's contributed a lot of great stuff to our industry,  produced the first performant open source evolutionary fuzzer called American Fuzzy Lot.  It uses a pretty comprehensive list of types of fuzzing strategies,  whether it's bit flipping, byte flipping, d-word flipping, and so on, crossovers, and various types of mutation.  It uses, originally it used block coverage via a plugin, a post process of the GCC compilation.  It would compile your code to assembler and then annotate the assembler to add callback hooks at every basic block entry point.  And then that's, through that modified source code, that's how you got your code coverage.  And then, as I mentioned, he takes those edge transitions, basically shifts and zoars them together,  and then increments offsets into this bloom filter or byte map that is basically able to track, you know, whether or not you've seen this edge before.  Now, you can't get out what those addresses were originally because it's simply an offset into this mapping,  but you can very quickly look up, have I been here before?  And that's all we care about as far as whether or not to keep that sample.  And then, of course, it was written on top of the POSIX API, so it wasn't Windows-compatible out the gate.  The benefits that it has, it tracks edge transitions and not just block entry.  It uses the bloom filter. It had a fork server built into it.  And then, basically, after your process has initialized, it waited until all your libraries are loaded and all the linking and everything is done.  And then once you get to the parser code, then it would fork.  And so, you skip all that initialization time, which is an optimization.  And then, very importantly, he introduced persistent mode fuzzing, which is an in-memory type of fuzzing where you're not exiting and recreating the process every time.  So, you're giving it a pointer to a function and the number of arguments to that function and saying, okay, once you exit this section of code, start over again and take our new inputs as inputs to this function.  And so, that also reduces the amount of code that you're tracing and executing down to the minimal points as an optimization.  And then, importantly, you can use this to build a corpus on open-source software.  It's very fast, and so you can use those as inputs into your pipeline on maybe slower or more heavyweight analysis on other types of fuzzing.  So, the way they do their tracing is every block gets a unique ID.  The edges are indexed in that map.  It creates that hash using the shift and zore, and then we increment the map.  So, this was great.  I was looking into this and how to optimize this and bring this to the Windows platform.  Obviously, we can't use source code instrumentation for the majority of the software that we're targeting, so we needed something that could do binary targeting.  And so, this seemed ideal versus the other options of using PIN or Dynamo Rio and so on.  And so, last summer, we decided to start looking into this.  Well, around that same time of my talk at Recon last year, Ivan Fratric from Google, I think he's Project Zero or Google Security,  released Win AFL, which was a port of Michael Zeweski's AFL to Windows using Dynamo Rio as a backend.  Are you guys familiar with PIN or Dynamo Rio?  They're basically loaders for your program, and as you visit each new basic block of code, it caches that and allows you to modify it in real time.  So, it's a kind of, Valgrind works the same way if you're familiar with that.  So, it was using that as a backend.  It's really cool. It works. It was like the first thing that you could just go download right now and start fuzzing Windows GDI in five minutes.  Beautiful.  The biggest thing that's allowed it to be a performance is that it uses this persistent mode where it doesn't exit the process.  So, these loaders like PIN or Dynamo Rio, they have to disassemble your program to instrument them.  So, he was able to, and I did some experimentation on trying to do forking in Windows previous talks, and it turns out it's just real pain.  So, using persistent mode, you get things like, you don't have to worry about ASLR because you're not exiting the process,  and you don't have to re-disassemble the process every time because you're using that code cache.  So, Win AFL turned out to be pretty well engineered.  And basically, you can tell it how many iterations to persist, like, you know, maybe do a thousand iterations,  and then go ahead and exit and restart the process so that we can, if we have any memory leaks or we're not quite cleaning up properly,  we can just handle that through delaying the restart.  Now, persistence is key because every time you load this in the DBI, it's going to disassemble it.  So, if we were to just do it every time, we'd get two executions a second on this GDI plus demo I'm going to show you.  If we persisted 100 times before we restarted 72 executions a second and so on, it reaches its ceiling somewhere around a thousand or so iterations.  So, we have now integrated our Intel PT driver into Win AFL as an alternative tracer engine.  And now, this brings some problems because the reason I had him show you the text version of that dump was that we don't have all the addresses in the log file.  So, we have to recover some of those along the way.  We don't have persistence mode working quite yet, unfortunately.  I have done some experimentation here. It's around the corner.  By the next time we present it to Hack in the Box, this will be available.  I'm building my tooling on top of Alex Inescu's great work from last year at Recon on the application verifier hooking system.  And so, basically, as it is, I'm using the IP filtering mode so you can specify up to four DLLs that you want to trace or, you know, four modules or any address ranges in your process that you want to trace and so on.  The current status now is that we do accurately decode the full trace, so doing disassembly online and using cache for the control filigree that you recover.  So, I first looked to see if we've already resolved what this upcoming conditional branch is.  And if so, then it's a quick index.  If not, I have to disassemble forward to determine the targets for the conditional branch and then store those in a structure.  The edge and source destination are recorded as expected, so we're not reduced to basic blocks.  We actually do get the edges.  And I'm currently just using create process, so we're doing this iteratively rather than forking or persistence mode yet.  So, in order to determine the performance of this tracing mechanism, I first made a dummy looping benchmark.  So, basically, it was just creating the process and waiting for it to exit.  So, we get our kind of maximum bounds on how many iterations we'll be able to execute with a sample.  In this case, I found that we could get 85 executions a second without doing any tracing.  We're just generating the fuzzer input and running it without anything like that.  We're not parsing the log file. We're not doing anything.  So, once we enable the tracing, that was reduced to 72 executions a second, which is right in that sweet zone of 15% overhead that Intel was promising for this particular sample.  So, parsing the log file was an additional 22% overhead.  So, now we're down to 55 executions a second.  And I'll demo what this all means for you here.  So, we'll compare it to the...  I'm going to show you... So, we're going to fuzz GDI plus.  This is an experiment that comes with the Win AFL out the gate.  You just pass it some image files and it uses Windows to render them without rendering to the screen.  And this is a live demo of that working.  So, currently this is using Dynamo Rio in persistence mode with the maximum number of iterations possible.  And we see that it's getting 127 executions a second.  The lighting is not good here, unfortunately, but hopefully you guys can see that a little bit.  I can increase the font just quickly here.  So, what we're looking at here is this number specifically. So, we get 126 executions a second using Dynamo Rio versus GDI.  So, let's see how our Windows PC driver performs in comparison.  Sorry, that's me decoding the log here. So, let's redirect that out to Noel.  Okay, so we're seeing with that overhead of the 15% for tracing and then the additional overhead of 22% for decoding.  This will creep up a little bit, but we're only getting about 40 something executions a second.  That's a little bit disappointing. We're kind of hoping to see a little bit faster.  Now, you have to keep in mind again that this is just iterative tracing. We're not doing in-memory fuzzing.  So, once we get to doing in-memory fuzzing, this number will increase significantly.  However, this is not the end of the story. I originally was doing all my testing against their setup with the GDI plus wrapper.  And as you can see in my command line here, this is tracing only the Windows Codex DLL and the GDI plus DLL in the process.  So, I made another demo in order to compare the performance. And this time we'll trace libpng using WinAFL.  And so, this is just what libpng statically compiled into a small harness that will just load a png file.  It will fuzz it and then it will fix up check sums and then it will parse it through libpng.  And we see that the performance becomes quite abysmal, actually. So, this is the Dynamo Rio. That's expected.  But we're seeing only 0.5 executions per second. So, this is quite slow, obviously. This is not where we want to be.  And this is because since this statically compiled all of the code involved in libpng, including the encryption or compression and things like that,  are included here, which causes some issues in the Dynamo Rio backend.  Now, here's the fun part. We have a constant overhead when we use Intel PT.  So, using Intel PT, we can see that we're back up in that. This will creep up to about 55-60 executions a second.  So, instead of being only a half an execution a second, this is 100 times faster than the Dynamo Rio backend.  And this is doing the full code coverage tracing, finding new paths, and so on.  So, with this engine, we've done 100 times performance increase depending on your target application.  And this is not specifically chosen. This was just randomly I had this fuzzer laying around and uses the demo.  So, yeah. So, that's my demos for this plugged into 1AFL.  Thank you.  And so, just closing remarks now. Number one, all the code that we've written is already open source.  They're going to be open source. We're on GitHub slash Intel PT. Real simple.  The driver is already there. I actually have a pull request from Andrea from last night.  So, the latest version is on GitHub. It'll be merged tonight.  The 1AFL needs a little cleanup. I was up till 3.30 last night making my final preparation.  So, that code will be up next week. And then we just have a few more things that we need to address.  But obviously, you know, we see that code coverage is being finally harnessed to make our fuzzing better.  We can load this information into IDA to do our analysis of crash or malware or whatever it might be.  And using the hardware supported tracing engine, we don't have any issues like you would have with other software-based instrumentation and hooking engines.  You know, once, you know, one of our future plans is to get this into a hypervisor so that we can trace the guests inside of a hypervisor.  And then your, you know, your malware tracing will be basically, you know, unobservable and you won't be able to disable it in that method.  There are capabilities of deploying Intel PT to trace things like SGX mode and SMM and BMM.  And those are kind of future areas of work. And then also, my end goal is, of course, to get this fully supported with persistent mode and everything as well.  One thing that we need to do as part of that is to finish the ETW-based thread context switch awareness,  because we need to separate these logs out into a per-thread instance. Otherwise, you have to use the timing information to determine where the synchronization is between the threads,  and that slows down the parsing a lot. So our goal is to get the logs individualized before you do your parsing,  and so you only have to parse the thread that you care about that's doing your, you know, your file or network I.O.  And did you have any further comments, solutions?  Just wanted to say that the new version that we are going to release already support the XAVE feature,  and if you download it, you can test it directly, and it's quite a cool feature in my opinion.  Okay, so you can get this code and you can reach us on Twitter, and thank you very much.  Thank you.  Hopefully, I think we might have a minute for questions. We're just on time.  Yes.  Okay, Trace, inside the filter machine.  So currently, there are, we, there are any hypervisors that currently expose this and virtualize this.  For example, in the other hardware tracing modes, the hypervisor has to virtualize the support for writing those MSRs.  This does exist, like, for example, in VMware using the BTS mechanism, but Intel PT is rather new,  so there aren't any hypervisors that are available yet. So we are either going to have to modify Zen or KBM,  and we've been actually even just today talking about perhaps being able to trace the entirety of the hypervisor  and then later on pull out only the user mode processor kernel threads that you're interested in.  So currently, no, but absolutely, we'll continue working on this until we get there.  Thanks.  Yeah, and I know that that will hopefully be applicable for cuckoo sandbox, so.  Were there any other questions?  Okay, feel free to grab us. Thank you very much for your attention.
recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_0_71.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72_119.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_120_179.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_180_227.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_228_299.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_300_37271.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_37272_37331.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_37332_37463.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_37464_37511.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_37512_37871.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_37872_38231.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_38232_39383.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_39384_39455.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_39456_39515.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_39516_40451.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_40452_41531.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_41532_42023.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_42024_42251.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_42252_42359.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_42360_42407.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_42408_42731.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_42732_43091.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_43092_43415.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_43416_44087.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_44088_44495.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_44496_44543.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_44544_44651.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_44652_44699.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_44700_46607.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_46608_46691.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_46692_46739.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_46740_47267.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_47268_47843.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_47844_47915.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_47916_47963.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_47964_49247.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_49248_51167.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_51168_51371.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_51372_51539.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_51540_53531.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_53532_62075.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_62076_71627.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_71628_71843.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_71844_71891.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_71892_71975.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_71976_72071.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72072_72239.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72240_72287.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72288_72347.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72348_72395.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72396_72455.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72456_72575.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72576_72671.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72672_72743.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72744_72911.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72912_72995.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_72996_73055.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_73056_73355.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_73356_73535.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_73536_73583.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_73584_73847.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_73848_74099.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74100_74255.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74256_74327.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74328_74603.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74604_74723.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74724_74771.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74772_74891.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_74892_75263.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_75264_75683.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_75684_75731.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_75732_76199.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_76200_76295.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_76296_76991.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_76992_80171.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_80172_80483.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_80484_80999.png recon2017-brx-03-andrea-allievi-richard-johnson-Harnessing-Intel-Processor-Trace-on-Windows-for-fuzz_81000_83052.png