Asura: A huge PCAP file analyzer for anomaly packets detection using massive multithreading
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Ando, Ruo
Recently, the inspection of huge traffic log is imposing a great burden on security analysts. Unfortunately, there have been few research efforts focusing on scalablility in analyzing very large PCAP file with reasonable computing resources. Asura is a portable and scalable PCAP file analyzer for detecting anomaly packets using massive multithreading. Asura's parallel packet dump inspection is based on task-based decomposition and therefore can handle massive threads for large PCAP file without considering tidy parameter selection in adopting data decomposition. Asura is designed to scale out in processing large PCAP file by taking as many threads as possible. Asura takes two steps. First, Asura extracts feature vector represented by associative containers of sourceIP, destIP pair. By doing this, the feature vector can be drastically small compared with the size of original PCAP files. In other words, Asura can reduce packet dump data into the size of unique sourceIP, destIP pairs (for example, in experiment, Asura's output which is reduced in first step is about 2% compared with the size of original libpcap files). Second, a parallel clustering algorithm is applied for the feature vector r. In second step, Asura adopts an enhanced Kmeans algorithm. Concretely, two functions of Kmeans which are (1)calculating distance and (2)relabeling points are improved for parallel processing. In experiment, in processing public PCAP datasets, Asura can identified 750 packets which are labeled as malicious from among 70 million (about 18GB) normal packets. In a nutshell, Asura successfully found 750 malicious packets in about 18GB packet dump. For Asura to inspect 70 million packets, it took reasonable computing time of around 350-450 minutes with 1000-5000 multithreading by running commodity workstation. Asura will be released under MIT license and available at author's GitHub site on the first day of DEF CON 26.
 We'll start with Rui Oando. Hello everyone. I'm really excited to talk here. So thank you for listening to me. And in this presentation, I'm going to talk about weird application using multi-threading for analyzing huge pick up file. This is the two which take full advantage of multi-core processor and cheap high-performance improvement. Actually, is there anyone in this room that thinks that wire shack is a little bit slow? The design goal of these two is kind of multi-threaded wire shack with automated deduction. So I guess multi-threading can be one of the new frontier for packet inspection. My name is Rui Oando. I'm working in a governmental organization. So I'm not weird. So my talk is divided four parts. At first, I would like to talk about the current catastrophic situation of traffic analysis. The funny thing here is that we have too many packets to be inspected. However, for the program to have the solution, we have more packets. This is a kind of really helpless situation. I'll tell you what later. And the second one is the main part. When you build a tool for analyzing huge pick up file using massive threads, you have some selections on how to convert code into a concurrent version. The selection of futures and containers and synchronization mechanism such as mutex, block free, and something like that. And the third part is a demo and experimental result. Simply stated, speed up is a ratio of parallel execution time to serial execution time. So I'll show you the comparison. Then let me conclude this talk. This slide shows the catastrophic situation as everyone of the audience already know. Internet traffic is increasing at exponential rate. However, there are two huge professionals. Sorry. Sorry. I'm so excited to wait for what's going to happen. Thank you. Huge traffic is a great burden on security researchers and analysts. But traffic explosion is not similar to the hacking or exploit. Because hacking and exploit is impulsive and will be finished within several minutes. But this is not the case of traffic explosion. Unfortunately, traffic explosion keeps exploding like an accident of nuclear power plant. So in my case, in my laboratory, I have 200 to 300 log files to be stored in the server to be inspected. During this 20 minutes presentation, 325 gigabyte file is stored to be inspected. This is really helpful session for me. So automation is really important for me and everyone of the audience. But for my experience, open source data mining too doesn't work in many cases. Because in the world of advertisement and marketing, commercial tool is not going to find people trying to hide his activities. And to make things more worse, open source data mining too simply ignore people trying to hide and assume in-line behavior is a part of everyone else. So I would like to emphasize that packet dump is the last resort. Pick a file is rare and hard to find source to be trusted. And machine learning, this one is 1 million versus 1 trillion. Machine learning has table property. I tell you what, if machine learning doesn't work on the dataset comprising 1 million training dataset, what is needed? What is needed is much more bigger set. This is unexpected because machine learning failed to fail in the dataset comprising 1 million training dataset. The intuitive conclusion is that it doesn't work at all. But according to this paper, all we need is much more bigger packets. So the situation is very curious. So, Astra has four features. At first, Astra should be run on commodity workstation and laptops. It can run with reasonable computing resources. Because GPU and crafting system such as Spark is still expensive and high-cost and sometimes spout. And more importantly, Astra uses project speed thread, which is really old programming style. When writing a program, choosing appropriate labor of abstraction is really important. Usually, hardly anyone misses the old program method except hackers. What we are copying here is the real world packet stream, which is huge, not nice, not organized in regular pattern, and unfortunately unpredictable. Flexibility is important, like you use assembly language for analyzing malware binaries. So, LawsRes and MPI expose the control of parallel computing at the lowest level. But at the lowest level, we don't have rivalries, containers, and schedulers. So you have to implement these utilities in full scratch by yourselves, like era of 1980s or 1990s. I guess this field can be one of the new frontier for packet inspection. As a result, Astra is compact but powerful. Actually, Astra has about 2,000 lines of code but can process about 75 million packets in 200 to 400 minutes. These two are intuitively simple. Astra takes two steps. Reduction using task decomposition and clustering using data decomposition. As you know, reduction takes collection data and reduces it to single-scarer value. Clustering is a task of grouping data in the same group, in such a way that data in the same group is more similar than those in other groups. An important thing here is reduction passes container to clustering. The container is a really important key in this two-stage processing. Container is a class template of C++ and the future selection are almost all anomaly detection of packet based on futures. There are many research efforts and futures could be many. But the important thing here is to find proper representation for reducing massive pickup fire. We use this representation in the middle of this slide, key value, and we use two structures. This is a little bit complicated. Please see the source code in detail. Let me talk more about containers. Continent is a really important point for mass-reading. You have three options. First one is STL. STL is an old, basic, and regular programming style. But STL is not concurrent friendly. It is a standard practice to wrap up a lock around STL to make them safe for concurrent access. Second one is Intel TVB. Intel TVB provides highly-concurrent container. But high-concurrent container is sometimes high-cost. It takes a long time. I guess this one is mainly for scientific computation. But everyone here is doing science computation. The data is not well organized and I'm pretty predictable. In this case, TVB is not suitable. And the third one is emerging technology of thrust. Thrust is a TC++ template library for GPU. By using thrust, you can write a code to perform a reduce scan and something like that, accelerated by GPU. But unfortunately, as far as I know, there's no plan to implement the hash table, the associated container in GPU. So I guess it will be time to become a full-packet inspection. I guess this could be a future work. And this slide is the main architecture of other. If you have a case, when the computation time on individual PQA file is variable and unpredictable, you'd be better sucked by task decomposition. Specifically, if you have a case, the amount of computation time will vary. Dynamic scheduler will be based. And here, as with dynamic scheduler of task decomposition, load balancing is important to take into consideration. You have to implement scheduler by yourself. So please see the upper side of this slide. This is a shared container, which is Q. Dynamic scheduler involves setting up the shared container, which holds data and allows threads to pull out tasks when the previous task is completed. So you should protect shared container so that the thread can be assigned correctly and the task should not be lost through some corruption of shared container. Okay, let me show about the experimental result. To put it simply, the speed up here is the ratio of parallel computing time to sequential computing time. And scalability. So why scalability? I think scalability is a measure of how much speed up the program gets as you add more and more core and threads. With, I guess, this kernel tuning is proper. I'm not exactly, but with proper tuning of Linux kernel, as well as 75 million packets with 500 threads in about 287 minutes. To tell the truth, there are some rooms to be improved because the size of shared container is not proper. Lock intention and context switching occurred too much. But I guess it's reasonable that the can process more than 7 million packets in several hours. And I would like to skip the attack detected in detail because some issues of public data set. So instead, let me show you a demo. Let me show you demo. First of all, binary is compiled according to the configuration of number of threads. And reduction step one, this demo is too fast. I can't, I'll reduction step two. Building binaries for clustering. Clustering has five to seven dimensions. And the data is truncated for this demo, which is too short. Do you know what's going on? Sorry, I don't know what's going on. Because machine learning is too fast. The machine learning relies on so huge data set and processing speed is so fast. So machine learning might solve problems. We cannot expect it to solve. In a way such that we don't fully understand. So demo is too fast. So I cannot talk about it. So let me conclude this talk. I have talked about a little bit weird application, which is called ASRA, using mass threading. So copying with real world pick up stream, which is huge, not nice and sometimes evil. Flexibility is needed. Just like you should use assembly language for binary. But using raw thread and MPI takes advantage of retrieved performance of magical processors. And the piece rate can expose the control of parallel computing at the lowest level. But unfortunately or not, we should implement everything, rivals, content, schedules. This is really exciting for me. And as a result, they offer maximum flexibility. So as a result, ASRA is compact, but powerful. ASRA has thousands of code and can process more than 70 million packets in 200 to 5 million. So 200 to 5 minutes. For future work, ASRA must be speeded up because there's room to improve, okay, configure the size of containers and applying TVB and GPU. And I really recommend this mass threading, applying mass threading for packet inspection. It's really exciting. This can be one of the new frontiers for packet inspection. So thank you, everyone. That's all. Thank you very much.
 We'll start with Rui Oando.  Hello everyone.  I'm really excited to talk here.  So thank you for listening to me.  And in this presentation, I'm going to talk about weird application using  multi-threading for analyzing huge pick up file.  This is the two which take full advantage of multi-core processor and  cheap high-performance improvement.  Actually, is there anyone in this room that thinks that  wire shack is a little bit slow?  The design goal of these two is kind of multi-threaded wire shack  with automated deduction.  So I guess multi-threading can be one of the new frontier for packet inspection.  My name is Rui Oando.  I'm working in a governmental organization.  So I'm not weird.  So my talk is divided four parts.  At first, I would like to talk about the current catastrophic situation of traffic analysis.  The funny thing here is that we have too many packets to be inspected.  However, for the program to have the solution, we have more packets.  This is a kind of really helpless situation.  I'll tell you what later.  And the second one is the main part.  When you build a tool for analyzing huge pick up file using massive threads,  you have some selections on how to convert code into a concurrent version.  The selection of futures and containers and synchronization mechanism such as  mutex, block free, and something like that.  And the third part is a demo and experimental result.  Simply stated, speed up is a ratio of parallel execution time to serial execution time.  So I'll show you the comparison.  Then let me conclude this talk.  This slide shows the catastrophic situation as everyone of the audience already know.  Internet traffic is increasing at exponential rate.  However, there are two huge professionals.  Sorry.  Sorry.  I'm so excited to wait for what's going to happen.  Thank you.  Huge traffic is a great burden on security researchers and analysts.  But traffic explosion is not similar to the hacking or exploit.  Because hacking and exploit is impulsive and will be finished within several minutes.  But this is not the case of traffic explosion.  Unfortunately, traffic explosion keeps exploding like an accident of nuclear power plant.  So in my case, in my laboratory, I have 200 to 300 log files to be stored in the server to be inspected.  During this 20 minutes presentation, 325 gigabyte file is stored to be inspected.  This is really helpful session for me.  So automation is really important for me and everyone of the audience.  But for my experience, open source data mining too doesn't work in many cases.  Because in the world of advertisement and marketing, commercial tool is not going to find people trying to hide his activities.  And to make things more worse, open source data mining too simply ignore people trying to hide and assume in-line behavior is a part of everyone else.  So I would like to emphasize that packet dump is the last resort.  Pick a file is rare and hard to find source to be trusted.  And machine learning, this one is 1 million versus 1 trillion.  Machine learning has table property.  I tell you what, if machine learning doesn't work on the dataset comprising 1 million training dataset, what is needed?  What is needed is much more bigger set.  This is unexpected because machine learning failed to fail in the dataset comprising 1 million training dataset.  The intuitive conclusion is that it doesn't work at all.  But according to this paper, all we need is much more bigger packets.  So the situation is very curious.  So, Astra has four features.  At first, Astra should be run on commodity workstation and laptops.  It can run with reasonable computing resources.  Because GPU and crafting system such as Spark is still expensive and high-cost and sometimes spout.  And more importantly, Astra uses project speed thread, which is really old programming style.  When writing a program, choosing appropriate labor of abstraction is really important.  Usually, hardly anyone misses the old program method except hackers.  What we are copying here is the real world packet stream, which is huge, not nice, not organized in regular pattern, and unfortunately unpredictable.  Flexibility is important, like you use assembly language for analyzing malware binaries.  So, LawsRes and MPI expose the control of parallel computing at the lowest level.  But at the lowest level, we don't have rivalries, containers, and schedulers.  So you have to implement these utilities in full scratch by yourselves, like era of 1980s or 1990s.  I guess this field can be one of the new frontier for packet inspection.  As a result, Astra is compact but powerful.  Actually, Astra has about 2,000 lines of code but can process about 75 million packets in 200 to 400 minutes.  These two are intuitively simple. Astra takes two steps. Reduction using task decomposition and clustering using data decomposition.  As you know, reduction takes collection data and reduces it to single-scarer value.  Clustering is a task of grouping data in the same group, in such a way that data in the same group is more similar than those in other groups.  An important thing here is reduction passes container to clustering. The container is a really important key in this two-stage processing.  Container is a class template of C++ and the future selection are almost all anomaly detection of packet based on futures.  There are many research efforts and futures could be many. But the important thing here is to find proper representation for reducing massive pickup fire.  We use this representation in the middle of this slide, key value, and we use two structures. This is a little bit complicated. Please see the source code in detail.  Let me talk more about containers. Continent is a really important point for mass-reading. You have three options.  First one is STL. STL is an old, basic, and regular programming style. But STL is not concurrent friendly.  It is a standard practice to wrap up a lock around STL to make them safe for concurrent access.  Second one is Intel TVB. Intel TVB provides highly-concurrent container. But high-concurrent container is sometimes high-cost. It takes a long time.  I guess this one is mainly for scientific computation. But everyone here is doing science computation. The data is not well organized and I'm pretty predictable.  In this case, TVB is not suitable. And the third one is emerging technology of thrust. Thrust is a TC++ template library for GPU.  By using thrust, you can write a code to perform a reduce scan and something like that, accelerated by GPU. But unfortunately, as far as I know, there's no plan to implement the hash table, the associated container in GPU.  So I guess it will be time to become a full-packet inspection. I guess this could be a future work.  And this slide is the main architecture of other. If you have a case, when the computation time on individual PQA file is variable and unpredictable, you'd be better sucked by task decomposition.  Specifically, if you have a case, the amount of computation time will vary. Dynamic scheduler will be based.  And here, as with dynamic scheduler of task decomposition, load balancing is important to take into consideration. You have to implement scheduler by yourself.  So please see the upper side of this slide. This is a shared container, which is Q. Dynamic scheduler involves setting up the shared container, which holds data and allows threads to pull out tasks when the previous task is completed.  So you should protect shared container so that the thread can be assigned correctly and the task should not be lost through some corruption of shared container.  Okay, let me show about the experimental result.  To put it simply, the speed up here is the ratio of parallel computing time to sequential computing time. And scalability. So why scalability?  I think scalability is a measure of how much speed up the program gets as you add more and more core and threads.  With, I guess, this kernel tuning is proper. I'm not exactly, but with proper tuning of Linux kernel, as well as 75 million packets with 500 threads in about 287 minutes.  To tell the truth, there are some rooms to be improved because the size of shared container is not proper.  Lock intention and context switching occurred too much. But I guess it's reasonable that the can process more than 7 million packets in several hours.  And I would like to skip the attack detected in detail because some issues of public data set. So instead, let me show you a demo.  Let me show you demo.  First of all, binary is compiled according to the configuration of number of threads. And reduction step one, this demo is too fast.  I can't, I'll reduction step two. Building binaries for clustering. Clustering has five to seven dimensions. And the data is truncated for this demo, which is too short.  Do you know what's going on? Sorry, I don't know what's going on.  Because machine learning is too fast. The machine learning relies on so huge data set and processing speed is so fast. So machine learning might solve problems.  We cannot expect it to solve. In a way such that we don't fully understand. So demo is too fast. So I cannot talk about it.  So let me conclude this talk. I have talked about a little bit weird application, which is called ASRA, using mass threading.  So copying with real world pick up stream, which is huge, not nice and sometimes evil. Flexibility is needed. Just like you should use assembly language for binary.  But using raw thread and MPI takes advantage of retrieved performance of magical processors. And the piece rate can expose the control of parallel computing at the lowest level.  But unfortunately or not, we should implement everything, rivals, content, schedules. This is really exciting for me.  And as a result, they offer maximum flexibility. So as a result, ASRA is compact, but powerful. ASRA has thousands of code and can process more than 70 million packets in 200 to 5 million.  So 200 to 5 minutes. For future work, ASRA must be speeded up because there's room to improve, okay, configure the size of containers and applying TVB and GPU.  And I really recommend this mass threading, applying mass threading for packet inspection. It's really exciting. This can be one of the new frontiers for packet inspection.  So thank you, everyone. That's all. Thank you very much.
DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_0_4511.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_4512_4619.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_4620_7079.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_7080_9587.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_9588_9815.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_9816_11207.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_11208_14243.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_14244_16451.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_16452_17939.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_17940_21611.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_21612_21683.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_21684_24431.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_24432_24479.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_24480_27011.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_27012_27431.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_27432_27479.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_27480_27611.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_27612_27851.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_27852_28403.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_28404_28595.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_28596_28751.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_28752_28799.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_28800_29015.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_29016_29087.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_29088_29411.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_29412_29891.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_29892_30443.png DEF_CON_26_-_Ruo_Ando_-_Asura_PCAP_File_Analyzer_for_Anomaly_Packets_Detection_fJM-VWDRV6M_30444_33649.png