AI VILLAGE - Stop and Step Away from the Data: Rapid Anomaly Detection via Ransom Note File Classification
CC Attribution 3.0 Unported:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Mager, Mark
The proliferation of ransomware has become a widespread problem culminating in numerous incidents that have affected users worldwide. Current ransomware detection approaches are limited in that they either take too long to determine if a process is truly malicious or tend to miss certain processes due to focusing solely on static analysis of executables. To address these shortcomings, we developed a machine learning model to classify forensic artifacts common to ransomware infections: ransom notes. Leveraging this model, we built a ransomware detection capability that is more efficient and effective than the status quo. I will highlight the limitations to current ransomware detection technologies and how that instigated our new approach, including our research design, data collection, high value features, and how we performed testing to ensure acceptable detection rates while being resilient to false positives. I will also be conducting a live demonstration with ransomware samples to demonstrate our technology’s effectiveness. Additionally, we will be releasing all related source code and our model to the public, which will enable users to generate and test their own models, as we hope to further push innovative research on effective ransomware detection capabilities.
 Next up we have Mark Maeger on stop and step away from the data rapid anomaly detection Do-do-do by a random note file classification we'd like to thank our sponsors endgame silence tinder and so-forth and Reminder if you could please sit down in the seats. We don't want to have a fire code violation with that enjoy the talk All right morning everybody So just getting things a little bit about me. I'm not a data scientist So take whatever I say up here on stage with a very big grain of salt and please feel to Ridicule and embarrass me after the talk about the things that I get wrong But anyways a little bit about me. I'm a senior malware researcher at endgame to put the reverse engineering in since it developed And past two and a half years pretty much since I've been in endgame. I've been doing ransomware protection research Just to get into the agenda. I'm gonna provide a brief overview of ransomware What their current detection methodology looks like? ransom notes and then I'm gonna delve into some exploratory research about the detection research I did And then discuss in depth the proof concept framework that I came up with and then wrap things up inclusion Hopefully I have a little bit of time for questions So if you don't know about ransomware basically it's a software. It's written to deny users access to data on their host the most typical approach is through in encrypting individual files on a file system and The file extensions are what's going to be targeted So think of high-value documents like PDFs text files or document Excel spreadsheets things of that nature And so there's two typical types of output from ransomware the encryption encrypted files that I was just alluding to and the actual So detection methodology can be broken down pretty simply into two areas right now You have static detections which are either going to be signature based Signature or your career six base or machine learning base the main benefit to This approach is that all data is preserved If the detection is successful, but the drawback is that you essentially have one chance to detect whether a binary is ransomware or you know malware or not and if we miss that then all data is going to be compromised on the host and for dynamic detections Basically the way those work is its process is going to be running in the background That's going to monitor for any sort of anomalous behavior on the host There can be a focus on detecting encrypted files in certain cases some approaches leverage canary files which are files that are written to disk In you know kind of spread out in different locations and if they're modified in particular ways and that can be a trigger for alert so the main benefit for dynamic detections is that you know, hypothetically as a process is Is executing there will always be an ongoing chance that will be detected So it's not just there's one initial chance to detect and then You know your host after that But you know so should still be able to detect it later on the drawback to dynamic approaches is that essentially your sacrificing a large amount of files in order to determine whether or not there's grants more executing on the host Maybe in certain cases it's easier to detect the anomalous behavior In some cases it might you know either be possible or take a very long time So how can we improve what the current saving art is right now? Probably the best approach is to combine The benefits of static and dynamic detections in the ideal case. Yes, you would detect everything with machine learning Immediately, nothing would ever execute on the host But that's not always case. You know, there is definitely false negatives So you need a robust Dynamic detection to serve as fallback. So leveraging a learned security approach is a probably the most recommended way to make sure you're covered for ransomware I'm Optimizing your machine learning models to specifically classify ransomware as opposed to just now or specifically That can prove very beneficial for this problem. And then to go back to Dynamic detections Perhaps there might be a way to Boost the time or reduce the amount of time that's required for detecting this behavior So getting into ransom notes a little bit of background on this Since I've been doing ransomware research for about two years or so. I've executed detonated, you know, probably almost thousands of files manually in a virtualized environment and kind of studied how the output typically looks and then you know as sort of a Side I was seeing you know ransom notes being rented this in multiple ways multiple directories for formats so What kind of got the gears turning for research that I'm presenting today is that I Started kind of seeing a pattern and how the ransom notes looked and so I wanted to explore that to see if there was a way That we could kind of classify those and see if there was something there that kind of unites all of them and makes them easy to detect using AI So just go back a little bit ransom notes files that are in juice less of the ransom payments They come across in multiple file types the most typical format is TXC files plain text files, but you also see ones that are in more formatted text formats such as HTML or TF And there's also images or even like GUI based like little dot-net programs Ransomers are going to be one of the first files that are written to disk and sometimes they're even written to every directory essentially The adversaries trying to be as noisy as possible in the hopes that they frustrate the users enough and get the point across that Their data has been you know totally compromised and they have to provide the ransom to getting their data back So we'll go back through here and like look at a few ransom notes to kind of get a general idea of what I'm talking about so this one's from Cryptolocker and they kind of lead off with Just saying your files are encrypted and then they talk about You don't have access to the description key so you can't recover your files They want you to email them and they're providing a specific time window for how long The ransom will be valid essentially and then they even get into talking about the AES encryption that they're practically using Going on to the next sample it pretty much starts out with the exact same way all your files have been encrypted and Then they say something similar about all your documents are encrypted Can't recover please pay us point oh one Bitcoin, you know to a specific wall wallet ID And then they also provide a specific time window and then they also provide the email address and Finally here's the actual image base ransom note and if you'll pay particular attention They were requesting 100 Bitcoin, which is approximately $750,000 right now So not exactly sure how successful they were with this ransomware campaign, but They were at least pretty pricey So as we saw from even just looking at three very disparate samples of ransom notes You can kind of see a template kind of form They typically laid off was saying something about your files have been encrypted Sometimes they provide a family name and then they'll Sometimes get into talking about the actual encryption that was implemented as part of the ransomware they get a point across that That files can't be recovered without a printer that they'll provide only if a Only if the ransom is provided and then you know potentially provide email address and then a time window for when their ransom essentially will be developed So You know as I previously said in my intro. I'm not data scientist so Exploratory research for me in this case was just developing a better familiarity with data science data science concepts and ideas and then Moving on from there. I need to collect You know big enough corpus of ransom notes in order to do some training and And on the flip side that we need to put together a nice base representative benign data set you know to go with the ransom notes and You know the part the overarching goal of the exploratory research is to determine if You know this approach can possibly work for a classification So tools I'm using here pretty much is use an account for everything which came bundled with Python 3 Jufra notebook and you'll also use a psychic learning space So delving into the data sets a little bit benign data, I just ended up using the 20 newsgroups data set which probably most of you are familiar That's a list of the actual 20 news groups that are part of that And then for the ransom notes, it's definitely a little tougher to Put together a large collection of ransom notes not every Ransom or family writes them out to disk so Going through and kind of manually doing the research and figuring out which families actually drop notes It can be a little tedious. So a lot of this involved manually detonating ransom or samples over a period of years collecting the ransom notes You know soaring them off and then you know kind of digging them out for this project but also You know searching through blog posts Twitter and a lot of things like that. You know, he's able to collect enough samples to Feel confident. I had something who's representative instance in general So the actual approach that I was taking for the exploratory research is We're just gonna go with unlabeled data. We're gonna take the 20 newsgroups data set and then we'll combine that with the ransom notes and so we will take a clustering approach using k-means and and set it to 21 clusters and we write that that is newsgroups data set and ransom notes and we're gonna hope that with the 21 clusters the way they kind of Settle down this You know, they'll be distinctly each of the news groups will be in their own cluster and then ransom notes will stick together in the In order to analyze the data a little closely we'll Take a look at the data using a count factorizer in a TFI So getting started just to do some very basic data prep Before tokenization, we're just going to strip out new line characters Converting lowercase strip out null bytes just things like that to just get the data starting to Make a little bit more sense and then when we do the actual tokenization We're going to limit it to alphanumeric characters only Rely to strip out any stop words that are in the default species stop words list and we'll do limitization so Very quick example here encryption would actually turn you So here's just a you know very quick overview of how the tokenization worked and for this example actually took you know very small board for a ransom note and Pass it in and you can see how it actually breaks down to to very You know very core set words there file and trips and Bitcoin is a payment. I mean that pretty much is Very descriptive of exactly what they're going for So now not sure how well you can see up there Breaking down the most common features that we're seeing in the 170 he ransom notes we see a lot of the same sort of words we see things describing You know files with data is being encrypted Bitcoin course encrypt the crypt You know things along that sort of nature Like even just looking through those words you might be able to you know construct what the purpose of a bit without having any sort of concepts and then when we break it out to Bigrams things make a little more sense because you're working with phrasing so It's not just files in a vacuum. It's files being encrypted files be encrypted Private keys that kind of addresses things along that picture But just to give you a little bit of an idea of what the data looks like Then we don't reply tfIDVF You know looks pretty similar to what we're getting from the counter vectorizer So yeah, just gives you another view of what the data looks like Now Not sure how well you can see this here, but essentially with it with the 21 clusters they They broke out like quite nicely for us actually and in cluster 3 despite the ransom notes only consisting of a hundred seventy three unique samples Versus the eleven thousand Messages that were in the 20 newsgroups data set The ransom notes all clustered together extremely well the that that cluster the I believe that's the top ten features that are that are in that cluster Matches extremely well with what we just seen in the previous two slides and that actually is a good test for for the data set because if you'll see the top entry for the for the newsgroup in the in the image to the right inside our crypt which is the the encryption newsgroup at the time But yeah, if you see cluster 6 it might be a little tough to tell but it kind of you can get an idea of how old it is because They're talking about clever chips which you know or pop like that was around the mid 90s or so, but but either way distinguishing between newsgroup discussions around encryption versus ransom notes that do discuss encryption at a more high level That's a good initial test of how strong the data correlates and So delving into how You know how the cluster actually worked We want to like kind of get under the hood and in passing some sample data so it you know took another ransom note and Passage the K means predictor and if you break out the results for that Using square root of the sum of the squares we can calculate the distance From the centroid centroid for you know each of the classes here. So In our case with that ransom note it did end up in cluster 3 which is what we're hoping and That worked out very well for us For a second example, we kind of use something that's more generically just talking about encryption, but not specifically a ransom note In this case it actually ended up being closer matched to cluster 4 which is actually entries from computer graphics. So What did we learn from doing our sport research? Well, as I mentioned before we have a small set of data But the ransom notes do cluster together very well The second sample demonstrated that there is nuance in how the data was clustered together And you know from all that we learned that it appears to that the data is going to be appropriate for classification So, you know, we can actually go forward with an actual proof concept So for a POSC framework, we have a few requirements first and foremost we need to obtain the file change events in real time We need to Take the file paths that are being created and pass them to a model that we develop From there We're going to read in the actual text data from the file paths that we created We're going to read in the file contents and then pass that along to the classifier to determine whether or not the data consists of a ransom note and then if it is a ransom note, we need a way to mitigate the process So to reduce the problem space for this We're gonna put up a few restrictions here. We're gonna stick to English only and Docs.exe files only as I mentioned, they're the most common ransom notes But that doesn't cover, you know, the entire world of ransom notes But yeah, for Matatex, it's gonna require parsing and images We have to use OCR to extract the data and it's probably, you know, it would require a little bit of cleaning up beyond that So at least for the for this research, I figured that was out of scope for what I started to accomplish And then we're gonna stick to files that are only less than 20 kilobytes The reasoning for this is ransom notes are generally pretty small You know, kind of going back to the template I was discussing earlier They're not really trying to get across too much. They're very utilitarian just saying hey files encrypted, please send us a ransom That's basically it. So Reducing the problem space there, keeping you less than 20 kilobytes, you know, helps out with performance as well So we can break down the the framework into a couple of Components and just two pretty distinct processes. So we'll have a file change event listener and That's going to read in the events and place it into a queue for a second process Which will be the text extraction and the actual classification of notes and then If we determine that there's a ransom note There will be a process mitigation handler That will operate So here's kind of a high level diagram of how a typical sort of infection scenario would play out with the framework on disk So you'd have ransomware executing they drop a ransom note to the Ruben C drive the event listener is going to be You know pulling for events at that time It'll see a follow creation event for the ransom note and then it'll pass along that file path to text extractor and classifier Which will read in the contents of the ransom note and then do the actual classification Hopefully return yes, and then that works up result in the ransomware process space So for the POC framework we wanted to build out a more representative data set So for the benign side we'll still stick with the 20 news groups But we'll take a smaller slice of it instead of the overarching 11,000 and then to supplement that will Leverage some of the Windows text files that I was kind of able to scoop up sort of typically talking about log files reading files Any sort of like installer logs, you know things along those lines And then for the ransom notes did my best to collect as many Many more ransom notes as I could Ended up finding a bunch on pay spin and a few other sources. So that was a great source You know but but still we're left with only 350 ransom notes compared to 11,000 benign messages so For the classification approach here. We want to address a data set in balance, which is very Quite large So we can use that as a base for the data set Quite large so we can use smoke to generate synthetic data for us and see you know Hopefully back and kind of bridge gap for us and make up for that You know pretty big in balance So the approach for the classifier here we're going to use do feature selection via TF IDF and Essentially what we have is a You know supervised learning problem. We're going to label the data this time As either benign or ransom note And then then yeah, we're breaking down All this into a binary classification problem Does the text consists of a ransom note or is it benign data and For us a naive base classifier Is simple and straightforward and that's the approach that we you know went went for immediately and delve into the results So very high overview the high-level overview of data processing pipeline here we start with our label data set and We pass that along to the pre-tokenization where we're stripping out characters converting lower case things along those lines We do the actual tokenization and then we You know get into Sanitizing the data a little bit by stripping out Stop words anything is not helping America and then do lemonization For we pass it along to the TF IDF vectorizer to vectorize the data We'll use smoke to balance out the data sets and then we'll do the actual training with our So for testing here we're splitting the the data into a 80-20 split 80% of data will be training while 20% will be user testing we use a train test split from scikit learn to to handle that and just get into a brief overview of the terminology involved that Probably extremely common And known to most you guys but the accuracy score they're going to be referring to here is the actual accuracy Classification score F1 score is going to be a average of the precision and recall confusion matrix just a great way to represent true true and false positive and negative grades and For our cross validation, we're going to use a Monte Carlo approach Essentially where we're running multiple runs You know through through building and See you know training you can test data sets each side. So so we're in this case. We're just We're testing out the models ability to predict the data You know testing to see how this how this approaches is going to be flexible and not try to So for a single one single test here, we actually ended up doing extremely well I accuracy over 99% F1 score 91 and you can see from the confusion matrix zero false negatives, which is great a few false positives, but Nothing too crazy So You know that's encouraging, but how does that scale? So we need to do some cross validation to determine if that was just going to outlier or if it's a So predictor things come and so we ran through cross validation and separate runs very training and test data and ended up with actually very similar results Accuracy was over 99 and that point score was overnight The confusion matrix looked about the same. So I think that you know Vivid case, you know the The approach problem we're taking Just some graph data here to provide you a better representation of what we're looking at As I said, not data scientists, but it's good So breaking things out into the other components in the framework With the event listener, we do need to monitor file change events We're looking at all processes that are active on a host and we need a way to map each event to a specific process And focus specifically on file creates in our case There's a few approaches that you can take to to getting this data Including you know using Python watcher, but as I said before the most important thing that we need here is we need a We need the type of file event We need the type of Sorry the the process That's responsible for the particular event and we need the file path. So Python watcher in this case It's based off of the redirecting changes API that I believe Windows API that I believe Doesn't actually return any source source process data. So in our case, that's that's not gonna help so alternative approaches that you could come through event logs or You can write your own file and you filter driver You know both of those, you know would work the developing your own driver That's gonna take way too much work that So For our case here what I ended up wanting to do was leverage something that's gonna be pre-built and See if I can kind of sift through event log data for that to get our Our follow-up in real time and so for my case I was able to leverage sysmon If you're not familiar with sysmon, it's you know just a tool that's You know used for monitoring event data on Windows and so there's a specific file create event actually of an ID 11 that's Perfect for our purposes. So we don't have to worry about distinguishing between different types of File change events. We only have one type of event For us here. It's just great Very simple configuration file that I came up with and then I posted that to the give repository for this project We're limiting things just to dot txt files as I previously mentioned And just trying to sift out other data. So we're not trying to crowd the event logs There's a registry key that you have to add in order to Properly allow the event log to be queried at an in real time. So that's there and so basically what we're trying to do in this case is we're going to pull the Event log and we're going to use the W my query language and we're essentially just gonna be pulling every 10 milliseconds In order to try to get updates of new file change events that are coming in in the real time so we need to limit the size of the result set that we're getting and We're parsing any results we get with something That's what the query essentially looks like You know pretty self-expand over there and for the actual Approach for process mitigation very straightforward here. All we need to do to determine is Is that process currently active with that ID and process name? It is active will suspend it and we need a user that there was this activity on their host and giving the choice to Terminate process or All right, so we're gonna try a live demo here, so let's see what happens Okay, so I have the framework here running in a single Python file. I have process monitors set up with couple filters We're looking at volcano dot exe volcano is a common ransomware family And I renamed the executable to volcano dot exe to make this more simple and we're going to use We're just going to look strictly at right file events for a process with that name So as you can see no events at the moment And here is my volcano dot exe execute that And we get our problem So it provides us with a specific file path To the text file that it determined to be a ransom note and it went ahead and suspended Volcano dot exe with that specific pit and if we go back to here In process Explorer we can verify that that process has been suspended And if we go through here we can kind of look through how You know the progression of the ransomware as it's writing files to disk it's like at 540 22 that was the first activity and See around 540 25 is when it was when the process is suspended and there was no for their fence So detection time within three seconds or so, but we have to like for our purposes actually since we're not keying off of any other files what we're only keying off of is the text files so we can go through the process monitor and See if you do the data and only look at text files to get a better idea of how long it took for us to Decalate tomorrow And let's see if we want a path that actually ends with.txt files. And so here what we can see is that there are multiple ransom notes that are in the disk as I mentioned before. Ransomers are typically pretty noisy without their distributing ransom notes on disks. So in this case we actually have 22 of the same file that's going to be written out. Well, actually I think we're only looking for.txt so that might even be less and I think some of those were actual files are being directly encrypted. But that gives you an idea of just how noisy the ransomware was. So we still have that process suspended and we can go ahead and click terminate. And as we'll see here the process is done. Okay, so getting into some more testing that I did at the framework, I was able to test against nine samples that were essentially holdouts because the ransom notes weren't part of our training or test data set. So we were able to detect those nine specific samples from those families and as well as three samples I tested that already had notes in the training set from earlier. So in order to get some, you know, a better idea of how successful this approach is, I wanted to test against, you know, what's currently out there. And so for our cases, for our case for this, we just wanted to do testing against anything that was free or trial based. You know, I did one shot and I wanted you to test it here. And we want to break it out to two different tests. It doesn't detect the sample and if it does, we run it side by side with a classifier framework where you just came up with and we just want to give a rough estimate of what the detection speed looks like. There's definitely potential complicating factors in that for that particular test case because things like driver altitude can definitely affect how it leads to products are running side by side. But you know, just a way to get a rough idea of how the performance compares to actual stuff that's currently available for downward. And so the testing issue went extremely well. I kept things very generic. I don't want to call out any specific vendors or anything like that. But in our case, there was one specific product that did perform very well and was typically faster in detection than the classifier framework I developed. And that being said, the detections where the product feed did perform better, the framework was still close in performance and it lacked behind open by half a second or so. But surprisingly, there were two products that were very easily outperformed by the classifier framework. And I mean, if you even look at the A1 and A2, while it detects pretty much all of the time for the U12 samples that we saw, it only, I think, is unable to run a test for one of the samples. But it was outperformed nearly all the time by our framework. And that's actually pretty amazing considering the ad hoc approach that we took with sifting through event logs for data and then doing all this classification at runtime and doing it all in Python, essentially going head-to-head with something that's running native code and probably leveraging MiniFolk for driver 2 to obtain their input. So definitely validating the approach that we took. So that being said, those results are great, but there are definitely limitations with this approach. There are plenty of random samples that don't drop.txt to ransom notes. Some don't even drop notes at all. Some try to convey their ransom message just in a custom file extension that they apply to every single file. Some samples drop ransom notes much later in the game after all the files have been encrypted. And then there's also samples that leverage some sort of persistence and typically respond even if you suspend the process, terminate it, whatever. Yeah, we might be able to detect it, but it's going to just keep going over and over again. And of course, there are a ransomware that actually take different approaches to denying these as access to their data, NBR modifications, anything for raw disk, or just simple 3-mockers. And of course, as we mentioned, going in, we're sticking only to English. So future work, we'd like to improve the datasets. Definitely more ransom notes, less synthetic data would be nice as well. As well as new ransom notes as new ransomware families come up when they're made aware of. And we'd also like to build out a more representative 9-text dataset, more log files, more installer files, things that nature. If we could port our code base to a lower level language, that would be great and would lead to very significant performance improvements. We'd be able to improve our detection time as well. It would be nice to support other file types, like for the format in text as I mentioned before, as well as images and doing OCR to extract text. Expand language support would be nice as well, as well as experimenting with the actual approach to classification. So to wrap things up, Cluster8on gave us a good idea of the data being suitable for classification. And we saw that ransom notes do share enough features for our solution to be viable. And going into this, we do realize this isn't going to catch all ransomware, but it could be a very integral piece of a layer detection approach with static classifiers as well. So yeah, the proof of concept did work, but there are definitely many improvements to the game. All right. Thank you very much. Thank you.
 Next up we have Mark Maeger on stop and step away from the data rapid anomaly detection  Do-do-do by a random note file classification  we'd like to thank our sponsors endgame silence tinder and so-forth and  Reminder if you could please sit down in the seats. We don't want to have a fire code violation with that enjoy the talk  All right morning everybody  So just getting things a little bit about me. I'm not a data scientist  So take whatever I say up here on stage with a very big grain of salt and please feel to  Ridicule and embarrass me after the talk about the things that I get wrong  But anyways a little bit about me. I'm a senior malware researcher at endgame to put the reverse engineering in since it developed  And past two and a half years pretty much since I've been in endgame. I've been doing ransomware protection research  Just to get into the agenda. I'm gonna provide a brief overview of ransomware  What their current detection methodology looks like?  ransom notes and then I'm gonna delve into some exploratory research about the detection  research I did  And then discuss in depth the proof concept framework that I came up with and then wrap things up inclusion  Hopefully I have a little bit of time for questions  So if you don't know about ransomware basically it's a software. It's written to  deny users access to data on their host  the most typical approach is through in  encrypting individual files on a file system and  The file extensions are what's going to be targeted  So think of high-value documents like PDFs text files or document Excel spreadsheets  things of that nature  And so there's two typical types of output from ransomware the encryption encrypted files that I was just alluding to  and the actual  So detection methodology can be broken down pretty simply into two areas right now  You have static detections which are either going to be signature based  Signature or your career six base or machine learning base  the main benefit to  This approach is that all data is preserved  If the detection is successful, but the drawback is that you essentially have one chance to detect  whether a binary is  ransomware or you know malware or not and if we miss that then all data is going to be compromised on the host  and for dynamic detections  Basically the way those work is its process is going to be running in the background  That's going to monitor for any sort of anomalous behavior on the host  There can be a focus on detecting encrypted files in certain cases  some approaches leverage canary files which are files that are written to disk  In you know kind of spread out in different locations and if they're modified in particular ways and that can be a  trigger for alert  so the main benefit for  dynamic detections is that  you know, hypothetically as a process is  Is executing there will always be an ongoing chance that will be detected  So it's not just there's one initial chance to detect and then  You know your host after that  But you know so should still be able to detect it later on the drawback  to  dynamic approaches is that  essentially your sacrificing a  large amount of files in order to determine whether or not there's grants more executing on the host  Maybe in certain cases it's easier to detect the anomalous behavior  In some cases it might you know either be possible or take a very long time  So how can we improve what the current saving art is right now?  Probably the best approach is to combine  The benefits of static and dynamic detections in the ideal case. Yes, you would detect everything with machine learning  Immediately, nothing would ever execute on the host  But that's not always case. You know, there is definitely false negatives  So you need a robust  Dynamic detection to serve as fallback. So leveraging a learned security approach is a  probably the most recommended way to make sure you're covered for ransomware  I'm  Optimizing your machine learning models to specifically classify ransomware as opposed to just now or specifically  That can prove very beneficial for this problem. And then to go back to  Dynamic detections  Perhaps there might be a way to  Boost the time or reduce the amount of time that's required for detecting this behavior  So getting into ransom notes  a little bit of background on this  Since I've been doing ransomware research for about two years or so. I've executed  detonated, you know, probably almost thousands of files manually  in a virtualized environment and kind of studied how the output typically looks and then  you know as sort of a  Side I was seeing you know ransom notes being rented this in multiple ways multiple directories  for formats so  What kind of got the gears turning for research that I'm presenting today is that I  Started kind of seeing a pattern and how the ransom notes looked and so I wanted to explore that to see if there was a way  That we could kind of classify those and see if there was something there that kind of unites all of them and makes them easy to detect  using AI  So just go back a little bit ransom notes files that are in juice less of the ransom payments  They come across in multiple file types the most typical format is  TXC files plain text files, but you also see ones that are in more formatted text formats such as  HTML or TF  And there's also images or even like GUI based like little dot-net programs  Ransomers are going to be one of the first files that are written to disk and sometimes they're even written to every directory essentially  The adversaries trying to be as noisy as possible in the hopes that they frustrate the users enough and get the point across that  Their data has been you know totally compromised and they have to provide the ransom to getting their data back  So we'll go back through here and like look at a few ransom notes to kind of get a general idea of what I'm talking about  so this one's from Cryptolocker and they kind of lead off with  Just saying your files are encrypted and then they talk about  You don't have access to the description key so you can't recover your files  They want you to email them and they're providing a specific time window for how long  The ransom will be valid essentially and then they even get into talking about the AES encryption that they're practically using  Going on to the next sample it pretty much starts out with the exact same way all your files have been encrypted and  Then they say something similar about all your documents are encrypted  Can't recover please pay us point oh one Bitcoin, you know to a specific wall wallet ID  And then they also provide a specific time window  and then they also provide the email address and  Finally here's the actual image base ransom note and if you'll pay particular attention  They were requesting 100 Bitcoin, which is approximately $750,000 right now  So not exactly sure how successful they were with this ransomware campaign, but  They were at least pretty pricey  So as we saw from even just looking at three very disparate samples of ransom notes  You can kind of see a template kind of form  They typically laid off was saying something about your files have been encrypted  Sometimes they provide a family name and then they'll  Sometimes get into talking about the actual  encryption that was implemented as part of the ransomware  they get a point across that  That files can't be recovered without a printer that they'll provide only if a  Only if the ransom is provided and then you know potentially provide email address and then a time window for when their  ransom essentially will be developed  So  You know as I previously said in my intro. I'm not data scientist so  Exploratory research for me in this case was just developing a better familiarity with data science data science concepts and  ideas and then  Moving on from there. I need to collect  You know big enough corpus of ransom notes  in order to do some training and  And on the flip side that we need to put together a nice base representative benign data set  you know to go with the ransom notes and  You know the part the overarching goal of the exploratory research is to determine if  You know this approach can possibly work for a classification  So tools I'm using here pretty much is use an account for everything which came bundled with Python 3  Jufra notebook and you'll also use a psychic learning space  So delving into the data sets a little bit benign data, I just ended up using the 20 newsgroups data set which probably most of you are familiar  That's a list of the actual 20 news groups that are part of that  And then for the ransom notes, it's definitely a little tougher to  Put together a large collection of ransom notes  not every  Ransom or family writes them out  to disk so  Going through and kind of manually doing the research and figuring out which families actually drop notes  It can be a little tedious. So a lot of this involved manually detonating ransom or samples over a period of years  collecting the ransom notes  You know soaring them off and then you know kind of digging them out for this project  but also  You know searching through blog posts  Twitter and a lot of things like that. You know, he's able to collect enough  samples to  Feel confident. I had something who's representative  instance in general  So the actual approach that I was taking for the exploratory research is  We're just gonna go with unlabeled data. We're gonna take the 20 newsgroups data set and then we'll combine that with the ransom notes  and so we will take a clustering approach using k-means and  and  set it to 21 clusters and  we write that that is  newsgroups data set and ransom notes and we're gonna hope that  with the 21 clusters the way they kind of  Settle down this  You know, they'll be distinctly each of the news groups will be in their own cluster and then ransom notes will stick together in the  In order to analyze the data a little closely we'll  Take a look at the data using a count factorizer in a TFI  So getting started just to do some very basic data prep  Before tokenization, we're just going to strip out new line characters  Converting lowercase strip out null bytes just things like that to just get the data starting to  Make a little bit more sense and then when we do the actual tokenization  We're going to limit it to alphanumeric characters only  Rely to strip out any stop words that are in the default  species stop words list and we'll do limitization so  Very quick example here encryption would actually turn you  So here's just a you know very quick overview of how  the tokenization worked and  for this example actually took you know very small board for a ransom note and  Pass it in and you can see how it actually breaks down  to to very  You know very core set words there file and trips and Bitcoin is a payment. I mean that pretty much is  Very descriptive of exactly what they're going for  So now not sure how well you can see up there  Breaking down the most common features that we're seeing in the  170 he ransom notes we see a lot of the same sort of words we see  things describing  You know files with data is being encrypted  Bitcoin course encrypt the crypt  You know things along that sort of nature  Like even just looking through those words you might be able to you know construct what the purpose of  a bit without having any sort of concepts  and then when we break it out to  Bigrams things make a little more sense because you're working with phrasing so  It's not just files in a vacuum. It's files being encrypted files be encrypted  Private keys that kind of addresses things along that picture  But just to give you a little bit of an idea of what the data looks like  Then we don't reply tfIDVF  You know looks pretty similar to what we're getting from the counter vectorizer  So yeah, just gives you another view of what the data looks like  Now  Not sure how well you can see this here, but essentially with it with the 21 clusters they  They broke out like quite nicely for us actually and in cluster 3  despite the ransom notes only consisting of a hundred seventy three unique samples  Versus the eleven thousand  Messages that were in the 20 newsgroups data set  The ransom notes all clustered together extremely well  the that that cluster the I believe that's the top ten features that are that are in that cluster  Matches extremely well with what we just seen in the previous two slides  and that actually is a  good  test for for the data set because if you'll see the top  entry for the for the newsgroup in the in the image to the right inside our crypt which is the  the  encryption newsgroup at the time  But yeah, if you see cluster 6 it might be a little tough to tell  but it kind of you can get an idea of how old it is because  They're talking about clever chips which you know or pop like that was around the mid 90s or so, but but either way  distinguishing between newsgroup discussions around encryption versus  ransom notes that do discuss encryption at a more high level  That's a good initial test of how strong the data correlates  and  So delving into how  You know how the cluster actually worked  We want to like kind of get under the hood and in passing some sample data  so it you know took another ransom note and  Passage the K means predictor and if you break out the results for that  Using square root of the sum of the squares we can calculate the distance  From the centroid centroid for you know each of the classes here. So  In our case with that ransom note it did end up in cluster 3 which is what we're hoping and  That worked out very well for us  For a second example, we kind of use something that's more generically just talking about  encryption, but not specifically a ransom note  In this case it actually ended up  being  closer matched to  cluster 4 which is actually entries from computer graphics. So  What did we learn from doing our sport research? Well, as I mentioned before we have a small set of data  But the ransom notes do cluster together very well  The second sample demonstrated that there is nuance in how the data was clustered together  And you know from all that we learned that it appears to that the data is going to be appropriate for classification  So, you know, we can actually go forward with an actual proof concept  So for a POSC framework, we have a few requirements first and foremost we need to obtain the file change events in real time  We need to  Take the file paths that are being created and pass them to a model that we develop From there  We're going to read in the actual text data from the file paths that we created  We're going to read in the file contents and then pass that along to the classifier to determine whether or not  the data consists of a ransom note and then if it is a ransom note, we need a way to mitigate the process  So to reduce the problem space for this  We're gonna put up a few restrictions here. We're gonna stick to English only and  Docs.exe files only as I mentioned, they're the most common ransom notes  But that doesn't cover, you know, the entire world of ransom notes  But yeah, for Matatex, it's gonna require parsing and images  We have to use OCR to extract the data and it's probably, you know, it would require a little bit of cleaning up beyond that  So at least for the for this research, I figured that was out of scope for what I started to accomplish  And then we're gonna stick to files that are only less than 20 kilobytes  The reasoning for this is ransom notes are generally pretty small  You know, kind of going back to the template I was discussing earlier  They're not really trying to get across too much. They're very utilitarian just saying hey files encrypted, please send us a ransom  That's basically it. So  Reducing the problem space there, keeping you less than 20 kilobytes, you know, helps out with performance as well  So we can break down the the framework into a couple of  Components and just two pretty distinct processes. So we'll have a file change event listener and  That's going to read in the events and place it into a queue for a second process  Which will be the text extraction and the actual classification of notes and then  If we determine that there's a ransom note  There will be a process mitigation handler  That will operate  So here's kind of a high level diagram of how a typical sort of infection scenario would play out with the framework on disk  So you'd have ransomware executing they drop a ransom note to the Ruben C drive the event listener is going to be  You know pulling for events at that time  It'll see a follow creation event for the ransom note and then it'll pass along that file path to text extractor and classifier  Which will read in the contents of the ransom note and then do the actual classification  Hopefully return yes, and then that works up result in the ransomware process space  So for the POC framework we wanted to build out a more representative data set  So for the benign side we'll still stick with the 20 news groups  But we'll take a smaller slice of it instead of the  overarching 11,000 and then to supplement that will  Leverage some of the Windows text files that I was kind of able to scoop up sort of typically talking about log files reading files  Any sort of like installer logs, you know things along those lines  And then for the ransom notes did my best to collect as many  Many more ransom notes as I could  Ended up finding a bunch on pay spin and a few other sources. So that was a great source  You know but but still we're left with only 350 ransom notes compared to 11,000  benign messages so  For the classification approach here. We want to address a data set in balance, which is very  Quite large  So we can use that as a base for the data set  Quite large so we can use smoke to generate synthetic data for us and see you know  Hopefully back and kind of bridge gap for us and make up for that  You know pretty big in balance  So the approach for the classifier here we're going to use do feature selection via TF IDF and  Essentially what we have is a  You know supervised learning problem. We're going to label the data this time  As either benign or ransom note  And then then yeah, we're breaking down  All this into a binary classification problem  Does the text consists of a ransom note or is it benign data and  For us a naive base classifier  Is simple and straightforward and that's the approach that we  you know went went for immediately and  delve into the  results  So very high overview the high-level overview of data processing pipeline here we start with our  label data set and  We pass that along to the pre-tokenization where we're stripping out characters converting lower case things along those lines  We do the actual tokenization and then we  You know get into  Sanitizing the data a little bit by stripping out  Stop words anything is not helping America and then do lemonization  For we pass it along to the TF IDF vectorizer to vectorize the data  We'll use smoke to balance out the data sets and then we'll do the actual training with our  So for testing here we're splitting the  the data into a  80-20 split 80% of data will be training while 20% will be  user testing we use a train test split from scikit learn to  to handle that and just get into a brief overview of the terminology involved that  Probably extremely common  And known to most you guys but the accuracy score they're going to be referring to here is the actual accuracy  Classification score F1 score is going to be a average of the precision and recall  confusion matrix just  a great way to represent  true true and false positive and negative grades and  For our cross validation, we're going to use a Monte Carlo approach  Essentially where we're running multiple runs  You know through  through building and  See you know training you can test data sets each side. So so we're in this case. We're just  We're testing out the models ability to predict the data  You know testing to see how this how this approaches is going to be flexible and not try to  So for a single one single test here, we actually ended up doing extremely well  I accuracy over 99% F1 score 91 and you can see from the confusion matrix zero false negatives, which is great  a few false positives, but  Nothing too crazy  So  You know that's encouraging, but how does that scale?  So we need to do some cross validation to determine if that was just going to outlier  or if it's a  So predictor things come and so we ran through cross validation and separate runs  very training and test data and ended up with actually very similar results  Accuracy was over 99 and that point score was overnight  The confusion matrix looked about the same. So I think that you know  Vivid case, you know the  The approach problem we're taking  Just some graph data here to provide you a better representation of what we're looking at  As I said, not data scientists, but it's good  So breaking things out into the other components in the framework  With the event listener, we do need to monitor file change events  We're looking at all processes that are active on a host and we need a way to map each event to a specific process  And focus specifically on file creates in our case  There's a few approaches that you can take to to getting this data  Including you know using Python watcher, but as I said before the most important thing that we need here is we need a  We need the type of file event  We need the type of  Sorry the the process  That's responsible for the particular event and we need the file path. So Python watcher in this case  It's based off of the redirecting changes API that I believe Windows API that I believe  Doesn't actually return any source source process data. So in our case, that's that's not gonna help so  alternative approaches that you could come through event logs or  You can write your own file and you filter driver  You know both of those, you know would work the developing your own driver  That's gonna take way too much work that  So  For our case here what I ended up wanting to do was leverage something that's gonna be pre-built and  See if I can kind of sift through event log data for that to get our  Our follow-up in real time and so for my case I was able to leverage sysmon  If you're not familiar with sysmon, it's you know just a tool that's  You know used for monitoring  event data on  Windows and so there's a specific file create event actually of an ID 11 that's  Perfect for our purposes. So we don't have to worry about  distinguishing between different types of  File change events. We only have one type of event  For us here. It's just great  Very simple configuration file that I came up with and then I posted that to the give repository for this project  We're limiting things just to dot txt files as I previously mentioned  And just trying to sift out other data. So we're not trying to  crowd the event logs  There's a registry key that you have to add in order to  Properly allow the event log to be queried at an in real time. So that's there  and  so basically what we're trying to do in this case is we're going to pull the  Event log and we're going to use the W my query language and we're essentially just gonna be pulling every 10 milliseconds  In order to try to get updates of new file change events that are coming in in the real time  so we need to limit the size of the  result set that we're getting and  We're parsing any results we get with something  That's what the query essentially looks like  You know pretty self-expand over there and for the actual  Approach for process mitigation very straightforward here. All we need to do to determine is  Is that process currently active with that ID and process name?  It is active will suspend it and we need a user that there was this activity on their host and  giving the choice to  Terminate process or  All right, so we're gonna try a live demo here, so let's see what happens  Okay, so I have the framework here running in a single Python file. I have process monitors set up with  couple filters  We're looking at volcano dot exe volcano is a common ransomware family  And I renamed the executable to volcano dot exe to make this more simple and we're going to use  We're just going to look strictly at right file events for a process with that name  So as you can see no events at the moment  And here is my volcano dot exe  execute that  And we get our problem  So it provides us with a specific file path  To the text file that it determined to be a ransom note and it went ahead and suspended  Volcano dot exe with that specific pit and if we go back to here  In process Explorer we can verify that that process has been suspended  And if we go through here we can kind of look through how  You know the progression of the ransomware as it's writing files to disk  it's like at 540 22 that was the first activity and  See around 540 25 is when it was when the process is suspended and there was no for their fence  So detection time within three seconds or so, but we have to like for our purposes actually since we're not keying off of any other  files what we're only keying off of is the text files so we can go through the process monitor and  See if you do the data and only look at text files to get a better idea of how long it took for us to  Decalate tomorrow  And let's see if we want a path that actually ends with.txt files.  And so here what we can see is that there are multiple ransom notes that are in the  disk as I mentioned before.  Ransomers are typically pretty noisy without their distributing ransom notes on disks.  So in this case we actually have 22 of the same file that's going to be written out.  Well, actually I think we're only looking for.txt so that might even be less and I think  some of those were actual files are being directly encrypted.  But that gives you an idea of just how noisy the ransomware was.  So we still have that process suspended and we can go ahead and click terminate.  And as we'll see here the process is done.  Okay, so getting into some more testing that I did at the framework, I was able to test  against nine samples that were essentially holdouts because the ransom notes weren't  part of our training or test data set.  So we were able to detect those nine specific samples from those families and as well as  three samples I tested that already had notes in the training set from earlier.  So in order to get some, you know, a better idea of how successful this approach is, I  wanted to test against, you know, what's currently out there.  And so for our cases, for our case for this, we just wanted to do testing against anything  that was free or trial based.  You know, I did one shot and I wanted you to test it here.  And we want to break it out to two different tests.  It doesn't detect the sample and if it does, we run it side by side with a classifier framework  where you just came up with and we just want to give a rough estimate of what the detection  speed looks like.  There's definitely potential complicating factors in that for that particular test case  because things like driver altitude can definitely affect how it leads to products are running  side by side.  But you know, just a way to get a rough idea of how the performance compares to actual  stuff that's currently available for downward.  And so the testing issue went extremely well.  I kept things very generic.  I don't want to call out any specific vendors or anything like that.  But in our case, there was one specific product that did perform very well and was typically  faster in detection than the classifier framework I developed.  And that being said, the detections where the product feed did perform better, the framework  was still close in performance and it lacked behind open by half a second or so.  But surprisingly, there were two products that were very easily outperformed by the  classifier framework.  And I mean, if you even look at the A1 and A2, while it detects pretty much all of the  time for the U12 samples that we saw, it only, I think, is unable to run a test for one of  the samples.  But it was outperformed nearly all the time by our framework.  And that's actually pretty amazing considering the ad hoc approach that we took with sifting  through event logs for data and then doing all this classification at runtime and doing  it all in Python, essentially going head-to-head with something that's running native code  and probably leveraging MiniFolk for driver 2 to obtain their input.  So definitely validating the approach that we took.  So that being said, those results are great, but there are definitely limitations with  this approach.  There are plenty of random samples that don't drop.txt to ransom notes.  Some don't even drop notes at all.  Some try to convey their ransom message just in a custom file extension that they apply  to every single file.  Some samples drop ransom notes much later in the game after all the files have been encrypted.  And then there's also samples that leverage some sort of persistence and typically respond  even if you suspend the process, terminate it, whatever.  Yeah, we might be able to detect it, but it's going to just keep going over and over again.  And of course, there are a ransomware that actually take different approaches to denying  these as access to their data, NBR modifications, anything for raw disk, or just simple 3-mockers.  And of course, as we mentioned, going in, we're sticking only to English.  So future work, we'd like to improve the datasets.  Definitely more ransom notes, less synthetic data would be nice as well.  As well as new ransom notes as new ransomware families come up when they're made aware of.  And we'd also like to build out a more representative 9-text dataset, more log files, more installer  files, things that nature.  If we could port our code base to a lower level language, that would be great and would  lead to very significant performance improvements.  We'd be able to improve our detection time as well.  It would be nice to support other file types, like for the format in text as I mentioned  before, as well as images and doing OCR to extract text.  Expand language support would be nice as well, as well as experimenting with the actual approach  to classification.  So to wrap things up, Cluster8on gave us a good idea of the data being suitable for classification.  And we saw that ransom notes do share enough features for our solution to be viable.  And going into this, we do realize this isn't going to catch all ransomware, but it could  be a very integral piece of a layer detection approach with static classifiers as well.  So yeah, the proof of concept did work, but there are definitely many improvements to  the game.  All right.  Thank you very much.  Thank you.
DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_0_95.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_96_143.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_144_755.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_756_1451.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_1452_2171.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_2172_3227.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_3228_7883.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_7884_10595.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_10596_11555.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_11556_12059.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_12060_12587.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_12588_15275.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_15276_16895.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_16896_19139.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_19140_19859.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_19860_24095.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_24096_25055.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25056_25199.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25200_25247.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25248_25295.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25296_25715.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25716_25811.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_25812_29807.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_29808_30839.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_30840_33335.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_33336_33899.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_33900_34919.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_34920_37019.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_37020_37835.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_37836_38555.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_38556_38819.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_38820_40823.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_40824_41987.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_41988_42791.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_42792_43703.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_43704_44363.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_44364_44459.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_44460_44519.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_44520_44675.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_44676_46019.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_46020_49127.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_49128_49211.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_49212_49307.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_49308_49403.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_49404_49919.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_49920_50123.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_50124_51443.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_51444_51731.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_51732_54323.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_54324_55883.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_55884_57443.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_57444_58247.png DEF_CON_26_AI_VILLAGE_-_Mark_Mager_-_Rapid_Anomaly_Detection_via_Ransom_Note_File_C_58248_58450.png