pystemon
CC Attribution 2.0 Belgium:
You are free to use, adapt and copy, distribute and transmit the work or content in adapted or unchanged form for any legal purpose as long as the work is attributed to the author in the manner specified by the author or licensor.
Vandeplas, Christophe
Pastebin alike monitoring Pystemon is a multithreaded monitoring tool for PasteBin-alike sites written in Python. The presentation will quickly overview the main functionality of the tool and elaborate on interesting aspects: - complexities when monitoring pastebin alike sites - dynamic configuration by config file or variable classes to increase flexibility while keeping code small, readable and extensible.
 Okay, so I'm indeed also working in asserts and it's quite useful to know when data from your constituents has been leaked on the internet or when email accounts have been compromised or certain forums where your users use their same email addresses of the work and perhaps the same passwords are leaked. In other contexts, we also have Xavier Mechthans here who started this similar project before and he has a website called leakedin.com and over there he publishes automatically all information, fancy information that was leaked on on Pacebin. So I'm quickly going to do three steps. First, what is what is Pycemon? So as he said, it's a tool to monitor websites like Pacebin. The Pycemon itself monitors like seven or eight websites. Currently, it's built to be easily configurable so you can easily add new websites. But it's not only to monitor but it's also made to be able to archive the sites. So every paste you see on a website, you can keep it and store it on your local disk. That's kind of useful to keep copies of pastes which contains leaked information. They are usually deleted within hours or days. So when you have a copy, you can still do your work either analyzing the leaked data or incident handling. So the tool itself has an alerting feature. So when you can configure searches based on regular expressions and when it finds something that interests you, it can send you an email. You can also configure the multi-threaded parts. So automatically the tool will start different threads for each paste websites and you can configure how many parallel threads it will use to download each paste from each site. But some sites implement some security measures like troddling and etc. So there's also a feature to import a list of proxies and the tool will then automatically use random proxies from your configuration file. And when a proxy is not working after a few attempts, it will remove the proxy from the list so it will still work and download the pastes. You can also configure it to use random user agents. By default on the GitHub sites, I deliver a list of 7,000 user agents that are all real. So based on a local collection, the tool has been built to enable to add very easily new sites. So I will show a few examples. When you run the tool, the tool is command line based. But when you run the tool, you will see this output. The first when the program starts, you see that it immediately downloads the list of the last pastes published on the site. And then it puts everything in buckets. And each thread will process the buckets to download the paste in parallel. When you have a hit for a search you programs, you also see the output that it found a hit for what you were searching for. So a lot alerting parts. It's kind of easy for it's the configuration file is a YAML based. So we have the search section with two points in here. So many options, many parameters are optional. So the first is a simple search for domain name. It's based on a regular expression. So you can configure it to lower the chance of false positives. For example in here, I ask example.com. But everything before should not be a normal letter. So that means that the main domain and the subdomains are included, but not hello example.com. You can also put a description which is more user-friendly in emails and in logging. But for some cases you like to exclude things from the search. So if for example in the second line I have a search for download, but I do not want to see the words porn sex or teen. And I want to have at least four hits of the search keywords and only then it will alert me if it's available. You can also configure the regular expression flags. So for example, I had situations where I had to search for data which was multi-line. So you just set the regx flag and you can go even further. Site itself is extremely easy to configure. So you can add your own site and share them please. For Pacebin you had archive URL. That's the page that contains the list of the last. For Pacebin I think it's like 50 or 80 pastes. The regular expression you want to use to extract the unique paste ID and the URL where he should download that ID. There are also two parameters update, max and update min. That's to add some variable parts in the download. So it will download or check for updates of the archive URL in a range between 30 and 40 seconds. So that automated tools will have less chance to detect you. Unfortunately monitoring websites like Pacebin it's not easy. You have rate limiting. So the solution was to add proxies and user agents poofing. Some websites don't have lists of recent pastes. So they're your kind of a screwed except if you have another source where you can import the pastes. Some pastes are password protected. Then you also need the paste ID and a password. On some pastes sites there is no separate download page or download pages that require form submits. That means you can't simply download the page automatically. This the pastes site has this example and you can see that there is a unique confirmation key for each request. To solve this issue I enabled dynamic configuration of downloads features. So for the pastes site I have a separate class in my configuration. That's in the YAML configuration. And in the Python code I simply create a class that inherits from my main pasty class. And over there I override the fetch pasty function that downloads pasties. And I simply download the original pasty page, parse the right section, search for the confirmation string, submit the form and get the real text-based pasty. Why is it important to have text-based pasties? Because some pastes contain HTML characters and if it's shown in HTML view they are usually pre-parsed to be not interpreted by the browser. So then your regular expression searches won't work. So the tool has kind of a dynamic configuration, a dynamic class configuration. And Python is really nice language because it's extremely dynamic. You can use variable functions, you can use variable classes. So in here I simply check if the class name variable is set in my YAML configuration. And then I search in a global pool of all classes. If that class exists and I create an instance of the class. So it's extremely easy with in Python to use kind of a dynamic classes or dynamic functions. To solve one of the other problems, one of the other problems is that the some sites don't have a list of recent pastes. But for example they have a generation algorithm for the past ID. For example 1, 2, 3, 4, 5 or ABC. So I plan to add also a dynamic archive list last pasties class so that you could even program extra classes where algorithms could be programmed. So you could also download pastes from such sites. For the password protected pasties also have a solution in mind is that I will provide classes to download the pastes and an interface where you can just submit the URL. He will automatically detect of what pastes site the URL is and then he will download the pastes and search for the alert keywords and archive it if you want to have archived. So there was a very fast overview of Spiceman. Usually certs use this tool to have copies of the least that paste and be alerted automatically when something interesting happens on the pastes sites. So the code is on GitHub. Feel free to fork it to send in bug reports to fix bugs or to program new pastes sites. Inform me if you have some specific pastes sites you know that is not yet present in a configuration in the full configuration and feel free to contribute it. If you also have also monitor pastes sites and have a list of regular expressions that are interesting feel free also to share them and I could put them on the GitHub sites. So maybe there are any questions. Hi I'm gonna ask an annoying question. Where do you legally stand on downloading these contents and archiving them unless they're on a creative commons license or likewise license to be published for anyone's use? I'm going to answer with a standard I'm a little lawyer but there are two parts. First of all the code this code is just any software you can use it as you want. About the data that is published on websites when it's published on the internet it's kind of a public data you still have kind of a copyright. If you download the data on your own system it's kind of a your problem. If you share the data you downloaded then you have a more complex answer. So as long as you download things which are not illegal to possess I mean I don't know if it's illegal to possess a list of usernames and passwords. I don't know if it's illegal to possess some sample code someone pasted on the internet like programming code or HTML code. I don't think that's illegal. I think the only risk you have of archiving pastes sites is when you start sharing the pastes you saved on your disk. So I wouldn't expect any problems as long as you don't share what you saved on your disk. Any work published you have a copyright protection so when you publish something on past beans it's like if you publish a book you are protected for the copyright. So it's only the word distributions like you just explain that is problematic not the fact you are shifted. For example Xavier on leakedin.com has a kind of a bird press blog where he automatically posts leaked data and he has an abuse email address where he processes all abuse requests. So when someone asks please take this offline he takes it offline. So that's another way to still partially publish data but still be in the kind of a legal zone. But of course if you publish your data be careful you don't publish credit cards or other protected data. There's still over there. There was a question there or yeah over there. Have you already tried to predict URLs for the next I don't know 50 archives? Pardon have I tried to? On sites that don't have archives you said so they use generators or algorithms to generate the next IDs. Did you try to predict the next 50 pastes? There was one site which was kind of easy just an incrementing algorithm of AZ09 so that's something when you look at five pastes you immediately see it. I have not studied the algorithms of pastes bin or any other pastes sites. However I think combining it with a search engine like any favorite search engine and searching for specific strings on the search engine and feeding that into the Pistmon would be an interesting way to find things you can't find otherwise and also to search in pastes for URLs. Sometimes people publish pastes with links to other pastes so then you can also discover things you didn't know yet. Of course the tool also keeps in memory lists for efficiency lists of recent downloaded pastes so it doesn't download the same ID twice. You can also store it in a database not a complete paste but references of what has been found and what paste and what URL etc. So it's still a tool in progress but it's kind of a stable right now.
 Okay, so I'm indeed also working in asserts and it's quite useful to know  when data from your constituents has been leaked on the internet or when email  accounts have been compromised or certain forums where your users use their  same email addresses of the work and perhaps the same passwords are leaked.  In other contexts, we also have Xavier Mechthans here who started this similar  project before and he has a website called leakedin.com and over there he  publishes automatically all information, fancy information that was leaked on  on Pacebin. So I'm quickly going to do three steps. First, what is  what is Pycemon? So as he said, it's a tool to monitor websites like  Pacebin. The Pycemon itself monitors like seven or eight websites. Currently,  it's built to be easily configurable so you can easily add new websites. But  it's not only to monitor but it's also made to be able to archive the sites. So  every paste you see on a website, you can keep it and store it on your local  disk. That's kind of useful to keep copies of pastes which contains leaked  information. They are usually deleted within hours or days. So when you have a  copy, you can still do your work either analyzing the leaked data or incident  handling. So the tool itself has an alerting feature. So when you can  configure searches based on regular expressions and when it finds something  that interests you, it can send you an email. You can also configure the  multi-threaded parts. So automatically the tool will start different threads for  each paste websites and you can configure how many parallel threads it will  use to download each paste from each site. But some sites implement some  security measures like troddling and etc. So there's also a feature to import a  list of proxies and the tool will then automatically use random proxies from  your configuration file. And when a proxy is not working after a few attempts, it  will remove the proxy from the list so it will still work and download the  pastes. You can also configure it to use random user agents. By default on the  GitHub sites, I deliver a list of 7,000 user agents that are all real. So based  on a local collection, the tool has been built to enable to add very easily new  sites. So I will show a few examples. When you run the tool, the tool is  command line based. But when you run the tool, you will see this output. The first  when the program starts, you see that it immediately downloads the list of the  last pastes published on the site. And then it puts everything in buckets. And  each thread will process the buckets to download the paste in parallel. When you  have a hit for a search you programs, you also see the output that it found a  hit for what you were searching for. So a lot alerting parts. It's kind of easy for  it's the configuration file is a YAML based. So we have the search section  with two points in here. So many options, many parameters are optional. So the  first is a simple search for domain name. It's based on a regular expression. So  you can configure it to lower the chance of false positives. For example in here,  I ask example.com. But everything before should not be a normal letter. So that  means that the main domain and the subdomains are included, but not hello  example.com. You can also put a description which is more user-friendly  in emails and in logging. But for some cases you like to exclude things from  the search. So if for example in the second line I have a search for download,  but I do not want to see the words porn sex or teen. And I want to have at least  four hits of the search keywords and only then it will alert me if it's  available. You can also configure the regular expression flags. So for example,  I had situations where I had to search for data which was multi-line. So you  just set the regx flag and you can go even further.  Site itself is extremely easy to configure. So you can add your own  site and share them please. For Pacebin you had archive URL. That's the page  that contains the list of the last. For Pacebin I think it's like 50 or 80  pastes. The regular expression you want to use to extract the unique paste ID and  the URL where he should download that ID. There are also two parameters update,  max and update min. That's to add some variable parts in the download. So it  will download or check for updates of the archive URL in a range between 30 and  40 seconds. So that automated tools will have less chance to detect you.  Unfortunately monitoring websites like Pacebin it's not easy. You have rate  limiting. So the solution was to add proxies and user agents  poofing. Some websites don't have lists of recent pastes. So they're your kind of  a screwed except if you have another source where you can import the pastes.  Some pastes are password protected. Then you also need the paste ID and a password.  On some pastes sites there is no separate download page or download pages that  require form submits. That means you can't simply download the page  automatically. This the pastes site has this example and you can see that there  is a unique confirmation key for each request. To solve this issue I enabled  dynamic configuration of downloads features. So for the pastes site I have a  separate class in my configuration. That's in the YAML configuration. And in the  Python code I simply create a class that inherits from my main pasty class. And  over there I override the fetch pasty function that downloads pasties. And I  simply download the original pasty page, parse the right section, search for the  confirmation string, submit the form and get the real text-based pasty. Why is  it important to have text-based pasties? Because some pastes contain HTML  characters and if it's shown in HTML view they are usually pre-parsed to  be not interpreted by the browser. So then your regular expression searches  won't work. So the tool has kind of a dynamic configuration, a dynamic class  configuration. And Python is really nice language because it's extremely  dynamic. You can use variable functions, you can use variable classes. So in here  I simply check if the class name variable is set in my YAML configuration. And then  I search in a global pool of all classes. If that class exists and I create an  instance of the class. So it's extremely easy with in Python to use kind of a  dynamic classes or dynamic functions. To solve one of the other problems,  one of the other problems is that the some sites don't have a list of recent  pastes. But for example they have a generation algorithm for the past ID. For  example 1, 2, 3, 4, 5 or ABC. So I plan to add also a dynamic archive list last  pasties class so that you could even program extra classes where algorithms  could be programmed. So you could also download pastes from such sites. For the  password protected pasties also have a solution in mind is that I will provide  classes to download the pastes and an interface where you can just submit the  URL. He will automatically detect of what pastes site the URL is and then he  will download the pastes and search for the alert keywords and archive it if you  want to have archived. So there was a very fast overview of Spiceman. Usually  certs use this tool to have copies of the least that paste and be alerted  automatically when something interesting happens on the pastes sites. So the code  is on GitHub. Feel free to fork it to send in bug reports to fix bugs or to  program new pastes sites. Inform me if you have some specific pastes sites you  know that is not yet present in a configuration in the full configuration  and feel free to contribute it. If you also have also monitor pastes sites and  have a list of regular expressions that are interesting feel free also to share  them and I could put them on the GitHub sites. So maybe there are any questions.  Hi I'm gonna ask an annoying question. Where do you legally stand on downloading  these contents and archiving them unless they're on a creative commons license  or likewise license to be published for anyone's use? I'm going to answer with a  standard I'm a little lawyer but there are two parts. First of all the code this  code is just any software you can use it as you want. About the data that is  published on websites when it's published on the internet it's kind of a  public data you still have kind of a copyright. If you download the data on  your own system it's kind of a your problem. If you share the data you  downloaded then you have a more complex answer. So as long as you download  things which are not illegal to possess I mean I don't know if it's illegal to  possess a list of usernames and passwords. I don't know if it's illegal to  possess some sample code someone pasted on the internet like programming code or  HTML code. I don't think that's illegal. I think the only risk you have of  archiving pastes sites is when you start sharing the pastes you saved on your  disk. So I wouldn't expect any problems as long as you don't share what you  saved on your disk. Any work published you have a copyright protection so when  you publish something on past beans it's like if you publish a book you are  protected for the copyright. So it's only the word distributions like you just  explain that is problematic not the fact you are shifted. For example Xavier on  leakedin.com has a kind of a bird press blog where he automatically posts leaked  data and he has an abuse email address where he processes all abuse requests.  So when someone asks please take this offline he takes it offline. So that's  another way to still partially publish data but still be in the kind of a legal  zone. But of course if you publish your data be careful you don't publish  credit cards or other protected data.  There's still over there.  There was a question there or yeah over there.  Have you already tried to predict URLs for the next I don't know 50 archives?  Pardon have I tried to? On sites that don't have archives you said so they use  generators or algorithms to generate the next IDs. Did you try to predict the  next 50 pastes? There was one site which was kind of easy just an incrementing  algorithm of AZ09 so that's something when you look at five pastes you  immediately see it. I have not studied the algorithms of pastes bin or any other  pastes sites. However I think combining it with a search engine like any  favorite search engine and searching for specific strings on the search engine  and feeding that into the Pistmon would be an interesting way to find things you  can't find otherwise and also to search in pastes for URLs. Sometimes people  publish pastes with links to other pastes so then you can also discover things  you didn't know yet. Of course the tool also keeps in memory lists for efficiency  lists of recent downloaded pastes so it doesn't download the same ID twice. You  can also store it in a database not a complete paste but references of what  has been found and what paste and what URL etc. So it's still a tool in  progress but it's kind of a stable right now.
pystemon_V_vk5LPz2ms_2_0_155.png pystemon_V_vk5LPz2ms_2_156_1211.png pystemon_V_vk5LPz2ms_2_1212_13427.png pystemon_V_vk5LPz2ms_2_13428_16895.png pystemon_V_vk5LPz2ms_2_16896_16979.png pystemon_V_vk5LPz2ms_2_16980_17147.png pystemon_V_vk5LPz2ms_2_17148_17195.png pystemon_V_vk5LPz2ms_2_17196_17243.png pystemon_V_vk5LPz2ms_2_17244_17291.png pystemon_V_vk5LPz2ms_2_17292_18539.png pystemon_V_vk5LPz2ms_2_18540_20759.png pystemon_V_vk5LPz2ms_2_20760_20819.png pystemon_V_vk5LPz2ms_2_20820_20867.png pystemon_V_vk5LPz2ms_2_20868_20951.png pystemon_V_vk5LPz2ms_2_20952_21179.png pystemon_V_vk5LPz2ms_2_21180_21239.png pystemon_V_vk5LPz2ms_2_21240_22451.png pystemon_V_vk5LPz2ms_2_22452_22919.png pystemon_V_vk5LPz2ms_2_22920_23147.png pystemon_V_vk5LPz2ms_2_23148_23267.png pystemon_V_vk5LPz2ms_2_23268_23652.png