{"action":"created","comment":{"url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/comments/184499398","pull_request_review_id":115700634,"id":184499398,"diff_hunk":"@@ -30,66 +30,259 @@ From the GNU site:\n ## Dependencies\n \n * python\n-* tor \n+* tor\n \n-### pip install:\n+## Warning\n+The version of Elasticsearch needs to be less than 6.x. The maximum is 5.6.6 in the 5.x version. If you go higher of 5.x you will have problems. Also, if you decide to install Kibana or any extra functionalities link to Elasticsearch, install them with the same version because will not work.\n \n-pip install -r requirements.txt\n+Do not start too many instances of scraper/crawler because, with only 4 instances of tor proxy, it will be hard to connect to the onion site. If you create more than 3-4 instances, it could become really slow. In this situation,  the crawler will become so slow that they will not be able to crawl pages. So you will not progress with this method. Let the crawler run and you will create a bigger list of valid domains with information in it.\n \n-## Install\n+The Pastebin script works only if you are on the whitelist of Pastebin. If you're not, you will need to read the scraping API to understand how to activate it: https://pastebin.com/api_scraping_faq\n \n-Create mysql db from schema.sql\n+After booting, be sure that the link between Tor and Privoxy are working. To test it, use these commands.\n \n-Edit etc/database for your database setup\n+    curl --socks5-hostname 127.0.0.1:9050 http://http://workingOnionWebsite\n+    curl --proxy 127.0.0.1:3129 http://workingOnionWebsite\n+    \n+If it didn't work, fix the problem before crawling because all your onions will convert to a \"dead\" status. You can try to run the script:`start.sh` to reinitialize the links.\n \n-Edit etc/proxy for your TOR setup\n+### Tor browser Linux (Complementary)\n+To test websites before crawling (install this to have an interface and a better visualization) Important!! Tor browser needs an interface... Don't install this on a console.\n \n-    script/push.sh someoniondirectory.onion \n-    script/push.sh anotheroniondirectory.onion\n+* download https://www.torproject.org/download/download-easy.html.en\n+* extract the folder on your desktop.\n+* edit tor-browser_en-US/start-tor-browser .\n \n-Edit etc/uwsgi_only and set BASEDIR to wherever torscraper is installed (i.e. /home/user/torscraper)\n \n-Run:\n+Change this:\n+  \n+    if [ \"`id -u`\" -eq 0 ]; then\n+        complain \"The Tor Browser Bundle should not be run as root.  Exiting.\"\n+        exit 1\n+    fi  \n+    \n \n-    init/scraper_service.sh # to start crawling\n-    init/isup_service.sh # to keep site status up to date\n+To:\n \n-### Optional ElasticSearch Fulltext Search\n+    if [ \"`id -u`\" -eq 1 ]; then\n+      complain \"The Tor Browser Bundle should not be run as root.  Exiting.\"\n+      exit 1\n+    fi\n \n-The torscraper comes with optional elasticsearch capability (enabled by default). Edit etc/elasticsearch and set vars or set ELASTICSEARCH_ENABLED=false to disable. Run scripts/elasticsearch_migrate.sh to perform the initial setup after configuration. \n+* click on Tor Browser file and click on connect.\n \n-if elasticsearch is disabled there will be no fulltext search, however crawling and discovering new sites will still work.\n+### Tor service\n+To use the new version of tor, you should follow these steps: https://www.torproject.org/docs/debian.html.en\n+By using the last version of tor, you will be able to crawler the new generation of onions (V3).\n \n-### cronjobs\n+If you used a version older than 0.3.x, you can have a problem with the update to 0.3.x. I was missing two libraries: \n+ \n+ * libssl1.1\n+ * libzstd1\n \n-    # harvest onions from various sources\n-    1 18 * * * /home/scraper/torscraper/scripts/harvest.sh\n-    \n-    # get ssh fingerprints for new sites\n-    1 4,16 * * * /home/scraper/torscraper/scripts/update_fingerprints.sh\n+So, I installed them:\n+\n+        sudo apt-get install libzstd1\n+To install libssl1.1, I used a Debian package: https://packages.ubuntu.com/bionic/libssl1.1\n+\n+        lynx  https://packages.ubuntu.com/bionic/libssl1.1\n+Use the bottom arrow to go at the bottom of the page and select your \"Architecture Package Size\". When you had made your choice, click on the right arrow, it will redirect you to the download page. Now it's the same thing. Use the bottom arrow to go down and choose the one that you want. When you find the one, just click on the right arrow. At the bottom of your interface, you will see `D) Download or C) Cancel`. Press `D`. When you will see the text `Save to disk`, go on it. Press the right arrow and press on `Enter`. When it's done click on `q` and `y` to quit.\n+\n+        dpkg -i libssl1.1_1.1.0g-2ubuntu2_amd64.deb #the name of your debain package\n+\n+Finish the tor installation by looking to your version. If you have the last one (0.3.2 at the time that I wrote it).\n+\n+        tor --version\n+\n+\n+### Haproxy service\n+\n+    sudo apt-get install haproxy\n     \n-    # mark sites as genuine / fake from the /r/darknetmarkets superlist    \n-    1 9 * * 1 /home/scraper/torscraper/scripts/get_valid.sh\n+### Privoxy service\n+\n+    sudo apt-get install privoxy\n+\n+### Install Pip:\n+\n+    sudo apt-get install python-pip\n+    sudo pip install --upgrade pip\n+### Install Virtual environment    \n+    sudo pip install virtualenv\n+    sudo apt-get install python-virtualenv\n+\n+Go in your crawler/scraper folder and write.\n+\n+    virtualenv venv\n+\n+then activate it.\n+\n+    . venv/bin/activate\n+    # Run the next command when you're in your virtual environment because if you aren't, it will install in your normal environment\n+    pip install -r requirements.txt\n+### Install MariaDB\n+*** Mysql have a problem with some syntax in the code so I recommend you to install MariaDB ***\n+\n+    sudo apt-get install mariadb-server\n+    sudo apt-get install mariadb-client\n+\n+Now we will connect to MariaDB and create our database from `schema.sql`. We need to be in the folder to be able to see `schema.sql` because we will need it later.\n+\n+    mysql -u root\n+    CREATE DATABASE databaseName;\n+    use databaseName;\n+    source schema.sql\n+To know if all works well you should have \"Query OK\" on each row. You should have 20 tables if you do this command:\n+\n+    show tables;\n+\n+Need a modification to be able to connect Elasticsearh with our database.\n+\n+    use mysql;\n+    update user set plugin='mysql_native_password' where User='root';\n+    flush privileges;\n+    exit\n+    #To secure the installation. By default the password should be empty so just press enter. I recommand to put one.\n+    sudo mysql_secure_installation\n+    #To reconnect\n+    mysql -u root -p\n+\n+### Config your files\n+Edit `etc/database` for your database setup\n+\n+Edit `etc/tor/torrc` to uncomment the line : SocksPort 9050 (line 18)\n+\n+Edit `etc/uwsgi_only` and set BASEDIR to wherever torscraper is installed (i.e. /home/user/torscraper)\n+\n+Edit `etc/proxy` for your TOR setup\n+\n+    export TOR_PROXY_PORT=3129\n+    #export TOR_PROXY_PORT=3140\n+    export TOR_PROXY_HOST=localhost\n+    export http_proxy=http://localhost:3129\n+    #export http_proxy=http://localhost:3140\n+    export https_proxy=https://localhost:3129\n+    export SOCKS_PROXY=localhost:9050\n+    HIDDEN_SERVICE_PROXY_HOST=127.0.0.1\n+    HIDDEN_SERVICE_PROXY_PORT=9090\n+\n+ Now we will go in Privoxy config\n+\n+    cd /etc/privoxy/\n+    cp default.action default.action.orig\n+    cp default.filter default.filter.orig\n+    touch default.action (let file empty)\n+    touch default.filter (let file empty)\n+\n+### Start your services\n+\n+    service tor start\n+    service privoxy start\n+    service haproxy start\n+    service elasticsearch start\n+    service mysql start\n+\n+Go to the scripts folder and run this command\n+\n+    ./create_privoxy_confs.sh\n+   \n+Now it's time to try. Go to the directory:  .../freshonions-torscraper/scripts/. This directory is relative, you could have changed the name of the directory.\n+\n+    ./start.sh\n     \n-    # scrape pastebin for onions (needs paid account / IP whitelisting)                 \n-    */5 * * * * /home/scraper/torscraper/scripts/pastebin.sh\n+Now you can test if it works with the new generation of onions (V3) (test all ports 9051, 9052, 90... and 3129, 3130, 31...)\n+\n+    curl --socks5-hostname 127.0.0.1:9051 http://jamie3vkiwibfiwucd6vxijskbhpjdyajmzeor4mc4i7yopvpo4p7cyd.onion/\n+    curl --proxy 127.0.0.1:3129 http://jamie3vkiwibfiwucd6vxijskbhpjdyajmzeor4mc4i7yopvpo4p7cyd.onion/\n+\n+If you get something likes to Privoxy localhost port forwarding don't continue, it will not work.\n+\n+    ./push.sh someoniondirectory.onion\n+\n+To start the flask server to see our web interface.\n+\n+    ./scripts/web.sh\n+\n+To set up the port forwarding from your server to your browser, do this command on your computer to access server\n+\n+    ssh -L 5000:localhost:5000 username@IpAddressOfServer\n+\n+To try if it works well for now.\n+\n+    scripts/push.sh someoniondirectory.onion\n+    scripts/push.sh anotheroniondirectory.onion\n     \n-    # portscan new onions               \n-    1 */6 * * * /home/scraper/torscraper/scripts/portscan_up.sh\n+Run:\n \n-    # scrape stronghold paste\n-    32 */2 * * * /home/scraper/torscraper/scripts/stronghold_paste_rip.sh\n+    script/harvest.sh  #To get onions (just detect the onions, don't go deeper to find bitcoin address, emails, etc.)\n+    init/scraper_service.sh  #To start crawling (will get bitcoin address, emails, etc. if you already found onions with harvest.sh)\n+    init/isup_service.sh  #To keep site status up to date\n     \n-    # detect clones\n-    16 3 * * * /home/scraper/torscraper/scripts/detect_clones.sh\n+### Optional ElasticSearch Fulltext Search\n+\n+The Torscraper comes with optional Elasticsearch capability (enabled by default). Edit `etc/elasticsearch` and set vars or set `ELASTICSEARCH_ENABLED=false` to disable. \n+\n+Run `scripts/elasticsearch_migrate.sh` to perform the initial setup after configuration.\n+\n+If Elasticsearch is disabled there will be no full-text search, however crawling and discovering new sites will still work.\n+\n+\n+### ElasticSearch\n+You will need to install Elasticsearch(probably not only the pip package), this is the link to download the last version of 5.x. : https://www.elastic.co/downloads/past-releases/elasticsearch-5-6-6 . You can have problems with versions (like I said in the warning section). If you want to be sure you are using the right version, you can do this command :\n+\n+    curl -XGET 'http://localhost:9200'\n+\n+To enable Elasticsearch\n+\n+    service elasticsearch start\n+    ./elasticsearch_migrate.sh  #To perform the initial setup or if you want to reset Elasticsearch, but we need it at the beginning to start it. \n+\n+After restart :\n+\n+    . venv/bin/activate\n+    ./script/start.sh  #To start the instance of tor and privoxy\n+\n+### FLASK :\n+    ./scripts/web.sh  #Launch flask to have a web interface\n+\n+### Cronjobs\n+\n+    #Harvest onions from various sources\n+    1 18 * * * /home/freshonions-torscraper/scripts/harvest.sh\n+\n+    #Get ssh fingerprints for new sites\n+    1 4,16 * * * /home/freshonions-torscraper/scripts/update_fingerprints.sh\n+\n+    #Mark sites as genuine / fake from the /r/darknetmarkets superlist\n+    1 1 * * 1 /home/freshonions-torscraper/scripts/get_valid.sh\n+\n+    #Scrape pastebin for onions (needs paid account / IP whitelisting)\n+    */5 * * * * /home/freshonions-torscraper/scripts/pastebin.sh\n+\n+    #Portscan new onions\n+    1 13 * * * /home/freshonions-torscraper/scripts/portscan_up.sh\n+\n+    #Scrape stronghold paste\n+    32 */2 * * * /home/freshonions-torscraper/scripts/stronghold_paste_rip.sh\n+\n+    #Detect clones\n+    20 14 * * * /home/freshonions-torscraper/scripts/detect_clones.sh\n+\n+    #Keep a sql dump of data\n+    1 */1 * * * mysqldump -u username -ppassword --database tor --result-file=/home/dump.sql\n+    1 */8 * * * mysqldump -u username -ppassword --database tor --result-file=/home/dump_backup.sql\n \n \n ## Infrastructure\n \n-Fresh Onions runs on two servers, a frontend host running the database and hidden service web site, and a backend host running the crawler. Probably most interesting to the reader is the setup for the backend. TOR as a client is COMPLETELY SINGLETHREADED. I know! It's 2017, and along with a complete lack of flying cars, TOR runs in a single thread. What this means is that if you try to run a crawler on a single TOR instance you will quickly find you are maxing out your CPU at 100%.\n+Fresh Onions runs on two servers, a frontend host running the database and hidden service website, and a backend host running the crawler. Probably most interesting to the reader is the setup for the backend. T\n+OR as a client is COMPLETELY SINGLETHREADED. I know! It's 2017, and along with a complete lack of flying cars, TOR runs in a single thread. What this means is that if you try to run a crawler on a single TOR instance you will quickly find you are maxing out your CPU at 100%.\n \n The solution to this problem is running multiple TOR instances and connecting to them through some kind of frontend that will round-robin your requests. The Fresh Onions crawler runs eight Tor instances.\n \n-Debian (and ubuntu) comes with a useful program \"tor-instance-create\" for quickly creating multiple instances of TOR. I used Squid as my frontend proxy, but unfortunately it can't connect to SOCKS directly, so I used \"privoxy\" as an intermediate proxy. You will need one privoxy instance for every TOR instance. There is a script in \"scripts/create_privoxy.sh\" to help with creating privoxy instances on debian systems. It also helps to replace /etc/privoxy/default.filter with an empty file, to reduce CPU load by removing unnecessary regexes.\n+Debian (and Ubuntu) comes with a useful program \"tor-instance-create\" for quickly creating multiple instances of TOR. I used Squid as my frontend proxy, but unfortunately, it can't connect to SOCKS directly, so I\n+ used \"Privoxy\" as an intermediate proxy. You will need one Privoxy instance for every TOR instance. There is a script in \"scripts/create_privoxy.sh\" to help with creating Privoxy instances on Debian systems. It also helps to replace /etc/privoxy/default.filter with an empty file, to reduce CPU load by removing unnecessary regexes.","path":"README.md","position":324,"original_position":324,"commit_id":"85d343b501a4f18b0284265b57005e4ad55dd23d","original_commit_id":"740845d1fdff828a7fe7788342c9dc97ba8f7b40","user":{"login":"obilodeau","id":546325,"avatar_url":"https://avatars2.githubusercontent.com/u/546325?v=4","gravatar_id":"","url":"https://api.github.com/users/obilodeau","html_url":"https://github.com/obilodeau","followers_url":"https://api.github.com/users/obilodeau/followers","following_url":"https://api.github.com/users/obilodeau/following{/other_user}","gists_url":"https://api.github.com/users/obilodeau/gists{/gist_id}","starred_url":"https://api.github.com/users/obilodeau/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/obilodeau/subscriptions","organizations_url":"https://api.github.com/users/obilodeau/orgs","repos_url":"https://api.github.com/users/obilodeau/repos","events_url":"https://api.github.com/users/obilodeau/events{/privacy}","received_events_url":"https://api.github.com/users/obilodeau/received_events","type":"User","site_admin":false},"body":"whitespace changes warning!\r\n\r\nIt looks like a newline was added in TOR (after the T) if this is the only change, it should be reverted. Whitespace changes contribute to nothing except git diff noise.","created_at":"2018-04-26T19:06:29Z","updated_at":"2018-04-26T19:09:04Z","html_url":"https://github.com/GoSecure/freshonions-torscraper/pull/1#discussion_r184499398","pull_request_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/comments/184499398"},"html":{"href":"https://github.com/GoSecure/freshonions-torscraper/pull/1#discussion_r184499398"},"pull_request":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1"}}},"pull_request":{"url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1","id":174485857,"html_url":"https://github.com/GoSecure/freshonions-torscraper/pull/1","diff_url":"https://github.com/GoSecure/freshonions-torscraper/pull/1.diff","patch_url":"https://github.com/GoSecure/freshonions-torscraper/pull/1.patch","issue_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/1","number":1,"state":"open","locked":false,"title":"Update readme","user":{"login":"L3houx","id":32520954,"avatar_url":"https://avatars3.githubusercontent.com/u/32520954?v=4","gravatar_id":"","url":"https://api.github.com/users/L3houx","html_url":"https://github.com/L3houx","followers_url":"https://api.github.com/users/L3houx/followers","following_url":"https://api.github.com/users/L3houx/following{/other_user}","gists_url":"https://api.github.com/users/L3houx/gists{/gist_id}","starred_url":"https://api.github.com/users/L3houx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/L3houx/subscriptions","organizations_url":"https://api.github.com/users/L3houx/orgs","repos_url":"https://api.github.com/users/L3houx/repos","events_url":"https://api.github.com/users/L3houx/events{/privacy}","received_events_url":"https://api.github.com/users/L3houx/received_events","type":"User","site_admin":false},"body":"The last verification to be sure that it doesn't contain any errors.","created_at":"2018-03-12T20:30:27Z","updated_at":"2018-04-26T19:09:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"644282f7aeee54549f2fec9ba51ec38074be2605","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1/commits","review_comments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1/comments","review_comment_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/comments{/number}","comments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/1/comments","statuses_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/statuses/85d343b501a4f18b0284265b57005e4ad55dd23d","head":{"label":"GoSecure:update-readme","ref":"update-readme","sha":"85d343b501a4f18b0284265b57005e4ad55dd23d","user":{"login":"GoSecure","id":16034225,"avatar_url":"https://avatars1.githubusercontent.com/u/16034225?v=4","gravatar_id":"","url":"https://api.github.com/users/GoSecure","html_url":"https://github.com/GoSecure","followers_url":"https://api.github.com/users/GoSecure/followers","following_url":"https://api.github.com/users/GoSecure/following{/other_user}","gists_url":"https://api.github.com/users/GoSecure/gists{/gist_id}","starred_url":"https://api.github.com/users/GoSecure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoSecure/subscriptions","organizations_url":"https://api.github.com/users/GoSecure/orgs","repos_url":"https://api.github.com/users/GoSecure/repos","events_url":"https://api.github.com/users/GoSecure/events{/privacy}","received_events_url":"https://api.github.com/users/GoSecure/received_events","type":"Organization","site_admin":false},"repo":{"id":121563510,"name":"freshonions-torscraper","full_name":"GoSecure/freshonions-torscraper","owner":{"login":"GoSecure","id":16034225,"avatar_url":"https://avatars1.githubusercontent.com/u/16034225?v=4","gravatar_id":"","url":"https://api.github.com/users/GoSecure","html_url":"https://github.com/GoSecure","followers_url":"https://api.github.com/users/GoSecure/followers","following_url":"https://api.github.com/users/GoSecure/following{/other_user}","gists_url":"https://api.github.com/users/GoSecure/gists{/gist_id}","starred_url":"https://api.github.com/users/GoSecure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoSecure/subscriptions","organizations_url":"https://api.github.com/users/GoSecure/orgs","repos_url":"https://api.github.com/users/GoSecure/repos","events_url":"https://api.github.com/users/GoSecure/events{/privacy}","received_events_url":"https://api.github.com/users/GoSecure/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoSecure/freshonions-torscraper","description":"Fresh Onions is an open source TOR spider / hidden service onion crawler","fork":true,"url":"https://api.github.com/repos/GoSecure/freshonions-torscraper","forks_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/forks","keys_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/teams","hooks_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/hooks","issue_events_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/events{/number}","events_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/events","assignees_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/assignees{/user}","branches_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/branches{/branch}","tags_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/tags","blobs_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/statuses/{sha}","languages_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/languages","stargazers_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/stargazers","contributors_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/contributors","subscribers_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/subscribers","subscription_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/subscription","commits_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/contents/{+path}","compare_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/merges","archive_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/downloads","issues_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues{/number}","pulls_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls{/number}","milestones_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/milestones{/number}","notifications_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/labels{/name}","releases_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/releases{/id}","deployments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/deployments","created_at":"2018-02-14T21:30:54Z","updated_at":"2018-04-26T02:49:59Z","pushed_at":"2018-04-26T19:08:35Z","git_url":"git://github.com/GoSecure/freshonions-torscraper.git","ssh_url":"git@github.com:GoSecure/freshonions-torscraper.git","clone_url":"https://github.com/GoSecure/freshonions-torscraper.git","svn_url":"https://github.com/GoSecure/freshonions-torscraper","homepage":"","size":2631,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":1,"license":null,"forks":2,"open_issues":1,"watchers":2,"default_branch":"master"}},"base":{"label":"GoSecure:master","ref":"master","sha":"ab4fe991911e47b0e043ed228f9a9515b2183dd9","user":{"login":"GoSecure","id":16034225,"avatar_url":"https://avatars1.githubusercontent.com/u/16034225?v=4","gravatar_id":"","url":"https://api.github.com/users/GoSecure","html_url":"https://github.com/GoSecure","followers_url":"https://api.github.com/users/GoSecure/followers","following_url":"https://api.github.com/users/GoSecure/following{/other_user}","gists_url":"https://api.github.com/users/GoSecure/gists{/gist_id}","starred_url":"https://api.github.com/users/GoSecure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoSecure/subscriptions","organizations_url":"https://api.github.com/users/GoSecure/orgs","repos_url":"https://api.github.com/users/GoSecure/repos","events_url":"https://api.github.com/users/GoSecure/events{/privacy}","received_events_url":"https://api.github.com/users/GoSecure/received_events","type":"Organization","site_admin":false},"repo":{"id":121563510,"name":"freshonions-torscraper","full_name":"GoSecure/freshonions-torscraper","owner":{"login":"GoSecure","id":16034225,"avatar_url":"https://avatars1.githubusercontent.com/u/16034225?v=4","gravatar_id":"","url":"https://api.github.com/users/GoSecure","html_url":"https://github.com/GoSecure","followers_url":"https://api.github.com/users/GoSecure/followers","following_url":"https://api.github.com/users/GoSecure/following{/other_user}","gists_url":"https://api.github.com/users/GoSecure/gists{/gist_id}","starred_url":"https://api.github.com/users/GoSecure/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GoSecure/subscriptions","organizations_url":"https://api.github.com/users/GoSecure/orgs","repos_url":"https://api.github.com/users/GoSecure/repos","events_url":"https://api.github.com/users/GoSecure/events{/privacy}","received_events_url":"https://api.github.com/users/GoSecure/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/GoSecure/freshonions-torscraper","description":"Fresh Onions is an open source TOR spider / hidden service onion crawler","fork":true,"url":"https://api.github.com/repos/GoSecure/freshonions-torscraper","forks_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/forks","keys_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/teams","hooks_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/hooks","issue_events_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/events{/number}","events_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/events","assignees_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/assignees{/user}","branches_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/branches{/branch}","tags_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/tags","blobs_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/refs{/sha}","trees_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/statuses/{sha}","languages_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/languages","stargazers_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/stargazers","contributors_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/contributors","subscribers_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/subscribers","subscription_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/subscription","commits_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/commits{/sha}","git_commits_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/git/commits{/sha}","comments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/comments{/number}","issue_comment_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/comments{/number}","contents_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/contents/{+path}","compare_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/merges","archive_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/downloads","issues_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues{/number}","pulls_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls{/number}","milestones_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/milestones{/number}","notifications_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/labels{/name}","releases_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/releases{/id}","deployments_url":"https://api.github.com/repos/GoSecure/freshonions-torscraper/deployments","created_at":"2018-02-14T21:30:54Z","updated_at":"2018-04-26T02:49:59Z","pushed_at":"2018-04-26T19:08:35Z","git_url":"git://github.com/GoSecure/freshonions-torscraper.git","ssh_url":"git@github.com:GoSecure/freshonions-torscraper.git","clone_url":"https://github.com/GoSecure/freshonions-torscraper.git","svn_url":"https://github.com/GoSecure/freshonions-torscraper","homepage":"","size":2631,"stargazers_count":2,"watchers_count":2,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"archived":false,"open_issues_count":1,"license":null,"forks":2,"open_issues":1,"watchers":2,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1"},"html":{"href":"https://github.com/GoSecure/freshonions-torscraper/pull/1"},"issue":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/1"},"comments":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/issues/1/comments"},"review_comments":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1/comments"},"review_comment":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/pulls/1/commits"},"statuses":{"href":"https://api.github.com/repos/GoSecure/freshonions-torscraper/statuses/85d343b501a4f18b0284265b57005e4ad55dd23d"}},"author_association":"NONE"}}
{"actor":{"display_login":"obilodeau"}}