{"action":"created","issue":{"url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/16","repository_url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security","labels_url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/16/labels{/name}","comments_url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/16/comments","events_url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/16/events","html_url":"https://github.com/13o-bbr-bbq/machine_learning_security/issues/16","id":342065353,"node_id":"MDU6SXNzdWUzNDIwNjUzNTM=","number":16,"title":"When crawling a web port it goes bad","user":{"login":"cybert79","id":16208462,"node_id":"MDQ6VXNlcjE2MjA4NDYy","avatar_url":"https://avatars2.githubusercontent.com/u/16208462?v=4","gravatar_id":"","url":"https://api.github.com/users/cybert79","html_url":"https://github.com/cybert79","followers_url":"https://api.github.com/users/cybert79/followers","following_url":"https://api.github.com/users/cybert79/following{/other_user}","gists_url":"https://api.github.com/users/cybert79/gists{/gist_id}","starred_url":"https://api.github.com/users/cybert79/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cybert79/subscriptions","organizations_url":"https://api.github.com/users/cybert79/orgs","repos_url":"https://api.github.com/users/cybert79/repos","events_url":"https://api.github.com/users/cybert79/events{/privacy}","received_events_url":"https://api.github.com/users/cybert79/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-07-17T20:06:57Z","updated_at":"2018-07-19T23:26:09Z","closed_at":null,"author_association":"NONE","body":"2018-07-17 22:05:12 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: scrapybot)\r\n2018-07-17 22:05:12 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.5.3 (default, Jan 19 2017, 14:11:04) - [GCC 6.3.0 20170118], pyOpenSSL 18.0.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptographyform Linux-4.9.0-3-amd64-x86_64-with-debian-9.5\r\n2018-07-17 22:05:12 [scrapy.crawler] INFO: Overridden settings: {'FEED_FORMAT': 'json', 'SPIDER_LOADER_WARN_ONLY': True, 'FEED_URI': 'crawl_result/20180717220511_crawl_result.json'}\r\n2018-07-17 22:05:12 [scrapy.middleware] INFO: Enabled extensions:\r\n['scrapy.extensions.corestats.CoreStats',\r\n 'scrapy.extensions.feedexport.FeedExporter',\r\n 'scrapy.extensions.logstats.LogStats',\r\n 'scrapy.extensions.memusage.MemoryUsage',\r\n 'scrapy.extensions.telnet.TelnetConsole']\r\n[*] Save log to /opt/machine_learning_security/DeepExploit/crawl_result/some-ip1_80.log\r\n2018-07-17 22:05:13 [scrapy.middleware] INFO: Enabled downloader middlewares:\r\n['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',\r\n 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',\r\n 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',\r\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',\r\n 'scrapy.downloadermiddlewares.retry.RetryMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',\r\n 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',\r\n 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',\r\n 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',\r\n 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',\r\n 'scrapy.downloadermiddlewares.stats.DownloaderStats']\r\n2018-07-17 22:05:13 [scrapy.middleware] INFO: Enabled spider middlewares:\r\n['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',\r\n 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',\r\n 'scrapy.spidermiddlewares.referer.RefererMiddleware',\r\n 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',\r\n 'scrapy.spidermiddlewares.depth.DepthMiddleware']\r\n2018-07-17 22:05:13 [scrapy.middleware] INFO: Enabled item pipelines:\r\n[]\r\n2018-07-17 22:05:13 [scrapy.core.engine] INFO: Spider opened\r\n2018-07-17 22:05:13 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)\r\n2018-07-17 22:05:13 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023\r\n2018-07-17 22:05:13 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://some-ip:80/> (referer: None)\r\n2018-07-17 22:05:13 [scrapy.core.engine] INFO: Closing spider (finished)\r\n2018-07-17 22:05:13 [scrapy.statscollectors] INFO: Dumping Scrapy stats:\r\n{'downloader/request_bytes': 212,\r\n 'downloader/request_count': 1,\r\n 'downloader/request_method_count/GET': 1,\r\n 'downloader/response_bytes': 564,\r\n 'downloader/response_count': 1,\r\n 'downloader/response_status_count/200': 1,\r\n 'finish_reason': 'finished',\r\n 'finish_time': datetime.datetime(2018, 7, 17, 20, 5, 13, 150929),\r\n 'log_count/DEBUG': 2,\r\n 'log_count/INFO': 7,\r\n 'memusage/max': 55209984,\r\n 'memusage/startup': 55209984,\r\n 'response_received_count': 1,\r\n 'scheduler/dequeued': 1,\r\n 'scheduler/dequeued/memory': 1,\r\n 'scheduler/enqueued': 1,\r\n 'scheduler/enqueued/memory': 1,\r\n 'start_time': datetime.datetime(2018, 7, 17, 20, 5, 13, 10283)}\r\n2018-07-17 22:05:13 [scrapy.core.engine] INFO: Spider closed (finished)\r\nTraceback (most recent call last):\r\n  File \"DeepExploit.py\", line 2064, in <module>\r\n    target_tree = env.get_target_info(rhost, proto_list, info_list)\r\n  File \"DeepExploit.py\", line 525, in get_target_info\r\n    web_target_info = self.util.run_spider(rhost, web_port_list)\r\n  File \"/opt/machine_learning_security/DeepExploit/util.py\", line 159, in run_spider\r\n    dict_json = json.load(fin)\r\n  File \"/usr/lib/python3.5/json/__init__.py\", line 268, in load\r\n    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)\r\n  File \"/usr/lib/python3.5/json/__init__.py\", line 319, in loads\r\n    return _default_decoder.decode(s)\r\n  File \"/usr/lib/python3.5/json/decoder.py\", line 339, in decode\r\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\r\n  File \"/usr/lib/python3.5/json/decoder.py\", line 357, in raw_decode\r\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\r\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\r\n"},"comment":{"url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/comments/406443829","html_url":"https://github.com/13o-bbr-bbq/machine_learning_security/issues/16#issuecomment-406443829","issue_url":"https://api.github.com/repos/13o-bbr-bbq/machine_learning_security/issues/16","id":406443829,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjQ0MzgyOQ==","user":{"login":"13o-bbr-bbq","id":12124329,"node_id":"MDQ6VXNlcjEyMTI0MzI5","avatar_url":"https://avatars3.githubusercontent.com/u/12124329?v=4","gravatar_id":"","url":"https://api.github.com/users/13o-bbr-bbq","html_url":"https://github.com/13o-bbr-bbq","followers_url":"https://api.github.com/users/13o-bbr-bbq/followers","following_url":"https://api.github.com/users/13o-bbr-bbq/following{/other_user}","gists_url":"https://api.github.com/users/13o-bbr-bbq/gists{/gist_id}","starred_url":"https://api.github.com/users/13o-bbr-bbq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/13o-bbr-bbq/subscriptions","organizations_url":"https://api.github.com/users/13o-bbr-bbq/orgs","repos_url":"https://api.github.com/users/13o-bbr-bbq/repos","events_url":"https://api.github.com/users/13o-bbr-bbq/events{/privacy}","received_events_url":"https://api.github.com/users/13o-bbr-bbq/received_events","type":"User","site_admin":false},"created_at":"2018-07-19T23:26:09Z","updated_at":"2018-07-19T23:26:09Z","author_association":"OWNER","body":"thanks for providing your info.  \r\ni understood cause of this problem.  \r\nin this case, there are two problems.  \r\n\r\n1) why cause json exception?  \r\ndeepexploit didn't correspond to empty file.  \r\nin this result, deepexploit caused exception of reading json.  \r\n\r\n2) why json is empty?\r\ncurrently `Scrapy` (crawling module) doesn't correspond parsing javascript (only html).  \r\nin this result, `Scrapy` couldn't extract links.  \r\n\r\ni fixed the problem 1).  \r\nplease, update `util.py`.  \r\n\r\nhowever, i cannot quickly fix the problem 2).  \r\nbecause, i need to reconstruct design of `Scrapy`.  \r\nnear the future, i'll correspond the parsing javascript, one moment please.  "}}
{"actor":{"display_login":"13o-bbr-bbq"}}