{"action":"created","issue":{"url":"https://api.github.com/repos/OJ/gobuster/issues/79","repository_url":"https://api.github.com/repos/OJ/gobuster","labels_url":"https://api.github.com/repos/OJ/gobuster/issues/79/labels{/name}","comments_url":"https://api.github.com/repos/OJ/gobuster/issues/79/comments","events_url":"https://api.github.com/repos/OJ/gobuster/issues/79/events","html_url":"https://github.com/OJ/gobuster/pull/79","id":322669491,"node_id":"MDExOlB1bGxSZXF1ZXN0MTg3NzE3OTY3","number":79,"title":"implement progress, timeout and context","user":{"login":"FireFart","id":105281,"node_id":"MDQ6VXNlcjEwNTI4MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/105281?v=4","gravatar_id":"","url":"https://api.github.com/users/FireFart","html_url":"https://github.com/FireFart","followers_url":"https://api.github.com/users/FireFart/followers","following_url":"https://api.github.com/users/FireFart/following{/other_user}","gists_url":"https://api.github.com/users/FireFart/gists{/gist_id}","starred_url":"https://api.github.com/users/FireFart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FireFart/subscriptions","organizations_url":"https://api.github.com/users/FireFart/orgs","repos_url":"https://api.github.com/users/FireFart/repos","events_url":"https://api.github.com/users/FireFart/events{/privacy}","received_events_url":"https://api.github.com/users/FireFart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-05-14T04:32:42Z","updated_at":"2018-06-27T22:04:39Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/OJ/gobuster/pulls/79","html_url":"https://github.com/OJ/gobuster/pull/79","diff_url":"https://github.com/OJ/gobuster/pull/79.diff","patch_url":"https://github.com/OJ/gobuster/pull/79.patch"},"body":"## Changes\r\n* massive refactoring. libgobuster can now be used as a library from other programs too (but it's still not perfect because it still contains some command line option logic inside the library which should be extracted)\r\n* status on wordlist progress is now printed\r\n* a context is now passed to the main function to handle cancelation. Also all HTTP requests now receive the context so they will be canceled immediately. \r\n* connections are now actually reused. This speeds up gobuster by factor 3 (at least on HTTPS connections)\r\n* new `-to` parameter to specify request timeout (default 10s)\r\n* status codes are now sorted on output\r\n* a default user agent of `gobuster version` is set on every request (old: `Go-http-client/1.1`, changeable with command line option)\r\n* added status `403` to the default status list\r\n* new STDIN handling\r\n* lower memory usage\r\n\r\n## Memory usage comparison\r\n### current master\r\n#### HTOP\r\n![old](https://user-images.githubusercontent.com/105281/40357486-99bece5c-5dbc-11e8-905a-b278f0e9c0fe.png)\r\n#### ps\r\n```\r\nroot      2942 50.4  0.1 1127080 15416 pts/0   Sl+  11:19   0:07  |   |   \\_ /root/go/bin/gobuster -u https://firefart.at/ -w /usr/share/wordlists/dirb/common.txt\r\n```\r\n\r\n### this pr\r\n#### HTOP\r\n![new](https://user-images.githubusercontent.com/105281/40357494-9fe7e994-5dbc-11e8-864d-94e32f59432b.png)\r\n#### ps\r\n```\r\nroot      2970 47.3  0.1  14784  9860 pts/0    Sl+  11:20   0:04  |   |   \\_ ./gobuster -u https://firefart.at/ -w /usr/share/wordlists/dirb/common.txt\r\n```\r\n\r\n## Example runs\r\n### during execution\r\n![run](https://user-images.githubusercontent.com/105281/40380891-8cd8f09e-5dfa-11e8-8632-779a6e98f240.png)\r\n\r\n### finished\r\n![finished](https://user-images.githubusercontent.com/105281/40380906-93853e7a-5dfa-11e8-89fa-c7b7fb6c4c46.png)\r\n\r\n## Test instructions\r\n* `go get -u github.com/OJ/gobuster`\r\n* `cd $GOROOT/src/github.com/OJ/gobuster`\r\n* `git remote add firefart https://github.com/FireFart/gobuster.git`\r\n* `git fetch firefart`\r\n* `git checkout -b status firefart/status`\r\n* `go build`\r\n"},"comment":{"url":"https://api.github.com/repos/OJ/gobuster/issues/comments/400843924","html_url":"https://github.com/OJ/gobuster/pull/79#issuecomment-400843924","issue_url":"https://api.github.com/repos/OJ/gobuster/issues/79","id":400843924,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDg0MzkyNA==","user":{"login":"FireFart","id":105281,"node_id":"MDQ6VXNlcjEwNTI4MQ==","avatar_url":"https://avatars1.githubusercontent.com/u/105281?v=4","gravatar_id":"","url":"https://api.github.com/users/FireFart","html_url":"https://github.com/FireFart","followers_url":"https://api.github.com/users/FireFart/followers","following_url":"https://api.github.com/users/FireFart/following{/other_user}","gists_url":"https://api.github.com/users/FireFart/gists{/gist_id}","starred_url":"https://api.github.com/users/FireFart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FireFart/subscriptions","organizations_url":"https://api.github.com/users/FireFart/orgs","repos_url":"https://api.github.com/users/FireFart/repos","events_url":"https://api.github.com/users/FireFart/events{/privacy}","received_events_url":"https://api.github.com/users/FireFart/received_events","type":"User","site_admin":false},"created_at":"2018-06-27T22:04:39Z","updated_at":"2018-06-27T22:04:39Z","author_association":"NONE","body":"Jeah found it during tracing. The error is also present in the current released gobuster so it creates a lot of connections to the target slowing things down. If I apply this patch to the current master I also get a ~3 times speedup\r\n\r\n```patch\r\n--- a/libgobuster/dir.go\r\n+++ b/libgobuster/dir.go\r\n@@ -2,6 +2,7 @@ package libgobuster\r\n\r\n import (\r\n        \"fmt\"\r\n+       \"io\"\r\n        \"io/ioutil\"\r\n        \"net/http\"\r\n        \"net/url\"\r\n@@ -93,6 +94,8 @@ func MakeRequest(s *State, fullUrl, cookie string) (*int, *int                                                                                                                                                                                                                                                                                           64) {\r\n                } else {\r\n                        *length = resp.ContentLength\r\n                }\r\n+       } else {\r\n+               io.Copy(ioutil.Discard, resp.Body)\r\n        }\r\n\r\n        return &resp.StatusCode, length\r\n```"}}
{"actor":{"display_login":"FireFart"}}