{"action":"created","issue":{"url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/13","repository_url":"https://api.github.com/repos/bradfitz/go-issue-mirror","labels_url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/13/labels{/name}","comments_url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/13/comments","events_url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/13/events","html_url":"https://github.com/bradfitz/go-issue-mirror/issues/13","id":184741233,"number":13,"title":"Hint go tool not to look inside issues folder for Go packages.","user":{"login":"shurcooL","id":1924134,"avatar_url":"https://avatars.githubusercontent.com/u/1924134?v=3","gravatar_id":"","url":"https://api.github.com/users/shurcooL","html_url":"https://github.com/shurcooL","followers_url":"https://api.github.com/users/shurcooL/followers","following_url":"https://api.github.com/users/shurcooL/following{/other_user}","gists_url":"https://api.github.com/users/shurcooL/gists{/gist_id}","starred_url":"https://api.github.com/users/shurcooL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shurcooL/subscriptions","organizations_url":"https://api.github.com/users/shurcooL/orgs","repos_url":"https://api.github.com/users/shurcooL/repos","events_url":"https://api.github.com/users/shurcooL/events{/privacy}","received_events_url":"https://api.github.com/users/shurcooL/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-24T03:24:30Z","updated_at":"2016-10-24T17:22:13Z","closed_at":null,"body":"_(I'm reporting this issue, but no PR because it's not feasible to make a PR to resolve this. Only you, @bradfitz, the owner of this repo, can resolve this. It's easy.)_\r\n\r\n### Problem\r\n\r\nHaving this repository inside my GOPATH workspace makes operations such as `go list all` or `go list .../foo/bar` much, much slower, needlessly.\r\n\r\nI have a moderate/large amount of Go packages in my GOPATH (around 2000), and I have an SSD. With `go-issue-mirror` repository as is, doing `go list all` takes 8-10 seconds:\r\n\r\n```\r\n$ time go list all\r\nreal\t0m9.400s\r\n$ time go list all\r\nreal\t0m7.993s\r\n$ time go list all\r\nreal\t0m10.523s\r\n```\r\n\r\nWith a simple change I propose below in \"Solution\" section, that time can be restored to a much more reasonable 2.5~ seconds.\r\n\r\n`go list all` is just an example. It also affects queries such as `.../foo/bar`, which I use very often to navigate to specific Go packages.\r\n\r\n### Cause\r\n\r\nThe `cmd/go` command scans the GOPATH workspace(s) each time to find Go packages. It goes over all folders, and checks if they have .go files (i.e. it checks if [`build.ImportDir`](https://godoc.org/go/build#Context.ImportDir) successfully finds a package).\r\n\r\nThe [issues directory](https://github.com/bradfitz/go-issue-mirror/tree/master/issues) in this repository contains a single Go package, but it also has 1000 directories, each with many more directories and files.\r\n\r\nWe as humans know it's a waste of time looking for Go packages inside, but the `cmd/go` tool doesn't. It needs a hint.\r\n\r\n### Solution\r\n\r\nLuckily, it [supports](https://golang.org/cmd/go/#hdr-Description_of_package_lists) such hints:\r\n\r\n> Directory and file names that begin with \".\" or \"_\" are ignored by the go tool, as are directories named \"testdata\".\r\n\r\nUsing \".\" is not a good idea because it'll make the folder hidden. \"testdata\" isn't a great fit since this is just real \"data\", not \"data for tests\". But you can prepend the folder with an underscore, and `cmd/go` will know to skip it and not waste time scanning all those directories/files.\r\n\r\nI usually name such folders `_data` (it's like testdata, except not for tests, and starts with `_`). E.g.:\r\n\r\n- https://github.com/shurcooL/issuesapp/tree/master/assets/_data\r\n- https://github.com/shurcooL/notificationsapp/tree/master/assets/_data\r\n- https://github.com/shurcooL/home/tree/master/assets/_data\r\n\r\nSo, if I move all of issues subfolders into a directory named `_data`, then it becomes 2.5 seconds:\r\n\r\n```\r\n$ time go list all\r\nreal\t0m2.646s\r\n$ time go list all\r\nreal\t0m2.736s\r\n$ time go list all\r\nreal\t0m2.591s\r\n```\r\n\r\nI know you use `github.com/bradfitz/go-issue-mirror` import path to find the issues \"data\". You can move all that issues data into `_data` subfolder:\r\n\r\n```\r\n.\r\n├── README.md\r\n├── cmd\r\n│   └── servegoissues\r\n│       └── servegoissues.go\r\n└── issues\r\n    ├── issues.go\r\n    └── _data\r\n        ├── 000\r\n        │   ├── 1000.comments\r\n        │   │   ├── comment-66052206.json\r\n        │   │   ├── comment-66052207.json\r\n        │   │   ├── comment-66052208.json\r\n        │   │   ├── comment-66052209.json\r\n        │   │   ├── comment-66052210.json\r\n        │   │   └── comment-66052211.json\r\n        │   ├── 1000.json\r\n        │   ├── 10000.comments\r\n        │   │   ├── comment-76099228.json\r\n        │   │   ├── comment-76835959.json\r\n        │   │   ├── comment-82634364.json\r\n        │   │   └── comment-82636153.json\r\n        │   ├── 10000.json\r\n        │   ├── 11000.comments\r\n        │   │   ├── comment-107078804.json\r\n```\r\n\r\nAnd keep `issues.go` inside `github.com/bradfitz/go-issue-mirror/issues` as is, just modify [issues.go#L18](https://github.com/bradfitz/go-issue-mirror/blob/92a8d6d27c9ca0b2439036ec86e01194a5679aa3/issues/issues.go#L18) to point to the new subdirectory.\r\n\r\nAnyway, how exactly you resolve this is up to you, but please fix it... I am sad when Go is unnecessarily slow. :("},"comment":{"url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/comments/255806286","html_url":"https://github.com/bradfitz/go-issue-mirror/issues/13#issuecomment-255806286","issue_url":"https://api.github.com/repos/bradfitz/go-issue-mirror/issues/13","id":255806286,"user":{"login":"bradfitz","id":2621,"avatar_url":"https://avatars.githubusercontent.com/u/2621?v=3","gravatar_id":"","url":"https://api.github.com/users/bradfitz","html_url":"https://github.com/bradfitz","followers_url":"https://api.github.com/users/bradfitz/followers","following_url":"https://api.github.com/users/bradfitz/following{/other_user}","gists_url":"https://api.github.com/users/bradfitz/gists{/gist_id}","starred_url":"https://api.github.com/users/bradfitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradfitz/subscriptions","organizations_url":"https://api.github.com/users/bradfitz/orgs","repos_url":"https://api.github.com/users/bradfitz/repos","events_url":"https://api.github.com/users/bradfitz/events{/privacy}","received_events_url":"https://api.github.com/users/bradfitz/received_events","type":"User","site_admin":false},"created_at":"2016-10-24T17:22:13Z","updated_at":"2016-10-24T17:22:13Z","body":"SGTM.\r\n\r\nThis is especially bad with on OS X with its shitty VFS.\r\n\r\nI get:\r\n```\r\nbradfitz@laptop ~$ time go list all\r\n....\r\nreal\t1m19.141s\r\nuser\t0m6.622s\r\nsys\t0m6.329s\r\n```\r\n\r\nAnd on the second run:\r\n\r\n```\r\nreal\t0m20.448s\r\nuser\t0m5.820s\r\nsys\t0m4.794s\r\n```\r\n\r\n(Where Linux is 5 seconds on the second run.)\r\n\r\nBut both 5 secondsand 22 seconds are both slow.\r\n\r\n"}}
{"actor":{"display_login":"bradfitz"}}