{"action":"created","comment":{"url":"https://api.github.com/repos/qri-io/walk/pulls/comments/230603048","pull_request_review_id":171351165,"id":230603048,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzMDYwMzA0OA==","diff_hunk":"@@ -36,10 +50,24 @@ func DefaultConfig() *Config {\n \t\t\t},\n \t\t},\n \t\tResourceHandlers: []*ResourceHandlerConfig{},\n+\t\tBadger:           NewBadgerConfig(),\n \t\t// DestPath: \"sitemap.json\",\n \t}\n }\n \n+// ErrNoBadgerConfig is the result of attempting to connect to a badgerDB\n+// without one configured\n+var ErrNoBadgerConfig = fmt.Errorf(\"badger is not configured\")\n+\n+// BadgerDB returns the badger DB connection, creating a default-configured\n+// badger store if one doesn't exist\n+func (cfg *Config) BadgerDB() (*badger.DB, error) {\n+\tif cfg.Badger == nil {\n+\t\treturn nil, ErrNoBadgerConfig","path":"lib/config.go","position":49,"original_position":49,"commit_id":"ee74adeba9f616018952ba7ce3c599e9b5a34b53","original_commit_id":"ee74adeba9f616018952ba7ce3c599e9b5a34b53","user":{"login":"Mr0grog","id":74178,"node_id":"MDQ6VXNlcjc0MTc4","avatar_url":"https://avatars0.githubusercontent.com/u/74178?v=4","gravatar_id":"","url":"https://api.github.com/users/Mr0grog","html_url":"https://github.com/Mr0grog","followers_url":"https://api.github.com/users/Mr0grog/followers","following_url":"https://api.github.com/users/Mr0grog/following{/other_user}","gists_url":"https://api.github.com/users/Mr0grog/gists{/gist_id}","starred_url":"https://api.github.com/users/Mr0grog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Mr0grog/subscriptions","organizations_url":"https://api.github.com/users/Mr0grog/orgs","repos_url":"https://api.github.com/users/Mr0grog/repos","events_url":"https://api.github.com/users/Mr0grog/events{/privacy}","received_events_url":"https://api.github.com/users/Mr0grog/received_events","type":"User","site_admin":false},"body":"In a previous version of this code, we *created* a new Badger config here. I think this change is mostly reasonable except it exposes an issue I hadn’t noticed before: `JSONConfigFromFilepath` completely overwrites instead of adding to the existing config. https://github.com/qri-io/walk/blob/e13838a20af9a993969e689c7d581846df265f18/lib/config.go#L43-L59\r\n\r\nBecause of that, it took me a bit of hunting to realize I now need to add  a `Badger` section to my config file… and that that section needs more than just the `Dir` and `ValueDir` properties. It doesn’t seem to match up with what the [Badger docs say is required](https://godoc.org/github.com/dgraph-io/badger#Options), so a after a couple false starts, I just copied the entirety of `badger.DefaultOptions` in there:\r\n\r\n```js\r\n{\r\n    \"Coordinator\": /* ... */,\r\n    \"Queue\": /* ... */,\r\n    \"RequestStore\": /* ... */,\r\n    \"Workers\": /* ... */,\r\n    \"ResourceHandlers\": /* ... */,\r\n    \"Badger\": {\r\n      \"Dir\": \"badger\",\r\n      \"ValueDir\": \"badger\",\r\n      \"ValueLogFileSize\": 1073741823,\r\n      \"DoNotCompact\":        false,\r\n      \"LevelOneSize\":        268435456,\r\n      \"LevelSizeMultiplier\": 10,\r\n      \"TableLoadingMode\":    1,  // options.LoadToRAM, not so great to reference an enum/constant this way\r\n      \"ValueLogLoadingMode\": 2,  // options.MemoryMap, not so great to reference an enum/constant this way\r\n\r\n      \"MaxLevels\":               7,\r\n      \"MaxTableSize\":            67108864,\r\n      \"NumCompactors\":           3,\r\n      \"NumLevelZeroTables\":      5,\r\n      \"NumLevelZeroTablesStall\": 10,\r\n      \"NumMemtables\":            5,\r\n      \"SyncWrites\":              true,\r\n      \"NumVersionsToKeep\":       1,\r\n\r\n      \"ValueLogMaxEntries\": 1000000,\r\n      \"ValueThreshold\":     32,\r\n      \"Truncate\":           false\r\n    }\r\n}\r\n```\r\n\r\nAnyway, all that seems pretty painful. (Also: was there an easier way for me to have done this?) I think it’s probably `JSONConfigFromFilepath` that should change and the code you have here now should stay, but not really certain about that.\r\n\r\nBarring any other changes, it seems like we should at least document this in an example config file.","created_at":"2018-11-04T20:07:46Z","updated_at":"2018-11-04T20:42:28Z","html_url":"https://github.com/qri-io/walk/pull/13#discussion_r230603048","pull_request_url":"https://api.github.com/repos/qri-io/walk/pulls/13","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/qri-io/walk/pulls/comments/230603048"},"html":{"href":"https://github.com/qri-io/walk/pull/13#discussion_r230603048"},"pull_request":{"href":"https://api.github.com/repos/qri-io/walk/pulls/13"}}},"pull_request":{"url":"https://api.github.com/repos/qri-io/walk/pulls/13","id":209370114,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA5MzcwMTE0","html_url":"https://github.com/qri-io/walk/pull/13","diff_url":"https://github.com/qri-io/walk/pull/13.diff","patch_url":"https://github.com/qri-io/walk/pull/13.patch","issue_url":"https://api.github.com/repos/qri-io/walk/issues/13","number":13,"state":"open","locked":false,"title":"WIP: Sitemap","user":{"login":"b5","id":1154390,"node_id":"MDQ6VXNlcjExNTQzOTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1154390?v=4","gravatar_id":"","url":"https://api.github.com/users/b5","html_url":"https://github.com/b5","followers_url":"https://api.github.com/users/b5/followers","following_url":"https://api.github.com/users/b5/following{/other_user}","gists_url":"https://api.github.com/users/b5/gists{/gist_id}","starred_url":"https://api.github.com/users/b5/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/b5/subscriptions","organizations_url":"https://api.github.com/users/b5/orgs","repos_url":"https://api.github.com/users/b5/repos","events_url":"https://api.github.com/users/b5/events{/privacy}","received_events_url":"https://api.github.com/users/b5/received_events","type":"User","site_admin":false},"body":"Should get us ready to test our first milestone 😄.\r\n\r\nEasiest way to play with this one is via the restored `walk sitemap` command, which accepts a configuration and generates a sitemap.json file whenever you `SIGKILL` the process (control-C).\r\n\r\nThis PR introduces [badger](https://github.com/dgraph-io/badger) as a key-value store that sitemap can rely on to store requests that can be built up as Resources come in, and output the final sitemap by iterating keys with a matching prefix to generate the sitemap itself. This Approach also makes versioning different sitemap crawls by varying up the prefix a near-term possibility.\r\n\r\nBased on previous work with sitemap building & massive amounts of RAM consumption, I feel like Badger will provide a _lot_ of mileage for this project moving forward. This PR will eventually include a Badger implementation of the `RequestStore` interface.","created_at":"2018-08-19T17:17:18Z","updated_at":"2018-11-04T20:42:28Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6e133329b34dd150d32dd4f4ee97c2a375d86545","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":1021439039,"node_id":"MDU6TGFiZWwxMDIxNDM5MDM5","url":"https://api.github.com/repos/qri-io/walk/labels/feat","name":"feat","color":"9a4dd6","default":false},{"id":1021440259,"node_id":"MDU6TGFiZWwxMDIxNDQwMjU5","url":"https://api.github.com/repos/qri-io/walk/labels/test","name":"test","color":"9bffa5","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/qri-io/walk/pulls/13/commits","review_comments_url":"https://api.github.com/repos/qri-io/walk/pulls/13/comments","review_comment_url":"https://api.github.com/repos/qri-io/walk/pulls/comments{/number}","comments_url":"https://api.github.com/repos/qri-io/walk/issues/13/comments","statuses_url":"https://api.github.com/repos/qri-io/walk/statuses/ee74adeba9f616018952ba7ce3c599e9b5a34b53","head":{"label":"qri-io:sitemap","ref":"sitemap","sha":"ee74adeba9f616018952ba7ce3c599e9b5a34b53","user":{"login":"qri-io","id":19268065,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjY4MDY1","avatar_url":"https://avatars1.githubusercontent.com/u/19268065?v=4","gravatar_id":"","url":"https://api.github.com/users/qri-io","html_url":"https://github.com/qri-io","followers_url":"https://api.github.com/users/qri-io/followers","following_url":"https://api.github.com/users/qri-io/following{/other_user}","gists_url":"https://api.github.com/users/qri-io/gists{/gist_id}","starred_url":"https://api.github.com/users/qri-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qri-io/subscriptions","organizations_url":"https://api.github.com/users/qri-io/orgs","repos_url":"https://api.github.com/users/qri-io/repos","events_url":"https://api.github.com/users/qri-io/events{/privacy}","received_events_url":"https://api.github.com/users/qri-io/received_events","type":"Organization","site_admin":false},"repo":{"id":143492898,"node_id":"MDEwOlJlcG9zaXRvcnkxNDM0OTI4OTg=","name":"walk","full_name":"qri-io/walk","private":false,"owner":{"login":"qri-io","id":19268065,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjY4MDY1","avatar_url":"https://avatars1.githubusercontent.com/u/19268065?v=4","gravatar_id":"","url":"https://api.github.com/users/qri-io","html_url":"https://github.com/qri-io","followers_url":"https://api.github.com/users/qri-io/followers","following_url":"https://api.github.com/users/qri-io/following{/other_user}","gists_url":"https://api.github.com/users/qri-io/gists{/gist_id}","starred_url":"https://api.github.com/users/qri-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qri-io/subscriptions","organizations_url":"https://api.github.com/users/qri-io/orgs","repos_url":"https://api.github.com/users/qri-io/repos","events_url":"https://api.github.com/users/qri-io/events{/privacy}","received_events_url":"https://api.github.com/users/qri-io/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/qri-io/walk","description":null,"fork":false,"url":"https://api.github.com/repos/qri-io/walk","forks_url":"https://api.github.com/repos/qri-io/walk/forks","keys_url":"https://api.github.com/repos/qri-io/walk/keys{/key_id}","collaborators_url":"https://api.github.com/repos/qri-io/walk/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/qri-io/walk/teams","hooks_url":"https://api.github.com/repos/qri-io/walk/hooks","issue_events_url":"https://api.github.com/repos/qri-io/walk/issues/events{/number}","events_url":"https://api.github.com/repos/qri-io/walk/events","assignees_url":"https://api.github.com/repos/qri-io/walk/assignees{/user}","branches_url":"https://api.github.com/repos/qri-io/walk/branches{/branch}","tags_url":"https://api.github.com/repos/qri-io/walk/tags","blobs_url":"https://api.github.com/repos/qri-io/walk/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/qri-io/walk/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/qri-io/walk/git/refs{/sha}","trees_url":"https://api.github.com/repos/qri-io/walk/git/trees{/sha}","statuses_url":"https://api.github.com/repos/qri-io/walk/statuses/{sha}","languages_url":"https://api.github.com/repos/qri-io/walk/languages","stargazers_url":"https://api.github.com/repos/qri-io/walk/stargazers","contributors_url":"https://api.github.com/repos/qri-io/walk/contributors","subscribers_url":"https://api.github.com/repos/qri-io/walk/subscribers","subscription_url":"https://api.github.com/repos/qri-io/walk/subscription","commits_url":"https://api.github.com/repos/qri-io/walk/commits{/sha}","git_commits_url":"https://api.github.com/repos/qri-io/walk/git/commits{/sha}","comments_url":"https://api.github.com/repos/qri-io/walk/comments{/number}","issue_comment_url":"https://api.github.com/repos/qri-io/walk/issues/comments{/number}","contents_url":"https://api.github.com/repos/qri-io/walk/contents/{+path}","compare_url":"https://api.github.com/repos/qri-io/walk/compare/{base}...{head}","merges_url":"https://api.github.com/repos/qri-io/walk/merges","archive_url":"https://api.github.com/repos/qri-io/walk/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/qri-io/walk/downloads","issues_url":"https://api.github.com/repos/qri-io/walk/issues{/number}","pulls_url":"https://api.github.com/repos/qri-io/walk/pulls{/number}","milestones_url":"https://api.github.com/repos/qri-io/walk/milestones{/number}","notifications_url":"https://api.github.com/repos/qri-io/walk/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/qri-io/walk/labels{/name}","releases_url":"https://api.github.com/repos/qri-io/walk/releases{/id}","deployments_url":"https://api.github.com/repos/qri-io/walk/deployments","created_at":"2018-08-04T02:59:33Z","updated_at":"2018-11-02T16:28:52Z","pushed_at":"2018-11-02T19:52:01Z","git_url":"git://github.com/qri-io/walk.git","ssh_url":"git@github.com:qri-io/walk.git","clone_url":"https://github.com/qri-io/walk.git","svn_url":"https://github.com/qri-io/walk","homepage":null,"size":9092,"stargazers_count":1,"watchers_count":1,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":10,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":1,"open_issues":10,"watchers":1,"default_branch":"master"}},"base":{"label":"qri-io:master","ref":"master","sha":"5e8f3fe5e9da69d249b2c58a3a788c6722452dc5","user":{"login":"qri-io","id":19268065,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjY4MDY1","avatar_url":"https://avatars1.githubusercontent.com/u/19268065?v=4","gravatar_id":"","url":"https://api.github.com/users/qri-io","html_url":"https://github.com/qri-io","followers_url":"https://api.github.com/users/qri-io/followers","following_url":"https://api.github.com/users/qri-io/following{/other_user}","gists_url":"https://api.github.com/users/qri-io/gists{/gist_id}","starred_url":"https://api.github.com/users/qri-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qri-io/subscriptions","organizations_url":"https://api.github.com/users/qri-io/orgs","repos_url":"https://api.github.com/users/qri-io/repos","events_url":"https://api.github.com/users/qri-io/events{/privacy}","received_events_url":"https://api.github.com/users/qri-io/received_events","type":"Organization","site_admin":false},"repo":{"id":143492898,"node_id":"MDEwOlJlcG9zaXRvcnkxNDM0OTI4OTg=","name":"walk","full_name":"qri-io/walk","private":false,"owner":{"login":"qri-io","id":19268065,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE5MjY4MDY1","avatar_url":"https://avatars1.githubusercontent.com/u/19268065?v=4","gravatar_id":"","url":"https://api.github.com/users/qri-io","html_url":"https://github.com/qri-io","followers_url":"https://api.github.com/users/qri-io/followers","following_url":"https://api.github.com/users/qri-io/following{/other_user}","gists_url":"https://api.github.com/users/qri-io/gists{/gist_id}","starred_url":"https://api.github.com/users/qri-io/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/qri-io/subscriptions","organizations_url":"https://api.github.com/users/qri-io/orgs","repos_url":"https://api.github.com/users/qri-io/repos","events_url":"https://api.github.com/users/qri-io/events{/privacy}","received_events_url":"https://api.github.com/users/qri-io/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/qri-io/walk","description":null,"fork":false,"url":"https://api.github.com/repos/qri-io/walk","forks_url":"https://api.github.com/repos/qri-io/walk/forks","keys_url":"https://api.github.com/repos/qri-io/walk/keys{/key_id}","collaborators_url":"https://api.github.com/repos/qri-io/walk/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/qri-io/walk/teams","hooks_url":"https://api.github.com/repos/qri-io/walk/hooks","issue_events_url":"https://api.github.com/repos/qri-io/walk/issues/events{/number}","events_url":"https://api.github.com/repos/qri-io/walk/events","assignees_url":"https://api.github.com/repos/qri-io/walk/assignees{/user}","branches_url":"https://api.github.com/repos/qri-io/walk/branches{/branch}","tags_url":"https://api.github.com/repos/qri-io/walk/tags","blobs_url":"https://api.github.com/repos/qri-io/walk/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/qri-io/walk/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/qri-io/walk/git/refs{/sha}","trees_url":"https://api.github.com/repos/qri-io/walk/git/trees{/sha}","statuses_url":"https://api.github.com/repos/qri-io/walk/statuses/{sha}","languages_url":"https://api.github.com/repos/qri-io/walk/languages","stargazers_url":"https://api.github.com/repos/qri-io/walk/stargazers","contributors_url":"https://api.github.com/repos/qri-io/walk/contributors","subscribers_url":"https://api.github.com/repos/qri-io/walk/subscribers","subscription_url":"https://api.github.com/repos/qri-io/walk/subscription","commits_url":"https://api.github.com/repos/qri-io/walk/commits{/sha}","git_commits_url":"https://api.github.com/repos/qri-io/walk/git/commits{/sha}","comments_url":"https://api.github.com/repos/qri-io/walk/comments{/number}","issue_comment_url":"https://api.github.com/repos/qri-io/walk/issues/comments{/number}","contents_url":"https://api.github.com/repos/qri-io/walk/contents/{+path}","compare_url":"https://api.github.com/repos/qri-io/walk/compare/{base}...{head}","merges_url":"https://api.github.com/repos/qri-io/walk/merges","archive_url":"https://api.github.com/repos/qri-io/walk/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/qri-io/walk/downloads","issues_url":"https://api.github.com/repos/qri-io/walk/issues{/number}","pulls_url":"https://api.github.com/repos/qri-io/walk/pulls{/number}","milestones_url":"https://api.github.com/repos/qri-io/walk/milestones{/number}","notifications_url":"https://api.github.com/repos/qri-io/walk/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/qri-io/walk/labels{/name}","releases_url":"https://api.github.com/repos/qri-io/walk/releases{/id}","deployments_url":"https://api.github.com/repos/qri-io/walk/deployments","created_at":"2018-08-04T02:59:33Z","updated_at":"2018-11-02T16:28:52Z","pushed_at":"2018-11-02T19:52:01Z","git_url":"git://github.com/qri-io/walk.git","ssh_url":"git@github.com:qri-io/walk.git","clone_url":"https://github.com/qri-io/walk.git","svn_url":"https://github.com/qri-io/walk","homepage":null,"size":9092,"stargazers_count":1,"watchers_count":1,"language":"HTML","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":10,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0","node_id":"MDc6TGljZW5zZTk="},"forks":1,"open_issues":10,"watchers":1,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/qri-io/walk/pulls/13"},"html":{"href":"https://github.com/qri-io/walk/pull/13"},"issue":{"href":"https://api.github.com/repos/qri-io/walk/issues/13"},"comments":{"href":"https://api.github.com/repos/qri-io/walk/issues/13/comments"},"review_comments":{"href":"https://api.github.com/repos/qri-io/walk/pulls/13/comments"},"review_comment":{"href":"https://api.github.com/repos/qri-io/walk/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/qri-io/walk/pulls/13/commits"},"statuses":{"href":"https://api.github.com/repos/qri-io/walk/statuses/ee74adeba9f616018952ba7ce3c599e9b5a34b53"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"Mr0grog"}}