{"action":"closed","number":159,"pull_request":{"url":"https://api.github.com/repos/storj/storj/pulls/159","id":203097228,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAzMDk3MjI4","html_url":"https://github.com/storj/storj/pull/159","diff_url":"https://github.com/storj/storj/pull/159.diff","patch_url":"https://github.com/storj/storj/pull/159.patch","issue_url":"https://api.github.com/repos/storj/storj/issues/159","number":159,"state":"closed","locked":false,"title":"captplanet","user":{"login":"jtolds","id":109603,"node_id":"MDQ6VXNlcjEwOTYwMw==","avatar_url":"https://avatars0.githubusercontent.com/u/109603?v=4","gravatar_id":"","url":"https://api.github.com/users/jtolds","html_url":"https://github.com/jtolds","followers_url":"https://api.github.com/users/jtolds/followers","following_url":"https://api.github.com/users/jtolds/following{/other_user}","gists_url":"https://api.github.com/users/jtolds/gists{/gist_id}","starred_url":"https://api.github.com/users/jtolds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtolds/subscriptions","organizations_url":"https://api.github.com/users/jtolds/orgs","repos_url":"https://api.github.com/users/jtolds/repos","events_url":"https://api.github.com/users/jtolds/events{/privacy}","received_events_url":"https://api.github.com/users/jtolds/received_events","type":"User","site_admin":false},"body":"I kind of went overboard this weekend.\r\n\r\nThe major goal of this changeset is to provide an environment for local development where all of the various services can be easily run together. Developing on Storj v3 should be as easy as running a setup command and a run command!\r\n\r\nTo do this, this changeset introduces a new tool called captplanet, which combines the powers of the Overlay Cache, the PointerDB, the PieceStore, Kademlia, the Minio Gateway, etc.\r\n\r\nRunning 40 farmers and a heavy client inside the same process forced a rethinking of the \"services\" that we had. To avoid confusion by reusing prior terms, this changeset introduces two new types: Providers and Responsibilities. I wanted to avoid as many merge conflicts as possible, so I left the existing Services and code for now, but if people like this route we can clean up the duplication.\r\n\r\nA Responsibility is a collection of gRPC methods and corresponding state. The following systems are examples of Responsibilities:\r\n * Kademlia\r\n * OverlayCache\r\n * PointerDB\r\n * StatDB\r\n * PieceStore\r\n * etc.\r\n\r\nA Provider is a collection of Responsibilities that share an Identity, such as:\r\n * The heavy client\r\n * The farmer\r\n * The gateway\r\n\r\nAn Identity is a public/private key pair, a node id, etc. Farmers all need different Identities, so captplanet needs to support running multiple concurrent Providers with different Identities.\r\n\r\nEach Responsibility and Provider should allow for configuration of multiple copies on its own so creating Responsibilities and Providers use a new workflow.\r\n\r\nTo make a Responsibility, one should create a \"config\" struct, such as:\r\n\r\n```\r\ntype Config struct {\r\n  RepairThreshold int `help:\"If redundancy falls below this number of\r\npieces, repair is triggered\" default:\"30\"`\r\n  SuccessThreshold int `help:\"If redundancy is above this number then\r\nno additional uploads are needed\" default:\"40\"`\r\n}\r\n```\r\n\r\nTo use \"config\" structs, this changeset introduces another new library called 'cfgstruct', which allows for the configuration of arbitrary structs through flagsets, and thus through cobra and viper.\r\n\r\ncfgstruct relies on Go's \"struct tags\" feature to document help information and default values. Config structs can be configured via cfgstruct.Bind for binding the struct to a flagset.\r\n\r\nBecause this configuration system makes setup and configuration easier *in general*, additional commands are provided that allow for easy standup of separate Providers. Please make sure to\r\ncheck out:\r\n * cmd/captplanet/farmer/main.go (a new farmer binary)\r\n * cmd/captplanet/hc/main.go (a new heavy client binary)\r\n * cmd/captplanet/gw/main.go (a new minio gateway binary)\r\n\r\nUsage:\r\n\r\n```\r\n$ go install -v storj.io/storj/cmd/captplanet\r\n$ captplanet setup\r\n$ captplanet run\r\n```\r\n\r\nConfiguration is placed by default in `~/.storj/capt/`\r\n\r\nOther changes:\r\n\r\n * introduces new config structs for currently existing Responsibilities that conform to the new Responsibility interface. Please see the `pkg/*/config.go` files for examples.\r\n\r\n * integrates the PointerDB API key with other global configuration via flags, instead of through environment variables through viper like it's been doing. (ultimately this should also change to use the PointerDB config struct but this is an okay shortterm solution).\r\n\r\n * changes the Overlay cache to use a URL for database configuration instead of separate redis and bolt config settings.\r\n\r\n * stubs out some peer identity skeleton code (but not the meat).\r\n\r\n * Fixes the SegmentStore to use the overlay client and pointerdb clients instead of gRPC client code directly\r\n\r\n * Leaves a very clear spot where we need to tie the object to stream to segment store together. There's sort of a \"golden spike\" opportunity to connect all the train tracks together at the bottom of pkg/miniogw/config.go, labeled with a bunch of TODOs.\r\n\r\nFuture stuff:\r\n\r\n * I now prefer this design over the original pkg/process.Service thing I had been pushing before (sorry!)\r\n\r\n * The experience of trying to have multiple farmers configurable concurrently led me to prefer config structs over global flags (I finally came around) or using viper directly. I think global flags are okay sometimes but in general going forward we should try and get all relevant config into config structs.\r\n\r\n * If you all like this direction, I think we can go delete my old Service interfaces and a bunch of flags and clean up a bunch of stuff.\r\n\r\n * If you don't like this direction, it's no sweat at all, and despite how much code there is here I'm not very tied to any of this! Considering a lot of this was written between midnight and 6 am, it might not be any good!\n\n<!-- Reviewable:start -->\n---\nThis change is [<img src=\"https://reviewable.io/review_button.svg\" height=\"34\" align=\"absmiddle\" alt=\"Reviewable\"/>](https://reviewable.io/reviews/storj/storj/159)\n<!-- Reviewable:end -->\n","created_at":"2018-07-23T02:54:52Z","updated_at":"2018-07-24T16:08:29Z","closed_at":"2018-07-24T16:08:29Z","merged_at":"2018-07-24T16:08:29Z","merge_commit_sha":"5f6607935b9232c85e8df3b856ee99cb543f54a2","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":982933521,"node_id":"MDU6TGFiZWw5ODI5MzM1MjE=","url":"https://api.github.com/repos/storj/storj/labels/cla-signed","name":"cla-signed","color":"ededed","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/storj/storj/pulls/159/commits","review_comments_url":"https://api.github.com/repos/storj/storj/pulls/159/comments","review_comment_url":"https://api.github.com/repos/storj/storj/pulls/comments{/number}","comments_url":"https://api.github.com/repos/storj/storj/issues/159/comments","statuses_url":"https://api.github.com/repos/storj/storj/statuses/ff05f68764b712995f62cea448d8dd22fefb39ca","head":{"label":"storj:captplanet","ref":"captplanet","sha":"ff05f68764b712995f62cea448d8dd22fefb39ca","user":{"login":"storj","id":6308081,"node_id":"MDEyOk9yZ2FuaXphdGlvbjYzMDgwODE=","avatar_url":"https://avatars3.githubusercontent.com/u/6308081?v=4","gravatar_id":"","url":"https://api.github.com/users/storj","html_url":"https://github.com/storj","followers_url":"https://api.github.com/users/storj/followers","following_url":"https://api.github.com/users/storj/following{/other_user}","gists_url":"https://api.github.com/users/storj/gists{/gist_id}","starred_url":"https://api.github.com/users/storj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/storj/subscriptions","organizations_url":"https://api.github.com/users/storj/orgs","repos_url":"https://api.github.com/users/storj/repos","events_url":"https://api.github.com/users/storj/events{/privacy}","received_events_url":"https://api.github.com/users/storj/received_events","type":"Organization","site_admin":false},"repo":{"id":128089774,"node_id":"MDEwOlJlcG9zaXRvcnkxMjgwODk3NzQ=","name":"storj","full_name":"storj/storj","owner":{"login":"storj","id":6308081,"node_id":"MDEyOk9yZ2FuaXphdGlvbjYzMDgwODE=","avatar_url":"https://avatars3.githubusercontent.com/u/6308081?v=4","gravatar_id":"","url":"https://api.github.com/users/storj","html_url":"https://github.com/storj","followers_url":"https://api.github.com/users/storj/followers","following_url":"https://api.github.com/users/storj/following{/other_user}","gists_url":"https://api.github.com/users/storj/gists{/gist_id}","starred_url":"https://api.github.com/users/storj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/storj/subscriptions","organizations_url":"https://api.github.com/users/storj/orgs","repos_url":"https://api.github.com/users/storj/repos","events_url":"https://api.github.com/users/storj/events{/privacy}","received_events_url":"https://api.github.com/users/storj/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/storj/storj","description":null,"fork":false,"url":"https://api.github.com/repos/storj/storj","forks_url":"https://api.github.com/repos/storj/storj/forks","keys_url":"https://api.github.com/repos/storj/storj/keys{/key_id}","collaborators_url":"https://api.github.com/repos/storj/storj/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/storj/storj/teams","hooks_url":"https://api.github.com/repos/storj/storj/hooks","issue_events_url":"https://api.github.com/repos/storj/storj/issues/events{/number}","events_url":"https://api.github.com/repos/storj/storj/events","assignees_url":"https://api.github.com/repos/storj/storj/assignees{/user}","branches_url":"https://api.github.com/repos/storj/storj/branches{/branch}","tags_url":"https://api.github.com/repos/storj/storj/tags","blobs_url":"https://api.github.com/repos/storj/storj/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/storj/storj/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/storj/storj/git/refs{/sha}","trees_url":"https://api.github.com/repos/storj/storj/git/trees{/sha}","statuses_url":"https://api.github.com/repos/storj/storj/statuses/{sha}","languages_url":"https://api.github.com/repos/storj/storj/languages","stargazers_url":"https://api.github.com/repos/storj/storj/stargazers","contributors_url":"https://api.github.com/repos/storj/storj/contributors","subscribers_url":"https://api.github.com/repos/storj/storj/subscribers","subscription_url":"https://api.github.com/repos/storj/storj/subscription","commits_url":"https://api.github.com/repos/storj/storj/commits{/sha}","git_commits_url":"https://api.github.com/repos/storj/storj/git/commits{/sha}","comments_url":"https://api.github.com/repos/storj/storj/comments{/number}","issue_comment_url":"https://api.github.com/repos/storj/storj/issues/comments{/number}","contents_url":"https://api.github.com/repos/storj/storj/contents/{+path}","compare_url":"https://api.github.com/repos/storj/storj/compare/{base}...{head}","merges_url":"https://api.github.com/repos/storj/storj/merges","archive_url":"https://api.github.com/repos/storj/storj/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/storj/storj/downloads","issues_url":"https://api.github.com/repos/storj/storj/issues{/number}","pulls_url":"https://api.github.com/repos/storj/storj/pulls{/number}","milestones_url":"https://api.github.com/repos/storj/storj/milestones{/number}","notifications_url":"https://api.github.com/repos/storj/storj/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/storj/storj/labels{/name}","releases_url":"https://api.github.com/repos/storj/storj/releases{/id}","deployments_url":"https://api.github.com/repos/storj/storj/deployments","created_at":"2018-04-04T16:27:18Z","updated_at":"2018-07-24T15:21:40Z","pushed_at":"2018-07-24T16:08:29Z","git_url":"git://github.com/storj/storj.git","ssh_url":"git@github.com:storj/storj.git","clone_url":"https://github.com/storj/storj.git","svn_url":"https://github.com/storj/storj","homepage":null,"size":9229,"stargazers_count":88,"watchers_count":88,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":33,"mirror_url":null,"archived":false,"open_issues_count":16,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":33,"open_issues":16,"watchers":88,"default_branch":"master"}},"base":{"label":"storj:master","ref":"master","sha":"445044b8561a40c024e1e085e2ff9a13eaaa1a18","user":{"login":"storj","id":6308081,"node_id":"MDEyOk9yZ2FuaXphdGlvbjYzMDgwODE=","avatar_url":"https://avatars3.githubusercontent.com/u/6308081?v=4","gravatar_id":"","url":"https://api.github.com/users/storj","html_url":"https://github.com/storj","followers_url":"https://api.github.com/users/storj/followers","following_url":"https://api.github.com/users/storj/following{/other_user}","gists_url":"https://api.github.com/users/storj/gists{/gist_id}","starred_url":"https://api.github.com/users/storj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/storj/subscriptions","organizations_url":"https://api.github.com/users/storj/orgs","repos_url":"https://api.github.com/users/storj/repos","events_url":"https://api.github.com/users/storj/events{/privacy}","received_events_url":"https://api.github.com/users/storj/received_events","type":"Organization","site_admin":false},"repo":{"id":128089774,"node_id":"MDEwOlJlcG9zaXRvcnkxMjgwODk3NzQ=","name":"storj","full_name":"storj/storj","owner":{"login":"storj","id":6308081,"node_id":"MDEyOk9yZ2FuaXphdGlvbjYzMDgwODE=","avatar_url":"https://avatars3.githubusercontent.com/u/6308081?v=4","gravatar_id":"","url":"https://api.github.com/users/storj","html_url":"https://github.com/storj","followers_url":"https://api.github.com/users/storj/followers","following_url":"https://api.github.com/users/storj/following{/other_user}","gists_url":"https://api.github.com/users/storj/gists{/gist_id}","starred_url":"https://api.github.com/users/storj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/storj/subscriptions","organizations_url":"https://api.github.com/users/storj/orgs","repos_url":"https://api.github.com/users/storj/repos","events_url":"https://api.github.com/users/storj/events{/privacy}","received_events_url":"https://api.github.com/users/storj/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/storj/storj","description":null,"fork":false,"url":"https://api.github.com/repos/storj/storj","forks_url":"https://api.github.com/repos/storj/storj/forks","keys_url":"https://api.github.com/repos/storj/storj/keys{/key_id}","collaborators_url":"https://api.github.com/repos/storj/storj/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/storj/storj/teams","hooks_url":"https://api.github.com/repos/storj/storj/hooks","issue_events_url":"https://api.github.com/repos/storj/storj/issues/events{/number}","events_url":"https://api.github.com/repos/storj/storj/events","assignees_url":"https://api.github.com/repos/storj/storj/assignees{/user}","branches_url":"https://api.github.com/repos/storj/storj/branches{/branch}","tags_url":"https://api.github.com/repos/storj/storj/tags","blobs_url":"https://api.github.com/repos/storj/storj/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/storj/storj/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/storj/storj/git/refs{/sha}","trees_url":"https://api.github.com/repos/storj/storj/git/trees{/sha}","statuses_url":"https://api.github.com/repos/storj/storj/statuses/{sha}","languages_url":"https://api.github.com/repos/storj/storj/languages","stargazers_url":"https://api.github.com/repos/storj/storj/stargazers","contributors_url":"https://api.github.com/repos/storj/storj/contributors","subscribers_url":"https://api.github.com/repos/storj/storj/subscribers","subscription_url":"https://api.github.com/repos/storj/storj/subscription","commits_url":"https://api.github.com/repos/storj/storj/commits{/sha}","git_commits_url":"https://api.github.com/repos/storj/storj/git/commits{/sha}","comments_url":"https://api.github.com/repos/storj/storj/comments{/number}","issue_comment_url":"https://api.github.com/repos/storj/storj/issues/comments{/number}","contents_url":"https://api.github.com/repos/storj/storj/contents/{+path}","compare_url":"https://api.github.com/repos/storj/storj/compare/{base}...{head}","merges_url":"https://api.github.com/repos/storj/storj/merges","archive_url":"https://api.github.com/repos/storj/storj/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/storj/storj/downloads","issues_url":"https://api.github.com/repos/storj/storj/issues{/number}","pulls_url":"https://api.github.com/repos/storj/storj/pulls{/number}","milestones_url":"https://api.github.com/repos/storj/storj/milestones{/number}","notifications_url":"https://api.github.com/repos/storj/storj/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/storj/storj/labels{/name}","releases_url":"https://api.github.com/repos/storj/storj/releases{/id}","deployments_url":"https://api.github.com/repos/storj/storj/deployments","created_at":"2018-04-04T16:27:18Z","updated_at":"2018-07-24T15:21:40Z","pushed_at":"2018-07-24T16:08:29Z","git_url":"git://github.com/storj/storj.git","ssh_url":"git@github.com:storj/storj.git","clone_url":"https://github.com/storj/storj.git","svn_url":"https://github.com/storj/storj","homepage":null,"size":9229,"stargazers_count":88,"watchers_count":88,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":33,"mirror_url":null,"archived":false,"open_issues_count":16,"license":{"key":"agpl-3.0","name":"GNU Affero General Public License v3.0","spdx_id":"AGPL-3.0","url":"https://api.github.com/licenses/agpl-3.0","node_id":"MDc6TGljZW5zZTE="},"forks":33,"open_issues":16,"watchers":88,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/storj/storj/pulls/159"},"html":{"href":"https://github.com/storj/storj/pull/159"},"issue":{"href":"https://api.github.com/repos/storj/storj/issues/159"},"comments":{"href":"https://api.github.com/repos/storj/storj/issues/159/comments"},"review_comments":{"href":"https://api.github.com/repos/storj/storj/pulls/159/comments"},"review_comment":{"href":"https://api.github.com/repos/storj/storj/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/storj/storj/pulls/159/commits"},"statuses":{"href":"https://api.github.com/repos/storj/storj/statuses/ff05f68764b712995f62cea448d8dd22fefb39ca"}},"author_association":"MEMBER","merged":true,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":{"login":"jtolds","id":109603,"node_id":"MDQ6VXNlcjEwOTYwMw==","avatar_url":"https://avatars0.githubusercontent.com/u/109603?v=4","gravatar_id":"","url":"https://api.github.com/users/jtolds","html_url":"https://github.com/jtolds","followers_url":"https://api.github.com/users/jtolds/followers","following_url":"https://api.github.com/users/jtolds/following{/other_user}","gists_url":"https://api.github.com/users/jtolds/gists{/gist_id}","starred_url":"https://api.github.com/users/jtolds/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jtolds/subscriptions","organizations_url":"https://api.github.com/users/jtolds/orgs","repos_url":"https://api.github.com/users/jtolds/repos","events_url":"https://api.github.com/users/jtolds/events{/privacy}","received_events_url":"https://api.github.com/users/jtolds/received_events","type":"User","site_admin":false},"comments":5,"review_comments":43,"maintainer_can_modify":false,"commits":2,"additions":1558,"deletions":77,"changed_files":38}}
{"actor":{"display_login":"jtolds"}}