{"push_id":2771759997,"size":19,"distinct_size":11,"ref":"refs/heads/dev_minio_unittests","head":"4d18af4c2d32d55d9554db64aa06845c90abce59","before":"bdbe8eada3c8504ef087a32f1d85b736e18d40ed","commits":[{"sha":"85eee948145a9f64fcc14ba0c82c7a8f65cee4e2","author":{"name":"Kaloyan Raev","email":"8309452d65536b05642541a30b309d395f9a6d5c@users.noreply.github.com"},"message":"Fixes V3-173 - ECClient check if given nodes are unique (#186)","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/85eee948145a9f64fcc14ba0c82c7a8f65cee4e2"},{"sha":"f6fb41c5cc5e684308e586114588f0aed02fa7db","author":{"name":"Yaroslav Vorobiov","email":"12954da0b6a115181905567e4f555b0d1824de09@storj.io"},"message":"Testcoverage kademlia (#154)\n\n* Unit test covarege increased for kademlia pkg\r\n\r\ngo style formatting added\r\n\r\nRemoved DHT param from newTestKademlia method, added comments for Bucket methods that informs that these tests will need to be updated\r\n\r\nunnecessary comment deleted from newTestKademlia\r\n\r\nAdjust Segment Store to the updated interface (#160)\r\n\r\n* Adjust Segment Store to the updated interface\r\n\r\n* Move /pkg/storage/segment to /pkg/storage/segments\r\n\r\n* Fix overlay client tests\r\n\r\n* Revert changes in NewOverlayClient return value\r\n\r\n* Rename `rem` to `seg`\r\n\r\n* Implement Meta()\r\n\r\ncaptplanet (#159)\r\n\r\n* captplanet\r\n\r\nI kind of went overboard this weekend.\r\n\r\nThe major goal of this changeset is to provide an environment\r\nfor local development where all of the various services can\r\nbe easily run together. Developing on Storj v3 should be as\r\neasy as running a setup command and a run command!\r\n\r\nTo do this, this changeset introduces a new tool called\r\ncaptplanet, which combines the powers of the Overlay Cache,\r\nthe PointerDB, the PieceStore, Kademlia, the Minio Gateway,\r\netc.\r\n\r\nRunning 40 farmers and a heavy client inside the same process\r\nforced a rethinking of the \"services\" that we had. To\r\navoid confusion by reusing prior terms, this changeset\r\nintroduces two new types: Providers and Responsibilities.\r\nI wanted to avoid as many merge conflicts as possible, so\r\nI left the existing Services and code for now, but if people\r\nlike this route we can clean up the duplication.\r\n\r\nA Responsibility is a collection of gRPC methods and\r\ncorresponding state. The following systems are examples of\r\nResponsibilities:\r\n * Kademlia\r\n * OverlayCache\r\n * PointerDB\r\n * StatDB\r\n * PieceStore\r\n * etc.\r\n\r\nA Provider is a collection of Responsibilities that\r\nshare an Identity, such as:\r\n * The heavy client\r\n * The farmer\r\n * The gateway\r\n\r\nAn Identity is a public/private key pair, a node id, etc.\r\nFarmers all need different Identities, so captplanet\r\nneeds to support running multiple concurrent Providers\r\nwith different Identities.\r\n\r\nEach Responsibility and Provider should allow for configuration\r\nof multiple copies on its own so creating Responsibilities and\r\nProviders use a new workflow.\r\n\r\nTo make a Responsibility, one should create a \"config\"\r\nstruct, such as:\r\n\r\n```\r\ntype Config struct {\r\n  RepairThreshold int `help:\"If redundancy falls below this number of\r\npieces, repair is triggered\" default:\"30\"`\r\n  SuccessThreshold int `help:\"If redundancy is above this number then\r\nno additional uploads are needed\" default:\"40\"`\r\n}\r\n```\r\n\r\nTo use \"config\" structs, this changeset introduces another\r\nnew library called 'cfgstruct', which allows for the configuration\r\nof arbitrary structs through flagsets, and thus through cobra and\r\nviper.\r\n\r\ncfgstruct relies on Go's \"struct tags\" feature to document\r\nhelp information and default values. Config structs can be\r\nconfigured via cfgstruct.Bind for binding the struct to a flagset.\r\n\r\nBecause this configuration system makes setup and configuration\r\neasier *in general*, additional commands are provided that allow\r\nfor easy standup of separate Providers. Please make sure to\r\ncheck out:\r\n * cmd/captplanet/farmer/main.go (a new farmer binary)\r\n * cmd/captplanet/hc/main.go (a new heavy client binary)\r\n * cmd/captplanet/gw/main.go (a new minio gateway binary)\r\n\r\nUsage:\r\n\r\n```\r\n$ go install -v storj.io/storj/cmd/captplanet\r\n$ captplanet setup\r\n$ captplanet run\r\n```\r\n\r\nConfiguration is placed by default in `~/.storj/capt/`\r\n\r\nOther changes:\r\n\r\n * introduces new config structs for currently existing\r\n   Responsibilities that conform to the new Responsibility\r\n   interface. Please see the `pkg/*/config.go` files for\r\n   examples.\r\n\r\n * integrates the PointerDB API key with other global\r\n   configuration via flags, instead of through environment\r\n   variables through viper like it's been doing. (ultimately\r\n   this should also change to use the PointerDB config\r\n   struct but this is an okay shortterm solution).\r\n\r\n * changes the Overlay cache to use a URL for database\r\n   configuration instead of separate redis and bolt config\r\n   settings.\r\n\r\n * stubs out some peer identity skeleton code (but not the\r\n   meat).\r\n\r\n * Fixes the SegmentStore to use the overlay client and\r\n   pointerdb clients instead of gRPC client code directly\r\n\r\n * Leaves a very clear spot where we need to tie the object to\r\n   stream to segment store together. There's sort of a \"golden\r\n   spike\" opportunity to connect all the train tracks together\r\n   at the bottom of pkg/miniogw/config.go, labeled with a\r\n   bunch of TODOs.\r\n\r\nFuture stuff:\r\n\r\n * I now prefer this design over the original\r\n   pkg/process.Service thing I had been pushing before (sorry!)\r\n\r\n * The experience of trying to have multiple farmers\r\n   configurable concurrently led me to prefer config structs\r\n   over global flags (I finally came around) or using viper\r\n   directly. I think global flags are okay sometimes but in\r\n   general going forward we should try and get all relevant\r\n   config into config structs.\r\n\r\n * If you all like this direction, I think we can go delete my\r\n   old Service interfaces and a bunch of flags and clean up a\r\n   bunch of stuff.\r\n\r\n * If you don't like this direction, it's no sweat at all, and\r\n   despite how much code there is here I'm not very tied to any\r\n   of this! Considering a lot of this was written between midnight\r\n   and 6 am, it might not be any good!\r\n\r\n* bind tests\r\n\r\nAdd files for testing builds in docker (#161)\r\n\r\n* Add files for testing builds in docker\r\n\r\n* Make tests check for redis running before trying to start redis-server, which may not exist.\r\n\r\n* Clean redis server before any tests use it.\r\n\r\n* Add more debugging for travis\r\n\r\n* Explicitly requiring redis for travis\r\n\r\npkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)\r\n\r\n* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services\r\n\r\n* add setup to gw binary too\r\n\r\n* captplanet: output what addresses everything is listening on\r\n\r\n* revert peertls/io_util changes\r\n\r\n* define config flag across all commands\r\n\r\n* use trimsuffix\r\n\r\nfix docker makefile (#170)\r\n\r\n* fix makefile\r\n\r\nprotos: update protobufs with go generate (#169)\r\n\r\nthe import for timestamp and duration should use\r\nthe path provided by a standard protocol buffer library\r\ninstallation\r\n\r\nRefactor List in PointerDB (#163)\r\n\r\n* Refactor List in Pointer DB\r\n\r\n* Fix pointerdb-client example\r\n\r\n* Fix issue in Path type related to empty paths\r\n\r\n* Test for the PointerDB service with some fixes\r\n\r\n* Fixed debug message in example: trancated --> more\r\n\r\n* GoDoc comments for unexported methods\r\n\r\n* TODO comment to check if Put is overwriting\r\n\r\n* Log warning if protobuf timestamp cannot be converted\r\n\r\n* TODO comment to make ListPageLimit configurable\r\n\r\n* Rename 'segment' package to 'segments' to reflect folder name\r\n\r\nMinio integration with Object store (#156)\r\n\r\n* initial WIP integration with Object store\r\n\r\n* List WIP\r\n\r\n* minio listobject function changes complete\r\n\r\n* Code review changes and work in progress for the mock objectstore unit testing cases\r\n\r\n* Warning fix redeclaration of err\r\n\r\n* Warning fix redeclaration of err\r\n\r\n*  code review comments & unit testing inprogress\r\n\r\n* fix compilation bug\r\n\r\n* Fixed code review comments & added GetObject Mock test case\r\n\r\n* rearraged the mock test file and gateway storj test file in to the proper directory\r\n\r\n* added the missing file\r\n\r\n* code clean up\r\n\r\n* fix lint error on the mock generated code\r\n\r\n* modified per code review comments\r\n\r\n* added the PutObject mock test case\r\n\r\n* added the GetObjectInfo  mock test case\r\n\r\n* added listobject mock test case\r\n\r\n* fixed package from storj to miniogw\r\n\r\n* resolved the gateway-storj.go initialization merge conflict\r\n\r\nupdate readme (#174)\r\n\r\nadded assertion for unused errors (#152)\r\n\r\nmerging this PR to avoid future issues\r\n\r\nupdating github user to personal account (#171)\r\n\r\nTest coverage ranger (#168)\r\n\r\n* Fixed go panic for corner case\r\n\r\n* Initial test coverage for ranger pkg\r\n\r\nstreamstore: add passthrough implementation (#176)\r\n\r\nthis doesn't implement streamstore, this just allows us to try and\r\nget the june demo working again in the meantime\r\n\r\nStatDB (#144)\r\n\r\n* add statdb proto and example client\r\n\r\n* server logic\r\n\r\n* update readme\r\n\r\n* remove boltdb from service.go\r\n\r\n* sqlite3\r\n\r\n* add statdb server executable file\r\n\r\n* create statdb node table if it does not exist already\r\n\r\n* get UpdateBatch working\r\n\r\n* update based on jt review\r\n\r\n* remove some commented lines\r\n\r\n* fix linting issues\r\n\r\n* reformat\r\n\r\n* apiKey -> APIKey\r\n\r\n* update statdb client apiKey->APIKey\r\n\r\nUpdate README.md\r\n\r\nUpdate README.md\r\n\r\noverlay: correct dockerfile db (#179)\r\n\r\ncmd/hc, cmd/gw, cmd/captplanet: simplify setup/run commands (#178)\r\n\r\nalso allows much more customization of services within captain planet,\r\nsuch as reconfiguring the overlay service to use redis\r\n\r\npkg/process: don't require json formatting (#177)\r\n\r\nCleanup metadata across layers (#180)\r\n\r\n* Cleanup metadata across layers\r\n\r\n* Fix pointer db tests\r\n\r\nKademlia Routing Table (#164)\r\n\r\n* adds comment\r\n\r\n* runs deps\r\n\r\n* creates boltdb kademlia routing table\r\n\r\n* protobuf updates\r\n\r\n* adds reverselist to mockkeyvaluestore interface\r\n\r\n* xor wip\r\n\r\n* xor wip\r\n\r\n* fixes xor sort\r\n\r\n* runs go fmt\r\n\r\n* fixes\r\n\r\n* goimports again\r\n\r\n* trying to fix travis tests\r\n\r\n* fixes mock tests\r\n\r\nRanger refactoring (#158)\r\n\r\n* Fixed go panic for corner case\r\n\r\n* Cosmetic changes, and small error fixes\r\n\r\nminiogw: log all errors (#182)\r\n\r\n* miniogw: log all errors\r\n\r\n* tests added\r\n\r\n* doc comment to satisfy linter\r\n\r\n* fix test failure\r\n\r\nJennifer added to CLA list\r\n\r\n* Temporary fix for storage/redis list method test","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/f6fb41c5cc5e684308e586114588f0aed02fa7db"},{"sha":"d934733c4dfa89dfd76ed1aeaa00fd1c92eb2f19","author":{"name":"Maximillian von Briesen","email":"3300903d5d47eaed9eab80cefb2c11cf7d476dcb@gmail.com"},"message":"Move api key from pdb client method calls to pdb client constructor (#194)\n\n* move api key from pdb client method calls to pdb client constructor\r\n\r\n* update pdb example client","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/d934733c4dfa89dfd76ed1aeaa00fd1c92eb2f19"},{"sha":"30e3b503a49f84a58b134bf77460a98bf9192f8c","author":{"name":"JT Olio","email":"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d@jtolio.com"},"message":"captplanet: configure api key (#195)","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/30e3b503a49f84a58b134bf77460a98bf9192f8c"},{"sha":"5170057c7f7c31005b430f3869385e19b7685bcf","author":{"name":"JT Olio","email":"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d@jtolio.com"},"message":"log grpc server errors (#196)","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/5170057c7f7c31005b430f3869385e19b7685bcf"},{"sha":"e66a0d289a72c767f8c6ef9e3a253666d653c7ce","author":{"name":"JT Olio","email":"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d@jtolio.com"},"message":"miniogw: make a Config.NewGateway helper, bypass overlay for now (#197)","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/e66a0d289a72c767f8c6ef9e3a253666d653c7ce"},{"sha":"7348bf1e09640f49755e1419e077ac1c8ddaab89","author":{"name":"Dennis Coyle","email":"c856b7fbc4b2d24f601c64270065fe3937aaef25@users.noreply.github.com"},"message":"Adjustments to how find storage nodes handles limit/amout requested (#192)\n\n* adjusted based on feedback from @jtolds and @kaloyan-raev\r\n\r\n* generated protobuf\r\n\r\n* fix crash\r\n\r\n* WIP\r\n\r\n* cleanup\r\n\r\n* v >>found\r\n\r\n* pointer\r\n\r\n* test\r\n\r\n* copyrights + cleanup\r\n\r\n* fix tests","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/7348bf1e09640f49755e1419e077ac1c8ddaab89"},{"sha":"c9c2a93c31bbb0fb54023c8ffd5edc16a1fddb65","author":{"name":"JT Olio","email":"aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d@jtolio.com"},"message":"Update README.md","distinct":false,"url":"https://api.github.com/repos/storj/storj/commits/c9c2a93c31bbb0fb54023c8ffd5edc16a1fddb65"},{"sha":"0d5c87e739ed169a371736884b086d3bfb047f24","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"added byteranger unit test cases to test minio.GetObject","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/0d5c87e739ed169a371736884b086d3bfb047f24"},{"sha":"b711d19d1671c1e27215c26af0f473994826d89f","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"added TestPutObject testcases","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/b711d19d1671c1e27215c26af0f473994826d89f"},{"sha":"77f76717b4515a8698c0385cf841c3aa9597fa3a","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"added TestGetOjectInfo testcases","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/77f76717b4515a8698c0385cf841c3aa9597fa3a"},{"sha":"3e86dda172de5c8d73d0722ecd7a08e4010907b4","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"added TestGetOjectInfo testcases","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/3e86dda172de5c8d73d0722ecd7a08e4010907b4"},{"sha":"395647bb937bcb850d60282c7b5ca026519f9806","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"added TestListInfo testcases","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/395647bb937bcb850d60282c7b5ca026519f9806"},{"sha":"e1e27869bf0063cac243ff093afe53955a0d8707","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"TestGetObject() refined and added different set of combinations","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/e1e27869bf0063cac243ff093afe53955a0d8707"},{"sha":"f89a1ec295797a1c6cc4a9aa36e63c0988050a47","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"TestPutObject test case refined and added more test scenarios","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/f89a1ec295797a1c6cc4a9aa36e63c0988050a47"},{"sha":"7c4e27be243fd3ff0b9c98414a893191df1efb03","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"TestGetObjectInfo test case refined and added more test scenarios","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/7c4e27be243fd3ff0b9c98414a893191df1efb03"},{"sha":"ce6c1842746d9cb23e3a72b0ea8b7df5eed2faeb","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":" TestListObject test case refned and added more test scenarios","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/ce6c1842746d9cb23e3a72b0ea8b7df5eed2faeb"},{"sha":"8cb0434b480e4da2175327daa374bea12cdd4b0d","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"modified to the best of the code review comments","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/8cb0434b480e4da2175327daa374bea12cdd4b0d"},{"sha":"4d18af4c2d32d55d9554db64aa06845c90abce59","author":{"name":"kishore","email":"7e0f32d5d98ee70646f8f945e725680bf7ae474d@storj.io"},"message":"code review comments","distinct":true,"url":"https://api.github.com/repos/storj/storj/commits/4d18af4c2d32d55d9554db64aa06845c90abce59"}]}
{"actor":{"display_login":"aligeti"}}