{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/21680","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/21680/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/21680/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/21680/events","html_url":"https://github.com/cockroachdb/cockroach/issues/21680","id":290623200,"node_id":"MDU6SXNzdWUyOTA2MjMyMDA=","number":21680,"title":"Build roachprod/fixture based infrastructure for nightly tests","user":{"login":"tschottdorf","id":5076964,"node_id":"MDQ6VXNlcjUwNzY5NjQ=","avatar_url":"https://avatars1.githubusercontent.com/u/5076964?v=4","gravatar_id":"","url":"https://api.github.com/users/tschottdorf","html_url":"https://github.com/tschottdorf","followers_url":"https://api.github.com/users/tschottdorf/followers","following_url":"https://api.github.com/users/tschottdorf/following{/other_user}","gists_url":"https://api.github.com/users/tschottdorf/gists{/gist_id}","starred_url":"https://api.github.com/users/tschottdorf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tschottdorf/subscriptions","organizations_url":"https://api.github.com/users/tschottdorf/orgs","repos_url":"https://api.github.com/users/tschottdorf/repos","events_url":"https://api.github.com/users/tschottdorf/events{/privacy}","received_events_url":"https://api.github.com/users/tschottdorf/received_events","type":"User","site_admin":false},"labels":[{"id":914190256,"node_id":"MDU6TGFiZWw5MTQxOTAyNTY=","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/A-testing","name":"A-testing","color":"37cea3","default":false},{"id":78928347,"node_id":"MDU6TGFiZWw3ODkyODM0Nw==","url":"https://api.github.com/repos/cockroachdb/cockroach/labels/C-enhancement","name":"C-enhancement","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13","html_url":"https://github.com/cockroachdb/cockroach/milestone/13","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/13/labels","id":2745639,"node_id":"MDk6TWlsZXN0b25lMjc0NTYzOQ==","number":13,"title":"2.1","description":"","creator":{"login":"knz","id":642886,"node_id":"MDQ6VXNlcjY0Mjg4Ng==","avatar_url":"https://avatars3.githubusercontent.com/u/642886?v=4","gravatar_id":"","url":"https://api.github.com/users/knz","html_url":"https://github.com/knz","followers_url":"https://api.github.com/users/knz/followers","following_url":"https://api.github.com/users/knz/following{/other_user}","gists_url":"https://api.github.com/users/knz/gists{/gist_id}","starred_url":"https://api.github.com/users/knz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/knz/subscriptions","organizations_url":"https://api.github.com/users/knz/orgs","repos_url":"https://api.github.com/users/knz/repos","events_url":"https://api.github.com/users/knz/events{/privacy}","received_events_url":"https://api.github.com/users/knz/received_events","type":"User","site_admin":false},"open_issues":582,"closed_issues":968,"state":"open","created_at":"2017-09-05T15:30:49Z","updated_at":"2018-07-21T14:20:24Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-01-22T21:34:39Z","updated_at":"2018-07-21T15:18:03Z","closed_at":null,"author_association":"MEMBER","body":"As noticed repeatedly during recent dev cycles, there's a big gap between our unit testing (including the unit acceptance testing) and our manual production testing:\r\n\r\n- unit tests run for short periods of time and exercise lots of variety. They answer the question: do all of this features work in small toy cases?\r\n- production tests run for long periods of time (weeks or months). They are severely limited in variety and basically set up ad-hoc, and as such are hard to maintain. They test very little: most of our clusters run a simple load generator and maybe some cronjob-driven chaos testing. They want to answer the question \"do all of these features work in realistic settings?\" but fail to do so, and in particular require a human to look at graphs and manually determine problems (though there is an amount of reasonable monitoring).\r\n\r\nThe proposition here is to build a maintainable bit of infrastructure that gives us the flexibility to run a variety of workloads and operations on real clusters for any amount of time (it's expected that standard durations are on the order of hours).\r\n\r\nWe built [such infrastructure (via terraform)](https://github.com/cockroachdb/cockroach/tree/master/pkg/acceptance/terrafarm) before and it is used in some nightly tests:\r\n https://github.com/cockroachdb/cockroach/blob/8c0f19014167787ab14e7bd526afbf9caa70d46e/pkg/acceptance/continuous_load_test.go#L85\r\n\r\nSee also [our teamcity config](https://teamcity.cockroachdb.com/project.html?projectId=Cockroach_Nightlies&tab=projectOverview).\r\n\r\nTerrafarm has some shortcomings that are problematic. It uses terraform, which is a complicated beast that is too general for what we need, and has been a barrier of entry. It was forced into the existing acceptance tests which force the provisioned system into a rigid cage that isn't flexible enough to comfortably write the kinds of versatile tests that we need in the future. It's been unpleasant to use it for long-running tests, as it requires a long-running test runner. Additionally, it's a huge hassle to run these tests manually, or even more ambitiously, to run their logic against an experimental cluster.\r\n\r\nOn the other hand, we have the tools https://github.com/cockroachlabs/roachprod and https://github.com/cockroachdb/roachperf. The former sets up clusters (i.e. roughly what terraform did, except it doesn't put any binaries anywhere) and the latter was conceived to run benchmarks against clusters created by the former via an incantation of `roachperf <clustername> test <testname>` (after having run another `roachperf` incantation that deploys binaries, and starts and configures the cluster).\r\n\r\nThese tools are useful and if you squint, you can see how to run tests via\r\n\r\n```\r\nroachprod create x\r\nroachperf put x ./cockroach\r\n# either (use the local machine as test runner)\r\nroachperf test x mytest\r\n# or (using one of the nodes as test runner)\r\nroachprod ssh x 1 \"roachperf test x mytest --background\"\r\n```\r\n\r\nWe should work towards making this a reality and port the existing terraform tests over to this new infrastructure (which in the process needs to be hashed out).\r\n\r\nSome items and ideas to keep in mind here are:\r\n\r\n1. A bias for less abstractions. These tests currently run against GCE and that is fine for now, but in the medium term we'll also want to run against AWS. We'll likely end up with a small subset of tests that tie somewhat deeply into the specific system used (imagine testing file system corruption or having to install certain packages to generate i/o pressure on the hard drives), but it's preferable to let these tests figure out the differences at first rather than trying to build abstractions into the test harness too early. (Go templates could go a long way in removing duplication here).\r\n\r\n2. mixed version testing (https://airtable.com/tblD3oZPLJgGhCmch/viwOj1Qb7D4gjtemc/recom67pixgSeyDIw). It's desirable to have tests that can work against mixed version clusters do so. This is somewhat in opposition to 1) which suggests leaving it up to the test to place the binaries on the nodes. My suggestion would be to make sure that the terraform-ported tests (which can surely run against mixed version clusters) come in a flavor that does so, and that guidelines are put into place that suggest doing so for all appropriate tests forward.\r\n\r\n3. Preference for SQL over cli commands. We are in the process of making sure that all `cli` functionality is available through SQL. Tests should almost never run random `./cockroach X` incantations on the remote servers.\r\n\r\n4. Nemeses. Some nemeses will require an upfront setup, for example the toxiproxy network nemesis, or a disk corruption nemesis. Others can be started when desired (a crash nemesis, for example, is just a `kill -9` via ssh and a later restart). In keeping with 1), an initial preference for not baking these nemeses into the infrastructure is suggested.\r\n\r\n5. Crash protection. Tests should generally be able to tell when nodes crashed half-way through. An easy way to do so is to periodically measure the uptime of the nodes (note that when nemeses are running, this must be relaxed as crashes are expected). Again, this should be developed as an auxiliary mechanism a test can chose to start (and by default does). Baking such a mechanism deep into the acceptance test framework in our existing infrastructure was unfortunate since it turned quite complicated when chaos was added.\r\n\r\n6. Integration with https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/workload (in the sense, that this tool is readily available to tests and is the go-to staple for running load and loading fixtures). This is a tool that creates and caches large data sets. Getting large data sets during tests was historically painful; it is no more."},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/406803256","html_url":"https://github.com/cockroachdb/cockroach/issues/21680#issuecomment-406803256","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/21680","id":406803256,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNjgwMzI1Ng==","user":{"login":"petermattis","id":6201335,"node_id":"MDQ6VXNlcjYyMDEzMzU=","avatar_url":"https://avatars0.githubusercontent.com/u/6201335?v=4","gravatar_id":"","url":"https://api.github.com/users/petermattis","html_url":"https://github.com/petermattis","followers_url":"https://api.github.com/users/petermattis/followers","following_url":"https://api.github.com/users/petermattis/following{/other_user}","gists_url":"https://api.github.com/users/petermattis/gists{/gist_id}","starred_url":"https://api.github.com/users/petermattis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/petermattis/subscriptions","organizations_url":"https://api.github.com/users/petermattis/orgs","repos_url":"https://api.github.com/users/petermattis/repos","events_url":"https://api.github.com/users/petermattis/events{/privacy}","received_events_url":"https://api.github.com/users/petermattis/received_events","type":"User","site_admin":false},"created_at":"2018-07-21T15:18:03Z","updated_at":"2018-07-21T15:18:03Z","author_association":"CONTRIBUTOR","body":"@tschottdorf Ping. I think we should close this issue and rely on more specific issues regarding enhancing our nightly test infrastructure."}}
{"actor":{"display_login":"petermattis"}}