{"action":"created","comment":{"url":"https://api.github.com/repos/deis/workflow-e2e/pulls/comments/57750685","id":57750685,"diff_hunk":"@@ -381,20 +304,101 @@ func cmdWithRetry(cmd Cmd, expectedResult string, timeout int) bool {\n \t\t}\n \t\ttime.Sleep(1 * time.Second)\n \t}\n-\tfmt.Printf(\"FAIL: '%s' does not match expected result of '%s'\\n\", result, expectedResult)\n+\tfmt.Fprintf(GinkgoWriter, \"FAIL: '%s' does not match expected result of '%s'\\n\", result, expectedResult)\n \treturn false\n }\n \n // gitInit simply invokes 'git init' and verifies the command is successful\n func gitInit() {\n-\tcmd, err := start(\"git init\")\n+\tcmd, err := start(\"git init\", \"\")\n \tExpect(err).NotTo(HaveOccurred())\n \tEventually(cmd).Should(Say(\"Initialized empty Git repository\"))\n }\n \n // gitClean destroys the .git directory and verifies the command is successful\n func gitClean() {\n-\tcmd, err := start(\"rm -rf .git\")\n+\tcmd, err := start(\"rm -rf .git\", \"\")\n \tExpect(err).NotTo(HaveOccurred())\n \tEventually(cmd).Should(Exit(0))\n }\n+\n+func registerHTTP(url string, username string, password string, email string) string {\n+\tregisterURL := fmt.Sprintf(\"%s/v2/auth/register/\", url)","path":"tests/tests_suite_test.go","position":null,"original_position":367,"commit_id":"922bbf3ec895cd2398c97e6a07d6c55524b96a75","original_commit_id":"4fa9c73d344a6c49c768df97ff05fbd3dcc9e2a6","user":{"login":"jchauncey","id":430360,"avatar_url":"https://avatars.githubusercontent.com/u/430360?v=3","gravatar_id":"","url":"https://api.github.com/users/jchauncey","html_url":"https://github.com/jchauncey","followers_url":"https://api.github.com/users/jchauncey/followers","following_url":"https://api.github.com/users/jchauncey/following{/other_user}","gists_url":"https://api.github.com/users/jchauncey/gists{/gist_id}","starred_url":"https://api.github.com/users/jchauncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchauncey/subscriptions","organizations_url":"https://api.github.com/users/jchauncey/orgs","repos_url":"https://api.github.com/users/jchauncey/repos","events_url":"https://api.github.com/users/jchauncey/events{/privacy}","received_events_url":"https://api.github.com/users/jchauncey/received_events","type":"User","site_admin":false},"body":"k pushed this change","created_at":"2016-03-29T16:07:35Z","updated_at":"2016-03-29T16:07:35Z","html_url":"https://github.com/deis/workflow-e2e/pull/135#discussion_r57750685","pull_request_url":"https://api.github.com/repos/deis/workflow-e2e/pulls/135","_links":{"self":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/comments/57750685"},"html":{"href":"https://github.com/deis/workflow-e2e/pull/135#discussion_r57750685"},"pull_request":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/135"}}},"pull_request":{"url":"https://api.github.com/repos/deis/workflow-e2e/pulls/135","id":64348736,"html_url":"https://github.com/deis/workflow-e2e/pull/135","diff_url":"https://github.com/deis/workflow-e2e/pull/135.diff","patch_url":"https://github.com/deis/workflow-e2e/pull/135.patch","issue_url":"https://api.github.com/repos/deis/workflow-e2e/issues/135","number":135,"state":"open","locked":false,"title":"feature: Run the e2e test suite in parallel","user":{"login":"jchauncey","id":430360,"avatar_url":"https://avatars.githubusercontent.com/u/430360?v=3","gravatar_id":"","url":"https://api.github.com/users/jchauncey","html_url":"https://github.com/jchauncey","followers_url":"https://api.github.com/users/jchauncey/followers","following_url":"https://api.github.com/users/jchauncey/following{/other_user}","gists_url":"https://api.github.com/users/jchauncey/gists{/gist_id}","starred_url":"https://api.github.com/users/jchauncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchauncey/subscriptions","organizations_url":"https://api.github.com/users/jchauncey/orgs","repos_url":"https://api.github.com/users/jchauncey/repos","events_url":"https://api.github.com/users/jchauncey/events{/privacy}","received_events_url":"https://api.github.com/users/jchauncey/received_events","type":"User","site_admin":false},"body":"This PR gives us the ability to run the e2e test suite in parallel. This dramatically speeds up the test run making it easier on the end user who is trying to get a crucial change. For example here is a sequential test run performed against my 4 node m4.xlarge cluster -\r\n\r\n```\r\nRan 100 of 113 Specs in 2003.951 seconds\r\nSUCCESS! -- 100 Passed | 0 Failed | 13 Pending | 0 Skipped PASS\r\n```\r\n\r\nThat test run took almost 33 minutes to complete.\r\n\r\nBut when running the tests in parallel we can see results like the following:\r\n\r\n```\r\nRan 100 of 113 Specs in 367.949 seconds\r\nSUCCESS! -- 100 Passed | 0 Failed | 13 Pending | 0 Skipped\r\n```\r\nThat is just over 6 minutes.\r\n\r\nTo accomplish this several changes were made to the test suite:\r\n\r\n1. Use http to register and login all users. This allows us to store the login token in a `user.json` file within the `$TEST_HOME` directory. \r\n\r\n1. Use `DEIS_PROFILE` to ensure we use the correct user for each action. When running tests in parallel we cannot be certain that the right fork/exec of the deis cli command is using the correct logged in user. \r\n\r\n1. Removed (commented out) the reset all user tokens test. So this test was wreaking havoc on my test runs because it basically blanked out all login tokens I had generated up until that point. This meant other tests that were inflight would suddenly start getting 401's. Until we can figure out a better way to test this I suggest we leave it commented out.\r\n\r\n1.  Made most tests use the `Eventually` matcher. This is more of a consistency thing than for the parallel tests.\r\n\r\n1. Removed lots of clean up code. Because we reset the cluster after each run and each test generates its own data we no longer need to cleanup after ourselves. The one notable exception to this is the domains/certs tests as I wasnt sure if we needed to clean up between tests for them to behave properly. However, I am pretty sure that I can remove that as well @helgi, @mboersma?\r\n\r\n1.  `defaultMaxTimeout` is now at 600 seconds and `DefaultEventuallyTimeout` is 60 seconds. For most of last week I was dealing with timeouts when running these tests in parallel. I eventually found out that my cluster was mostly to blame. That being said I found having a longer timeout eased some of the failures I was seeing.\r\n\r\n1. Use `fmt.Fprintf(GinkgoWriter...)`. Because we are running tests in parallel we can no longer use the standard `fmt.Printf` or `fmt.Println` calls because the output is jumbled. The `GinkgoWriter` is setup to handle parallel test runs.","created_at":"2016-03-28T17:42:17Z","updated_at":"2016-03-29T16:07:35Z","closed_at":null,"merged_at":null,"merge_commit_sha":"27f99472daa86600b64617b333fa5f033e3c04fc","assignee":{"login":"jchauncey","id":430360,"avatar_url":"https://avatars.githubusercontent.com/u/430360?v=3","gravatar_id":"","url":"https://api.github.com/users/jchauncey","html_url":"https://github.com/jchauncey","followers_url":"https://api.github.com/users/jchauncey/followers","following_url":"https://api.github.com/users/jchauncey/following{/other_user}","gists_url":"https://api.github.com/users/jchauncey/gists{/gist_id}","starred_url":"https://api.github.com/users/jchauncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchauncey/subscriptions","organizations_url":"https://api.github.com/users/jchauncey/orgs","repos_url":"https://api.github.com/users/jchauncey/repos","events_url":"https://api.github.com/users/jchauncey/events{/privacy}","received_events_url":"https://api.github.com/users/jchauncey/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/deis/workflow-e2e/milestones/3","html_url":"https://github.com/deis/workflow-e2e/milestones/v2.0-rc1","labels_url":"https://api.github.com/repos/deis/workflow-e2e/milestones/3/labels","id":1598101,"number":3,"title":"v2.0-rc1","description":null,"creator":{"login":"technosophos","id":89193,"avatar_url":"https://avatars.githubusercontent.com/u/89193?v=3","gravatar_id":"","url":"https://api.github.com/users/technosophos","html_url":"https://github.com/technosophos","followers_url":"https://api.github.com/users/technosophos/followers","following_url":"https://api.github.com/users/technosophos/following{/other_user}","gists_url":"https://api.github.com/users/technosophos/gists{/gist_id}","starred_url":"https://api.github.com/users/technosophos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/technosophos/subscriptions","organizations_url":"https://api.github.com/users/technosophos/orgs","repos_url":"https://api.github.com/users/technosophos/repos","events_url":"https://api.github.com/users/technosophos/events{/privacy}","received_events_url":"https://api.github.com/users/technosophos/received_events","type":"User","site_admin":false},"open_issues":14,"closed_issues":4,"state":"open","created_at":"2016-02-22T20:27:50Z","updated_at":"2016-03-29T14:38:08Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/deis/workflow-e2e/pulls/135/commits","review_comments_url":"https://api.github.com/repos/deis/workflow-e2e/pulls/135/comments","review_comment_url":"https://api.github.com/repos/deis/workflow-e2e/pulls/comments{/number}","comments_url":"https://api.github.com/repos/deis/workflow-e2e/issues/135/comments","statuses_url":"https://api.github.com/repos/deis/workflow-e2e/statuses/922bbf3ec895cd2398c97e6a07d6c55524b96a75","head":{"label":"jchauncey:random-user","ref":"random-user","sha":"922bbf3ec895cd2398c97e6a07d6c55524b96a75","user":{"login":"jchauncey","id":430360,"avatar_url":"https://avatars.githubusercontent.com/u/430360?v=3","gravatar_id":"","url":"https://api.github.com/users/jchauncey","html_url":"https://github.com/jchauncey","followers_url":"https://api.github.com/users/jchauncey/followers","following_url":"https://api.github.com/users/jchauncey/following{/other_user}","gists_url":"https://api.github.com/users/jchauncey/gists{/gist_id}","starred_url":"https://api.github.com/users/jchauncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchauncey/subscriptions","organizations_url":"https://api.github.com/users/jchauncey/orgs","repos_url":"https://api.github.com/users/jchauncey/repos","events_url":"https://api.github.com/users/jchauncey/events{/privacy}","received_events_url":"https://api.github.com/users/jchauncey/received_events","type":"User","site_admin":false},"repo":{"id":52101171,"name":"workflow-e2e","full_name":"jchauncey/workflow-e2e","owner":{"login":"jchauncey","id":430360,"avatar_url":"https://avatars.githubusercontent.com/u/430360?v=3","gravatar_id":"","url":"https://api.github.com/users/jchauncey","html_url":"https://github.com/jchauncey","followers_url":"https://api.github.com/users/jchauncey/followers","following_url":"https://api.github.com/users/jchauncey/following{/other_user}","gists_url":"https://api.github.com/users/jchauncey/gists{/gist_id}","starred_url":"https://api.github.com/users/jchauncey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchauncey/subscriptions","organizations_url":"https://api.github.com/users/jchauncey/orgs","repos_url":"https://api.github.com/users/jchauncey/repos","events_url":"https://api.github.com/users/jchauncey/events{/privacy}","received_events_url":"https://api.github.com/users/jchauncey/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jchauncey/workflow-e2e","description":"End-to-end tests for Deis Workflow","fork":true,"url":"https://api.github.com/repos/jchauncey/workflow-e2e","forks_url":"https://api.github.com/repos/jchauncey/workflow-e2e/forks","keys_url":"https://api.github.com/repos/jchauncey/workflow-e2e/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jchauncey/workflow-e2e/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jchauncey/workflow-e2e/teams","hooks_url":"https://api.github.com/repos/jchauncey/workflow-e2e/hooks","issue_events_url":"https://api.github.com/repos/jchauncey/workflow-e2e/issues/events{/number}","events_url":"https://api.github.com/repos/jchauncey/workflow-e2e/events","assignees_url":"https://api.github.com/repos/jchauncey/workflow-e2e/assignees{/user}","branches_url":"https://api.github.com/repos/jchauncey/workflow-e2e/branches{/branch}","tags_url":"https://api.github.com/repos/jchauncey/workflow-e2e/tags","blobs_url":"https://api.github.com/repos/jchauncey/workflow-e2e/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jchauncey/workflow-e2e/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jchauncey/workflow-e2e/git/refs{/sha}","trees_url":"https://api.github.com/repos/jchauncey/workflow-e2e/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jchauncey/workflow-e2e/statuses/{sha}","languages_url":"https://api.github.com/repos/jchauncey/workflow-e2e/languages","stargazers_url":"https://api.github.com/repos/jchauncey/workflow-e2e/stargazers","contributors_url":"https://api.github.com/repos/jchauncey/workflow-e2e/contributors","subscribers_url":"https://api.github.com/repos/jchauncey/workflow-e2e/subscribers","subscription_url":"https://api.github.com/repos/jchauncey/workflow-e2e/subscription","commits_url":"https://api.github.com/repos/jchauncey/workflow-e2e/commits{/sha}","git_commits_url":"https://api.github.com/repos/jchauncey/workflow-e2e/git/commits{/sha}","comments_url":"https://api.github.com/repos/jchauncey/workflow-e2e/comments{/number}","issue_comment_url":"https://api.github.com/repos/jchauncey/workflow-e2e/issues/comments{/number}","contents_url":"https://api.github.com/repos/jchauncey/workflow-e2e/contents/{+path}","compare_url":"https://api.github.com/repos/jchauncey/workflow-e2e/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jchauncey/workflow-e2e/merges","archive_url":"https://api.github.com/repos/jchauncey/workflow-e2e/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jchauncey/workflow-e2e/downloads","issues_url":"https://api.github.com/repos/jchauncey/workflow-e2e/issues{/number}","pulls_url":"https://api.github.com/repos/jchauncey/workflow-e2e/pulls{/number}","milestones_url":"https://api.github.com/repos/jchauncey/workflow-e2e/milestones{/number}","notifications_url":"https://api.github.com/repos/jchauncey/workflow-e2e/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jchauncey/workflow-e2e/labels{/name}","releases_url":"https://api.github.com/repos/jchauncey/workflow-e2e/releases{/id}","deployments_url":"https://api.github.com/repos/jchauncey/workflow-e2e/deployments","created_at":"2016-02-19T16:33:41Z","updated_at":"2016-02-19T16:33:42Z","pushed_at":"2016-03-29T16:03:34Z","git_url":"git://github.com/jchauncey/workflow-e2e.git","ssh_url":"git@github.com:jchauncey/workflow-e2e.git","clone_url":"https://github.com/jchauncey/workflow-e2e.git","svn_url":"https://github.com/jchauncey/workflow-e2e","homepage":null,"size":18086,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"deis:master","ref":"master","sha":"33f133df5be7ab3ff34a76d71d14a8a4dbefee7d","user":{"login":"deis","id":6117407,"avatar_url":"https://avatars.githubusercontent.com/u/6117407?v=3","gravatar_id":"","url":"https://api.github.com/users/deis","html_url":"https://github.com/deis","followers_url":"https://api.github.com/users/deis/followers","following_url":"https://api.github.com/users/deis/following{/other_user}","gists_url":"https://api.github.com/users/deis/gists{/gist_id}","starred_url":"https://api.github.com/users/deis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deis/subscriptions","organizations_url":"https://api.github.com/users/deis/orgs","repos_url":"https://api.github.com/users/deis/repos","events_url":"https://api.github.com/users/deis/events{/privacy}","received_events_url":"https://api.github.com/users/deis/received_events","type":"Organization","site_admin":false},"repo":{"id":48503688,"name":"workflow-e2e","full_name":"deis/workflow-e2e","owner":{"login":"deis","id":6117407,"avatar_url":"https://avatars.githubusercontent.com/u/6117407?v=3","gravatar_id":"","url":"https://api.github.com/users/deis","html_url":"https://github.com/deis","followers_url":"https://api.github.com/users/deis/followers","following_url":"https://api.github.com/users/deis/following{/other_user}","gists_url":"https://api.github.com/users/deis/gists{/gist_id}","starred_url":"https://api.github.com/users/deis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deis/subscriptions","organizations_url":"https://api.github.com/users/deis/orgs","repos_url":"https://api.github.com/users/deis/repos","events_url":"https://api.github.com/users/deis/events{/privacy}","received_events_url":"https://api.github.com/users/deis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/deis/workflow-e2e","description":"End-to-end tests for Deis Workflow","fork":false,"url":"https://api.github.com/repos/deis/workflow-e2e","forks_url":"https://api.github.com/repos/deis/workflow-e2e/forks","keys_url":"https://api.github.com/repos/deis/workflow-e2e/keys{/key_id}","collaborators_url":"https://api.github.com/repos/deis/workflow-e2e/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/deis/workflow-e2e/teams","hooks_url":"https://api.github.com/repos/deis/workflow-e2e/hooks","issue_events_url":"https://api.github.com/repos/deis/workflow-e2e/issues/events{/number}","events_url":"https://api.github.com/repos/deis/workflow-e2e/events","assignees_url":"https://api.github.com/repos/deis/workflow-e2e/assignees{/user}","branches_url":"https://api.github.com/repos/deis/workflow-e2e/branches{/branch}","tags_url":"https://api.github.com/repos/deis/workflow-e2e/tags","blobs_url":"https://api.github.com/repos/deis/workflow-e2e/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/deis/workflow-e2e/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/deis/workflow-e2e/git/refs{/sha}","trees_url":"https://api.github.com/repos/deis/workflow-e2e/git/trees{/sha}","statuses_url":"https://api.github.com/repos/deis/workflow-e2e/statuses/{sha}","languages_url":"https://api.github.com/repos/deis/workflow-e2e/languages","stargazers_url":"https://api.github.com/repos/deis/workflow-e2e/stargazers","contributors_url":"https://api.github.com/repos/deis/workflow-e2e/contributors","subscribers_url":"https://api.github.com/repos/deis/workflow-e2e/subscribers","subscription_url":"https://api.github.com/repos/deis/workflow-e2e/subscription","commits_url":"https://api.github.com/repos/deis/workflow-e2e/commits{/sha}","git_commits_url":"https://api.github.com/repos/deis/workflow-e2e/git/commits{/sha}","comments_url":"https://api.github.com/repos/deis/workflow-e2e/comments{/number}","issue_comment_url":"https://api.github.com/repos/deis/workflow-e2e/issues/comments{/number}","contents_url":"https://api.github.com/repos/deis/workflow-e2e/contents/{+path}","compare_url":"https://api.github.com/repos/deis/workflow-e2e/compare/{base}...{head}","merges_url":"https://api.github.com/repos/deis/workflow-e2e/merges","archive_url":"https://api.github.com/repos/deis/workflow-e2e/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/deis/workflow-e2e/downloads","issues_url":"https://api.github.com/repos/deis/workflow-e2e/issues{/number}","pulls_url":"https://api.github.com/repos/deis/workflow-e2e/pulls{/number}","milestones_url":"https://api.github.com/repos/deis/workflow-e2e/milestones{/number}","notifications_url":"https://api.github.com/repos/deis/workflow-e2e/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/deis/workflow-e2e/labels{/name}","releases_url":"https://api.github.com/repos/deis/workflow-e2e/releases{/id}","deployments_url":"https://api.github.com/repos/deis/workflow-e2e/deployments","created_at":"2015-12-23T17:50:37Z","updated_at":"2016-03-24T21:04:41Z","pushed_at":"2016-03-29T16:03:35Z","git_url":"git://github.com/deis/workflow-e2e.git","ssh_url":"git@github.com:deis/workflow-e2e.git","clone_url":"https://github.com/deis/workflow-e2e.git","svn_url":"https://github.com/deis/workflow-e2e","homepage":null,"size":18112,"stargazers_count":4,"watchers_count":4,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13,"mirror_url":null,"open_issues_count":33,"forks":13,"open_issues":33,"watchers":4,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/135"},"html":{"href":"https://github.com/deis/workflow-e2e/pull/135"},"issue":{"href":"https://api.github.com/repos/deis/workflow-e2e/issues/135"},"comments":{"href":"https://api.github.com/repos/deis/workflow-e2e/issues/135/comments"},"review_comments":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/135/comments"},"review_comment":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/deis/workflow-e2e/pulls/135/commits"},"statuses":{"href":"https://api.github.com/repos/deis/workflow-e2e/statuses/922bbf3ec895cd2398c97e6a07d6c55524b96a75"}}}}