{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/51336558","id":51336558,"diff_hunk":"@@ -112,19 +112,58 @@ We are working on implementing clearer partitioning of our e2e tests to make run\n - `[Disruptive]`: If a test restarts components that might cause other tests to fail or break the cluster completely, it is labeled `[Disruptive]`.  Any `[Disruptive]` test is also assumed to qualify for the `[Serial]` label, but need not be labeled as both.  These tests are not run against soak clusters to avoid restarting components.\n - `[Flaky]`: If a test is found to be flaky, it receives the `[Flaky]` label until it is fixed.  A `[Flaky]` label should be accompanied with a reference to the issue for de-flaking the test, because while a test remains labeled `[Flaky]`, it is not monitored closely in CI. `[Flaky]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.\n - `[Skipped]`: `[Skipped]` is a legacy label that we're phasing out.  If a test is marked `[Skipped]`, there should be an issue open to label it properly.  `[Skipped]` tests are by default not run, unless a `focus` or `skip` argument is explicitly given.\n-- `[Feature:...]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:...]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`.  `[Feature:...]` tests are not run in our core suites, instead running in custom suites.  There are a few use-cases for `[Feature:...]` tests:\n-  - If a feature is experimental or alpha and is not enabled by default due to being incomplete or potentially subject to breaking changes, it should *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s).\n-  - If a feature is in beta or GA, it *should* block the merge-queue.  In moving from experimental to beta or GA, tests that are expected to pass by default should simply remove the `[Feature:...]` label, and will be incorporated into our core suites.  If tests are not expected to pass by default, (e.g. they require a special environment such as added quota,) they should remain with the `[Feature:...]` label, and the suites that run them should be incorporated into our merge-queue, owned by the Build Cop.\n+- `[Feature:.+]`: If a test has non-default requirements to run or targets some non-core functionality, and thus should not be run as part of the standard suite, it receives a `[Feature:.+]` label, e.g. `[Feature:Performance]` or `[Feature:Ingress]`.  `[Feature:.+]` tests are not run in our core suites, instead running in custom suites.  There are a few use-cases for `[Feature:.+]` tests:\n+  - If a feature is experimental or alpha and is not enabled by default due to being incomplete or potentially subject to breaking changes, it does *not* block the merge-queue, and thus should run in some separate test suites owned by the feature owner(s) (see #continuous_integration below).\n \n Finally, `[Conformance]` tests are tests we expect to pass on **any** Kubernetes cluster.  The `[Conformance]` label does not supersede any other labels.  `[Conformance]` test policies are a work-in-progress; see #18162.\n \n-## Adding a New Test\n+## Continuous Integration\n+\n+A quick overview of how we run e2e CI on Kubernetes.\n+\n+### What is CI?\n+\n+We run a battery of `e2e` tests against `HEAD` of the master branch on a continuous basis, and block merges via the [submit queue](http://submit-queue.k8s.io/) on a subset of those tests if they fail (the subset is defined in the [munger config](https://github.com/kubernetes/contrib/blob/master/mungegithub/mungers/submit-queue.go) via the `jenkins-jobs` flag; note we also block on\t`kubernetes-build` and `kubernetes-test-go` jobs for build and unit and integration tests).\n+\n+CI results can be found at [ci-test.k8s.io](ci-test.k8s.io), e.g. [ci-test.k8s.io/kubernetes-e2e-gce/10594](ci-test.k8s.io/kubernetes-e2e-gce/10594).","path":"docs/devel/e2e-tests.md","position":21,"original_position":21,"commit_id":"89dfad4087d00c5c4576d0a055a9e0551c591fd0","original_commit_id":"89dfad4087d00c5c4576d0a055a9e0551c591fd0","user":{"login":"ixdy","id":8061296,"avatar_url":"https://avatars.githubusercontent.com/u/8061296?v=3","gravatar_id":"","url":"https://api.github.com/users/ixdy","html_url":"https://github.com/ixdy","followers_url":"https://api.github.com/users/ixdy/followers","following_url":"https://api.github.com/users/ixdy/following{/other_user}","gists_url":"https://api.github.com/users/ixdy/gists{/gist_id}","starred_url":"https://api.github.com/users/ixdy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ixdy/subscriptions","organizations_url":"https://api.github.com/users/ixdy/orgs","repos_url":"https://api.github.com/users/ixdy/repos","events_url":"https://api.github.com/users/ixdy/events{/privacy}","received_events_url":"https://api.github.com/users/ixdy/received_events","type":"User","site_admin":false},"body":"link needs http://","created_at":"2016-01-30T01:20:45Z","updated_at":"2016-01-30T01:20:45Z","html_url":"https://github.com/kubernetes/kubernetes/pull/20354#discussion_r51336558","pull_request_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments/51336558"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/20354#discussion_r51336558"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354"}}},"pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354","id":57718294,"html_url":"https://github.com/kubernetes/kubernetes/pull/20354","diff_url":"https://github.com/kubernetes/kubernetes/pull/20354.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/20354.patch","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20354","number":20354,"state":"open","locked":false,"title":"CI testing guidelines","user":{"login":"ihmccreery","id":368416,"avatar_url":"https://avatars.githubusercontent.com/u/368416?v=3","gravatar_id":"","url":"https://api.github.com/users/ihmccreery","html_url":"https://github.com/ihmccreery","followers_url":"https://api.github.com/users/ihmccreery/followers","following_url":"https://api.github.com/users/ihmccreery/following{/other_user}","gists_url":"https://api.github.com/users/ihmccreery/gists{/gist_id}","starred_url":"https://api.github.com/users/ihmccreery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ihmccreery/subscriptions","organizations_url":"https://api.github.com/users/ihmccreery/orgs","repos_url":"https://api.github.com/users/ihmccreery/repos","events_url":"https://api.github.com/users/ihmccreery/events{/privacy}","received_events_url":"https://api.github.com/users/ihmccreery/received_events","type":"User","site_admin":false},"body":"Branch off of #17791.\r\n\r\ncc @bprashanth ","created_at":"2016-01-29T23:39:25Z","updated_at":"2016-01-30T01:20:45Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8c8b81ab78bd55ba71dde580e8f3a09fc9d90b6a","assignee":{"login":"bprashanth","id":10927820,"avatar_url":"https://avatars.githubusercontent.com/u/10927820?v=3","gravatar_id":"","url":"https://api.github.com/users/bprashanth","html_url":"https://github.com/bprashanth","followers_url":"https://api.github.com/users/bprashanth/followers","following_url":"https://api.github.com/users/bprashanth/following{/other_user}","gists_url":"https://api.github.com/users/bprashanth/gists{/gist_id}","starred_url":"https://api.github.com/users/bprashanth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bprashanth/subscriptions","organizations_url":"https://api.github.com/users/bprashanth/orgs","repos_url":"https://api.github.com/users/bprashanth/repos","events_url":"https://api.github.com/users/bprashanth/events{/privacy}","received_events_url":"https://api.github.com/users/bprashanth/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354/commits","review_comments_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354/comments","review_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/20354/comments","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/89dfad4087d00c5c4576d0a055a9e0551c591fd0","head":{"label":"ihmccreery:ci_doc","ref":"ci_doc","sha":"89dfad4087d00c5c4576d0a055a9e0551c591fd0","user":{"login":"ihmccreery","id":368416,"avatar_url":"https://avatars.githubusercontent.com/u/368416?v=3","gravatar_id":"","url":"https://api.github.com/users/ihmccreery","html_url":"https://github.com/ihmccreery","followers_url":"https://api.github.com/users/ihmccreery/followers","following_url":"https://api.github.com/users/ihmccreery/following{/other_user}","gists_url":"https://api.github.com/users/ihmccreery/gists{/gist_id}","starred_url":"https://api.github.com/users/ihmccreery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ihmccreery/subscriptions","organizations_url":"https://api.github.com/users/ihmccreery/orgs","repos_url":"https://api.github.com/users/ihmccreery/repos","events_url":"https://api.github.com/users/ihmccreery/events{/privacy}","received_events_url":"https://api.github.com/users/ihmccreery/received_events","type":"User","site_admin":false},"repo":{"id":32747275,"name":"kubernetes","full_name":"ihmccreery/kubernetes","owner":{"login":"ihmccreery","id":368416,"avatar_url":"https://avatars.githubusercontent.com/u/368416?v=3","gravatar_id":"","url":"https://api.github.com/users/ihmccreery","html_url":"https://github.com/ihmccreery","followers_url":"https://api.github.com/users/ihmccreery/followers","following_url":"https://api.github.com/users/ihmccreery/following{/other_user}","gists_url":"https://api.github.com/users/ihmccreery/gists{/gist_id}","starred_url":"https://api.github.com/users/ihmccreery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ihmccreery/subscriptions","organizations_url":"https://api.github.com/users/ihmccreery/orgs","repos_url":"https://api.github.com/users/ihmccreery/repos","events_url":"https://api.github.com/users/ihmccreery/events{/privacy}","received_events_url":"https://api.github.com/users/ihmccreery/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ihmccreery/kubernetes","description":"Container Cluster Manager from Google","fork":true,"url":"https://api.github.com/repos/ihmccreery/kubernetes","forks_url":"https://api.github.com/repos/ihmccreery/kubernetes/forks","keys_url":"https://api.github.com/repos/ihmccreery/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ihmccreery/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ihmccreery/kubernetes/teams","hooks_url":"https://api.github.com/repos/ihmccreery/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/ihmccreery/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/ihmccreery/kubernetes/events","assignees_url":"https://api.github.com/repos/ihmccreery/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/ihmccreery/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/ihmccreery/kubernetes/tags","blobs_url":"https://api.github.com/repos/ihmccreery/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ihmccreery/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ihmccreery/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/ihmccreery/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ihmccreery/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/ihmccreery/kubernetes/languages","stargazers_url":"https://api.github.com/repos/ihmccreery/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/ihmccreery/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/ihmccreery/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/ihmccreery/kubernetes/subscription","commits_url":"https://api.github.com/repos/ihmccreery/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/ihmccreery/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/ihmccreery/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/ihmccreery/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/ihmccreery/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/ihmccreery/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ihmccreery/kubernetes/merges","archive_url":"https://api.github.com/repos/ihmccreery/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ihmccreery/kubernetes/downloads","issues_url":"https://api.github.com/repos/ihmccreery/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/ihmccreery/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/ihmccreery/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/ihmccreery/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ihmccreery/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/ihmccreery/kubernetes/releases{/id}","deployments_url":"https://api.github.com/repos/ihmccreery/kubernetes/deployments","created_at":"2015-03-23T17:20:55Z","updated_at":"2016-01-11T23:32:20Z","pushed_at":"2016-01-30T00:54:35Z","git_url":"git://github.com/ihmccreery/kubernetes.git","ssh_url":"git@github.com:ihmccreery/kubernetes.git","clone_url":"https://github.com/ihmccreery/kubernetes.git","svn_url":"https://github.com/ihmccreery/kubernetes","homepage":"http://kubernetes.io","size":155123,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"0069c901d9f952defa80c69b4f78664ee5c16ce4","user":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":20580498,"name":"kubernetes","full_name":"kubernetes/kubernetes","owner":{"login":"kubernetes","id":13629408,"avatar_url":"https://avatars.githubusercontent.com/u/13629408?v=3","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/kubernetes/kubernetes","description":"Container Cluster Manager from Google","fork":false,"url":"https://api.github.com/repos/kubernetes/kubernetes","forks_url":"https://api.github.com/repos/kubernetes/kubernetes/forks","keys_url":"https://api.github.com/repos/kubernetes/kubernetes/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/kubernetes/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/kubernetes/teams","hooks_url":"https://api.github.com/repos/kubernetes/kubernetes/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/kubernetes/events","assignees_url":"https://api.github.com/repos/kubernetes/kubernetes/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/kubernetes/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/kubernetes/tags","blobs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/kubernetes/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/kubernetes/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/kubernetes/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/kubernetes/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/kubernetes/languages","stargazers_url":"https://api.github.com/repos/kubernetes/kubernetes/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/kubernetes/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/kubernetes/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/kubernetes/subscription","commits_url":"https://api.github.com/repos/kubernetes/kubernetes/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/kubernetes/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/kubernetes/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/kubernetes/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/kubernetes/merges","archive_url":"https://api.github.com/repos/kubernetes/kubernetes/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/kubernetes/downloads","issues_url":"https://api.github.com/repos/kubernetes/kubernetes/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/kubernetes/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/kubernetes/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/kubernetes/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes/kubernetes/deployments","created_at":"2014-06-06T22:56:04Z","updated_at":"2016-01-29T23:38:24Z","pushed_at":"2016-01-30T01:18:29Z","git_url":"git://github.com/kubernetes/kubernetes.git","ssh_url":"git@github.com:kubernetes/kubernetes.git","clone_url":"https://github.com/kubernetes/kubernetes.git","svn_url":"https://github.com/kubernetes/kubernetes","homepage":"http://kubernetes.io","size":313193,"stargazers_count":12419,"watchers_count":12419,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":3516,"mirror_url":null,"open_issues_count":2844,"forks":3516,"open_issues":2844,"watchers":12419,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354"},"html":{"href":"https://github.com/kubernetes/kubernetes/pull/20354"},"issue":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/20354"},"comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/issues/20354/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/kubernetes/pulls/20354/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/kubernetes/statuses/89dfad4087d00c5c4576d0a055a9e0551c591fd0"}}}}