{"action":"created","comment":{"url":"https://api.github.com/repos/kubernetes/community/pulls/comments/219267651","pull_request_review_id":157403953,"id":219267651,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxOTI2NzY1MQ==","diff_hunk":"@@ -0,0 +1,360 @@\n+---\n+kep-number: 25\n+title: Migrating API objects to latest storage version\n+authors:\n+  - \"@xuchao\"\n+owning-sig: sig-api-machinery\n+reviewers:\n+  - \"@deads2k\"\n+  - \"@lavalamp\"\n+approvers:\n+  - \"@deads2k\"\n+  - \"@lavalamp\"\n+creation-date: 2018-08-06\n+last-updated: 2018-08-06\n+status: provisional\n+---\n+\n+# Migrating API objects to latest storage version\n+\n+## Table of Contents\n+\n+   * [Migrating API objects to latest storage version](#migrating-api-objects-to-latest-storage-version)\n+      * [Table of Contents](#table-of-contents)\n+      * [Summary](#summary)\n+      * [Motivation](#motivation)\n+         * [Goals](#goals)\n+      * [Proposal](#proposal)\n+         * [Alpha workflow](#alpha-workflow)\n+         * [API](#api)\n+         * [Failure recovery](#failure-recovery)\n+         * [Beta workflow - Automation](#beta-workflow---automation)\n+            * [a. Exposing the configured storage versions via API](#a-exposing-the-configured-storage-versions-via-api)\n+            * [b. Using the apiserver version to trigger migration](#b-using-the-apiserver-version-to-trigger-migration)\n+         * [Risks and Mitigations](#risks-and-mitigations)\n+      * [Graduation Criteria](#graduation-criteria)\n+      * [Alternatives](#alternatives)\n+         * [“Active management”](#active-management)\n+         * [update-storage-objects.sh](#update-storage-objectssh)\n+\n+\n+## Summary\n+\n+We propose to extend and improve the [oc adm migrate storage][] command line\n+tool to migrate the stored API objects in Kubernetes clusters. We will\n+deliver a standalone tool of alpha quality in 2018 Q3. We will integrate the\n+storage migration into the Kubernetes upgrade process in 2018 Q4. We will make\n+the migration automatically triggered in 2019.\n+\n+[oc adm migrate storage]:https://www.mankier.com/1/oc-adm-migrate-storage\n+\n+## Motivation\n+\n+\"Today it is possible to create API objects (e.g., HPAs) in one version of\n+Kubernetes, go through multiple upgrade cycles without touching those objects,\n+and eventually arrive at a version of Kubernetes that can’t interpret the stored\n+resource and crashes. See k8s.io/pr/52185.\"[1][]. We propose a solution to the\n+problem.\n+\n+[1]:https://docs.google.com/document/d/1eoS1K40HLMl4zUyw5pnC05dEF3mzFLp5TPEEt4PFvsM\n+\n+### Goals\n+\n+A successful storage version migration tool must:\n+* work for Kubernetes built-in APIs, custom resources (CR), and aggregated APIs.\n+* do not add burden to cluster administrators or Kubernetes distributions.\n+* only cause insignificant load to apiservers. For example, if the master has\n+  10GB memory, the migration tool should generate less than 10 qps of single\n+  object operations(TODO: measure the memory consumption of PUT operations;\n+  study how well the default 10 Mbps bandwidth limit in the oc command work).\n+* work for big clusters that have ~10^6 instances of some resource types.\n+* make progress in flaky environment, e.g., flaky apiservers, or the migration\n+  process get preempted.\n+* allow system administrators to track the migration progress.\n+\n+As to the deliverables,\n+* in the short term, providing system administrators with a tool to migrate\n+  the API objects to the proper storage versions.\n+* in the long term, automating the migration of Kubernetes built-in APIs, CR,\n+  aggregated APIs without further burdening system administrators or Kubernetes\n+  distributions.\n+\n+## Proposal\n+\n+### Alpha workflow\n+\n+We will create a **kube-migrator** command line tool. At the alpha stage,\n+**kube-migrator** needs to be manually invoked, and does not handle custom\n+resources. \n+\n+After all the apiservers are at the desired version, including kube-apiservers\n+and aggregated apiservers, the cluster administrator runs `kube-migrator\n+install` command. It\n+* deletes any existing `deployment` of **kube-migrator controller** \n+* creates a `deployment` of **kube-migrator controller** \n+* uses the same kubeconfig file as **kubectl** does.\n+\n+On start-up, the **kube-migrator controller** does the following,\n+* discovers all supported APIs.\n+* lists all custom resource definitions, excludes them from the API list\n+  retrieved in the previous step.\n+* for each non-custom resource, creates a `migration` CR (see the [API\n+  section][] for the schema) if it doesn't exist already.\n+* for each newly created `migration` instance, increments the\n+  `migration.spec.generation`, and logs the reason.\n+* the `ownerReferences` of the `migration` objects are set to the **kube-migrator\n+  controller** `deployment`. Thus, when user invokes the **kube-migrator** tool,\n+  the old `migration`s are deleted with the old `deployment`.\n+\n+The control loop of **kube-migrator controller** does the following:\n+* watches the instances of the `migration` CR.\n+* if `migration.spec.generation != migration.status.observedGeneration`\n+    * deletes existing **migration worker** `job` for the resource type.\n+    * launches new **migration worker** `job` to migrate the resource type. The\n+      `job` is annotated with the generation.\n+    * makes sure there is only one **migration worker** `job` running at a time,\n+      to avoid overloading the apiserver.\n+    * updates `migration.status.observedGeneration`.\n+* watches the **migration worker** jobs. Updates\n+  `migration.status.completedGeneration` when a job completes.\n+* short-circuit: if `migration.spec.groupVersions` shows that the resource has\n+  only one version, then its `.status.observedGeneration` and\n+  `.status.completedGeneration` are updated directly without launching the\n+  worker `job`.\n+\n+The **migrator worker** `job` runs the equivalence of `oc adm migrate storage\n+--include=<resource type>` to migrate a resource type. The **migrator worker**\n+uses API chunking to retrieve partial lists of a resource type and thus can\n+migrate a small chunk at a time. It stores the [continue token] in a `configmap`\n+to preserve progress. With the inconsistent continue token introduced in\n+[#67284][], the **migrator worker** does not need to worry about expired continue\n+token.\n+\n+[#67284]:https://github.com/kubernetes/kubernetes/pull/67284\n+[API section]:#api\n+\n+The cluster admin can run the `kube-migrator progress` command to check if the\n+migration has completed. The command lists all `migrations` instances and checks\n+if all of them have `.spec.generation == .status.observedGeneration`.\n+\n+### API\n+\n+We introduce the `migration` API to record the intention and progress of a\n+migration. The API will be a CRD defined in the `migration.k8s.io` group.\n+\n+Read the [workflow section][] to understand how the API is used.\n+\n+```golang\n+type StorageVersionMigration struct {\n+  metav1.TypeMeta\n+  metav1.ObjectMeta\n+  Spec StorageVersionMigrationSpec\n+  Status StorageVersionMigrationStatus\n+}\n+\n+type StorageVersionMigrationSpec {\n+  // Resource is the identifier of the resource that's being migrated.\n+  // TBD: maybe resource=hash(etcd instance, etcd path prefix).\n+  // Immuntable.\n+  Resource string \n+  // GroupVersions are all the groups and versions this resource is served by\n+  // the apiserver. The server preferred version should be groupVersions[0].\n+  // **Migration workers** GET & PUT via the preferred version by default, to\n+  // avoid conversions in the apiserver.\n+  GroupVersions []GroupVersion\n+  // Generation is the generation of the spec. Incrementing generation triggers\n+  // migration. We do not use the objectMeta.generation because its semantic is\n+  // not clear yet (e.g., objectMeta.generation might be incremented when\n+  // labels/annotations are updated [#67428][])\n+  Generation int64\n+  // Reason of the migration. Examples are \"manually triggered\", \"apiserver\n+  // updated\", etc.\n+  Reason string\n+}\n+\n+type GroupVersion struct {\n+  Group string\n+  Version string\n+}\n+\n+type StorageVersionMigrationStatus {\n+  // ObeservedGeneration is the spec generation that has been observed by the\n+  // **migration controller** It indicates a **migration worker** for this\n+  // generation has been launched.\n+  ObeservedGeneration int64\n+  // CompletedGeneration is the greatest spec generation whose **migration worker**\n+  // has completed.\n+  CompletedGeneration int64\n+}\n+```\n+\n+[#67428]:https://github.com/kubernetes/kubernetes/issues/67428\n+[continue token]:https://github.com/kubernetes/kubernetes/blob/972e1549776955456d9808b619d136ee95ebb388/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L82\n+[workflow section]:#alpha-workflow\n+\n+We need to expand the discovery API to recognize APIs in different groups\n+that share the same storage, e.g., `extensions/v1beta1/deployment` and\n+`apps/v1/deployment` share the same key space in etcd. Specifically, we expand\n+the [APIResource][] API:\n+\n+```golang\n+type APIResource struct {\n+  Name string\n+  ...\n+  // StorageUID=SHA-256(etcd server in the form of scheme://ip:port, etcd path),\n+  // obfuscated with a random key. E.g., for pods, it is\n+  // SHA-256(scheme://ip:port/registry/pods).\n+  // The string is obfuscated to avoid exposing server configuration details.\n+  StorageUID string\n+}\n+```\n+\n+[APIResource]:https://github.com/kubernetes/kubernetes/blob/d38fb2c6451447ad0ecbdfed971a563419d60883/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L874\n+\n+\n+### Failure recovery\n+\n+As stated in the goals section, the migration has to make progress even if the\n+environment is flaky. This section describes how the migrator recovers from\n+failure.\n+\n+Kubernetes **replicaset controller** restarts the **migration controller** `pod`\n+if it fails. The restarted **migration controller** follows the same bootstrap\n+workflow as described in the [workflow section][]. Because the **migration\n+controller** adopts existing `migration` objects, it can resume from where\n+it left off.\n+\n+Kubernetes **job controller** ensures the completion of a **migration worker**.\n+The **migration worker** uses the continue token as the checkpoint.\n+* the worker stores the latest continue token returned by the apiserver in a\n+  `configmap`.\n+* when the worker starts, it resumes listing with the continue token.\n+* the **migration controller** creates the configmap when creates the worker `job`,\n+  and sets the worker job as the owner. When the **migration controller** deletes\n+  the completed `job`, the `configmap` is garbage collected.\n+\n+[workflow section]:#alpha-workflow\n+\n+### Beta workflow - Automation\n+\n+It is a beta goal to automate the migration workflow. That is, migration does\n+not need to be triggered manually by cluster admins, or by custom control loops\n+of Kubernetes distributions.\n+\n+The automated migration should work for Kubernetes built-in resource types,\n+custom resources, and aggregated resources.\n+\n+Here are two alternatives and their requirements.\n+\n+#### a. Exposing the configured storage versions via API\n+\n+This is what the [Active Management][] design proposes. It requires:\n+* exposing the configured [storage versions][] of Kubernetes built-in resource\n+  types and aggregated resources. \n+\n+Note that custom resources already exposes this information. \n+\n+The migration of built-in resource types, aggregated resources, and custom\n+resources can then all be triggered by changes to the exposed storage versions\n+configurations.\n+\n+[Active Management]:https://docs.google.com/document/d/1eoS1K40HLMl4zUyw5pnC05dEF3mzFLp5TPEEt4PFvsM\n+[storage versions]:https://github.com/kubernetes/kubernetes/blob/db9545e69e630e5bc8c3c2dab14f9339dd3b871c/pkg/kubeapiserver/options/storage_versions.go#L98\n+\n+#### b. Using the apiserver version to trigger migration\n+\n+If the Kubernetes deprecation [policy][] is followed by every apiserver, and\n+every cluster uses the default [storage version configs][], then we can use the\n+apiserver version to drive the migration. More specifically,\n+* kube-apiserver removes the `--storagve-version` flag, so that every deployment\n+  of kube-apiserver uses the default storage versions. [#67678][]\n+* aggregated apiservers follow semantic versioning, and expose their version via\n+  API.\n+* aggregated apiservers follow the same API deprecation [policy][].\n+\n+Then the migration of built-in resources and aggregated resources can be driven\n+by the changes of apiservers versions.","path":"keps/sig-api-machinery/storage-migration.md","position":276,"original_position":276,"commit_id":"5cb23daf7978d25d73fb7a33cdec09b2a5f91e52","original_commit_id":"5cb23daf7978d25d73fb7a33cdec09b2a5f91e52","user":{"login":"caesarxuchao","id":2823529,"node_id":"MDQ6VXNlcjI4MjM1Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/2823529?v=4","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false},"body":"> Many times when it changes there's no storage change needed.\r\n\r\nMigrating objects whose storage version hasn't changed doesn't impact correctness. The extra workload is also tolerable.\r\n\r\nIf storage versions can change when apiserver binary versions do *not* change, that will impact correctness. Can you suggest cases where we need to do that?","created_at":"2018-09-20T18:18:16Z","updated_at":"2018-09-20T18:18:16Z","html_url":"https://github.com/kubernetes/community/pull/2524#discussion_r219267651","pull_request_url":"https://api.github.com/repos/kubernetes/community/pulls/2524","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments/219267651"},"html":{"href":"https://github.com/kubernetes/community/pull/2524#discussion_r219267651"},"pull_request":{"href":"https://api.github.com/repos/kubernetes/community/pulls/2524"}},"in_reply_to_id":217566145},"pull_request":{"url":"https://api.github.com/repos/kubernetes/community/pulls/2524","id":208042263,"node_id":"MDExOlB1bGxSZXF1ZXN0MjA4MDQyMjYz","html_url":"https://github.com/kubernetes/community/pull/2524","diff_url":"https://github.com/kubernetes/community/pull/2524.diff","patch_url":"https://github.com/kubernetes/community/pull/2524.patch","issue_url":"https://api.github.com/repos/kubernetes/community/issues/2524","number":2524,"state":"open","locked":false,"title":"[KEP] storage api version migration","user":{"login":"caesarxuchao","id":2823529,"node_id":"MDQ6VXNlcjI4MjM1Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/2823529?v=4","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false},"body":"This KEP proposes a tool that migrates the stored data in etcd to the latest storage version. The KEP is based on the [doc](https://docs.google.com/document/d/16HWEjqnlEc2PJnS4D7TVbHoNyVlyWpjx0zRsFNyxDsI/edit#heading=h.fap52hhsdgdt) which was discussed in the Aug. 1, 2018 sig apimachinery meeting. The KEP added more details on the tool's workflow and development timeline.\r\n\r\n/sig api-machinery\r\n/assign @lavalamp @deads2k ","created_at":"2018-08-13T17:03:04Z","updated_at":"2018-09-20T18:18:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bae3a80b42c16a462af2709f9fe1a13bdc311a54","assignee":{"login":"lavalamp","id":647318,"node_id":"MDQ6VXNlcjY0NzMxOA==","avatar_url":"https://avatars0.githubusercontent.com/u/647318?v=4","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},"assignees":[{"login":"lavalamp","id":647318,"node_id":"MDQ6VXNlcjY0NzMxOA==","avatar_url":"https://avatars0.githubusercontent.com/u/647318?v=4","gravatar_id":"","url":"https://api.github.com/users/lavalamp","html_url":"https://github.com/lavalamp","followers_url":"https://api.github.com/users/lavalamp/followers","following_url":"https://api.github.com/users/lavalamp/following{/other_user}","gists_url":"https://api.github.com/users/lavalamp/gists{/gist_id}","starred_url":"https://api.github.com/users/lavalamp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lavalamp/subscriptions","organizations_url":"https://api.github.com/users/lavalamp/orgs","repos_url":"https://api.github.com/users/lavalamp/repos","events_url":"https://api.github.com/users/lavalamp/events{/privacy}","received_events_url":"https://api.github.com/users/lavalamp/received_events","type":"User","site_admin":false},{"login":"deads2k","id":8225098,"node_id":"MDQ6VXNlcjgyMjUwOTg=","avatar_url":"https://avatars0.githubusercontent.com/u/8225098?v=4","gravatar_id":"","url":"https://api.github.com/users/deads2k","html_url":"https://github.com/deads2k","followers_url":"https://api.github.com/users/deads2k/followers","following_url":"https://api.github.com/users/deads2k/following{/other_user}","gists_url":"https://api.github.com/users/deads2k/gists{/gist_id}","starred_url":"https://api.github.com/users/deads2k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deads2k/subscriptions","organizations_url":"https://api.github.com/users/deads2k/orgs","repos_url":"https://api.github.com/users/deads2k/repos","events_url":"https://api.github.com/users/deads2k/events{/privacy}","received_events_url":"https://api.github.com/users/deads2k/received_events","type":"User","site_admin":false},{"login":"yliaog","id":32884155,"node_id":"MDQ6VXNlcjMyODg0MTU1","avatar_url":"https://avatars0.githubusercontent.com/u/32884155?v=4","gravatar_id":"","url":"https://api.github.com/users/yliaog","html_url":"https://github.com/yliaog","followers_url":"https://api.github.com/users/yliaog/followers","following_url":"https://api.github.com/users/yliaog/following{/other_user}","gists_url":"https://api.github.com/users/yliaog/gists{/gist_id}","starred_url":"https://api.github.com/users/yliaog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yliaog/subscriptions","organizations_url":"https://api.github.com/users/yliaog/orgs","repos_url":"https://api.github.com/users/yliaog/repos","events_url":"https://api.github.com/users/yliaog/events{/privacy}","received_events_url":"https://api.github.com/users/yliaog/received_events","type":"User","site_admin":false}],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":486032367,"node_id":"MDU6TGFiZWw0ODYwMzIzNjc=","url":"https://api.github.com/repos/kubernetes/community/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":653942861,"node_id":"MDU6TGFiZWw2NTM5NDI4NjE=","url":"https://api.github.com/repos/kubernetes/community/labels/sig/api-machinery","name":"sig/api-machinery","color":"d2b48c","default":false},{"id":653942863,"node_id":"MDU6TGFiZWw2NTM5NDI4NjM=","url":"https://api.github.com/repos/kubernetes/community/labels/sig/architecture","name":"sig/architecture","color":"d2b48c","default":false},{"id":653942910,"node_id":"MDU6TGFiZWw2NTM5NDI5MTA=","url":"https://api.github.com/repos/kubernetes/community/labels/size/L","name":"size/L","color":"ee9900","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/kubernetes/community/pulls/2524/commits","review_comments_url":"https://api.github.com/repos/kubernetes/community/pulls/2524/comments","review_comment_url":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}","comments_url":"https://api.github.com/repos/kubernetes/community/issues/2524/comments","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/5cb23daf7978d25d73fb7a33cdec09b2a5f91e52","head":{"label":"caesarxuchao:storage","ref":"storage","sha":"5cb23daf7978d25d73fb7a33cdec09b2a5f91e52","user":{"login":"caesarxuchao","id":2823529,"node_id":"MDQ6VXNlcjI4MjM1Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/2823529?v=4","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false},"repo":{"id":76486227,"node_id":"MDEwOlJlcG9zaXRvcnk3NjQ4NjIyNw==","name":"community","full_name":"caesarxuchao/community","private":false,"owner":{"login":"caesarxuchao","id":2823529,"node_id":"MDQ6VXNlcjI4MjM1Mjk=","avatar_url":"https://avatars2.githubusercontent.com/u/2823529?v=4","gravatar_id":"","url":"https://api.github.com/users/caesarxuchao","html_url":"https://github.com/caesarxuchao","followers_url":"https://api.github.com/users/caesarxuchao/followers","following_url":"https://api.github.com/users/caesarxuchao/following{/other_user}","gists_url":"https://api.github.com/users/caesarxuchao/gists{/gist_id}","starred_url":"https://api.github.com/users/caesarxuchao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caesarxuchao/subscriptions","organizations_url":"https://api.github.com/users/caesarxuchao/orgs","repos_url":"https://api.github.com/users/caesarxuchao/repos","events_url":"https://api.github.com/users/caesarxuchao/events{/privacy}","received_events_url":"https://api.github.com/users/caesarxuchao/received_events","type":"User","site_admin":false},"html_url":"https://github.com/caesarxuchao/community","description":"Kubernetes community content","fork":true,"url":"https://api.github.com/repos/caesarxuchao/community","forks_url":"https://api.github.com/repos/caesarxuchao/community/forks","keys_url":"https://api.github.com/repos/caesarxuchao/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/caesarxuchao/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/caesarxuchao/community/teams","hooks_url":"https://api.github.com/repos/caesarxuchao/community/hooks","issue_events_url":"https://api.github.com/repos/caesarxuchao/community/issues/events{/number}","events_url":"https://api.github.com/repos/caesarxuchao/community/events","assignees_url":"https://api.github.com/repos/caesarxuchao/community/assignees{/user}","branches_url":"https://api.github.com/repos/caesarxuchao/community/branches{/branch}","tags_url":"https://api.github.com/repos/caesarxuchao/community/tags","blobs_url":"https://api.github.com/repos/caesarxuchao/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/caesarxuchao/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/caesarxuchao/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/caesarxuchao/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/caesarxuchao/community/statuses/{sha}","languages_url":"https://api.github.com/repos/caesarxuchao/community/languages","stargazers_url":"https://api.github.com/repos/caesarxuchao/community/stargazers","contributors_url":"https://api.github.com/repos/caesarxuchao/community/contributors","subscribers_url":"https://api.github.com/repos/caesarxuchao/community/subscribers","subscription_url":"https://api.github.com/repos/caesarxuchao/community/subscription","commits_url":"https://api.github.com/repos/caesarxuchao/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/caesarxuchao/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/caesarxuchao/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/caesarxuchao/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/caesarxuchao/community/contents/{+path}","compare_url":"https://api.github.com/repos/caesarxuchao/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/caesarxuchao/community/merges","archive_url":"https://api.github.com/repos/caesarxuchao/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/caesarxuchao/community/downloads","issues_url":"https://api.github.com/repos/caesarxuchao/community/issues{/number}","pulls_url":"https://api.github.com/repos/caesarxuchao/community/pulls{/number}","milestones_url":"https://api.github.com/repos/caesarxuchao/community/milestones{/number}","notifications_url":"https://api.github.com/repos/caesarxuchao/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/caesarxuchao/community/labels{/name}","releases_url":"https://api.github.com/repos/caesarxuchao/community/releases{/id}","deployments_url":"https://api.github.com/repos/caesarxuchao/community/deployments","created_at":"2016-12-14T18:31:22Z","updated_at":"2016-12-14T18:31:24Z","pushed_at":"2018-09-11T21:52:21Z","git_url":"git://github.com/caesarxuchao/community.git","ssh_url":"git@github.com:caesarxuchao/community.git","clone_url":"https://github.com/caesarxuchao/community.git","svn_url":"https://github.com/caesarxuchao/community","homepage":"","size":29815,"stargazers_count":0,"watchers_count":0,"language":"Makefile","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"kubernetes:master","ref":"master","sha":"0ec68604d00fe99b4be6519800a806df15d55674","user":{"login":"kubernetes","id":13629408,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzNjI5NDA4","avatar_url":"https://avatars2.githubusercontent.com/u/13629408?v=4","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"repo":{"id":57939112,"node_id":"MDEwOlJlcG9zaXRvcnk1NzkzOTExMg==","name":"community","full_name":"kubernetes/community","private":false,"owner":{"login":"kubernetes","id":13629408,"node_id":"MDEyOk9yZ2FuaXphdGlvbjEzNjI5NDA4","avatar_url":"https://avatars2.githubusercontent.com/u/13629408?v=4","gravatar_id":"","url":"https://api.github.com/users/kubernetes","html_url":"https://github.com/kubernetes","followers_url":"https://api.github.com/users/kubernetes/followers","following_url":"https://api.github.com/users/kubernetes/following{/other_user}","gists_url":"https://api.github.com/users/kubernetes/gists{/gist_id}","starred_url":"https://api.github.com/users/kubernetes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kubernetes/subscriptions","organizations_url":"https://api.github.com/users/kubernetes/orgs","repos_url":"https://api.github.com/users/kubernetes/repos","events_url":"https://api.github.com/users/kubernetes/events{/privacy}","received_events_url":"https://api.github.com/users/kubernetes/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/kubernetes/community","description":"Kubernetes community content","fork":false,"url":"https://api.github.com/repos/kubernetes/community","forks_url":"https://api.github.com/repos/kubernetes/community/forks","keys_url":"https://api.github.com/repos/kubernetes/community/keys{/key_id}","collaborators_url":"https://api.github.com/repos/kubernetes/community/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/kubernetes/community/teams","hooks_url":"https://api.github.com/repos/kubernetes/community/hooks","issue_events_url":"https://api.github.com/repos/kubernetes/community/issues/events{/number}","events_url":"https://api.github.com/repos/kubernetes/community/events","assignees_url":"https://api.github.com/repos/kubernetes/community/assignees{/user}","branches_url":"https://api.github.com/repos/kubernetes/community/branches{/branch}","tags_url":"https://api.github.com/repos/kubernetes/community/tags","blobs_url":"https://api.github.com/repos/kubernetes/community/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/kubernetes/community/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/kubernetes/community/git/refs{/sha}","trees_url":"https://api.github.com/repos/kubernetes/community/git/trees{/sha}","statuses_url":"https://api.github.com/repos/kubernetes/community/statuses/{sha}","languages_url":"https://api.github.com/repos/kubernetes/community/languages","stargazers_url":"https://api.github.com/repos/kubernetes/community/stargazers","contributors_url":"https://api.github.com/repos/kubernetes/community/contributors","subscribers_url":"https://api.github.com/repos/kubernetes/community/subscribers","subscription_url":"https://api.github.com/repos/kubernetes/community/subscription","commits_url":"https://api.github.com/repos/kubernetes/community/commits{/sha}","git_commits_url":"https://api.github.com/repos/kubernetes/community/git/commits{/sha}","comments_url":"https://api.github.com/repos/kubernetes/community/comments{/number}","issue_comment_url":"https://api.github.com/repos/kubernetes/community/issues/comments{/number}","contents_url":"https://api.github.com/repos/kubernetes/community/contents/{+path}","compare_url":"https://api.github.com/repos/kubernetes/community/compare/{base}...{head}","merges_url":"https://api.github.com/repos/kubernetes/community/merges","archive_url":"https://api.github.com/repos/kubernetes/community/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/kubernetes/community/downloads","issues_url":"https://api.github.com/repos/kubernetes/community/issues{/number}","pulls_url":"https://api.github.com/repos/kubernetes/community/pulls{/number}","milestones_url":"https://api.github.com/repos/kubernetes/community/milestones{/number}","notifications_url":"https://api.github.com/repos/kubernetes/community/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/kubernetes/community/labels{/name}","releases_url":"https://api.github.com/repos/kubernetes/community/releases{/id}","deployments_url":"https://api.github.com/repos/kubernetes/community/deployments","created_at":"2016-05-03T03:36:05Z","updated_at":"2018-09-20T17:17:03Z","pushed_at":"2018-09-20T17:16:45Z","git_url":"git://github.com/kubernetes/community.git","ssh_url":"git@github.com:kubernetes/community.git","clone_url":"https://github.com/kubernetes/community.git","svn_url":"https://github.com/kubernetes/community","homepage":"","size":32136,"stargazers_count":2498,"watchers_count":2498,"language":"Go","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1456,"mirror_url":null,"archived":false,"open_issues_count":214,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1456,"open_issues":214,"watchers":2498,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/kubernetes/community/pulls/2524"},"html":{"href":"https://github.com/kubernetes/community/pull/2524"},"issue":{"href":"https://api.github.com/repos/kubernetes/community/issues/2524"},"comments":{"href":"https://api.github.com/repos/kubernetes/community/issues/2524/comments"},"review_comments":{"href":"https://api.github.com/repos/kubernetes/community/pulls/2524/comments"},"review_comment":{"href":"https://api.github.com/repos/kubernetes/community/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/kubernetes/community/pulls/2524/commits"},"statuses":{"href":"https://api.github.com/repos/kubernetes/community/statuses/5cb23daf7978d25d73fb7a33cdec09b2a5f91e52"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"caesarxuchao"}}