{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24501","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24501/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24501/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24501/events","html_url":"https://github.com/kubernetes/kubernetes/issues/24501","id":149592396,"number":24501,"title":"Content type for watch serialization in protobuf","user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-04-19T21:42:34Z","updated_at":"2016-04-19T21:46:09Z","closed_at":null,"body":"Currently, we have two serialization protocols in Kube:\r\n\r\n1. JSON encoded runtime.Objects - a single JSON object `{...}` on disk or on the wire - GET/LIST/.../error cases.  \r\n     This is clearly content type `application/json`\r\n\r\n2. JSON encoded stream of watch.Events separated by whitespace, returned by watch.  \r\n    This, according to all formal definitions, is not JSON (because it's > 1 object) but since the specs do not define a content type it is currently returned as content type `application/json`, and clients have to infer the separation by virtue of calling a \"watch\" API (GET /resource?watch=1).  \r\n\r\n    However, we also have the case where we get an admission / security / auth / format error prior to starting the watch, in which case we return a JSON encoded status object and end the stream (on a non 2xx status code). Clients don't have to work hard to distinguish the object, although they probably should be able to.\r\n\r\nHTTP Accept header negotiation is pretty clear cut - a client requests a content-type, and the server must error out or select from their list a matching value.  We *should* unambiguously define the content-type for watch serialization, but if we select a new content-type (application/vnd.kubernetes.json) we will break old clients, which we cannot do.\r\n\r\nOptions:\r\n\r\n1. Introduce a new optional watch JSON type - `application/vnd.kubernetes.watchjson` that well behaved clients can request.  Clients that asked for this type on normal requests would get an error.\r\n2. Modify the returned content type for watch with a mime-qualifier: `application/json;kubernetes=watch` or similar.  Might have backwards compat impacts to JSON.\r\n\r\nThe problem is compounded in protobuf - the default protobuf serialization will be `application/vnd.kubernetes.protobuf` with the following form:\r\n\r\n```\r\nBytes  0-3    \r\n\"k8s\" 0x00 <protobuf encoded runtime.Unknown ... <protobuf encoded core object>>\r\n```\r\n\r\nHowever, in order to properly do a watch, we need a streaming serialization (protobuf is not self describing) to identify object boundaries - our format there is a simple 4 byte integer length prefix for each \"frame\" (like GRPC), and each object will be a protobuf encoded watch.Event.  Every object returned by a watch is an event, so we don't need individual type info\r\n\r\n```\r\n4 byte integer defining length\r\n0x00 0x00 0x00 0x00 | <protobuf encoded watch.Event field=object <protobuf encoded runtime.Unknown ... <protobuf encoded core object> >\r\n```\r\n\r\nThe watch protocol is demonstrably different than the normal serialization.  If the server returns an error, we can't distinguish between the two (since the first frame could be \"k8s\" 0x00 bytes long).  The standard mechanism is to use the content-type to distinguish the two serializations - and like JSON above we should probably be consistent and define a mechanism.\r\n\r\nThe advantage of option 2 above is that the client can ask for `application/vnd.kubernetes.protobuf` in the Accept header and get back `application/vnd.kubernetes.protobuf` (Status) or `application/vnd.kubernetes.protobuf;stream=events` (watch stream).  If we used option 1, clients that are making watch calls would have to ask for `application/vnd.kubernetes.protobuf.watch` in a distinct fashion than other cases.\r\n\r\nIn general, I would expect each \"content-type\" to have a standard streaming format and a standard non-streaming format.  We do not serialize the streaming format to disk (today), instead we serialize Lists or other types."},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/212142386","html_url":"https://github.com/kubernetes/kubernetes/issues/24501#issuecomment-212142386","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24501","id":212142386,"user":{"login":"smarterclayton","id":1163175,"avatar_url":"https://avatars.githubusercontent.com/u/1163175?v=3","gravatar_id":"","url":"https://api.github.com/users/smarterclayton","html_url":"https://github.com/smarterclayton","followers_url":"https://api.github.com/users/smarterclayton/followers","following_url":"https://api.github.com/users/smarterclayton/following{/other_user}","gists_url":"https://api.github.com/users/smarterclayton/gists{/gist_id}","starred_url":"https://api.github.com/users/smarterclayton/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/smarterclayton/subscriptions","organizations_url":"https://api.github.com/users/smarterclayton/orgs","repos_url":"https://api.github.com/users/smarterclayton/repos","events_url":"https://api.github.com/users/smarterclayton/events{/privacy}","received_events_url":"https://api.github.com/users/smarterclayton/received_events","type":"User","site_admin":false},"created_at":"2016-04-19T21:46:09Z","updated_at":"2016-04-19T21:46:09Z","body":"@liggitt "}}