{"action":"created","issue":{"url":"https://api.github.com/repos/basho/riak_kv/issues/1378","repository_url":"https://api.github.com/repos/basho/riak_kv","labels_url":"https://api.github.com/repos/basho/riak_kv/issues/1378/labels{/name}","comments_url":"https://api.github.com/repos/basho/riak_kv/issues/1378/comments","events_url":"https://api.github.com/repos/basho/riak_kv/issues/1378/events","html_url":"https://github.com/basho/riak_kv/pull/1378","id":145257072,"number":1378,"title":"READY: Perf/query ttb encoding","user":{"login":"erikleitch","id":9666459,"avatar_url":"https://avatars.githubusercontent.com/u/9666459?v=3","gravatar_id":"","url":"https://api.github.com/users/erikleitch","html_url":"https://github.com/erikleitch","followers_url":"https://api.github.com/users/erikleitch/followers","following_url":"https://api.github.com/users/erikleitch/following{/other_user}","gists_url":"https://api.github.com/users/erikleitch/gists{/gist_id}","starred_url":"https://api.github.com/users/erikleitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erikleitch/subscriptions","organizations_url":"https://api.github.com/users/erikleitch/orgs","repos_url":"https://api.github.com/users/erikleitch/repos","events_url":"https://api.github.com/users/erikleitch/events{/privacy}","received_events_url":"https://api.github.com/users/erikleitch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-04-01T18:52:02Z","updated_at":"2016-04-07T18:35:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/basho/riak_kv/pulls/1378","html_url":"https://github.com/basho/riak_kv/pull/1378","diff_url":"https://github.com/basho/riak_kv/pull/1378.diff","patch_url":"https://github.com/basho/riak_kv/pull/1378.patch"},"body":"<hr>\r\n**Context**\r\n\r\nThis is part of a set of related PRs to rework handling of PB vs TTB encoding for client/server messages:\r\n* https://github.com/basho/riak-erlang-client/pull/267 \r\n* https://github.com/basho/riak_pb/pull/182\r\n* https://github.com/basho/riak_api/pull/109\r\n* https://github.com/basho/riak_kv/pull/1378\r\n\r\nThe original addition of alternate encodings was a hybrid implementation which 1) required the client and server to agree via a handshaking protocol about what encoding would be used, 2) used messages for both encodings that were generated from the .proto files, but some were only used for TTB encoding, and 3) allowed the server to send PB-encoded responses to TTB-encoded requests.\r\n\r\nThis work is an attempt to rationalize this situation, and to simplify how differently-encoded messages are handled within riak.  The guiding principles are:\r\n\r\n1. Even as we optimize encodings for RiakTS performance, the server should continue to support PB-encoding as before.  We do _not_, for example,  want to break custom PB connections made outside our supported client codebase\r\n\r\n2. We should be able to optimize messages for whatever encoding is imposed on them.  \r\n\r\n3. The server should handle interleaved messages with different encodings seamlessly, without any handshaking about encoding type\r\n\r\n4. We should guarantee that responses are encoded in the same way as the requests that generated them\r\n\r\n<hr>\r\n**Notes**\r\n\r\n* Regarding point 2. above, although we currently pack data into the same containers differently for PB vs TTB encoding, we do not currently have a use-case that requires separate message _structures_, so the decision was made to keep the existing .proto-generated messages until we have such a use-case.\r\n\r\n* Because TTB encoding is self-describing, we do not need to switch on per-message 8-bit MsgCode, as with PB-encoded messages, to know how to decode them.  This work therefor adds a single new MsgCode (104) to indicate that a message is TTB-encoded.   This allows the server to determine what encoding is being used per-message and without handshaking, addressing point 3. above.  It also preserves the remaining 8-bit space for future messages if needed.\r\n\r\n* The server now implements separate services for PB and TTB-encoded messages, selected on MsgCode.  Because the processing is done per-service, and not per-message, this also guarantees that responses are encoded in the same way as the requests, addressing point 4 above.  This is important, for example, for tsquery requests, where it is the encoding of the _responses_, not the _requests_ that matters for performance.\r\n\r\n<hr>\r\n**Changes in this repo:**\r\n\r\n* **src/riak_kv_ts_svc.erl, include/riak_kv_ts_svc.hrl\r\n * The service formerly known as riak_kv_pb_timeseries has been split into two services: riak_kv_pb_ts and riak_kv_ttb_ts, for handling the respective encodings.  These files have been added to collect code that is common between them.\r\n\r\n, src/riak_kv_pb_ts.erl, src/riak_kv_ttb_ts.erl** \r\n\r\n*  The service formerly known as riak_kv_pb_timeseries has been split into two services: riak_kv_pb_ts and riak_kv_ttb_ts, for handling the respective encodings. These files now collect the service api specific to each encoding type into separate modules, while the common code is in riak_kv_ts_svc module.\r\n \r\n The corresponding functions to build these records have also been modified to allow the originating Request to be passed as an argument\r\n\r\n* **src/riak_kv_app.erl** The range of messages handled by the riak_kv_pb_ts service has been truncated at 103, while the new service riak_kv_ttb_ts is registered to handle the new TTB message code, 104."},"comment":{"url":"https://api.github.com/repos/basho/riak_kv/issues/comments/207040968","html_url":"https://github.com/basho/riak_kv/pull/1378#issuecomment-207040968","issue_url":"https://api.github.com/repos/basho/riak_kv/issues/1378","id":207040968,"user":{"login":"alexmoore","id":1438,"avatar_url":"https://avatars.githubusercontent.com/u/1438?v=3","gravatar_id":"","url":"https://api.github.com/users/alexmoore","html_url":"https://github.com/alexmoore","followers_url":"https://api.github.com/users/alexmoore/followers","following_url":"https://api.github.com/users/alexmoore/following{/other_user}","gists_url":"https://api.github.com/users/alexmoore/gists{/gist_id}","starred_url":"https://api.github.com/users/alexmoore/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexmoore/subscriptions","organizations_url":"https://api.github.com/users/alexmoore/orgs","repos_url":"https://api.github.com/users/alexmoore/repos","events_url":"https://api.github.com/users/alexmoore/events{/privacy}","received_events_url":"https://api.github.com/users/alexmoore/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T18:35:08Z","updated_at":"2016-04-07T18:35:08Z","body":"+1 ec60260"}}