{"action":"created","issue":{"url":"https://api.github.com/repos/ably/docs/issues/473","repository_url":"https://api.github.com/repos/ably/docs","labels_url":"https://api.github.com/repos/ably/docs/issues/473/labels{/name}","comments_url":"https://api.github.com/repos/ably/docs/issues/473/comments","events_url":"https://api.github.com/repos/ably/docs/issues/473/events","html_url":"https://github.com/ably/docs/pull/473","id":344395432,"node_id":"MDExOlB1bGxSZXF1ZXN0MjAzNzk5NDA0","number":473,"title":"Idempotent REST publishing spec","user":{"login":"SimonWoolf","id":5908687,"node_id":"MDQ6VXNlcjU5MDg2ODc=","avatar_url":"https://avatars1.githubusercontent.com/u/5908687?v=4","gravatar_id":"","url":"https://api.github.com/users/SimonWoolf","html_url":"https://github.com/SimonWoolf","followers_url":"https://api.github.com/users/SimonWoolf/followers","following_url":"https://api.github.com/users/SimonWoolf/following{/other_user}","gists_url":"https://api.github.com/users/SimonWoolf/gists{/gist_id}","starred_url":"https://api.github.com/users/SimonWoolf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SimonWoolf/subscriptions","organizations_url":"https://api.github.com/users/SimonWoolf/orgs","repos_url":"https://api.github.com/users/SimonWoolf/repos","events_url":"https://api.github.com/users/SimonWoolf/events{/privacy}","received_events_url":"https://api.github.com/users/SimonWoolf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-07-25T10:59:16Z","updated_at":"2018-07-25T14:02:19Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/ably/docs/pulls/473","html_url":"https://github.com/ably/docs/pull/473","diff_url":"https://github.com/ably/docs/pull/473.diff","patch_url":"https://github.com/ably/docs/pull/473.patch"},"body":"Fixes https://github.com/ably/docs/issues/469 . \r\n\r\nTakes https://github.com/ably/wiki/issues/191#issuecomment-397741807 as a starting point, with modifications from discussions on [the realtime pr](https://github.com/ably/realtime/pull/1811#discussion_r204420926) and [in slack](https://ably-real-time.slack.com/archives/CB285667N/p1532362702000417).\r\n\r\nSo: Paddy's original proposal had that client-lib-set ids were structured `<randomStr>:<index>`, with the same randomStr for all Messages in a ProtocolMessage. I wasn't a big fan of that as if you have a ProtocolMessage going around with an id of the form, say, `sIAwterTfU:0`, the `0` might be either a msgSerial (in the case of a realtime publish) or an index (in the case of a rest idempotent publish where it's the index of the first Message and the ProtocolMessage takes its id from the first Message), and you can't know which without looking to see if `messages[0].id` exists.\r\n\r\nMy proposal is that, if we're not going to enforce a consistent relationship between Message and ProtocolMessage ids (because the PM id is just going to take its id verbatim from the first Message), why not just have it be free text for client-set ids? No structure, no colon, no index, just a new random string for every Message, where the ProtocolMessage picks the first of those for its id. Then the ProtocolMessage id wouldn't get confused with connectionid:msgSerial, and it wouldn't look like it's requiring/enforcing a consistency that it actually doesn't/isn't respectively.\r\n\r\nThen we can also drop the restriction that when using user-supplied ids you can only publish in batches -- they would work exactly the same as for client-lib-set ids. (The only caveat is that the user is responsible for setting some id either in all messages or none of them, i.e. if they only set some ids then the behaviour is formally undefined, which allows the client lib to assume that if the first message doesn't have an id then none of them do and vice versa)\r\n\r\nwdyt @mattheworiordan @paddybyers "},"comment":{"url":"https://api.github.com/repos/ably/docs/issues/comments/407764836","html_url":"https://github.com/ably/docs/pull/473#issuecomment-407764836","issue_url":"https://api.github.com/repos/ably/docs/issues/473","id":407764836,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNzc2NDgzNg==","user":{"login":"mattheworiordan","id":43789,"node_id":"MDQ6VXNlcjQzNzg5","avatar_url":"https://avatars3.githubusercontent.com/u/43789?v=4","gravatar_id":"","url":"https://api.github.com/users/mattheworiordan","html_url":"https://github.com/mattheworiordan","followers_url":"https://api.github.com/users/mattheworiordan/followers","following_url":"https://api.github.com/users/mattheworiordan/following{/other_user}","gists_url":"https://api.github.com/users/mattheworiordan/gists{/gist_id}","starred_url":"https://api.github.com/users/mattheworiordan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattheworiordan/subscriptions","organizations_url":"https://api.github.com/users/mattheworiordan/orgs","repos_url":"https://api.github.com/users/mattheworiordan/repos","events_url":"https://api.github.com/users/mattheworiordan/events{/privacy}","received_events_url":"https://api.github.com/users/mattheworiordan/received_events","type":"User","site_admin":false},"created_at":"2018-07-25T14:02:19Z","updated_at":"2018-07-25T14:02:19Z","author_association":"MEMBER","body":"@SimonWoolf, @paddybyers and I sat down earlier and talked through idempotent publishing.  Unfortunately we don't agree with your proposal.  Rough notes below, and I will add relevant comments beneath:\r\n\r\n---\r\n\r\nIdempotent publishing is technically part of the 1.1 spec, however we can add this functionality now as a patch release into 1.0 so long as automatic idempotent publishing is disabled by default.\r\n\r\nUse cases we wish to support:\r\n\r\n- Implicit idempotency without client IDs\r\n  - For realtime & REST this works\r\n  - The limit is 2 minutes though for idempotency, whether realtime or REST i.e. if you reconnect a realtime connection after 2 minutes and republish, we can't offer idempotent publishes.  Same applies for REST if you retry the REST publish after 2 minutes.\r\n\r\n- Client provided IDs with idempotency\r\n  - Does not work with realtime (we have no API for this)\r\n  - Works with REST, but only one message a time. Whilst we could explore publishes with batches of idempotent publishing, we believe it's OK to leave this out for now to keep simplicity down.  We will however have a support article on how to \"hack\" the IDs so that it follows the implicit idempotent ID format for customers who really want to work around this.  However, the key thing here is that as a publisher, you need to understand in detail how the protocol works, and the fact that idempotent publishing only works as a batch, and not for individual messages within the batch.  For example, even if we supported arbitrary IDs, a user could publish `[a, b, c]` as a single batch, and then a second later publish `b`, which would publish again i.e. it's not idempotent at a single message level. Equally, they could publish `[a, b, c, d]` and with @SimonWoolf' proposal I believe that too would introduce issues in that we now need to reject that batched message outright.\r\n\r\nHowever, for customers who do need to publish a high volume of idempotent messages, they have the following options:\r\n\r\n- Our bulk publish API (https://github.com/ably/wiki/issues/37), but this is not yet supported in client libs, experimental API\r\n- HTTP/2 (https://github.com/ably/wiki/issues/309)\r\n\r\nExample for an implicit idempotent publish from a client (i.e. no message ID specified in any messages)\r\n```\r\nPOST /channel/messages/\r\n[id: 'is2mDOHSn30fUDFhY/Bnxw==:0',\r\nid: 'is2mDOHSn30fUDFhY/Bnxw==:1',\r\nid: 'is2mDOHSn30fUDFhY/Bnxw==:2']\r\n\r\n# response => 201\r\n```\r\n\r\nWhilst it's a big ugly to inspect every id of the messages and check they match the expected format, we decided this was the best solution.  We considered sending a transaction ID as a param / header for publish requests and leaving the messages without IDs, but then we'd end up having different approaches for different types of idempotent publishes.  It adds complexity in the protocol and for the user that is arguably a bit unnecessary.\r\n\r\nGiven above, if a user then tried to publish a single message:\r\n\r\n```\r\nPOST /channel/messages/\r\n[id: '/publish: is2mDOHSn30fUDFhY/Bnxw==:1']\r\n\r\n# response => 201\r\n```\r\n\r\nThe ID would include the `:1` part because a single message publish can have any arbitrary ID.  We only treat the `:\\d+` part differently when a publish is an array of messages.\r\n\r\n---\r\n\r\n@SimonWoolf I will comment now on the actual spec in line based on above."}}
{"actor":{"display_login":"mattheworiordan"}}