{"action":"created","issue":{"url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/2","repository_url":"https://api.github.com/repos/mozilla-services/go-syncstorage","labels_url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/2/labels{/name}","comments_url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/2/comments","events_url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/2/events","html_url":"https://github.com/mozilla-services/go-syncstorage/issues/2","id":163970578,"number":2,"title":"Implement the new Batch API ","user":{"login":"mostlygeek","id":83972,"avatar_url":"https://avatars.githubusercontent.com/u/83972?v=3","gravatar_id":"","url":"https://api.github.com/users/mostlygeek","html_url":"https://github.com/mostlygeek","followers_url":"https://api.github.com/users/mostlygeek/followers","following_url":"https://api.github.com/users/mostlygeek/following{/other_user}","gists_url":"https://api.github.com/users/mostlygeek/gists{/gist_id}","starred_url":"https://api.github.com/users/mostlygeek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mostlygeek/subscriptions","organizations_url":"https://api.github.com/users/mostlygeek/orgs","repos_url":"https://api.github.com/users/mostlygeek/repos","events_url":"https://api.github.com/users/mostlygeek/events{/privacy}","received_events_url":"https://api.github.com/users/mostlygeek/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-05T23:46:25Z","updated_at":"2016-07-06T22:26:40Z","closed_at":null,"body":"The Batch API allows multiple POST requests to be atomically committed. BSOs, like those that container bookmark folder structure, can point to other BSOs. If these are changed and their references are not consistent bad things happen. Sometimes, bad, unrecoverable things. \r\n\r\nThe discussion started in [Bug 1250189](https://bugzilla.mozilla.org/show_bug.cgi?id=1250189) and the python implementation is happening in PR [mozilla-services/server-syncstorage/#36](https://github.com/mozilla-services/server-syncstorage/pull/36).\r\n\r\nThe batch api works like this: \r\n\r\n1. `POST /storage/collection/{name}?batch=true` - starts a new batch upload. Returns a batch id number\r\n  * returns a JSON: `{\"batch\":\"{batch_id}\"}`\r\n1. `POST /storage/collection/{name}?batch={batch_id}` - appends more BSOs to the batch\r\n1. `POST /storage/collection/{name}?batch={batch_id}&commit=true` - commits a batch as a single atomic transaction\r\n  * `POST .../?batch=true&commit=true` should work like a regular POST. Weird. \r\n1. `GET /info/configuration` (edit: added) [python implementation](https://github.com/mozilla-services/server-syncstorage/blob/25556c07aaa5eefe307bdad1ac5b785ce99b9c56/syncstorage/views/__init__.py#L188-L201). Returns a json object with keys:  \r\n  * max_request_bytes\r\n  * max_post_records\r\n  * max_post_bytes\r\n  * max_total_records\r\n  * max_total_bytes\r\n1. `X-Weave-*` batch headers from client\r\n  * `X-Weave-Total-Records` - total records in batch\r\n  * `X-Weave-Total-Bytes` - total amount of bytes in batch\r\n  * `X-Weave-Bytes` - size of upload \r\n\r\nSome implementation notes: \r\n\r\n* Validate all BSOs in the POST, so don't have to wait until everything is uploaded before failing\r\n* Keep pending batch BSOs in a `batch_jobs` table. Keep BSOs as text blobs\r\n* When committing, just load the raw blobs, and commit them with `syncstorage.DB.putBSO` to reuse the same DB logic.\r\n* Use the new batch Python functional tests to validate I got it right :)"},"comment":{"url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/comments/230925999","html_url":"https://github.com/mozilla-services/go-syncstorage/issues/2#issuecomment-230925999","issue_url":"https://api.github.com/repos/mozilla-services/go-syncstorage/issues/2","id":230925999,"user":{"login":"mostlygeek","id":83972,"avatar_url":"https://avatars.githubusercontent.com/u/83972?v=3","gravatar_id":"","url":"https://api.github.com/users/mostlygeek","html_url":"https://github.com/mostlygeek","followers_url":"https://api.github.com/users/mostlygeek/followers","following_url":"https://api.github.com/users/mostlygeek/following{/other_user}","gists_url":"https://api.github.com/users/mostlygeek/gists{/gist_id}","starred_url":"https://api.github.com/users/mostlygeek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mostlygeek/subscriptions","organizations_url":"https://api.github.com/users/mostlygeek/orgs","repos_url":"https://api.github.com/users/mostlygeek/repos","events_url":"https://api.github.com/users/mostlygeek/events{/privacy}","received_events_url":"https://api.github.com/users/mostlygeek/received_events","type":"User","site_admin":false},"created_at":"2016-07-06T22:26:40Z","updated_at":"2016-07-06T22:26:40Z","body":"> ISTM the only other sensible thing it could do would be return an error.\r\n\r\nTook a look at the [current implementation](https://github.com/mozilla-services/server-syncstorage/blob/25556c07aaa5eefe307bdad1ac5b785ce99b9c56/syncstorage/views/__init__.py#L347-L430) and it seems less weird to me. \r\n\r\nI still think that each call for the batch functionality should be discrete. So create, append and commit would each require a separate HTTP call. That way the current POST would be *only* way POST and commit in a single call, by design. "}}
{"actor":{"display_login":"mostlygeek"}}