{"action":"created","issue":{"url":"https://api.github.com/repos/Kinto/kinto/issues/759","repository_url":"https://api.github.com/repos/Kinto/kinto","labels_url":"https://api.github.com/repos/Kinto/kinto/issues/759/labels{/name}","comments_url":"https://api.github.com/repos/Kinto/kinto/issues/759/comments","events_url":"https://api.github.com/repos/Kinto/kinto/issues/759/events","html_url":"https://github.com/Kinto/kinto/issues/759","id":171438666,"number":759,"title":"error with batch posting with default backend (maybe wrong use)","user":{"login":"elelay","id":222469,"avatar_url":"https://avatars.githubusercontent.com/u/222469?v=3","gravatar_id":"","url":"https://api.github.com/users/elelay","html_url":"https://github.com/elelay","followers_url":"https://api.github.com/users/elelay/followers","following_url":"https://api.github.com/users/elelay/following{/other_user}","gists_url":"https://api.github.com/users/elelay/gists{/gist_id}","starred_url":"https://api.github.com/users/elelay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elelay/subscriptions","organizations_url":"https://api.github.com/users/elelay/orgs","repos_url":"https://api.github.com/users/elelay/repos","events_url":"https://api.github.com/users/elelay/events{/privacy}","received_events_url":"https://api.github.com/users/elelay/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Kinto/kinto/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/Kinto/kinto/labels/in%20progress","name":"in progress","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"leplatrem","id":546692,"avatar_url":"https://avatars.githubusercontent.com/u/546692?v=3","gravatar_id":"","url":"https://api.github.com/users/leplatrem","html_url":"https://github.com/leplatrem","followers_url":"https://api.github.com/users/leplatrem/followers","following_url":"https://api.github.com/users/leplatrem/following{/other_user}","gists_url":"https://api.github.com/users/leplatrem/gists{/gist_id}","starred_url":"https://api.github.com/users/leplatrem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leplatrem/subscriptions","organizations_url":"https://api.github.com/users/leplatrem/orgs","repos_url":"https://api.github.com/users/leplatrem/repos","events_url":"https://api.github.com/users/leplatrem/events{/privacy}","received_events_url":"https://api.github.com/users/leplatrem/received_events","type":"User","site_admin":false},"assignees":[{"login":"leplatrem","id":546692,"avatar_url":"https://avatars.githubusercontent.com/u/546692?v=3","gravatar_id":"","url":"https://api.github.com/users/leplatrem","html_url":"https://github.com/leplatrem","followers_url":"https://api.github.com/users/leplatrem/followers","following_url":"https://api.github.com/users/leplatrem/following{/other_user}","gists_url":"https://api.github.com/users/leplatrem/gists{/gist_id}","starred_url":"https://api.github.com/users/leplatrem/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leplatrem/subscriptions","organizations_url":"https://api.github.com/users/leplatrem/orgs","repos_url":"https://api.github.com/users/leplatrem/repos","events_url":"https://api.github.com/users/leplatrem/events{/privacy}","received_events_url":"https://api.github.com/users/leplatrem/received_events","type":"User","site_admin":false}],"milestone":null,"comments":6,"created_at":"2016-08-16T15:06:06Z","updated_at":"2016-08-18T12:41:56Z","closed_at":null,"body":"I've written a nodejs program to batch insert ~4000 tickets in a collection.\r\nI've batched them by 200 or less.\r\nI force the id to the ticket id, but there are no duplicate ids.\r\nEach batch is in a Promise, calling request. I've first combined the Promises with Promise.all, then serialized them, but get the same error.\r\n\r\nSome posts fail with code 500, with following error on the docker run terminal.\r\nIs this a known limitation of the default backend? An error in my code?\r\n\r\n```\r\n2016-08-16 14:57:02,332 ERROR [kinto.core.views.errors][waitress] \"POST  /v1/buckets/e3e5dfdd-b5de-e9dd-e2ad-637fc2c6481f/collections/tickets/records\" ? (? ms) dictionary changed size during iteration agent=None authn_type=basicauth collection_id=record collection_timestamp=1471359417323 errno=None exception=Traceback (most recent call last):\r\n  File \"/code/kinto/core/utils.py\", line 322, in follow_subrequest\r\n    return request.invoke_subrequest(subrequest, **kwargs), subrequest\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 211, in invoke_subrequest\r\n    response = handle_request(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 158, in handle_request\r\n    view_name\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/view.py\", line 547, in _call_view\r\n    response = view_callable(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 413, in viewresult_to_response\r\n    result = view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 147, in _requestonly_view\r\n    response = view(request)\r\n  File \"/code/kinto/plugins/default_bucket/__init__.py\", line 159, in default_bucket\r\n    response = request.invoke_subrequest(subrequest)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 211, in invoke_subrequest\r\n    response = handle_request(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 158, in handle_request\r\n    view_name\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/view.py\", line 547, in _call_view\r\n    response = view_callable(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/config/views.py\", line 182, in __call__\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 393, in attr_view\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 371, in predicate_wrapper\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 302, in _secured_view\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 442, in rendered_view\r\n    result = view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 147, in _requestonly_view\r\n    response = view(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/cornice/service.py\", line 571, in wrapper\r\n    response = view_()\r\n  File \"/code/kinto/core/resource/__init__.py\", line 311, in collection_post\r\n    existing = self._get_record_or_404(_id)\r\n  File \"/code/kinto/core/resource/__init__.py\", line 681, in _get_record_or_404\r\n    return self.model.get_record(record_id)\r\n  File \"/code/kinto/core/resource/model.py\", line 276, in get_record\r\n    permissions = self.permission.get_object_permissions(perm_object_id)\r\n  File \"/code/kinto/core/permission/__init__.py\", line 136, in get_object_permissions\r\n    perms = self.get_objects_permissions([object_id], permissions)\r\n  File \"/code/kinto/core/permission/memory.py\", line 114, in get_objects_permissions\r\n    aces = [k for k in self._store.keys()\r\n  File \"/code/kinto/core/permission/memory.py\", line 114, in <listcomp>\r\n    aces = [k for k in self._store.keys()\r\nRuntimeError: dictionary changed size during iteration lang=None uid=84ffd9ad5044b2b77b40e0f96fee3112f02894fcbe0eec6dfb8008394955ab7f\r\n2016-08-16 14:57:02,351 INFO  [kinto.core.views.batch][waitress] \"POST  /v1/buckets/e3e5dfdd-b5de-e9dd-e2ad-637fc2c6481f/collections/tickets/records\" 200 (22 ms) subrequest.summary agent=None authn_type=basicauth collection_id=record collection_timestamp=1471359422318 errno=None lang=None time=2016-08-16T14:57:02.351000 uid=84ffd9ad5044b2b77b40e0f96fee3112f02894fcbe0eec6dfb8008394955ab7f\r\n2016-08-16 14:57:02,376 INFO  [kinto.core.views.batch][waitress] \"POST  /v1/buckets/e3e5dfdd-b5de-e9dd-e2ad-637fc2c6481f/collections/tickets/records\" 201 (33 ms) subrequest.summary agent=None authn_type=basicauth collection_id=record collection_timestamp=1471359422318 errno=110 lang=None time=2016-08-16T14:57:02.376000 uid=84ffd9ad5044b2b77b40e0f96fee3112f02894fcbe0eec6dfb8008394955ab7f\r\n2016-08-16 14:57:02,381 INFO  [kinto.core.views.batch][waitress] \"POST  /v1/buckets/e3e5dfdd-b5de-e9dd-e2ad-637fc2c6481f/collections/tickets/records\" 200 (30 ms) subrequest.summary agent=None authn_type=basicauth collection_id=record collection_timestamp=1471359422318 errno=None lang=None time=2016-08-16T14:57:02.381000 uid=84ffd9ad5044b2b77b40e0f96fee3112f02894fcbe0eec6dfb8008394955ab7f\r\n2016-08-16 14:57:02,381 ERROR [kinto.core.views.errors][waitress] \"POST  /v1/buckets/e3e5dfdd-b5de-e9dd-e2ad-637fc2c6481f/collections/tickets/records\" ? (? ms) dictionary changed size during iteration agent=None authn_type=basicauth collection_id=record collection_timestamp=1471359417323 errno=999 exception=Traceback (most recent call last):\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/tweens.py\", line 22, in excview_tween\r\n    response = handler(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid_tm/__init__.py\", line 101, in tm_tween\r\n    reraise(*exc_info)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid_tm/compat.py\", line 15, in reraise\r\n    raise value\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid_tm/__init__.py\", line 83, in tm_tween\r\n    response = handler(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 158, in handle_request\r\n    view_name\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/view.py\", line 547, in _call_view\r\n    response = view_callable(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/config/views.py\", line 182, in __call__\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 393, in attr_view\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 371, in predicate_wrapper\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 442, in rendered_view\r\n    result = view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 147, in _requestonly_view\r\n    response = view(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/cornice/service.py\", line 573, in wrapper\r\n    response = view_(request)\r\n  File \"/code/kinto/core/views/batch.py\", line 100, in post_batch\r\n    use_tweens=False)\r\n  File \"/code/kinto/core/utils.py\", line 326, in follow_subrequest\r\n    raise e\r\n  File \"/code/kinto/core/utils.py\", line 322, in follow_subrequest\r\n    return request.invoke_subrequest(subrequest, **kwargs), subrequest\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 211, in invoke_subrequest\r\n    response = handle_request(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 158, in handle_request\r\n    view_name\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/view.py\", line 547, in _call_view\r\n    response = view_callable(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 413, in viewresult_to_response\r\n    result = view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 147, in _requestonly_view\r\n    response = view(request)\r\n  File \"/code/kinto/plugins/default_bucket/__init__.py\", line 159, in default_bucket\r\n    response = request.invoke_subrequest(subrequest)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 211, in invoke_subrequest\r\n    response = handle_request(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/router.py\", line 158, in handle_request\r\n    view_name\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/view.py\", line 547, in _call_view\r\n    response = view_callable(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/config/views.py\", line 182, in __call__\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 393, in attr_view\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 371, in predicate_wrapper\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 302, in _secured_view\r\n    return view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 442, in rendered_view\r\n    result = view(context, request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/pyramid/viewderivers.py\", line 147, in _requestonly_view\r\n    response = view(request)\r\n  File \"/usr/local/lib/python3.5/dist-packages/cornice/service.py\", line 571, in wrapper\r\n    response = view_()\r\n  File \"/code/kinto/core/resource/__init__.py\", line 311, in collection_post\r\n    existing = self._get_record_or_404(_id)\r\n  File \"/code/kinto/core/resource/__init__.py\", line 681, in _get_record_or_404\r\n    return self.model.get_record(record_id)\r\n  File \"/code/kinto/core/resource/model.py\", line 276, in get_record\r\n    permissions = self.permission.get_object_permissions(perm_object_id)\r\n  File \"/code/kinto/core/permission/__init__.py\", line 136, in get_object_permissions\r\n    perms = self.get_objects_permissions([object_id], permissions)\r\n  File \"/code/kinto/core/permission/memory.py\", line 114, in get_objects_permissions\r\n    aces = [k for k in self._store.keys()\r\n  File \"/code/kinto/core/permission/memory.py\", line 114, in <listcomp>\r\n    aces = [k for k in self._store.keys()\r\nRuntimeError: dictionary changed size during iteration lang=None uid=84ffd9ad5044b2b77b40e0f96fee3112f02894fcbe0eec6dfb8008394955ab7f\r\n```"},"comment":{"url":"https://api.github.com/repos/Kinto/kinto/issues/comments/240711016","html_url":"https://github.com/Kinto/kinto/issues/759#issuecomment-240711016","issue_url":"https://api.github.com/repos/Kinto/kinto/issues/759","id":240711016,"user":{"login":"elelay","id":222469,"avatar_url":"https://avatars.githubusercontent.com/u/222469?v=3","gravatar_id":"","url":"https://api.github.com/users/elelay","html_url":"https://github.com/elelay","followers_url":"https://api.github.com/users/elelay/followers","following_url":"https://api.github.com/users/elelay/following{/other_user}","gists_url":"https://api.github.com/users/elelay/gists{/gist_id}","starred_url":"https://api.github.com/users/elelay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elelay/subscriptions","organizations_url":"https://api.github.com/users/elelay/orgs","repos_url":"https://api.github.com/users/elelay/repos","events_url":"https://api.github.com/users/elelay/events{/privacy}","received_events_url":"https://api.github.com/users/elelay/received_events","type":"User","site_admin":false},"created_at":"2016-08-18T12:41:56Z","updated_at":"2016-08-18T12:41:56Z","body":"This may be too much mutex!\r\nI've switched to running from source:\r\n - default config, simply increased max_requests to 200\r\n - make serve\r\n\r\nWith master I get the exceptions listed in description but my program terminates.\r\n\r\nWith 759-fix-memory-mutual-exclusion I don't get the exceptions BUT my program doesn't terminate: cpu goes idle, but 3 out of 48 batch requests don't return. As soon as I interrupt kinto (Ctrl-C in its console) my program terminates (with socket errors from the returning requests).\r\n\r\nThere are no errors in the log file. Counting \"201\" lines, I get a total of 8438, while there are 8647 items to insert, so the program is stuck before it has inserted everything."}}
{"actor":{"display_login":"elelay"}}