{"action":"opened","issue":{"url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/5866","repository_url":"https://api.github.com/repos/rethinkdb/rethinkdb","labels_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/5866/labels{/name}","comments_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/5866/comments","events_url":"https://api.github.com/repos/rethinkdb/rethinkdb/issues/5866/events","html_url":"https://github.com/rethinkdb/rethinkdb/issues/5866","id":160846721,"number":5866,"title":"Duplicate primary key not thrown on all machines","user":{"login":"drejc","id":1780375,"avatar_url":"https://avatars.githubusercontent.com/u/1780375?v=3","gravatar_id":"","url":"https://api.github.com/users/drejc","html_url":"https://github.com/drejc","followers_url":"https://api.github.com/users/drejc/followers","following_url":"https://api.github.com/users/drejc/following{/other_user}","gists_url":"https://api.github.com/users/drejc/gists{/gist_id}","starred_url":"https://api.github.com/users/drejc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/drejc/subscriptions","organizations_url":"https://api.github.com/users/drejc/orgs","repos_url":"https://api.github.com/users/drejc/repos","events_url":"https://api.github.com/users/drejc/events{/privacy}","received_events_url":"https://api.github.com/users/drejc/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-17T09:22:50Z","updated_at":"2016-06-17T09:22:50Z","closed_at":null,"body":"I have the following set up.\r\n\r\n3 different machines run simultaneously listen to database changes on a table (this is a queue for tasks).\r\nOnce a record is inserted they all try to take this task and execute it, therefore I have created a new table that inserts a lock record setting the task id (as key) and the executor (the machine contending for the task) as additional data. \r\n\r\nLocally I have tested it out with 10 threads and 100 tasks on 2 nodes and everything works as expected. Only one executor gets one task.\r\n\r\nBut on production I can see that two executors get the same task, meaning the duplicate primary key is not thrown. Here are the logs I see ...\r\n\r\n\r\n2016-06-17 08:21:46 [pool-9-thread-104] ubuntu-sb68 Task: a95246d5-502d-4de7-95b9-16c112c77817, locked for executor: WZLm8B7cjL\r\n2016-06-17 08:21:45 [pool-9-thread-114] ubuntu-sb88 Task: a95246d5-502d-4de7-95b9-16c112c77817, locked for executor: Y65eeYfN0X\r\n2016-06-17 08:21:45 [pool-9-thread-93] ubuntu-sb78 DB error response: DB error happened: deleted:0 inserted:0 unchanged:0 replaced:0 first_error:Duplicate primary key `id`: { \"executor\":\t\"Y65eeYfN0X\", \"id\":\t\"a95246d5-502d-4de7-95b9-16c112c77817\", \"timestamp\":\t{ \"$reql_type$\":\t\"TIME\", \"epoch_time\":\t1466144505.658, \"timezone\":\t\"+00:00\" } } { \"executor\":\t\"MF2ORuquFx\", \"id\":\t\"a95246d5-502d-4de7-95b9-16c112c77817\", \"timestamp\":\t{ \"$reql_type$\":\t\"TIME\", \"epoch_time\":\t1466144505.659, \"timezone\":\t\"+00:00\" } } errors:1 skipped:0 \r\n2016-06-17 08:21:45 [pool-8-thread-1] ubuntu-sb88 Queue 'queue' new task: a95246d5-502d-4de7-95b9-16c112c77817 in state: queued, Y65eeYfN0X contending for task!\r\n2016-06-17 08:21:45 [pool-8-thread-1] ubuntu-sb78 Queue 'queue' new task: a95246d5-502d-4de7-95b9-16c112c77817 in state: queued, MF2ORuquFx contending for task!\r\n2016-06-17 08:21:45 [pool-8-thread-1] ubuntu-sb68 Queue 'queue' new task: a95246d5-502d-4de7-95b9-16c112c77817 in state: queued, WZLm8B7cjL contending for task! \r\n\r\n\r\nLet me explain how to read this ... so on the bottom we have 3 executors WZLm8B7cjL, MF2ORuquFx and Y65eeYfN0X, contending for the same task: a95246d5-502d-4de7-95b9-16c112c77817 \r\n\r\nMF2ORuquFx recieves a duplicate primary key error (which is OK) so he is out of the game. Y65eeYfN0X does not. So task gets locked twice, which is not what should be happening.\r\n\r\nWhat bugs me ... the error send to MF2ORuquFx, contains both errors ... for MF2ORuquFx and Y65eeYfN0X - he recieves both errors ... Y65eeYfN0X on the other hand gets none ... \r\n\r\nIt seem to me ... that once the duplicate primary key error is consumed it is considered obsolete, so the third thread never gets informed.\r\n\r\nI was trying to replicate this situation locally, but was unable to. It only happens in production with 3 separate machines running 3 RDB nodes, and 3 threads inserting a new record simultaneously.\r\n\r\nWe are currently running RethinkDB 2.3.2\r\nTable config and write to table is set to: \"durability\":  \"hard\", \"write_acks\":  \"majority\"\r\n\r\nHave I maybe found and bug, or is there some setting that I'm not aware of to fix this situation."}}
{"actor":{"display_login":"drejc"}}