{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb/issues/1096","repository_url":"https://api.github.com/repos/apache/couchdb","labels_url":"https://api.github.com/repos/apache/couchdb/issues/1096/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb/issues/1096/comments","events_url":"https://api.github.com/repos/apache/couchdb/issues/1096/events","html_url":"https://github.com/apache/couchdb/issues/1096","id":286597975,"number":1096,"title":"Sporadic 500 on writing attachment","user":{"login":"jvegaseg","id":12508470,"avatar_url":"https://avatars0.githubusercontent.com/u/12508470?v=4","gravatar_id":"","url":"https://api.github.com/users/jvegaseg","html_url":"https://github.com/jvegaseg","followers_url":"https://api.github.com/users/jvegaseg/followers","following_url":"https://api.github.com/users/jvegaseg/following{/other_user}","gists_url":"https://api.github.com/users/jvegaseg/gists{/gist_id}","starred_url":"https://api.github.com/users/jvegaseg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jvegaseg/subscriptions","organizations_url":"https://api.github.com/users/jvegaseg/orgs","repos_url":"https://api.github.com/users/jvegaseg/repos","events_url":"https://api.github.com/users/jvegaseg/events{/privacy}","received_events_url":"https://api.github.com/users/jvegaseg/received_events","type":"User","site_admin":false},"labels":[{"id":594290317,"url":"https://api.github.com/repos/apache/couchdb/labels/bug","name":"bug","color":"EE3F46","default":true},{"id":594289841,"url":"https://api.github.com/repos/apache/couchdb/labels/dbcore","name":"dbcore","color":"BFD4F2","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":16,"created_at":"2018-01-07T20:49:15Z","updated_at":"2018-02-15T21:58:44Z","closed_at":null,"author_association":"NONE","body":"<!--- Provide a general summary of the issue in the Title above -->\r\nWe have a 6-node cluster configured with q=16. We are migrating attachments from our old storage system to CouchDB. We first create the document in CouchDB database and immediately we issue a request in order to add the attachment associated to the document previously created.\r\n\r\nEverything work well but sometimes we receive a 500 Internal Server Error when trying to add an attachment. After the error, we wait 10 seconds and retry the same request and it works without problems.\r\n\r\nWe would like to know the cause of these sporadic errors. \r\n\r\nServers are very idle (CPU and disk activity).\r\n\r\n## Expected Behavior\r\n<!--- If you're describing a bug, tell us what should happen -->\r\n<!--- If you're suggesting a change/improvement, tell us how it should work -->\r\nIt should add the attachment to the document without errors.\r\n\r\n## Current Behavior\r\n<!--- If describing a bug, tell us what happens instead of the expected behavior -->\r\n<!--- If suggesting a change/improvement, explain the difference from current behavior -->\r\nSporadically, it gets an 500 error due to a \"mfa: fabric_rpc:update_docs/3 exit:timeout\". \r\n\r\n## CouchDB logs\r\n\r\nIn server who answer the request:\r\n\r\n> [error] 2018-01-07T19:12:15.517475Z couchdb@X.X.X.122 <0.14999.1143> f0ef070d10 fabric_worker_timeout update_docs,'couchdb@X.X.X.124',<<\"shards/10000000-1fffffff/ficheros.1514665102\">>\r\n\r\n\r\nIn other server for same request:\r\n\r\n> [error] 2018-01-07T19:21:15.517744Z couchdb@X.X.X.123 <0.25589.1086> f0ef070d10 rexi_server: from: couchdb@X.X.X.122(<13938.14999.1143>) mfa: fabric_rpc:update_docs/3 exit:timeout [{fabric_doc_attachments,'-receiver/2-fun-3-',1,[{file,\"src/fabric_doc_attachments.erl\"},{line,43}]},{couch_att,write_streamed_attachment,3,[{file,\"src/couch_att.erl\"},{line,577}]},{couch_db,with_stream,3,[{file,\"src/couch_db.erl\"},{line,1335}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,\"src/couch_db.erl\"},{line,1286}]},{couch_db,doc_flush_atts,2,[{file,\"src/couch_db.erl\"},{line,1286}]},{couch_db,'-update_docs/4-lc$^8/1-8-',2,[{file,\"src/couch_db.erl\"},{line,1141}]},{couch_db,'-update_docs/4-lc$^7/1-7-',2,[{file,\"src/couch_db.erl\"},{line,1140}]},{couch_db,update_docs,4,[{file,\"src/couch_db.erl\"},{line,1140}]}]\r\n\r\n\r\nIn other server for same request:\r\n\r\n> [error] 2018-01-07T19:21:15.518778Z couchdb@X.X.X.124 <0.17893.1127> f0ef070d10 rexi_server: from: couchdb@X.X.X.122(<13938.14999.1143>) mfa: fabric_rpc:update_docs/3 exit:timeout [{fabric_doc_attachments,'-receiver/2-fun-3-',1,[{file,\"src/fabric_doc_attachments.erl\"},{line,43}]},{couch_att,write_streamed_attachment,3,[{file,\"src/couch_att.erl\"},{line,577}]},{couch_db,with_stream,3,[{file,\"src/couch_db.erl\"},{line,1335}]},{couch_db,'-doc_flush_atts/2-lc$^0/1-0-',2,[{file,\"src/couch_db.erl\"},{line,1286}]},{couch_db,doc_flush_atts,2,[{file,\"src/couch_db.erl\"},{line,1286}]},{couch_db,'-update_docs/4-lc$^8/1-8-',2,[{file,\"src/couch_db.erl\"},{line,1141}]},{couch_db,'-update_docs/4-lc$^7/1-7-',2,[{file,\"src/couch_db.erl\"},{line,1140}]},{couch_db,update_docs,4,[{file,\"src/couch_db.erl\"},{line,1140}]}]\r\n\r\n\r\n \r\n\r\n## Possible Solution\r\n<!--- Not obligatory, but suggest a fix/reason for the bug, -->\r\n<!--- or ideas how to implement the addition or change -->\r\nRetrying the request works well.\r\n\r\n## Configuration\r\nWe use default settings except:\r\n\r\nlocal.ini\r\n```\r\n[cluster]\r\nq = 16\r\n\r\n[log]\r\nlevel = warning\r\n```\r\n\r\nvm.args\r\n```\r\n-name couchdb@X.X.X.120\r\n-setcookie XXXXXXXXXXXXXX\r\n-kernel inet_dist_listen_min 9100\r\n-kernel inet_dist_listen_max 9200\r\n\r\n-kernel error_logger silent\r\n-sasl sasl_error_logger false\r\n\r\n+K true\r\n+A 16\r\n+Bd -noinput\r\n```\r\n\r\nsys.config\r\n\r\n```\r\n[\r\n    {lager, [\r\n        {error_logger_hwm, 1000},\r\n        {error_logger_redirect, true},\r\n        {handlers, [\r\n            {lager_console_backend, [debug, {\r\n                lager_default_formatter,\r\n                [\r\n                    date, \" \", time,\r\n                    \" [\", severity, \"] \",\r\n                    node, \" \", pid, \" \",\r\n                    message,\r\n                    \"\\n\"\r\n                ]\r\n            }]}\r\n        ]},\r\n        {inet_dist_listen_min, 9100},\r\n        {inet_dist_listen_max, 9200}\r\n    ]}\r\n].\r\n```\r\n\r\n## Environment\r\nSoftware is running on virtualized Centos 7 using OpenVZ 7.\r\nEach node has same configuration.\r\nEach node has 16 cores.\r\n\r\nSome command outputs.\r\n**free -h**\r\n```\r\n              total        used        free      shared  buff/cache   available\r\nMem:            40G        319M         28K        1,3M         39G         36G\r\nSwap:          100G        4,6M         99G\r\n```\r\n\r\n**top**\r\n```\r\ntop - 22:02:37 up 9 days, 22:35,  1 user,  load average: 0,41, 0,25, 0,23\r\nTasks:  37 total,   1 running,  36 sleeping,   0 stopped,   0 zombie\r\n%Cpu(s):  0,5 us,  0,1 sy,  0,0 ni, 99,4 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st\r\nKiB Mem : 41943040 total,    70868 free,   284816 used, 41587356 buff/cache\r\nKiB Swap: 10485760+total, 10485286+free,     4740 used. 38381588 avail Mem  \r\n\r\n```\r\n\r\n**df -h**\r\n```\r\nS.ficheros        Tamaño Usados  Disp Uso% Montado en\r\n/dev/ploop23569p1   9,9T    72G  9,4T   1% /\r\nnone                 31G      0   31G   0% /sys/fs/cgroup\r\nnone                 31G      0   31G   0% /dev\r\ntmpfs                20G      0   20G   0% /dev/shm\r\ntmpfs                20G   144K   20G   1% /run\r\ntmpfs               4,0G      0  4,0G   0% /run/user/0\r\n```\r\n  "},"comment":{"url":"https://api.github.com/repos/apache/couchdb/issues/comments/366075429","html_url":"https://github.com/apache/couchdb/issues/1096#issuecomment-366075429","issue_url":"https://api.github.com/repos/apache/couchdb/issues/1096","id":366075429,"user":{"login":"wohali","id":112292,"avatar_url":"https://avatars2.githubusercontent.com/u/112292?v=4","gravatar_id":"","url":"https://api.github.com/users/wohali","html_url":"https://github.com/wohali","followers_url":"https://api.github.com/users/wohali/followers","following_url":"https://api.github.com/users/wohali/following{/other_user}","gists_url":"https://api.github.com/users/wohali/gists{/gist_id}","starred_url":"https://api.github.com/users/wohali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wohali/subscriptions","organizations_url":"https://api.github.com/users/wohali/orgs","repos_url":"https://api.github.com/users/wohali/repos","events_url":"https://api.github.com/users/wohali/events{/privacy}","received_events_url":"https://api.github.com/users/wohali/received_events","type":"User","site_admin":false},"created_at":"2018-02-15T21:58:44Z","updated_at":"2018-02-15T21:58:44Z","author_association":"MEMBER","body":"@jvegaseg can you confirm that the `?w=3` workaround provided by @davisp was insufficient to solve the problem?"}}
{"actor":{"display_login":"wohali"}}