{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb/issues/1119","repository_url":"https://api.github.com/repos/apache/couchdb","labels_url":"https://api.github.com/repos/apache/couchdb/issues/1119/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb/issues/1119/comments","events_url":"https://api.github.com/repos/apache/couchdb/issues/1119/events","html_url":"https://github.com/apache/couchdb/issues/1119","id":290443077,"number":1119,"title":"Could not open shard","user":{"login":"sergey-safarov","id":2562241,"avatar_url":"https://avatars2.githubusercontent.com/u/2562241?v=4","gravatar_id":"","url":"https://api.github.com/users/sergey-safarov","html_url":"https://github.com/sergey-safarov","followers_url":"https://api.github.com/users/sergey-safarov/followers","following_url":"https://api.github.com/users/sergey-safarov/following{/other_user}","gists_url":"https://api.github.com/users/sergey-safarov/gists{/gist_id}","starred_url":"https://api.github.com/users/sergey-safarov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergey-safarov/subscriptions","organizations_url":"https://api.github.com/users/sergey-safarov/orgs","repos_url":"https://api.github.com/users/sergey-safarov/repos","events_url":"https://api.github.com/users/sergey-safarov/events{/privacy}","received_events_url":"https://api.github.com/users/sergey-safarov/received_events","type":"User","site_admin":false},"labels":[{"id":594290869,"url":"https://api.github.com/repos/apache/couchdb/labels/docker","name":"docker","color":"FAD8C7","default":false},{"id":594291716,"url":"https://api.github.com/repos/apache/couchdb/labels/waiting%20on%20user","name":"waiting on user","color":"FBCA04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-01-22T12:08:03Z","updated_at":"2018-02-02T07:08:52Z","closed_at":null,"author_association":"NONE","body":"## Expected Behavior\r\nCouchDB must open shards without any error.\r\n\r\n## Current Behavior\r\nWhen replicated database from other server to standalone CouchDB time to time i see errors like this\r\n```\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [notice] 2018-01-15T10:34:47.577647Z couchdb@couchdb1.kazoo <0.25708.3> -------- 10.0.9.10 - - GET / 200\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [notice] 2018-01-15T10:34:47.880163Z couchdb@couchdb1.kazoo <0.25433.3> -------- Retrying GET request to http://10.0.9.9:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/ in 4.0 seconds due to error {code,500}\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.893174Z couchdb@couchdb1.kazoo <0.25726.3> -------- Could not open file ./data/shards/00000000-1fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.893399Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/00000000-1fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.894595Z couchdb@couchdb1.kazoo <0.25730.3> -------- Could not open file ./data/shards/20000000-3fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.895033Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/20000000-3fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.895562Z couchdb@couchdb1.kazoo <0.25734.3> -------- Could not open file ./data/shards/40000000-5fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.895868Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/40000000-5fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.896352Z couchdb@couchdb1.kazoo <0.25738.3> -------- Could not open file ./data/shards/60000000-7fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.896630Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/60000000-7fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.897154Z couchdb@couchdb1.kazoo <0.25742.3> -------- Could not open file ./data/shards/80000000-9fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.897454Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/80000000-9fffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.897923Z couchdb@couchdb1.kazoo <0.25746.3> -------- Could not open file ./data/shards/a0000000-bfffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.898239Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/a0000000-bfffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.898754Z couchdb@couchdb1.kazoo <0.25750.3> -------- Could not open file ./data/shards/c0000000-dfffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.899083Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/c0000000-dfffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.899571Z couchdb@couchdb1.kazoo <0.25754.3> -------- Could not open file ./data/shards/e0000000-ffffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479.couch: no such file or directory\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [info] 2018-01-15T10:34:47.899888Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/e0000000-ffffffff/account/51/b3/d5b4883ebe3261b1250e401deca0.1516012479\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.900597Z couchdb@couchdb1.kazoo <0.25433.3> -------- Replicator, request GET to \"http://10.0.9.9:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/\" failed due to error {code,500}\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.900935Z couchdb@couchdb1.kazoo <0.25433.3> -------- exit:{http_error_code,500,<<\"GET http://10.0.9.9:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/\">>}: Replication 77d0d1cc310aa875647f25011de294cf+create_target failed to start \"http://217.12.247.210:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/\" -> \"http://10.0.9.9:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/\" doc null:null stack:[{couch_replicator_api_wrap,db_open,3,[{file,\"src/couch_replicator_api_wrap.erl\"},{line,119}]},{couch_replicator_scheduler_job,init_state,1,[{file,\"src/couch_replicator_scheduler_job.erl\"},{line,590}]}]\r\nJan 15 10:34:47 node0.docker.rcsnet.ru sh[1690]: [error] 2018-01-15T10:34:47.901167Z couchdb@couchdb1.kazoo <0.357.0> -------- couch_replicator_scheduler : Transient job {\"77d0d1cc310aa875647f25011de294cf\",\"+create_target\"} failed, removing. Error: <<\"{http_error_code,500,\\n                 <<\\\"GET http://10.0.9.9:5984/account%2f51%2fb3%2fd5b4883ebe3261b1250e401deca0/\\\">>}\">>\r\nJan 15 10:34:48 node0.docker.rcsnet.ru sh[1690]: [notice] 2018-01-15T10:34:48.682942Z couchdb@couchdb1.kazoo <0.25757.3> 8f8d36eae3 undefined 10.0.9.10 undefined GET / 200 ok 0\r\nJan 15 10:34:49 node0.docker.rcsnet.ru sh[1690]: [notice] 2018-01-15T10:34:49.578731Z couchdb@couchdb1.kazoo <0.25774.3> -------- 10.0.9.10 - - GET / 200\r\n```\r\n\r\n## Possible Solution\r\nNot sure but this may by thread sync issue. Similar error is described here for other erlang program.\r\n[link 1](https://github.com/2600hz/kazoo/pull/3809)\r\n[link2](https://2600hz.atlassian.net/browse/KAZOO-5354)\r\n\r\n## Steps to Reproduce (for bugs)\r\n\r\n1. start docker container\r\n2. start replication about 1500 databases (total 25 Gb) from other server. \r\n\r\n## Context\r\nI want migrate date from other CouchDB server.\r\n\r\n## Your Environment\r\n* Version used: [docker image](https://hub.docker.com/r/apache/couchdb/) tag 2.1.1\r\n* Browser Name and version: Google Chrome 63.0.3239.132 (Official Build) (64-bit)\r\n* Operating System and version (desktop or mobile): Fedora 27 + Docker 17.12.0-ce\r\n* Link to your project: none\r\n"},"comment":{"url":"https://api.github.com/repos/apache/couchdb/issues/comments/362504677","html_url":"https://github.com/apache/couchdb/issues/1119#issuecomment-362504677","issue_url":"https://api.github.com/repos/apache/couchdb/issues/1119","id":362504677,"user":{"login":"sergey-safarov","id":2562241,"avatar_url":"https://avatars2.githubusercontent.com/u/2562241?v=4","gravatar_id":"","url":"https://api.github.com/users/sergey-safarov","html_url":"https://github.com/sergey-safarov","followers_url":"https://api.github.com/users/sergey-safarov/followers","following_url":"https://api.github.com/users/sergey-safarov/following{/other_user}","gists_url":"https://api.github.com/users/sergey-safarov/gists{/gist_id}","starred_url":"https://api.github.com/users/sergey-safarov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sergey-safarov/subscriptions","organizations_url":"https://api.github.com/users/sergey-safarov/orgs","repos_url":"https://api.github.com/users/sergey-safarov/repos","events_url":"https://api.github.com/users/sergey-safarov/events{/privacy}","received_events_url":"https://api.github.com/users/sergey-safarov/received_events","type":"User","site_admin":false},"created_at":"2018-02-02T07:08:52Z","updated_at":"2018-02-02T07:08:52Z","author_association":"NONE","body":"Joan (@wohali)\r\nFirst i measured disk io operation when `tmpfs` mounted as volume inside container.\r\n```\r\n[root@c0ed791b0e7d /]# rm -f /workspace/* && time dd if=/dev/zero of=/opt/couchdb/data/test_container1.img bs=512 count=1000 oflag=dsync\r\n1000+0 records in\r\n1000+0 records out\r\n512000 bytes (512 kB, 500 KiB) copied, 0.00277375 s, 185 MB/s\r\n\r\nreal\t0m0.004s\r\nuser\t0m0.003s\r\nsys\t0m0.001s\r\n[root@c0ed791b0e7d /]# rm -f /workspace/* && time dd if=/dev/zero of=/opt/couchdb/data/test_container1.img bs=512 count=1000 oflag=dsync\r\n1000+0 records in\r\n1000+0 records out\r\n512000 bytes (512 kB, 500 KiB) copied, 0.00277794 s, 184 MB/s\r\n\r\nreal\t0m0.004s\r\nuser\t0m0.001s\r\nsys\t0m0.003s\r\n[root@c0ed791b0e7d /]# rm -f /workspace/* && time dd if=/dev/zero of=/opt/couchdb/data/test_container1.img bs=512 count=1000 oflag=dsync\r\n1000+0 records in\r\n1000+0 records out\r\n512000 bytes (512 kB, 500 KiB) copied, 0.00283199 s, 181 MB/s\r\n\r\nreal\t0m0.004s\r\nuser\t0m0.001s\r\nsys\t0m0.003s\r\n[root@c0ed791b0e7d /]# rm -f /workspace/* && time dd if=/dev/zero of=/opt/couchdb/data/test_container1.img bs=512 count=1000 oflag=dsync\r\n1000+0 records in\r\n1000+0 records out\r\n512000 bytes (512 kB, 500 KiB) copied, 0.00279983 s, 183 MB/s\r\n\r\nreal\t0m0.004s\r\nuser\t0m0.001s\r\nsys\t0m0.003s\r\n```\r\nThen i started couchdb container with mounter `tmpfs` as volume.  Added arg `--mount type=tmpfs,destination=/opt/couchdb/data`\r\n ```\r\ndocker run -t --rm=true --log-driver=none --network kazoo --name couchdb1 \\\r\n                 --hostname couchdb1.kazoo \\\r\n                 --ip 10.0.9.8 \\\r\n                 --ulimit nofile=999999 \\\r\n                 --mount type=tmpfs,destination=/opt/couchdb/data \\\r\n                 -v /etc/kazoo/couchdb/vm.args.node1:/opt/couchdb/etc/vm.args \\\r\n                 apache/couchdb:2.1.1\r\n```\r\n`vm.args` is same to default with added option `name`\r\n```\r\n[root@node0 ~]# docker exec -it couchdb1 cat /opt/couchdb/etc/vm.args\r\n# Licensed under the Apache License, Version 2.0 (the \"License\"); you may not\r\n# use this file except in compliance with the License. You may obtain a copy of\r\n# the License at\r\n#\r\n#   http://www.apache.org/licenses/LICENSE-2.0\r\n#\r\n# Unless required by applicable law or agreed to in writing, software\r\n# distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\r\n# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\r\n# License for the specific language governing permissions and limitations under\r\n# the License.\r\n\r\n# Ensure that the Erlang VM listens on a known port\r\n-kernel inet_dist_listen_min 9100\r\n-kernel inet_dist_listen_max 9100\r\n\r\n# Tell kernel and SASL not to log anything\r\n-kernel error_logger silent\r\n-sasl sasl_error_logger false\r\n\r\n# Use kernel poll functionality if supported by emulator\r\n+K true\r\n\r\n# Start a pool of asynchronous IO threads\r\n+A 16\r\n\r\n# Comment this line out to enable the interactive Erlang shell on startup\r\n+Bd -noinput\r\n-setcookie monster\r\n-name couchdb@couchdb1.kazoo\r\n\r\n[root@node0 ~]# \r\n```\r\nNow disk layout inside container\r\n```\r\n[root@node0 ~]# docker exec -it couchdb1 df\r\nFilesystem                  1K-blocks     Used Available Use% Mounted on\r\noverlay                     268304384 76159032 192145352  29% /\r\ntmpfs                           65536        0     65536   0% /dev\r\ntmpfs                        65968072        0  65968072   0% /sys/fs/cgroup\r\n/dev/mapper/vgRAID10-root     2086912   157692   1929220   8% /etc/couchdb\r\n/dev/mapper/vgRAID10-docker 268304384 76159032 192145352  29% /etc/hosts\r\nshm                             65536        0     65536   0% /dev/shm\r\ntmpfs                        65968072       48  65968024   1% /opt/couchdb/data\r\ntmpfs                        65968072        0  65968072   0% /proc/scsi\r\ntmpfs                        65968072        0  65968072   0% /sys/firmware\r\n```\r\nHost total memory. Host have 128Gb of memory and used about 160 Mb of swap.\r\n```\r\ntop - 01:41:43 up 17 days, 15:46,  4 users,  load average: 1.78, 1.54, 0.87\r\nTasks: 900 total,   1 running, 283 sleeping,   0 stopped, 486 zombie\r\n%Cpu(s):  0.8 us,  0.8 sy,  0.0 ni, 98.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st\r\nKiB Mem : 13193614+total, 12358560 free,  2934528 used, 11664305+buff/cache\r\nKiB Swap: 31457280+total, 31441177+free,   161036 used. 12743585+avail Mem \r\n\r\n  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                \r\n 1332 root      20   0 1469936 452596   4084 S   6.3  0.3 587:28.21 beam.smp                                                                                                                               \r\n14915 root      20   0 1297128 295484   3928 S   5.6  0.2 510:51.97 beam.smp                                                                                                                               \r\n20715 root      20   0   54392   5120   3812 R   0.7  0.0   0:00.09 top\r\n```\r\nThen i started replication and after 5 hours i have folowing results.\r\nDisk layout inside couchdb container. CoucDB data size about 10Gb\r\n```\r\n[root@node0 ~]# docker exec -it couchdb1 df\r\nFilesystem                  1K-blocks     Used Available Use% Mounted on\r\noverlay                     268304384 76134184 192170200  29% /\r\ntmpfs                           65536        0     65536   0% /dev\r\ntmpfs                        65968072        0  65968072   0% /sys/fs/cgroup\r\n/dev/mapper/vgRAID10-root     2086912   157692   1929220   8% /etc/couchdb\r\n/dev/mapper/vgRAID10-docker 268304384 76134184 192170200  29% /etc/hosts\r\nshm                             65536        0     65536   0% /dev/shm\r\ntmpfs                        65968072 10083456  55884616  16% /opt/couchdb/data\r\ntmpfs                        65968072        0  65968072   0% /proc/scsi\r\ntmpfs                        65968072        0  65968072   0% /sys/firmware\r\n```\r\nHost memory usage. Swap usage not changed. All CouchDB data is located inside memory.\r\n```\r\ntop - 06:35:51 up 17 days, 20:40,  6 users,  load average: 1.03, 0.95, 0.82\r\nTasks: 938 total,   1 running, 322 sleeping,   0 stopped, 486 zombie\r\n%Cpu(s):  1.4 us,  1.1 sy,  0.0 ni, 97.2 id,  0.1 wa,  0.1 hi,  0.1 si,  0.0 st\r\nKiB Mem : 13193614+total, 87901344 free,  3644528 used, 40390272 buff/cache\r\nKiB Swap: 31457280+total, 31441177+free,   161036 used. 11659586+avail Mem \r\n\r\n  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                \r\n 5269 root      20   0   56704   5360   3876 R  11.8  0.0   0:00.03 top                                                                                                                                    \r\n14915 root      20   0 1315680 307456   5852 S  11.8  0.2 550:18.26 beam.smp                                                                                                                               \r\n 1332 root      20   0 1463408 452628   5444 S   5.9  0.3 615:53.69 beam.smp                                                                                                                               \r\n23036 root      20   0 1859228  63316  31484 S   5.9  0.0  30:43.66 dockerd \r\n```\r\nAfter 5 hours of replication to database located on memory disk (`tmpfs`) couchdb have following errors in log\r\n**Could not open shards** - total 2 error\r\n```\r\n2ea5274cd52c5bf0-201802/_local/75d3dc6ddfdfcae76e87a7acdfc5ed89 201 ok 42\r\nFeb 02 03:50:57 node0.docker.rcsnet.ru sh[28220]: [notice] 2018-02-02T03:50:57.866252Z couchdb@couchdb1.kazoo <0.357.0> -------- couch_replicator_scheduler: Job {\"75d3dc6ddfdfcae76e87a7acdfc5ed89\",\"+create_target\"} completed normally\r\nFeb 02 03:50:57 node0.docker.rcsnet.ru sh[28220]: [notice] 2018-02-02T03:50:57.866613Z couchdb@couchdb1.kazoo <0.22131.14> 8a931c0f9d 10.0.9.8:5984 10.0.9.8 undefined POST /_replicate 200 ok 2805\r\nFeb 02 03:50:57 node0.docker.rcsnet.ru sh[28220]: [notice] 2018-02-02T03:50:57.916222Z couchdb@couchdb1.kazoo <0.357.0> -------- couch_replicator_scheduler: Job {\"80f52a6352af0019a08ff5bf307e6192\",\"+create_target\"} started as <0.10587.13>\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.599271Z couchdb@couchdb1.kazoo <0.14885.14> 4c046843ba Request to create N=3 DB but only 1 node(s)\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [notice] 2018-02-02T03:50:59.629560Z couchdb@couchdb1.kazoo <0.14885.14> 4c046843ba 10.0.9.8:5984 10.0.9.8 undefined PUT /account%2f94%2f93%2f45d6587d8f195f1620b65b1eb063-201802/ 201 ok 32\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.655204Z couchdb@couchdb1.kazoo <0.2117.13> -------- Could not open file ./data/shards/60000000-7fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459.couch: no such file or directory\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.655216Z couchdb@couchdb1.kazoo <0.2121.13> -------- Could not open file ./data/shards/40000000-5fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459.couch: no such file or directory\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [info] 2018-02-02T03:50:59.666163Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/60000000-7fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [info] 2018-02-02T03:50:59.666210Z couchdb@couchdb1.kazoo <0.206.0> -------- open_result error {not_found,no_db_file} for shards/40000000-5fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [warning] 2018-02-02T03:50:59.666204Z couchdb@couchdb1.kazoo <0.2139.13> d9f2f7b03a creating missing database: shards/60000000-7fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [warning] 2018-02-02T03:50:59.666235Z couchdb@couchdb1.kazoo <0.2149.13> d9f2f7b03a creating missing database: shards/40000000-5fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.666644Z couchdb@couchdb1.kazoo <0.303.0> -------- mem3_shards tried to create shards/40000000-5fffffff/account/94/93/45d6587d8f195f1620b65b1eb063-201802.1517543459, got file_exists\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.932281Z couchdb@couchdb1.kazoo <0.16410.7> -------- rexi_server: from: couchdb@couchdb1.kazoo(<0.16407.7>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,\"src/rexi.erl\"},{line,256}]},{rexi,stream2,3,[{file,\"src/rexi.erl\"},{line,204}]},{fabric_rpc,view_cb,2,[{file,\"src/fabric_rpc.erl\"},{line,308}]},{couch_mrview,finish_fold,2,[{file,\"src/couch_mrview.erl\"},{line,642}]},{rexi_server,init_p,3,[{file,\"src/rexi_server.erl\"},{line,139}]}]\r\nFeb 02 03:50:59 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T03:50:59.932290Z couchdb@couchdb1.kazoo <0.16409.7> -------- rexi_server: from: couchdb@couchdb1.kazoo(<0.16407.7>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,\"src/rexi.erl\"},{line,256}]},{rexi,stream2,3,[{file,\"src/rexi.erl\"},{line,204}]},{fabric_rpc,view_cb,2,[{file,\"src/fabric_rpc.erl\"},{line,308}]},{couch_mrview,finish_fold,2,[{file,\"src/couch_mrview.erl\"},{line,642}]},{rexi_server,init_p,3,[{file,\"src/rexi_server.erl\"},{line,139}]}]\r\n```\r\n**Replicator error. Cannot put document** - about 20 errors\r\n```\r\nFeb 02 04:04:51 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T04:04:51.673222Z couchdb@couchdb1.kazoo <0.8179.40> -------- rexi_server: from: couchdb@couchdb1.kazoo(<0.10464.36>) mfa: fabric_rpc:all_docs/3 exit:timeout [{rexi,init_stream,1,[{file,\"src/rexi.erl\"},{line,256}]},{rexi,stream2,3,[{file,\"src/rexi.erl\"},{line,204}]},{fabric_rpc,view_cb,2,[{file,\"src/fabric_rpc.erl\"},{line,308}]},{couch_mrview,finish_fold,2,[{file,\"src/couch_mrview.erl\"},{line,642}]},{rexi_server,init_p,3,[{file,\"src/rexi_server.erl\"},{line,139}]}]\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T04:04:55.012851Z couchdb@couchdb1.kazoo emulator -------- Error in process <0.10651.59> on node 'couchdb@couchdb1.kazoo' with exit value: {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,\"src/couch_att.erl\"},{line,613}]},{couch_att,fold_streamed_data,4,[{file,\"src/couch_att.erl\"},{line,664}]},{couch_att,foldl,4,[{file,\"src/couch_att.erl\"},...\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]: [info] 2018-02-02T04:04:55.013064Z couchdb@couchdb1.kazoo <0.353.0> -------- Replication connection to: \"10.0.9.8\":5984 died with reason {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,\"src/couch_att.erl\"},{line,613}]},{couch_att,fold_streamed_data,4,[{file,\"src/couch_att.erl\"},{line,664}]},{couch_att,foldl,4,[{file,\"src/couch_att.erl\"},{line,617}]},{couch_httpd_multipart,atts_to_mp,4,[{file,\"src/couch_httpd_multipart.erl\"},{line,208}]}]}\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T04:04:55.014366Z couchdb@couchdb1.kazoo <0.7570.55> 8967ff8316 req_err(4199105376) badmatch : ok\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:     [<<\"chttpd_db:db_doc_req/3 L782\">>,<<\"chttpd:process_request/1 L295\">>,<<\"chttpd:handle_request_int/1 L231\">>,<<\"mochiweb_http:headers/6 L91\">>,<<\"proc_lib:init_p_do_apply/3 L237\">>]\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]: [notice] 2018-02-02T04:04:55.014881Z couchdb@couchdb1.kazoo <0.7570.55> 8967ff8316 10.0.9.8:5984 10.0.9.8 undefined PUT /account%2f0e%2fe2%2fb3fccd72d86299cf8f66f6caa6bd-201801/201801-b089783f4734b57244ea8b5613f8375a?new_edits=false 500 ok 2\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]: [error] 2018-02-02T04:04:55.015761Z couchdb@couchdb1.kazoo <0.5748.60> -------- Replicator, request PUT to \"http://10.0.9.8:5984/account%2f0e%2fe2%2fb3fccd72d86299cf8f66f6caa6bd-201801/201801-b089783f4734b57244ea8b5613f8375a?new_edits=false\" failed due to error {error,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:     {'EXIT',\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:         {{{nocatch,{mp_parser_died,noproc}},\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:           [{couch_att,'-foldl/4-fun-0-',3,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:                [{file,\"src/couch_att.erl\"},{line,613}]},\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:            {couch_att,fold_streamed_data,4,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:                [{file,\"src/couch_att.erl\"},{line,664}]},\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:            {couch_att,foldl,4,[{file,\"src/couch_att.erl\"},{line,617}]},\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:            {couch_httpd_multipart,atts_to_mp,4,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:                [{file,\"src/couch_httpd_multipart.erl\"},{line,208}]}]},\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:          {gen_server,call,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:              [<0.13476.55>,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:               {send_req,\r\nFeb 02 04:04:55 node0.docker.rcsnet.ru sh[28220]:                   {{url,\r\n```\r\nAbout replication error is important server IP `10.0.9.8`. This of destination CouchDB server.\r\n\r\nJoan (@wohali), I think issue not related to disk IO speed inside container. Test is made on RAM disk (`tmpfs`) with speed about 180 MB/s. That value is more greater then on real hard disk.\r\n "}}
{"actor":{"display_login":"sergey-safarov"}}