{"action":"opened","issue":{"url":"https://api.github.com/repos/Storj/core/issues/136","repository_url":"https://api.github.com/repos/Storj/core","labels_url":"https://api.github.com/repos/Storj/core/issues/136/labels{/name}","comments_url":"https://api.github.com/repos/Storj/core/issues/136/comments","events_url":"https://api.github.com/repos/Storj/core/issues/136/events","html_url":"https://github.com/Storj/core/issues/136","id":156927833,"number":136,"title":"High i/o on farmer start with existing shards","user":{"login":"BlackDuck888","id":17740762,"avatar_url":"https://avatars.githubusercontent.com/u/17740762?v=3","gravatar_id":"","url":"https://api.github.com/users/BlackDuck888","html_url":"https://github.com/BlackDuck888","followers_url":"https://api.github.com/users/BlackDuck888/followers","following_url":"https://api.github.com/users/BlackDuck888/following{/other_user}","gists_url":"https://api.github.com/users/BlackDuck888/gists{/gist_id}","starred_url":"https://api.github.com/users/BlackDuck888/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/BlackDuck888/subscriptions","organizations_url":"https://api.github.com/users/BlackDuck888/orgs","repos_url":"https://api.github.com/users/BlackDuck888/repos","events_url":"https://api.github.com/users/BlackDuck888/events{/privacy}","received_events_url":"https://api.github.com/users/BlackDuck888/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-05-26T08:10:23Z","updated_at":"2016-05-26T08:10:23Z","closed_at":null,"body":"### Package Versions\r\n\r\nReplace the values below using the output from `npm list storj`.\r\n\r\n```\r\nFarmer: v0.4.3\r\nCore:   v0.6.16\r\n```\r\n\r\nReplace the values below using the output from `node --version`.\r\n\r\n```\r\nv4.4.3\r\n```\r\n\r\n### Expected Behavior\r\n\r\nI expect that the system is responsive after a short periode after the farmer starts.\r\n\r\n### Actual Behavior\r\n\r\nIt looks like, thats the full farmer.db is reread/rewritten after the farmer start/restart. On my system this process takes up to 20 Minutes @ ~44GB Shards. For dedicated servers this problen is secundary, but for users who run Windows and have to use there Computer for \"normal\" things this could be a Problem. If there 500GB Shards on disk, and a reboot is caused/forced by Windows Update, the system will need 3h to be useable.\r\n\r\nI attach the [dstat.csv.txt](https://github.com/Storj/core/files/283700/dstat.csv.txt), in this file you see the start and the high i/o over time.\r\n\r\nAn a Screenshot from Diskusage.\r\n\r\n![diskusage_state](https://cloud.githubusercontent.com/assets/17740762/15567971/ca81e2ce-2329-11e6-9833-4e606d7eb73b.gif)\r\n\r\n\r\n\r\n```\r\nduck@vlan2-farmer:/$ date;pm2 restart all\r\nDo 26. Mai 09:11:48 CEST 2016\r\n[PM2] Applying action restartProcessId on app [all](ids: 0,1,10,20,30)\r\n[PM2] [17EAA_1](0) ?\r\n[PM2] [17EAA_2](1) ?\r\n[PM2] [1AKRj_1](10) ?\r\n[PM2] [1Gtpb_1](20) ?\r\n[PM2] [1Mh32_1](30) ?\r\n+-----------------------------------------------------------------------------------+\r\n¦ App name ¦ id ¦ mode ¦ pid   ¦ status ¦ restart ¦ uptime ¦ memory      ¦ watching ¦\r\n+----------+----+------+-------+--------+---------+--------+-------------+----------¦\r\n¦ 17EAA_1  ¦ 0  ¦ fork ¦ 25863 ¦ online ¦ 1       ¦ 0s     ¦ 29.258 MB   ¦ disabled ¦\r\n¦ 17EAA_2  ¦ 1  ¦ fork ¦ 25868 ¦ online ¦ 1       ¦ 0s     ¦ 25.469 MB   ¦ disabled ¦\r\n¦ 1AKRj_1  ¦ 10 ¦ fork ¦ 25873 ¦ online ¦ 1       ¦ 0s     ¦ 28.008 MB   ¦ disabled ¦\r\n¦ 1Gtpb_1  ¦ 20 ¦ fork ¦ 25878 ¦ online ¦ 2       ¦ 0s     ¦ 33.805 MB   ¦ disabled ¦\r\n¦ 1Mh32_1  ¦ 30 ¦ fork ¦ 25899 ¦ online ¦ 1       ¦ 0s     ¦ 17.184 MB   ¦ disabled ¦\r\n+-----------------------------------------------------------------------------------+\r\n\r\n\r\n\r\ntop - 09:13:31 up 19:42,  2 users,  load average: 4,34, 1,70, 0,98\r\nTasks: 550 gesamt,   2 laufend, 548 schlafend,   0 gestoppt,   0 Zombie\r\n%CPU(s): 27,4 be,  3,3 sy,  0,0 ni, 15,5 un, 53,7 wa,  0,0 hi,  0,1 si,  0,0 st\r\nKiB Spch :  8175356 gesamt,   994004 frei,  4668628 belegt,  2512724 Puff/Cache\r\nKiB Swap:  1044476 gesamt,  1016836 frei,    27640 belegt.  3049396 verfü Spch\r\n\r\n  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     ZEIT+ BEFEHL\r\n25878 duck      20   0 2089024 153184  89536 S   1,0  1,9   0:02.53 node /usr/local\r\n25863 duck      20   0 2854300 256532 176692 S   0,7  3,1   0:03.14 node /usr/local\r\n25873 duck      20   0 2297024 190036 134892 S   0,7  2,3   0:02.52 node /usr/local\r\n\r\n\r\n[pid 25909] open(\"/mnt/disk1/17EAA_1/farmer.db\", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 18\r\n[pid 25909] fstat(18, {st_mode=S_IFDIR|0755, st_size=7517, ...}) = 0\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32744\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32768\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32768\r\n[pid 25909] getdents(18, /* 1023 entries */, 32768) = 32744\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32760\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32768\r\n[pid 25909] getdents(18, /* 1024 entries */, 32768) = 32768\r\n[pid 25909] getdents(18, /* 350 entries */, 32768) = 11200\r\n[pid 25909] getdents(18, /* 0 entries */, 32768) = 0\r\n[pid 25909] close(18)                   = 0\r\n[pid 25909] munmap(0x7f7702ff5000, 330999) = 0\r\n[pid 25909] write(12, \"2016/05/26-09:15:19.393209 7f776\"..., 62) = 62\r\n[pid 25909] unlink(\"/mnt/disk1/17EAA_1/farmer.db/610262.ldb\") = 0\r\n[pid 25909] futex(0x31e0d64, FUTEX_WAIT_PRIVATE, 23, NULL <unfinished ...>\r\n[pid 25890] open(\"/mnt/disk1/17EAA_1/farmer.db/569556.ldb\", O_RDONLY) = 18\r\n[pid 25890] stat(\"/mnt/disk1/17EAA_1/farmer.db/569556.ldb\", {st_mode=S_IFREG|0664, st_size=2105873, ...}) = 0\r\n[pid 25890] mmap(NULL, 2105873, PROT_READ, MAP_SHARED, 18, 0) = 0x7f7724de3000\r\n[pid 25890] close(18)                   = 0\r\n[pid 25890] open(\"/mnt/disk1/17EAA_1/farmer.db/569557.ldb\", O_RDONLY) = 18\r\n[pid 25890] stat(\"/mnt/disk1/17EAA_1/farmer.db/569557.ldb\", {st_mode=S_IFREG|0664, st_size=2104675, ...}) = 0\r\n[pid 25890] mmap(NULL, 2104675, PROT_READ, MAP_SHARED, 18, 0) = 0x7f773ddc7000\r\n[pid 25890] close(18)                   = 0\r\n[pid 25890] munmap(0x7f7728a33000, 2108476) = 0\r\n[pid 25890] open(\"/mnt/disk1/17EAA_1/farmer.db/569558.ldb\", O_RDONLY) = 18\r\n[pid 25890] stat(\"/mnt/disk1/17EAA_1/farmer.db/569558.ldb\", {st_mode=S_IFREG|0664, st_size=2104652, ...}) = 0\r\n[pid 25890] mmap(NULL, 2104652, PROT_READ, MAP_SHARED, 18, 0) = 0x7f7728a34000\r\n[pid 25890] close(18)                   = 0\r\n[pid 25890] munmap(0x7f7719c90000, 2105555) = 0\r\n[pid 25890] open(\"/mnt/disk1/17EAA_1/farmer.db/569559.ldb\", O_RDONLY) = 18\r\n[pid 25890] stat(\"/mnt/disk1/17EAA_1/farmer.db/569559.ldb\", {st_mode=S_IFREG|0664, st_size=2104651, ...}) = 0\r\n[pid 25890] mmap(NULL, 2104651, PROT_READ, MAP_SHARED, 18, 0) = 0x7f77263fe000\r\n[pid 25890] close(18)                   = 0\r\n\r\n\r\nduck@vlan2-farmer:~$ date;pm2 list\r\nDo 26. Mai 09:16:14 CEST 2016\r\n+------------------------------------------------------------------------------------+\r\n¦ App name ¦ id ¦ mode ¦ pid   ¦ status ¦ restart ¦ uptime ¦ memory       ¦ watching ¦\r\n+----------+----+------+-------+--------+---------+--------+--------------+----------¦\r\n¦ 17EAA_1  ¦ 0  ¦ fork ¦ 25863 ¦ online ¦ 1       ¦ 4m     ¦ 329.820 MB   ¦ disabled ¦\r\n¦ 17EAA_2  ¦ 1  ¦ fork ¦ 25868 ¦ online ¦ 1       ¦ 4m     ¦ 71.723 MB    ¦ disabled ¦\r\n¦ 1AKRj_1  ¦ 10 ¦ fork ¦ 25873 ¦ online ¦ 1       ¦ 4m     ¦ 240.191 MB   ¦ disabled ¦\r\n¦ 1Gtpb_1  ¦ 20 ¦ fork ¦ 25878 ¦ online ¦ 2       ¦ 4m     ¦ 234.816 MB   ¦ disabled ¦\r\n¦ 1Mh32_1  ¦ 30 ¦ fork ¦ 25899 ¦ online ¦ 1       ¦ 4m     ¦ 283.184 MB   ¦ disabled ¦\r\n+------------------------------------------------------------------------------------+\r\n\r\nncdu 1.11 ~ Use the arrow keys to navigate, press ? for help\r\n--- /mnt/disk1 -------------------------------------------------------------\r\n   14,2 GiB [##########] /17EAA_1\r\n   14,2 GiB [######### ] /1Gtpb_1\r\n   11,5 GiB [########  ] /1Mh32_1\r\n    4,5 GiB [###       ] /1AKRj_1\r\n   16,0 MiB [          ] /17EAA_2\r\n    8,5 KiB [          ] /1AKRj_10\r\n    8,5 KiB [          ] /1Gtpb_6\r\n    8,5 KiB [          ] /1AKRj_8\r\n    8,5 KiB [          ] /1AKRj_6\r\n    8,5 KiB [          ] /1AKRj_3\r\n    8,5 KiB [          ] /17EAA_10\r\n    8,5 KiB [          ] /1Mh32_10\r\n    8,5 KiB [          ] /1Gtpb_10\r\n    8,5 KiB [          ] /1Mh32_9\r\n Total disk usage:  44,4 GiB  Apparent size:  44,5 GiB  Items: 24011\r\n```\r\n\r\n### Steps to Reproduce\r\n\r\nPlease include the steps the reproduce the issue, numbered below. Include as\r\nmuch detail as possible.\r\n\r\n1. You need a node with some GB of shards\r\n2. Restart the farmer app\r\n3. Watch dstat/top and monitor the load and wait (wait indicates that the system waits for disk i/o) values\r\n"}}
{"actor":{"display_login":"BlackDuck888"}}