{"action":"reopened","issue":{"url":"https://api.github.com/repos/leo-project/leofs/issues/443","repository_url":"https://api.github.com/repos/leo-project/leofs","labels_url":"https://api.github.com/repos/leo-project/leofs/issues/443/labels{/name}","comments_url":"https://api.github.com/repos/leo-project/leofs/issues/443/comments","events_url":"https://api.github.com/repos/leo-project/leofs/issues/443/events","html_url":"https://github.com/leo-project/leofs/issues/443","id":130910894,"number":443,"title":"[v1.2][v1.4][leo_storage] Node Stopped with Many Small Write Requests","user":{"login":"windkit","id":360884,"avatar_url":"https://avatars.githubusercontent.com/u/360884?v=3","gravatar_id":"","url":"https://api.github.com/users/windkit","html_url":"https://github.com/windkit","followers_url":"https://api.github.com/users/windkit/followers","following_url":"https://api.github.com/users/windkit/following{/other_user}","gists_url":"https://api.github.com/users/windkit/gists{/gist_id}","starred_url":"https://api.github.com/users/windkit/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/windkit/subscriptions","organizations_url":"https://api.github.com/users/windkit/orgs","repos_url":"https://api.github.com/users/windkit/repos","events_url":"https://api.github.com/users/windkit/events{/privacy}","received_events_url":"https://api.github.com/users/windkit/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/leo-project/leofs/labels/Bug","name":"Bug","color":"fc2929"},{"url":"https://api.github.com/repos/leo-project/leofs/labels/Test","name":"Test","color":"e102d8"},{"url":"https://api.github.com/repos/leo-project/leofs/labels/_leo_storage","name":"_leo_storage","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"yosukehara","id":148770,"avatar_url":"https://avatars.githubusercontent.com/u/148770?v=3","gravatar_id":"","url":"https://api.github.com/users/yosukehara","html_url":"https://github.com/yosukehara","followers_url":"https://api.github.com/users/yosukehara/followers","following_url":"https://api.github.com/users/yosukehara/following{/other_user}","gists_url":"https://api.github.com/users/yosukehara/gists{/gist_id}","starred_url":"https://api.github.com/users/yosukehara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosukehara/subscriptions","organizations_url":"https://api.github.com/users/yosukehara/orgs","repos_url":"https://api.github.com/users/yosukehara/repos","events_url":"https://api.github.com/users/yosukehara/events{/privacy}","received_events_url":"https://api.github.com/users/yosukehara/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/leo-project/leofs/milestones/51","html_url":"https://github.com/leo-project/leofs/milestones/1.2.20","labels_url":"https://api.github.com/repos/leo-project/leofs/milestones/51/labels","id":1548513,"number":51,"title":"1.2.20","description":"* Maintenance","creator":{"login":"yosukehara","id":148770,"avatar_url":"https://avatars.githubusercontent.com/u/148770?v=3","gravatar_id":"","url":"https://api.github.com/users/yosukehara","html_url":"https://github.com/yosukehara","followers_url":"https://api.github.com/users/yosukehara/followers","following_url":"https://api.github.com/users/yosukehara/following{/other_user}","gists_url":"https://api.github.com/users/yosukehara/gists{/gist_id}","starred_url":"https://api.github.com/users/yosukehara/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yosukehara/subscriptions","organizations_url":"https://api.github.com/users/yosukehara/orgs","repos_url":"https://api.github.com/users/yosukehara/repos","events_url":"https://api.github.com/users/yosukehara/events{/privacy}","received_events_url":"https://api.github.com/users/yosukehara/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":0,"state":"open","created_at":"2016-01-28T08:07:41Z","updated_at":"2016-02-04T13:02:25Z","due_on":"2016-02-04T15:00:00Z","closed_at":null},"comments":5,"created_at":"2016-02-03T04:46:41Z","updated_at":"2016-02-04T13:02:25Z","closed_at":null,"body":"### Description\r\nDuring a benchmark with cosbench, storage node stopped in the middle.\r\nThe workload was configured as 100% write of 4KB Objects with 64 Workers across 4 Drivers\r\nAt first, it ran well with good throughput >15000 ops (60MB/s) but then failed to serve the requests\r\n![cosbench](https://cloud.githubusercontent.com/assets/360884/12773017/a30dd3a4-ca7a-11e5-9ee0-7e4c6c5aebe1.jpg)\r\n\r\n### Environment\r\nLeoFS Version: 1.4.0-pre3 (Erlang 17.5)\r\nLeoFS Setup: 1x Gateway, 5 x Storage\r\nLeoFS Configuration: [conf.tar.gz](https://github.com/leo-project/leofs/files/115121/conf.tar.gz)\r\nOS: Ubuntu 14.04.3 x64\r\nHardware: Xeon E5-2630 v3, 32GB Ram, (Storage) 6x 15k sas HDD, RAID-0\r\n\r\n### Error Logs / Crashdump\r\nError Log: [error.txt](https://github.com/leo-project/leofs/files/115122/error.txt)\r\nTrimmed Crashdump (Uncompressed 3GB): [snap.dump](https://drive.google.com/file/d/0B26vAI5J5Wrhbm03OFNzYnFvMW8/view?usp=sharing)\r\n\r\n### Observation\r\nMany processes are staled at `mnesia_tm:rec/2` which initialized from `leo_sync_remote_cluster:defer_stack/1`.\r\nCommented out the whole method actually get rid of the issue and the benchmark could complete"}}