{"action":"created","issue":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/7074","repository_url":"https://api.github.com/repos/influxdata/influxdb","labels_url":"https://api.github.com/repos/influxdata/influxdb/issues/7074/labels{/name}","comments_url":"https://api.github.com/repos/influxdata/influxdb/issues/7074/comments","events_url":"https://api.github.com/repos/influxdata/influxdb/issues/7074/events","html_url":"https://github.com/influxdata/influxdb/issues/7074","id":167812321,"number":7074,"title":"General stability issues influxdb v1.0beta3 (possible compaction problems, still)? ","user":{"login":"sysadminsemantics","id":7550416,"avatar_url":"https://avatars.githubusercontent.com/u/7550416?v=3","gravatar_id":"","url":"https://api.github.com/users/sysadminsemantics","html_url":"https://github.com/sysadminsemantics","followers_url":"https://api.github.com/users/sysadminsemantics/followers","following_url":"https://api.github.com/users/sysadminsemantics/following{/other_user}","gists_url":"https://api.github.com/users/sysadminsemantics/gists{/gist_id}","starred_url":"https://api.github.com/users/sysadminsemantics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sysadminsemantics/subscriptions","organizations_url":"https://api.github.com/users/sysadminsemantics/orgs","repos_url":"https://api.github.com/users/sysadminsemantics/repos","events_url":"https://api.github.com/users/sysadminsemantics/events{/privacy}","received_events_url":"https://api.github.com/users/sysadminsemantics/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-07-27T09:38:59Z","updated_at":"2016-07-27T10:05:56Z","closed_at":null,"body":"# Bug report\r\n\r\n## System info:\r\n\r\n4 of these (non clustered, just mirroring all metrics data x 4 via telegraf, might shard or truly cluster these at some point):\r\nProLiant XL170r Gen9\r\nIntel(R) Xeon(R) CPU E5-2660 (40 cores hyperthreaded)\r\n256GB of RAM\r\n5 x 960GB SSD's (HP MK0960GECQK, apparently these are refurbished Samsungs), out of which:\r\n  - 4 of these are in RAID1+0, since the machines don't have a RAID controller, we're using software raid... -> xfs\r\n  - 1 of these drives, following design guidelines, used as a dedicated location for the WAL -> xfs\r\nInfluxDB v1.0.0beta3\r\nDebian Jessie 8.2\r\n\r\n## Steps to reproduce:\r\n- InfluxDB will be happily parsing all our metrics for a while. Current stats:\r\n  -> 54 shards\r\n  -> numMeasurements: 90\r\n  -> numSeries: 259848\r\n- a big event will occur, something like a power outage in a datacentre, stopping submission of metrics collection for a while\r\n- once metrics collection is resumed, the box in question will explode. io utilisation will rise to about 100%, the machine consuming all of its 40 threads\r\n- a restart of influxDB usually mitigates the issue (for a while). Current cluster has been running in a stable fashion for nearly 22 hours.\r\n- it \"feels\" like there is some queue'ing going on. WAL looks normal; the only thing that looks out of the ordinary is that the full compaction time suddenly spikes to be about 13 minutes on average. this could however also be a symptom of 100 io util\r\n- i've not been able so far to collect statistics about socket utilisation; amount of TIME_WAIT or CLOSE_WAIT sockets; which might be an interesting metric, but could also be highly symptomatic\r\n- i've collected a few reports of the cluster when it is actually behaving, but not when it is misbehaving. I will update this issue with that once this issue reoccurs (current reports are attached)\r\n- you'll see in a few reports that the max speed of the SSD's seems to be around 60 MB / sec. (and ioutil then being @ 100%) obviously that is very low, which is a separate hardware (and maybe software raid) issue we are investigating separately\r\n\r\n## Expected behavior:\r\n- no inexplicable loadspike / io lockup after metric submission has been stopped for a while\r\n\r\n## Actual behavior:\r\n- loadspike / io lockup\r\n\r\nI've attached a few screenshots of my influxDB instance metrics; as well as the reports per node (both during and after the occurrence). Also; separate bug, did we remove UDP metrics from _internal?\r\n\r\n![screen shot 2016-07-27 at 11 32 28](https://cloud.githubusercontent.com/assets/7550416/17170798/fed8c5d4-53ed-11e6-9778-2af8627f23ff.png)\r\n![screen shot 2016-07-27 at 11 33 25](https://cloud.githubusercontent.com/assets/7550416/17170799/fedbc306-53ed-11e6-80ac-07aecd51e46b.png)\r\n\r\nThe gist for the bugreport is here:\r\nhttps://gist.github.com/sysadminsemantics/18a9692acfeb417deba5b2ab7f67e825\r\n\r\nYou'll probably need more info, but if you have any pointers I could work with in the meanwhile, that would be greatly appreciated. Forgot to mention (but should be apparent in the graphs), that the bad state occurs around 12:30.\r\n"},"comment":{"url":"https://api.github.com/repos/influxdata/influxdb/issues/comments/235543274","html_url":"https://github.com/influxdata/influxdb/issues/7074#issuecomment-235543274","issue_url":"https://api.github.com/repos/influxdata/influxdb/issues/7074","id":235543274,"user":{"login":"e-dard","id":501993,"avatar_url":"https://avatars.githubusercontent.com/u/501993?v=3","gravatar_id":"","url":"https://api.github.com/users/e-dard","html_url":"https://github.com/e-dard","followers_url":"https://api.github.com/users/e-dard/followers","following_url":"https://api.github.com/users/e-dard/following{/other_user}","gists_url":"https://api.github.com/users/e-dard/gists{/gist_id}","starred_url":"https://api.github.com/users/e-dard/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/e-dard/subscriptions","organizations_url":"https://api.github.com/users/e-dard/orgs","repos_url":"https://api.github.com/users/e-dard/repos","events_url":"https://api.github.com/users/e-dard/events{/privacy}","received_events_url":"https://api.github.com/users/e-dard/received_events","type":"User","site_admin":false},"created_at":"2016-07-27T10:05:56Z","updated_at":"2016-07-27T10:05:56Z","body":"Oliver's gist contains:\r\n\r\n - vars.txt\r\n - iostat.txt\r\n - goroutine profile\r\n - blocking profile"}}
{"actor":{"display_login":"e-dard"}}