{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/logstash/issues/5831","repository_url":"https://api.github.com/repos/elastic/logstash","labels_url":"https://api.github.com/repos/elastic/logstash/issues/5831/labels{/name}","comments_url":"https://api.github.com/repos/elastic/logstash/issues/5831/comments","events_url":"https://api.github.com/repos/elastic/logstash/issues/5831/events","html_url":"https://github.com/elastic/logstash/issues/5831","id":173228596,"number":5831,"title":"Large logs filling up disks","user":{"login":"morganchristiansson","id":66844,"avatar_url":"https://avatars.githubusercontent.com/u/66844?v=3","gravatar_id":"","url":"https://api.github.com/users/morganchristiansson","html_url":"https://github.com/morganchristiansson","followers_url":"https://api.github.com/users/morganchristiansson/followers","following_url":"https://api.github.com/users/morganchristiansson/following{/other_user}","gists_url":"https://api.github.com/users/morganchristiansson/gists{/gist_id}","starred_url":"https://api.github.com/users/morganchristiansson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/morganchristiansson/subscriptions","organizations_url":"https://api.github.com/users/morganchristiansson/orgs","repos_url":"https://api.github.com/users/morganchristiansson/repos","events_url":"https://api.github.com/users/morganchristiansson/events{/privacy}","received_events_url":"https://api.github.com/users/morganchristiansson/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/logstash/labels/design","name":"design","color":"eb6420"},{"url":"https://api.github.com/repos/elastic/logstash/labels/enhancement","name":"enhancement","color":"009800"},{"url":"https://api.github.com/repos/elastic/logstash/labels/P3","name":"P3","color":"000000"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-08-25T15:05:13Z","updated_at":"2016-09-01T08:55:31Z","closed_at":null,"body":"Hello.\r\n\r\nWe run a logrotate policy where logstash logs are rotated at 100MB every 5 minutes. We recently had a new service come online which caused logstash generate error logs sized at 1.3GB uncompressed before being rotated, i.e. 260MB per minute. Causing an unintentional DoS attack on our ELK cluster.\r\n\r\nAll these errors was caused by a single source of data, an oversized field where a base64 encoded file was included as a logged response body, and managed to take our entire logstash cluster offline for a brief period, and caused significant stress on our logstash buffer.\r\n\r\nEach line in the log was sized at 1.5+ megabytes so a 1GB of log would be generated by 650 invalid documents.\r\n\r\nHere's an example error:\r\n\r\n```\r\n{:timestamp=>\"20XX-XX-XXTXX:XX:XX.XXXXXX+0000\", :message=>\"Failed action. \", :status=>400, :action=>[\"index\", {:_id=>nil, :_index=>\"logstash-20XX.XX.XX\", :_type=>\".....\", :_routing=>nil}, #<LogStash::Event:0x2e6bb3d5 @metadata_accessors=#<LogStash::Util::Accessors:0x50440b08 @store={\"index\"=>\"logstash-20XX.XX.XX\"}, @lut={\"[index]\"=>[{\"index\"=>\"logstash-20XX.XX.XX\"}, \"index\"]}>, @cancelled=false, @data={\"message\"=>\"....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................\"}>>], :response=>{\"create\"=>{\"_index\"=>\"logstash-20XX.XX.XX\", \"_type\"=>\"XXXXXX\", \"_id\"=>\"XXXXXXXXXX\", \"status\"=>400, \"error\"=>{\"type\"=>\"illegal_argument_exception\", \"reason\"=>\"Document contains at least one immense term in field=\\\"message\\\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[XX, XXX, XX, XX, XXX, XXX, XXX, XXX, XXX, XXX, XXX, XX, XXX, XXX, XXX, XXX, XXX, XX, XXX, XX, XX, XXX, XXX, XXX, XXX, XX, XX, XX, XXX, XX]...', original message: bytes can be at most 32766 in length; got 105611\", \"caused_by\"=>{\"type\"=>\"max_bytes_length_exceeded_exception\", \"reason\"=>\"max_bytes_length_exceeded_exception: bytes can be at most 32766 in length; got 105611\"}}}}, :level=>:warn}\r\n```\r\n\r\nPrivate data redacted. The oversized field is replaced with 32kb+ of dots to show representative error message. (Reading logstash logs is really difficult when each line is 1.5million+ characters wide).\r\n\r\nCrossposting this from logstash-plugins/logstash-output-elasticsearch#465 as the actual logging is not handled by the plugin.\r\n\r\nRunning Logstash 2.3.4."},"comment":{"url":"https://api.github.com/repos/elastic/logstash/issues/comments/244017862","html_url":"https://github.com/elastic/logstash/issues/5831#issuecomment-244017862","issue_url":"https://api.github.com/repos/elastic/logstash/issues/5831","id":244017862,"user":{"login":"morganchristiansson","id":66844,"avatar_url":"https://avatars.githubusercontent.com/u/66844?v=3","gravatar_id":"","url":"https://api.github.com/users/morganchristiansson","html_url":"https://github.com/morganchristiansson","followers_url":"https://api.github.com/users/morganchristiansson/followers","following_url":"https://api.github.com/users/morganchristiansson/following{/other_user}","gists_url":"https://api.github.com/users/morganchristiansson/gists{/gist_id}","starred_url":"https://api.github.com/users/morganchristiansson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/morganchristiansson/subscriptions","organizations_url":"https://api.github.com/users/morganchristiansson/orgs","repos_url":"https://api.github.com/users/morganchristiansson/repos","events_url":"https://api.github.com/users/morganchristiansson/events{/privacy}","received_events_url":"https://api.github.com/users/morganchristiansson/received_events","type":"User","site_admin":false},"created_at":"2016-09-01T08:55:31Z","updated_at":"2016-09-01T08:55:31Z","body":"We run logrotate aggressively but it wasn't enough with the volume of logs being written out. IO bandwidth is possibly also an issue when logs are spewed out at 4megabyte per second over an extended period of time.\r\n\r\nI agree an increased root disk is appropriate due to logstash and the high volume of data we send through it. We will most likely use EC2 ephemeral storage for /var/log which has both size and IO in abundance."}}
{"actor":{"display_login":"morganchristiansson"}}