{"action":"created","issue":{"url":"https://api.github.com/repos/sirensolutions/sentinl/issues/445","repository_url":"https://api.github.com/repos/sirensolutions/sentinl","labels_url":"https://api.github.com/repos/sirensolutions/sentinl/issues/445/labels{/name}","comments_url":"https://api.github.com/repos/sirensolutions/sentinl/issues/445/comments","events_url":"https://api.github.com/repos/sirensolutions/sentinl/issues/445/events","html_url":"https://github.com/sirensolutions/sentinl/issues/445","id":328480651,"node_id":"MDU6SXNzdWUzMjg0ODA2NTE=","number":445,"title":"Limit of total fields in index [watcher] has been exceeded","user":{"login":"varun0002","id":35331077,"node_id":"MDQ6VXNlcjM1MzMxMDc3","avatar_url":"https://avatars2.githubusercontent.com/u/35331077?v=4","gravatar_id":"","url":"https://api.github.com/users/varun0002","html_url":"https://github.com/varun0002","followers_url":"https://api.github.com/users/varun0002/followers","following_url":"https://api.github.com/users/varun0002/following{/other_user}","gists_url":"https://api.github.com/users/varun0002/gists{/gist_id}","starred_url":"https://api.github.com/users/varun0002/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/varun0002/subscriptions","organizations_url":"https://api.github.com/users/varun0002/orgs","repos_url":"https://api.github.com/users/varun0002/repos","events_url":"https://api.github.com/users/varun0002/events{/privacy}","received_events_url":"https://api.github.com/users/varun0002/received_events","type":"User","site_admin":false},"labels":[{"id":372324964,"node_id":"MDU6TGFiZWwzNzIzMjQ5NjQ=","url":"https://api.github.com/repos/sirensolutions/sentinl/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-06-01T11:21:30Z","updated_at":"2018-06-11T07:30:43Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"<!--\r\nThanks for taking time to report an issue. Please try to:\r\n\r\n* Be precise: describe expected results vs. actual results\r\n* Be clear: explain how to reproduce the problem, step by step\r\n* Include only one problem per report\r\n* Include details about your setup/system/os where possible\r\n\r\nThis section will NOT appear in your Issue report!\r\n-->\r\n\r\n1. Sentinl and Kibana version: 5.4.1\r\n\r\n2. Issue description: Getting following error:\r\n_Jun  1 09:59:08 <host> kibana: {\"type\":\"error\",\"@timestamp\":\"XXXXX\",\"tags\":[],\"pid\":DDDD,\"level\":\"error\",\"message\":\"[illegal_argument_exception] Limit of total fields [1000] in index [watcher] has been exceeded\",\"error\":{\"message\":\"[illegal_argument_exception] Limit of total fields [1000] in index [watcher] has been exceeded\",\"name\":\"Error\",\"stack\":\"[illegal_argument_exception] Limit of total fields [1000] in index [watcher] has been exceeded :: {\\\"path\\\":\\\"/watcher/sentinl-watcher/-vc514r4b588a-a9jaqg10a933\\\",\\\"query\\\":{},\\\"body\\\":\\\"{\\\\\\\"title\\\\\\\":\\\\\\\"watcher_title\\\\\\\",\\\\\\\"disable\\\\\\\":false,\\\\\\\"report\\\\\\\":false,\\\\\\\"trigger\\\\\\\":{\\\\\\\"schedule\\\\\\\":{\\\\\\\"later\\\\\\\":\\\\\\\"every 5 mins\\\\\\\"}},\\\\\\\"input\\\\\\\":{\\\\\\\"search\\\\\\\":{\\\\\\\"request\\\\\\\":{\\\\\\\"index\\\\\\\":[],\\\\\\\"body\\\\\\\":{}}}},\\\\\\\"condition\\\\\\\":{\\\\\\\"script\\\\\\\":{\\\\\\\"script\\\\\\\":\\\\\\\"payload.hits.total > 100\\\\\\\"}},\\\\\\\"actions\\\\\\\":{\\\\\\\"email_admin\\\\\\\":{\\\\\\\"throttle_period\\\\\\\":\\\\\\\"0h15m0s\\\\\\\",\\\\\\\"email\\\\\\\":{\\\\\\\"to\\\\\\\":\\\\\\\"alarm@localhost\\\\\\\",\\\\\\\"from\\\\\\\":\\\\\\\"sentinl@localhost\\\\\\\",\\\\\\\"subject\\\\\\\":\\\\\\\"Sentinl Alarm\\\\\\\",\\\\\\\"priority\\\\\\\":\\\\\\\"high\\\\\\\",\\\\\\\"body\\\\\\\":\\\\\\\"Found {{payload.hits.total}} Events\\\\\\\"}},\\\\\\\"New HTML email action bdjzgrztl9bh\\\\\\\":{\\\\\\\"throttle_period\\\\\\\":\\\\\\\"0h0m1s\\\\\\\",\\\\\\\"email_html\\\\\\\":{\\\\\\\"to\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"from\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"subject\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"body\\\\\\\":\\\\\\\"\\\\\\\",\\\\\\\"html\\\\\\\":\\\\\\\"\\\\\\\"}}}}\\\",\\\"statusCode\\\":400,\\\"response\\\":\\\"{\\\\\\\"error\\\\\\\":{\\\\\\\"root_cause\\\\\\\":[{\\\\\\\"type\\\\\\\":\\\\\\\"illegal_argument_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"Limit of total fields [1000] in index [watcher] has been exceeded\\\\\\\"}],\\\\\\\"type\\\\\\\":\\\\\\\"illegal_argument_exception\\\\\\\",\\\\\\\"reason\\\\\\\":\\\\\\\"Limit of total fields [1000] in index [watcher] has been exceeded\\\\\\\"},\\\\\\\"status\\\\\\\":400}\\\"}\\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:295:15)\\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:254:7)\\n    at HttpConnector.<anonymous>_ \r\n\r\n3. Workaround: \r\nFor now we increased field limits to 1500. However, this is not sustainable solution. it appears sentinl doesn't delete index fields even if watcher is deleted. Can you please suggest any possible ways to clean-up fields associated with deleted watchers?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/sirensolutions/sentinl/issues/comments/396148733","html_url":"https://github.com/sirensolutions/sentinl/issues/445#issuecomment-396148733","issue_url":"https://api.github.com/repos/sirensolutions/sentinl/issues/445","id":396148733,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjE0ODczMw==","user":{"login":"varun0002","id":35331077,"node_id":"MDQ6VXNlcjM1MzMxMDc3","avatar_url":"https://avatars2.githubusercontent.com/u/35331077?v=4","gravatar_id":"","url":"https://api.github.com/users/varun0002","html_url":"https://github.com/varun0002","followers_url":"https://api.github.com/users/varun0002/followers","following_url":"https://api.github.com/users/varun0002/following{/other_user}","gists_url":"https://api.github.com/users/varun0002/gists{/gist_id}","starred_url":"https://api.github.com/users/varun0002/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/varun0002/subscriptions","organizations_url":"https://api.github.com/users/varun0002/orgs","repos_url":"https://api.github.com/users/varun0002/repos","events_url":"https://api.github.com/users/varun0002/events{/privacy}","received_events_url":"https://api.github.com/users/varun0002/received_events","type":"User","site_admin":false},"created_at":"2018-06-11T07:30:43Z","updated_at":"2018-06-11T07:30:43Z","author_association":"CONTRIBUTOR","body":"We generally use curator to remove old data. However, this case is special as we are not dealing with old data rather unused fields created in watcher index by Sentinl alarm. \r\nIs there any way we can figure-out which fields in watcher index are no more in use? Any recommendation would be helpful."}}
{"actor":{"display_login":"varun0002"}}