{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/15267","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15267/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15267/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15267/events","html_url":"https://github.com/elastic/elasticsearch/issues/15267","id":120644830,"number":15267,"title":"Field data loading is forbidden on [FIELDNAME]","user":{"login":"velion","id":11274206,"avatar_url":"https://avatars.githubusercontent.com/u/11274206?v=3","gravatar_id":"","url":"https://api.github.com/users/velion","html_url":"https://github.com/velion","followers_url":"https://api.github.com/users/velion/followers","following_url":"https://api.github.com/users/velion/following{/other_user}","gists_url":"https://api.github.com/users/velion/gists{/gist_id}","starred_url":"https://api.github.com/users/velion/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/velion/subscriptions","organizations_url":"https://api.github.com/users/velion/orgs","repos_url":"https://api.github.com/users/velion/repos","events_url":"https://api.github.com/users/velion/events{/privacy}","received_events_url":"https://api.github.com/users/velion/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":23,"created_at":"2015-12-06T16:45:25Z","updated_at":"2016-02-13T20:11:01Z","closed_at":"2015-12-11T09:55:59Z","body":"when i Click “visualize” and then ,display error \"Visualize:java.lang.lllegalStateException:Field daVisualize\",or i used dashboard module ,before logstash auto update index,it was normal;when time is 8:00 am,dashboard module will not work,and display the error.\r\n\r\nerror details:\r\n\r\n1.kibana page error:\r\nError: Request to Elasticsearch failed: {\"error\":{\"root_cause\":[{\"type\":\"exception\",\"reason\":\"java.lang.IllegalStateException: Field data loading is forbidden on path\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"logstash-2015.11.29\",\"node\":\"ODyds4KgQsqrxczx1ana8A\",\"reason\":{\"type\":\"exception\",\"reason\":\"java.lang.IllegalStateException: Field data loading is forbidden on path\",\"caused_by\":{\"type\":\"unchecked_execution_exception\",\"reason\":\"java.lang.IllegalStateException: Field data loading is forbidden on path\",\"caused_by\":{\"type\":\"illegal_state_exception\",\"reason\":\"Field data loading is forbidden on path\"}}}}]}} KbnError@http://www.xxx.cn:80/bundles/c ... 64:30 RequestFailure@http://www.xxx.cn:80/bundles/c ... 97:19 http://www.xxx.cn:80/bundles/k ... 05:57 http://www.xxx.cn:80/bundles/c ... 91:28 http://www.xxx.cn:80/bundles/c ... 60:31 map@[native code] map@http://www.xxx.cn:80/bundles/c ... 59:34 callResponseHandlers@http://www.xxx.cn:80/bundles/k ... 77:26 http://www.xxx.cn:80/bundles/k ... 84:37 processQueue@http://www.xxx.cn:80/bundles/c ... 09:31 http://www.xxx.cn:80/bundles/c ... 25:40 $eval@http://www.xxx.cn:80/bundles/c ... 53:29 $digest@http://www.xxx.cn:80/bundles/c ... 64:37 $apply@http://www.xxx.cn:80/bundles/c ... 61:32 done@http://www.xxx.cn:80/bundles/c ... 10:54 completeRequest@http://www.xxx.cn:80/bundles/c ... 08:16 requestLoaded@http://www.xxx.cn:80/bundles/commons.bundle.js:37749:25\r\n\r\n2.elasticsearch error log:\r\n\r\n[2015-11-30 00:04:53,395][DEBUG][action.search.type       ] [Thumbelina] [logstash-2015.11.29][2], node[acrTX4O0RciN8ppbSdfoww], [P], v[4], s[STARTED], a[id=ogwkZP0yQCCgatW0_tnvnw]: Failed to execute [org.elasticsearch.action.search.SearchRequest@aef2ffe] lastShard [true]\r\nRemoteTransportException[[Thumbelina][192.168.1.76:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];\r\nCaused by: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];\r\n        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:343)\r\n        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:106)\r\n        at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:363)\r\n        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:375)\r\n        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)\r\n        at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)\r\n        at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:165)\r\n        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350)\r\n        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: ElasticsearchException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: UncheckedExecutionException[java.lang.IllegalStateException: Field data loading is forbidden on response]; nested: IllegalStateException[Field data loading is forbidden on response];\r\n        at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:82)\r\n        at org.elasticsearch.search.aggregations.support.ValuesSource$Bytes$FieldData.bytesValues(ValuesSource.java:195)\r\n        at org.elasticsearch.search.aggregations.bucket.terms.StringTermsAggregator.getLeafCollector(StringTermsAggregator.java:73)\r\n        at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:132)\r\n        at org.elasticsearch.search.aggregations.AggregatorBase.getLeafCollector(AggregatorBase.java:38)\r\n        at org.apache.lucene.search.MultiCollector.getLeafCollector(MultiCollector.java:117)\r\n        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:763)\r\n        at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:486)\r\n        at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:324)\r\n        ... 11 more\r\nVian  00:18:36\r\nCaused by: com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Field data loading is forbidden on response\r\n        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)\r\n        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)\r\n        at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)\r\n        at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache.load(IndicesFieldDataCache.java:156)\r\n        at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:76)\r\n        … 19 more\r\nCaused by: java.lang.IllegalStateException: Field data loading is forbidden on response\r\n        at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.fail(DisabledIndexFieldData.java:68)\r\n        at org.elasticsearch.index.fielddata.plain.DisabledIndexFieldData.loadDirect(DisabledIndexFieldData.java:54)\r\n        at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:163)\r\n        at org.elasticsearch.indices.fielddata.cache.IndicesFieldDataCache$IndexFieldCache$1.call(IndicesFieldDataCache.java:156)\r\n        at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)\r\n        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)\r\n        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)\r\n        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)\r\n        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)\r\n        … 23 more\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/183746431","html_url":"https://github.com/elastic/elasticsearch/issues/15267#issuecomment-183746431","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/15267","id":183746431,"user":{"login":"clintongormley","id":56599,"avatar_url":"https://avatars.githubusercontent.com/u/56599?v=3","gravatar_id":"","url":"https://api.github.com/users/clintongormley","html_url":"https://github.com/clintongormley","followers_url":"https://api.github.com/users/clintongormley/followers","following_url":"https://api.github.com/users/clintongormley/following{/other_user}","gists_url":"https://api.github.com/users/clintongormley/gists{/gist_id}","starred_url":"https://api.github.com/users/clintongormley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/clintongormley/subscriptions","organizations_url":"https://api.github.com/users/clintongormley/orgs","repos_url":"https://api.github.com/users/clintongormley/repos","events_url":"https://api.github.com/users/clintongormley/events{/privacy}","received_events_url":"https://api.github.com/users/clintongormley/received_events","type":"User","site_admin":false},"created_at":"2016-02-13T20:11:01Z","updated_at":"2016-02-13T20:11:01Z","body":"OK, to explain what all this means...  Analyzed string fields are for full text search.  Not-analyzed string fields are for concrete values eg HTTP method, URI, status code, etc.  Typically these fields are used for filtering (rather than full text search) and are often used in aggregations.\r\n\r\nIt seldom makes sense to aggregate on an analyzed string field, eg an analyzed string field containing \"Quick brown fox\" indexes the terms \"quick\", \"brown\", and \"fox\", and a terms aggregation on this field would aggregate on each of these terms (instead of the full \"Quick brown fox\", which is probably what you wanted).\r\n\r\nOn top of that, not_analyzed strings can use \"doc_values\", ie their values get stored on a disk in a way that they can be looked up for each document in a very efficient way while running aggregations.  Analyzed strings do not support doc values.  If you try to aggregate on an analyzed string, it reads the full index, \"uninverts the values\", and stores the result in memory.  This can use an ENORMOUS amount of memory and even crash your server.  You don't want to do this.\r\n\r\nThe change in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/309 means that:\r\n\r\n1. You can no longer run aggregations on the analyzed `message` field.\r\n2. All dynamically added strings are added with two forms: `name` (an analyzed string field to be used for full text search but not for aggregations) and `name.raw` which can be used for filtering and aggregations.\r\n\r\nSo, instead of aggregating on field `foo`,  try aggregation on `foo.raw` instead."}}