{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18917","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18917/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18917/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18917/events","html_url":"https://github.com/elastic/elasticsearch/issues/18917","id":160644346,"number":18917,"title":"MapperParsingException and field mapping conflict all out of the sudden","user":{"login":"defkev","id":5265345,"avatar_url":"https://avatars.githubusercontent.com/u/5265345?v=3","gravatar_id":"","url":"https://api.github.com/users/defkev","html_url":"https://github.com/defkev","followers_url":"https://api.github.com/users/defkev/followers","following_url":"https://api.github.com/users/defkev/following{/other_user}","gists_url":"https://api.github.com/users/defkev/gists{/gist_id}","starred_url":"https://api.github.com/users/defkev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/defkev/subscriptions","organizations_url":"https://api.github.com/users/defkev/orgs","repos_url":"https://api.github.com/users/defkev/repos","events_url":"https://api.github.com/users/defkev/events{/privacy}","received_events_url":"https://api.github.com/users/defkev/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-06-16T11:59:14Z","updated_at":"2016-06-16T13:40:16Z","closed_at":"2016-06-16T13:40:16Z","body":"**Elasticsearch version**:\r\n> {\r\n>   \"name\" : \"Alpha Ray\",\r\n>   \"cluster_name\" : \"elasticsearch\",\r\n>   \"version\" : {\r\n>     \"number\" : \"2.3.3\",\r\n>     \"build_hash\" : \"218bdf10790eef486ff2c41a3df5cfa32dadcfde\",\r\n>     \"build_timestamp\" : \"2016-05-17T15:40:04Z\",\r\n>     \"build_snapshot\" : false,\r\n>     \"lucene_version\" : \"5.5.0\"\r\n>   },\r\n>   \"tagline\" : \"You Know, for Search\"\r\n> }\r\n\r\n**JVM version**:\r\n> openjdk version \"1.8.0_91\"\r\n> OpenJDK Runtime Environment (build 1.8.0_91-b14)\r\n> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)\r\n\r\n**OS version**:\r\n> CentOS Linux release 7.2.1511 (Core)\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nOne of my syslog inputs from logstash stopped working this morning exactly at 2:00 am\r\nElasticsearch didn't add any new shards for this very type past this point but started throwing a MapperParsingException in elasticsearch.log for every new shard like:\r\n\r\n```\r\nMapperParsingException[failed to parse [timestamp]]; nested: IllegalArgumentException[Invalid format: \"Jun 16 02:00:28\"];\r\n        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:329)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseObjectOrField(DocumentParser.java:309)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseAndMergeUpdate(DocumentParser.java:738)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:625)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:442)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseObject(DocumentParser.java:262)\r\n        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:122)\r\n        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:309)\r\n        at org.elasticsearch.index.shard.IndexShard.prepareCreate(IndexShard.java:529)\r\n        at org.elasticsearch.index.shard.IndexShard.prepareCreateOnPrimary(IndexShard.java:506)\r\n        at org.elasticsearch.action.index.TransportIndexAction.prepareIndexOperationOnPrimary(TransportIndexAction.java:215)\r\n        at org.elasticsearch.action.index.TransportIndexAction.executeIndexRequestOnPrimary(TransportIndexAction.java:224)\r\n        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:326)\r\n        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:119)\r\n        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:68)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.doRun(TransportReplicationAction.java:639)\r\n        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:279)\r\n        at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:271)\r\n        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)\r\n        at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)\r\n        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n        at java.lang.Thread.run(Thread.java:745)\r\nCaused by: java.lang.IllegalArgumentException: Invalid format: \"Jun 16 02:00:28\"\r\n        at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187)\r\n        at org.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:780)\r\n        at org.elasticsearch.index.mapper.core.DateFieldMapper$DateFieldType.parseStringValue(DateFieldMapper.java:362)\r\n        at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:528)\r\n        at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:241)\r\n        at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:321)\r\n        ... 24 more\r\n```\r\nNote that this is the timestamp field from syslog, not the @timestamp added by elasticsearch.\r\nFurthermore the field in question now shows a mapping conflict in kibana, it was previously mapped as a string and the last shard created was at 1:59:36 am\r\n\r\nNow as i see it elasticsearch is suddenly trying to parse the said timestamp field as an actual timestamp instead of a string (which cannot be parsed as such since it is missing the year part i suppose) which in return is creating the mapping conflict.\r\n \r\nThe setup has been running for a while by now and nothing on the configuration has been changed recently, hence i am curious what has caused elasticsearch to thread \"Jun 16 01:59:36\" as a string and \"Jun 16 02:00:28\" as a timestamp."}}
{"actor":{"display_login":"defkev"}}