{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/kibana/issues/5213","labels_url":"https://api.github.com/repos/elastic/kibana/issues/5213/labels{/name}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/5213/comments","events_url":"https://api.github.com/repos/elastic/kibana/issues/5213/events","html_url":"https://github.com/elastic/kibana/pull/5213","id":113708640,"number":5213,"title":"Create an Index Pattern API","user":{"login":"Bargs","id":6239176,"avatar_url":"https://avatars.githubusercontent.com/u/6239176?v=3","gravatar_id":"","url":"https://api.github.com/users/Bargs","html_url":"https://github.com/Bargs","followers_url":"https://api.github.com/users/Bargs/followers","following_url":"https://api.github.com/users/Bargs/following{/other_user}","gists_url":"https://api.github.com/users/Bargs/gists{/gist_id}","starred_url":"https://api.github.com/users/Bargs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bargs/subscriptions","organizations_url":"https://api.github.com/users/Bargs/orgs","repos_url":"https://api.github.com/users/Bargs/repos","events_url":"https://api.github.com/users/Bargs/events{/privacy}","received_events_url":"https://api.github.com/users/Bargs/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/kibana/labels/review","name":"review","color":"eb6420"}],"state":"open","locked":false,"assignee":{"login":"rashidkpc","id":1250387,"avatar_url":"https://avatars.githubusercontent.com/u/1250387?v=3","gravatar_id":"","url":"https://api.github.com/users/rashidkpc","html_url":"https://github.com/rashidkpc","followers_url":"https://api.github.com/users/rashidkpc/followers","following_url":"https://api.github.com/users/rashidkpc/following{/other_user}","gists_url":"https://api.github.com/users/rashidkpc/gists{/gist_id}","starred_url":"https://api.github.com/users/rashidkpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rashidkpc/subscriptions","organizations_url":"https://api.github.com/users/rashidkpc/orgs","repos_url":"https://api.github.com/users/rashidkpc/repos","events_url":"https://api.github.com/users/rashidkpc/events{/privacy}","received_events_url":"https://api.github.com/users/rashidkpc/received_events","type":"User","site_admin":false},"milestone":null,"comments":40,"created_at":"2015-10-27T23:07:52Z","updated_at":"2016-01-05T20:28:27Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/elastic/kibana/pulls/5213","html_url":"https://github.com/elastic/kibana/pull/5213","diff_url":"https://github.com/elastic/kibana/pull/5213.diff","patch_url":"https://github.com/elastic/kibana/pull/5213.patch"},"body":"Closes #5199\r\n\r\n## Documentation\r\n\r\nIf you like Postman, import [this collection](https://www.getpostman.com/collections/0b9fffdc7bf629d786e6) with example requests to play around with.\r\n\r\n### Index Pattern resource schema\r\n\r\nThe API document format is somewhat based on the [JSON API specification](http://jsonapi.org/).\r\n\r\nEach payload is a JSON object with a `data` property containing the primary data (in this case, the index pattern), and an optional `included` property containing related data (in this case, an index template).\r\n\r\nAn index pattern resource has the following attributes:\r\n\r\n* `title`: String (Required) Note: this becomes the id of the document in elasticsearch.\r\n* `time_field_name`: String (optional, only needed for time based index patterns)\r\n* `interval_name`: String (optional)\r\n* `fields`: Array (optional)\r\n* `field_format_map`: Object (optional, map of non-default formatters to user for fields)\r\n* `not_expandable`: Boolean (optional)\r\n\r\nAn object in the `fields` array can have the following attributes:\r\n\r\n* `name`: String (required)\r\n* `type`: String (required)\r\n* `count`: Number\r\n* `scripted`: Boolean\r\n* `doc_values`: Boolean\r\n* `indexed`: Boolean\r\n* `analyzed`: Boolean\r\n\r\nThe optionally included index template follows the [format defined by Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html).\r\n\r\n\r\n### Endpoints\r\n\r\n#### GET /api/kibana/index_patterns[?include=template]\r\nRetrieve the list of all index patterns. \r\nAdding the query string parameter `include=template` will include any related index templates in the API response.\r\n\r\n\r\n##### Example\r\nRequest:\r\n```\r\nGET /api/kibana/index_patterns?include=template\r\n```\r\nResponse:\r\n```\r\nStatus 200\r\n\r\n{\r\n  \"data\": [\r\n    {\r\n      \"type\": \"index_patterns\",\r\n      \"id\": \"logstash-*\",\r\n      \"attributes\": {\r\n        \"title\": \"logstash-*\",\r\n        \"time_field_name\": \"@timestamp\",\r\n        \"fields\": [\r\n          {\r\n            \"name\": \"ip\",\r\n            \"count\": 2,\r\n            \"scripted\": false,\r\n            \"type\": \"ip\",\r\n            \"indexed\": true,\r\n            \"analyzed\": false,\r\n            \"doc_values\": true\r\n          },\r\n          ...\r\n        ]\r\n      },\r\n      \"relationships\": {\r\n        \"template\": {\r\n          \"data\": {\r\n            \"type\": \"index_templates\",\r\n            \"id\": \"kibana-logstash-*\"\r\n          }\r\n        }\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"index_patterns\",\r\n      \"id\": \"foo-*\",\r\n      \"attributes\": {\r\n        \"title\": \"foo-*\",\r\n        \"time_field_name\": \"@timestamp\",\r\n        \"fields\": [\r\n          {\r\n            \"name\": \"@timestamp\",\r\n            \"count\": 0,\r\n            \"scripted\": false,\r\n            \"type\": \"date\",\r\n            \"indexed\": true,\r\n            \"analyzed\": false,\r\n            \"doc_values\": true\r\n          },\r\n          ...\r\n        ]\r\n      },\r\n      \"relationships\": {\r\n        \"template\": {\r\n          \"data\": {\r\n            \"type\": \"index_templates\",\r\n            \"id\": \"kibana-foo-*\"\r\n          }\r\n        }\r\n      }\r\n    }\r\n  ],\r\n  \"included\": [\r\n    {\r\n      \"type\": \"index_templates\",\r\n      \"id\": \"kibana-logstash-*\",\r\n      \"attributes\": {\r\n        \"order\": 0,\r\n        \"template\": \"logstash-*\",\r\n        \"settings\": {},\r\n        \"mappings\": {\r\n          \"_default_\": {\r\n            \"properties\": {\r\n              \"ip\": {\r\n                \"index\": \"not_analyzed\",\r\n                \"type\": \"ip\",\r\n                \"doc_values\": true\r\n              },\r\n             ...\r\n            }\r\n          }\r\n        }\r\n      }\r\n    },\r\n    {\r\n      \"type\": \"index_templates\",\r\n      \"id\": \"kibana-foo-*\",\r\n      \"attributes\": {\r\n        \"order\": 0,\r\n        \"template\": \"foo-*\",\r\n        \"settings\": {},\r\n        \"mappings\": {\r\n          \"_default_\": {\r\n            \"properties\": {\r\n              \"@timestamp\": {\r\n                \"index\": \"not_analyzed\",\r\n                \"type\": \"date\",\r\n                \"doc_values\": true\r\n              },\r\n              ...\r\n            }\r\n          }\r\n        }\r\n      }\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\n\r\n#### GET /api/kibana/index_patterns/{id}[?include=template]\r\nRetrieve a specific index pattern.\r\nAdding the query string parameter `include=template` will include the related index template (if it exists) in the API response.\r\n\r\n##### Errors\r\n* 404 Not Found if the requested pattern does not exist\r\n\r\n##### Example\r\nRequest:\r\n```\r\nGET /api/kibana/index_patterns/logstash-*?include=template\r\n```\r\nResponse:\r\n```\r\nStatus 200\r\n\r\n{\r\n  \"data\": {\r\n    \"type\": \"index_patterns\",\r\n    \"id\": \"logstash-*\",\r\n    \"attributes\": {\r\n      \"title\": \"logstash-*\",\r\n      \"time_field_name\": \"@timestamp\",\r\n      \"fields\": [\r\n        {\r\n          \"name\": \"ip\",\r\n          \"count\": 2,\r\n          \"scripted\": false,\r\n          \"type\": \"ip\",\r\n          \"indexed\": true,\r\n          \"analyzed\": false,\r\n          \"doc_values\": true\r\n        },\r\n        {\r\n          \"name\": \"@timestamp\",\r\n          \"count\": 0,\r\n          \"scripted\": false,\r\n          \"type\": \"date\",\r\n          \"indexed\": true,\r\n          \"analyzed\": false,\r\n          \"doc_values\": true\r\n        },\r\n        ...\r\n      ]\r\n    },\r\n    \"relationships\": {\r\n      \"template\": {\r\n        \"data\": {\r\n          \"type\": \"index_templates\",\r\n          \"id\": \"kibana-logstash-*\"\r\n        }\r\n      }\r\n    }\r\n  },\r\n  \"included\": [\r\n    {\r\n      \"type\": \"index_templates\",\r\n      \"id\": \"kibana-logstash-*\",\r\n      \"attributes\": {\r\n        \"order\": 0,\r\n        \"template\": \"logstash-*\",\r\n        \"settings\": {},\r\n        \"mappings\": {\r\n          \"_default_\": {\r\n            \"properties\": {\r\n              \"@timestamp\": {\r\n                \"index\": \"not_analyzed\",\r\n                \"type\": \"date\",\r\n                \"doc_values\": true\r\n              },\r\n              \"ip\": {\r\n                \"index\": \"not_analyzed\",\r\n                \"type\": \"ip\",\r\n                \"doc_values\": true\r\n              },\r\n              ...\r\n            }\r\n          }\r\n        }\r\n      }\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\n\r\n#### POST /api/kibana/index_patterns\r\nCreate a new index pattern.\r\nRequires the JSON formatted index pattern resource in the body of the request. An index template can optionally be included and linked to the index pattern. You can only create a template if your pattern does not match existing indices.\r\n\r\nIf a template is included, the corresponding `fields` in the index pattern will have their mapping information (`type`, `indexed`, `analyzed`, `doc_values`) filled in based on the template. Fields will be de-duped and normalized so that the pattern stored in elasticsearch is backwards compatible with the existing Kibana frontend code.\r\n\r\n##### Errors\r\n* 400 Bad Request for an invalid payload\r\n* 409 Conflict when a pattern with the given ID already exists\r\n* 409 Conflict when an included template already exists or matches existing indices\r\n\r\n##### Example\r\nRequest:\r\n```\r\nPOST /api/kibana/index_patterns\r\n\r\n{\r\n      \"data\": {\r\n        \"type\": \"index_patterns\",\r\n        \"id\": \"logstash-*\",\r\n        \"attributes\": {\r\n          \"title\": \"logstash-*\",\r\n          \"time_field_name\": \"@timestamp\",\r\n          \"fields\": [{\r\n            \"name\": \"@timestamp\",\r\n            \"count\": 0,\r\n            \"scripted\": false\r\n          }]\r\n        },\r\n        \"relationships\": {\r\n          \"template\": {\r\n            \"data\": {\"type\": \"index_templates\", \"id\": \"kibana-logstash-*\"}\r\n          }\r\n        }\r\n      },\r\n      \"included\": [{\r\n        \"type\": \"index_templates\",\r\n        \"id\": \"kibana-logstash-*\",\r\n        \"attributes\": {\r\n          \"template\": \"logstash-*\",\r\n          \"order\": 0,\r\n          \"mappings\": {\r\n            \"_default_\": {\r\n              \"properties\": {\r\n                \"@timestamp\": {\"type\": \"date\", \"index\": \"not_analyzed\", \"doc_values\": true}\r\n              }\r\n            }\r\n          }\r\n        }\r\n      }]\r\n    }\r\n```\r\nResponse:\r\n```\r\nStatus 201 \r\n\r\nsuccess\r\n```\r\n\r\n#### PUT /api/kibana/index_patterns/{id}\r\nUpdate an existing index pattern.\r\nThe request body can contain some or all of the index-pattern attributes for updating. Anything that the elasticsearch update API can handle. Templates however cannot be included in a PUT request.\r\n\r\n##### Errors\r\n* 400 Bad Request if a template is included\r\n* 400 Bad Request for invalid payload\r\n* 404 Not Found if the given pattern does not exist\r\n\r\n##### Examples\r\nRequest:\r\n```\r\nPUT /api/kibana/index_patterns/logstash-*\r\n\r\n{\r\n    \"data\": {\r\n        \"type\": \"index_patterns\",\r\n        \"id\": \"logstash-*\",\r\n        \"attributes\": {\r\n          \"title\": \"logstash-*\",\r\n          \"time_field_name\": \"foo\"\r\n        }\r\n    }\r\n}\r\n```\r\nResponse:\r\n```\r\nStatus 200\r\n\r\nsuccess\r\n```\r\n\r\n#### DELETE /api/kibana/index_patterns/{id}[?include=template]\r\nDeletes the index pattern. If the `include=template` query string parameter is included, the index-pattern's related index template will also be deleted.\r\n\r\n##### Errors\r\n* 404 Not Found if the given pattern does not exist\r\n\r\n##### Example\r\nRequest:\r\n```\r\nDELETE /api/kibana/index_patterns/logstash-*\r\n```\r\nResponse:\r\n```\r\nStatus 200\r\n\r\nsuccess\r\n```\r\n\r\n\r\n\r\n\r\n\r\n\r\n### Done\r\n\r\n* Moved API endpoints into a plugin.\r\n* Implemented basic initial version of all the endpoints.\r\n* Error handling \r\nNeed to handle errors that the elasticsearch client might throw at me\r\nhttps://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/errors.html\r\n* Create an elasticsearch index template when an index pattern is created\r\n* Ignore type specific mappings, use [default](https://www.elastic.co/guide/en/elasticsearch/reference/master/default-mapping.html)\r\n  * Accept json payload that ES expects for mappings. \r\n  * If a pattern matches no existing indices, a template will be created.\r\n  * If a pattern matches existing indices, a template will not be created.\r\n  * If a user is trying to update an existing pattern with a matching template, we should only accept new field mappings and reject changes to existing ones (add a warning in the UI that uses this API to describe this behavior?) (going to punt on this until we feel like we actually need it because it's going to be rather complicated).\r\n* Payload validation\r\nCreate a Joi schema\r\n* Testing\r\nPrimarily going to rely on functional tests for API testing, do unit testing where appropriate. \r\n* Security\r\nShould accept API tokens for auth once #5267 is merged.\r\nUntil then, how should we secure this? It's a bit different than most ways we access elasticsearch right now because it accesses the .kibana index, but it will be initiated from the frontend. Should look at how the current pattern creation screen does it with a shielded cluster."},"comment":{"url":"https://api.github.com/repos/elastic/kibana/issues/comments/169122570","html_url":"https://github.com/elastic/kibana/pull/5213#issuecomment-169122570","issue_url":"https://api.github.com/repos/elastic/kibana/issues/5213","id":169122570,"user":{"login":"Bargs","id":6239176,"avatar_url":"https://avatars.githubusercontent.com/u/6239176?v=3","gravatar_id":"","url":"https://api.github.com/users/Bargs","html_url":"https://github.com/Bargs","followers_url":"https://api.github.com/users/Bargs/followers","following_url":"https://api.github.com/users/Bargs/following{/other_user}","gists_url":"https://api.github.com/users/Bargs/gists{/gist_id}","starred_url":"https://api.github.com/users/Bargs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Bargs/subscriptions","organizations_url":"https://api.github.com/users/Bargs/orgs","repos_url":"https://api.github.com/users/Bargs/repos","events_url":"https://api.github.com/users/Bargs/events{/privacy}","received_events_url":"https://api.github.com/users/Bargs/received_events","type":"User","site_admin":false},"created_at":"2016-01-05T20:28:27Z","updated_at":"2016-01-05T20:28:27Z","body":"Summary of what we talked about on the call, please comment if anyone feels I missed something:\r\n\r\n* I'll reduce this API to only include the feature set required by the ingest UI.\r\n* This API is intended for internal use only.\r\n* This API should not be seen as an example of best practices for future Kibana APIs.\r\n* Everyone agrees that ES specifics should be abstracted away as much as possible, but there's still a lot of debate about how much we can abstract and what those abstractions should look like.\r\n* While we'd all like to set a good precedent for future APIs, we can't allow perfection to get in the way of completion of features."}}