{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/825","repository_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop","labels_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/825/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/825/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/825/events","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/825","id":171324985,"number":825,"title":"Get fields with documents in Elasticsearch","user":{"login":"MSaifAsif","id":6280554,"avatar_url":"https://avatars.githubusercontent.com/u/6280554?v=3","gravatar_id":"","url":"https://api.github.com/users/MSaifAsif","html_url":"https://github.com/MSaifAsif","followers_url":"https://api.github.com/users/MSaifAsif/followers","following_url":"https://api.github.com/users/MSaifAsif/following{/other_user}","gists_url":"https://api.github.com/users/MSaifAsif/gists{/gist_id}","starred_url":"https://api.github.com/users/MSaifAsif/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MSaifAsif/subscriptions","organizations_url":"https://api.github.com/users/MSaifAsif/orgs","repos_url":"https://api.github.com/users/MSaifAsif/repos","events_url":"https://api.github.com/users/MSaifAsif/events{/privacy}","received_events_url":"https://api.github.com/users/MSaifAsif/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-08-16T04:58:13Z","updated_at":"2016-08-16T17:13:24Z","closed_at":null,"body":"<!--\r\nGitHub is reserved *only* for bug reports and feature requests. The best place\r\nto ask a general question is at the Elastic forums at\r\nhttps://discuss.elastic.co/c/elasticsearch-and-hadoop.\r\nIf you are in fact posting a bug report or a feature request, please \r\ninclude one and only one of the below blocks in your new issue.\r\n-->\r\n- [x] Feature Request. \r\n\r\n### Feature Request\r\n<!--\r\nIf you are filing a bug report, please fill provide responses for all \r\nof the below items.\r\nRemove the feature block.\r\n-->\r\n\r\n### Feature description\r\n\r\nI am using pig to read documents from my Elasticsearch servers. The documents look something like this\r\n\r\n    {\r\n        \"_index\": \"sa-test\",\r\n        \"_type\": \"logs\",\r\n        \"_id\": \"AUujDbzSR5FzDDhtC1LH\",\r\n        \"_score\": 1,\r\n        \"_source\": {\r\n            \"fieldA\": \"....\",\r\n            \"fieldB\": \"....\",\r\n            \"fieldC\": \".....\",\r\n            \"fieldE\": \".....\"\r\n        }\r\n    }, {\r\n        \"_index\": \"sa-test\",\r\n        \"_type\": \"logs\",\r\n        \"_id\": \"AUujDbzSR5FzXXXtC1LH\",\r\n        \"_score\": 1,\r\n        \"_source\": {\r\n            \"fieldB\": \"....\",\r\n            \"fieldC\": \".....\",\r\n            \"fieldD\": \".....\"\r\n        }\r\n    }\r\n\r\nwhich is fine. The problem occurs when I try to read this data in via Pig\r\n\r\n\r\n    -- load the data in\r\n    raw_logs = LOAD 'sa-test' USING org.elasticsearch.hadoop.pig.EsStorage('es.query=?q=someFilter', 'es.mapping.date.rich=false');\r\n    -- for now just dump it\r\n    DUMP raw_logs;\r\n\r\nthis yields data in the form\r\n\r\n    (fieldA,fieldB,fieldC,fieldE)\r\n    (fieldB,fieldC,fieldD)\r\n\r\nnow at this point, I don't have any idea which fields were missing or which ones are present since data in each row in inconsistent. This is proving a problem for further processing.\r\n\r\nIs there any way to get field values along with the rows or some sort of mapping that can tell me which value belongs to which field. Something of the sort\r\n\r\n    (A=fieldA,B=fieldB,C=fieldC,E=fieldE)\r\n    (A=fieldB,C=fieldC,D=fieldD)"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/comments/240171152","html_url":"https://github.com/elastic/elasticsearch-hadoop/issues/825#issuecomment-240171152","issue_url":"https://api.github.com/repos/elastic/elasticsearch-hadoop/issues/825","id":240171152,"user":{"login":"jbaiera","id":875779,"avatar_url":"https://avatars.githubusercontent.com/u/875779?v=3","gravatar_id":"","url":"https://api.github.com/users/jbaiera","html_url":"https://github.com/jbaiera","followers_url":"https://api.github.com/users/jbaiera/followers","following_url":"https://api.github.com/users/jbaiera/following{/other_user}","gists_url":"https://api.github.com/users/jbaiera/gists{/gist_id}","starred_url":"https://api.github.com/users/jbaiera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbaiera/subscriptions","organizations_url":"https://api.github.com/users/jbaiera/orgs","repos_url":"https://api.github.com/users/jbaiera/repos","events_url":"https://api.github.com/users/jbaiera/events{/privacy}","received_events_url":"https://api.github.com/users/jbaiera/received_events","type":"User","site_admin":false},"created_at":"2016-08-16T17:13:24Z","updated_at":"2016-08-16T17:13:24Z","body":"You can specify a schema when loading data into pig:\r\n\r\n    raw_logs = LOAD 'sa-test' USING org.elasticsearch.hadoop.pig.EsStorage('es.query=?q=someFilter', 'es.mapping.date.rich=false') AS (fieldA:...,fieldB:...,fieldC:...,fieldD:...fieldE:...);\r\n\r\nES-Hadoop will match up document fields from Elasticsearch with the provided schema for Pig. At that point, you can refer to the fields by name:\r\n\r\n    all_fieldA_values = FOREACH raw_logs GENERATE fieldA;\r\n\r\nAlso, just for future reference, Github issues are reserved for clearly defined bugs and actionable new feature requests. These sorts of questions are best asked on the [forum](https://discuss.elastic.co/c/elasticsearch-and-hadoop)."}}
{"actor":{"display_login":"jbaiera"}}