{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/2414","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/2414/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/2414/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/2414/events","html_url":"https://github.com/druid-io/druid/pull/2414","id":132256099,"number":2414,"title":"Support pulling values from complex avro data structures","user":{"login":"will-lauer","id":6795115,"avatar_url":"https://avatars.githubusercontent.com/u/6795115?v=3","gravatar_id":"","url":"https://api.github.com/users/will-lauer","html_url":"https://github.com/will-lauer","followers_url":"https://api.github.com/users/will-lauer/followers","following_url":"https://api.github.com/users/will-lauer/following{/other_user}","gists_url":"https://api.github.com/users/will-lauer/gists{/gist_id}","starred_url":"https://api.github.com/users/will-lauer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/will-lauer/subscriptions","organizations_url":"https://api.github.com/users/will-lauer/orgs","repos_url":"https://api.github.com/users/will-lauer/repos","events_url":"https://api.github.com/users/will-lauer/events{/privacy}","received_events_url":"https://api.github.com/users/will-lauer/received_events","type":"User","site_admin":false},"labels":[{"id":321808195,"url":"https://api.github.com/repos/druid-io/druid/labels/Improvement","name":"Improvement","color":"0052cc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/druid-io/druid/milestones/21","html_url":"https://github.com/druid-io/druid/milestones/0.9.3","labels_url":"https://api.github.com/repos/druid-io/druid/milestones/21/labels","id":1831000,"number":21,"title":"0.9.3","description":"","creator":{"login":"fjy","id":428325,"avatar_url":"https://avatars.githubusercontent.com/u/428325?v=3","gravatar_id":"","url":"https://api.github.com/users/fjy","html_url":"https://github.com/fjy","followers_url":"https://api.github.com/users/fjy/followers","following_url":"https://api.github.com/users/fjy/following{/other_user}","gists_url":"https://api.github.com/users/fjy/gists{/gist_id}","starred_url":"https://api.github.com/users/fjy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjy/subscriptions","organizations_url":"https://api.github.com/users/fjy/orgs","repos_url":"https://api.github.com/users/fjy/repos","events_url":"https://api.github.com/users/fjy/events{/privacy}","received_events_url":"https://api.github.com/users/fjy/received_events","type":"User","site_admin":false},"open_issues":58,"closed_issues":57,"state":"open","created_at":"2016-06-15T22:35:58Z","updated_at":"2016-11-08T22:10:46Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2016-02-08T21:03:15Z","updated_at":"2016-11-08T22:20:12Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/druid-io/druid/pulls/2414","html_url":"https://github.com/druid-io/druid/pull/2414","diff_url":"https://github.com/druid-io/druid/pull/2414.diff","patch_url":"https://github.com/druid-io/druid/pull/2414.patch"},"body":"Extending the Avro input row parsers so that complex avro datastructures\ncan be supported. By specifying an extra schemaMappings map in the row\nparser, references to avro fields in the parse spec are translated to\npaths through an avro structure. If the schemaMappings section is not\npresent, then the system works as before.\n\nAn example dataSchema using the new functionality would look like the\nfollowing:\n\n``` json\n\"dataSchema\" : {\n  \"dataSource\" : \"wikipedia\",\n  \"parser\" : {\n    \"type\" : \"avro_hadoop\",\n    \"parseSpec\" : {\n      \"format\" : \"timeAndDims\",\n      \"timestampSpec\" : {\n        \"column\" : \"timestamp\",\n        \"format\" : \"auto\"\n      },\n      \"dimensionsSpec\" : {\n        \"dimensions\":\n[\"page\",\"language\",\"user\",\"unpatrolled\",\"newPage\",\"robot\",\"anonymous\",\"namespace\",\"continent\",\"country\",\"region\",\"city\"],\n        \"dimensionExclusions\" : [],\n        \"spatialDimensions\" : []\n      }\n    },\n    \"schemaMappings\":{\n      \"user\":\"userInfo.userName\",\n      \"language\":\"userInfo.language[0]\",\n      \"country\":\"geoInfo(country)\",\n      \"region\":\"geoInfo(region)\",\n      \"city\":\"geoInfo(city)\",\n      \"added\":\"created\"\n    },\n    \"fromPigAvroStorage\":false\n  },\n  \"metricsSpec\" : [{\n    \"type\" : \"count\",\n    \"name\" : \"count\"\n  }, {\n    \"type\" : \"doubleSum\",\n    \"name\" : \"added\",\n    \"fieldName\" : \"added\"\n  }, {\n    \"type\" : \"doubleSum\",\n    \"name\" : \"deleted\",\n    \"fieldName\" : \"deleted\"\n  }, {\n    \"type\" : \"doubleSum\",\n    \"name\" : \"delta\",\n    \"fieldName\" : \"delta\"\n  }],\n  \"granularitySpec\" : {\n    \"segmentGranularity\" : \"DAY\",\n    \"queryGranularity\" : \"NONE\",\n    \"intervals\" : [ \"2013-08-31/2013-09-01\" ]\n  }\n}\n```\n\nIn this example, the avro record being ingested has a field called\nuserInfo which is a record that in turn has fields for user and\nlanguage. Similarly, the record has a field called geoInfo which is a\nmap with entries in it for city, region, and country. Mapping works\nfor both dimensions and metrics, as can be seen for the added metric,\nwhich is mapped to the created field in avro.\n\nThe language for schema mappings is:\n\nMAPPING : fieldName ( SEGMENT )*\nSEGMENT : '.' fieldName\n        | '(' mapKey ')'\n        | '[' arrayIndex ']'\n\nSo each of the following are valid mappings:\n- `A` - selects the field in the toplevel record\n- `A.B` - selects the subfield B from field A\n- `A(foo)` - selects the map value denoted by key foo from the field A\n  in\n  the top level record.\n- `A[1]` - selects the 2nd entry in the array from field A in the top\n  level record\n- `A.B[2](hello).D` - A complex path! Selects field A, then subfield B,\n  which is an array. Takes the third entry in the array, which is a map,\n  and finds the value denoted by key hello. This in turn is another\n  record, so takes field D.\n\nIf a mapping does not resolve to a valid entry in the Avro record, null\nis returned.\n"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/259277761","html_url":"https://github.com/druid-io/druid/pull/2414#issuecomment-259277761","issue_url":"https://api.github.com/repos/druid-io/druid/issues/2414","id":259277761,"user":{"login":"himanshug","id":246308,"avatar_url":"https://avatars.githubusercontent.com/u/246308?v=3","gravatar_id":"","url":"https://api.github.com/users/himanshug","html_url":"https://github.com/himanshug","followers_url":"https://api.github.com/users/himanshug/followers","following_url":"https://api.github.com/users/himanshug/following{/other_user}","gists_url":"https://api.github.com/users/himanshug/gists{/gist_id}","starred_url":"https://api.github.com/users/himanshug/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/himanshug/subscriptions","organizations_url":"https://api.github.com/users/himanshug/orgs","repos_url":"https://api.github.com/users/himanshug/repos","events_url":"https://api.github.com/users/himanshug/events{/privacy}","received_events_url":"https://api.github.com/users/himanshug/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T22:20:11Z","updated_at":"2016-11-08T22:20:11Z","body":"@fjy I'm removing the 0.9.3 milestone from this PR for now. @will-lauer 's team is currently using our internal avro module and plans to migrate to community avro module in future. We will re-think this PR afterwards."}}
{"actor":{"display_login":"himanshug"}}