{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/18798","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18798/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18798/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18798/events","html_url":"https://github.com/elastic/elasticsearch/pull/18798","id":159379069,"number":18798,"title":"Experiment: Module for computing prec@ on queries","user":{"login":"MaineC","id":70953,"avatar_url":"https://avatars.githubusercontent.com/u/70953?v=3","gravatar_id":"","url":"https://api.github.com/users/MaineC","html_url":"https://github.com/MaineC","followers_url":"https://api.github.com/users/MaineC/followers","following_url":"https://api.github.com/users/MaineC/following{/other_user}","gists_url":"https://api.github.com/users/MaineC/gists{/gist_id}","starred_url":"https://api.github.com/users/MaineC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaineC/subscriptions","organizations_url":"https://api.github.com/users/MaineC/orgs","repos_url":"https://api.github.com/users/MaineC/repos","events_url":"https://api.github.com/users/MaineC/events{/privacy}","received_events_url":"https://api.github.com/users/MaineC/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Search","name":":Search","color":"009800"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feature","name":"feature","color":"328a58"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v6.0.0","name":"v6.0.0","color":"dddddd"},{"url":"https://api.github.com/repos/elastic/elasticsearch/labels/WIP","name":"WIP","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2016-06-09T10:48:46Z","updated_at":"2016-06-30T18:33:28Z","closed_at":"2016-06-30T18:33:28Z","pull_request":{"url":"https://api.github.com/repos/elastic/elasticsearch/pulls/18798","html_url":"https://github.com/elastic/elasticsearch/pull/18798","diff_url":"https://github.com/elastic/elasticsearch/pull/18798.diff","patch_url":"https://github.com/elastic/elasticsearch/pull/18798.patch"},"body":"Before continuing to read: Consider this to be an experiment, WIP, an early\r\ndraft, including a few questions as even the tests will fail occasionally. Putting\r\nthis up as PR to start a discussion off of some code - so please do feel free to\r\ntake things apart ;)\r\n\r\nThe implementation itself has been lying around in my GitHub repo for a while which you can easily see from the commit history, I took some time to clean it up and adjust it to the current state of things recently - turns out that thanks to the query refactoring efforts, the introduction of modules and various other changes we made in past months the original code could be trimmed quite a bit.\r\n\r\nThis PR adds a module to add facilities to compute ranking quality metrics on a\r\nset of pre-annotated query/result sets. Currently only Prec@ is supported as a\r\nmetric, adding new metrics is as easy as extending the Evaluator/\r\nRankedListQualityMetric interface.\r\n\r\nAt the moment there's no REST end point implemented. For illustration purposes\r\non how to use this stuff look at PrecisionAtRequestTest. Essentially the way it\r\nworks is as following:\r\n\r\n* Assuming you've indexed a bunch of documents and want to understand how well\r\nyour chosen ranking function performs for a set of queries:\r\n* Create a set of queries, for each query add a set of documents you deem\r\nrelevant to each query.\r\n* Submit this stuff, queries are going to be executed against your index, for\r\neach query the returned documents are checked against the set of relevant\r\ndocuments you supplied, Prec@ is computed and each document that you\r\ndidn't explicitly submit as relevant that was still returned for the query is\r\nbeing returned as well.\r\n\r\nCaveats:\r\n\r\n* Currently the whole set of annotated queries and documents has to be sent over\r\nthe wire. I believe it would make sense to store this internally.\r\n* The naming of some classes, methods and fields could use quite a bit of love\r\nand consistency.\r\n* There's not REST endpoint yet.\r\n* There's no integration with the task mgmt framework whatsoever.\r\n* There's a dependency between this module and the :lang:mustache module that\r\nI'm pretty sure could be handled in a better way.\r\n* The integration test I'm referring to above fails in roughly half of the cases\r\nwith a NullPointerException in\r\noe.client.transport.support.TransportProxyClient:66 when looking for my\r\nRankedEvalAction - I assume I forgot to register that someplace important but\r\nafter looking around for some time this morning couldn't figure out where. Any\r\nenlightening appreciated.\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/229748885","html_url":"https://github.com/elastic/elasticsearch/pull/18798#issuecomment-229748885","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/18798","id":229748885,"user":{"login":"MaineC","id":70953,"avatar_url":"https://avatars.githubusercontent.com/u/70953?v=3","gravatar_id":"","url":"https://api.github.com/users/MaineC","html_url":"https://github.com/MaineC","followers_url":"https://api.github.com/users/MaineC/followers","following_url":"https://api.github.com/users/MaineC/following{/other_user}","gists_url":"https://api.github.com/users/MaineC/gists{/gist_id}","starred_url":"https://api.github.com/users/MaineC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MaineC/subscriptions","organizations_url":"https://api.github.com/users/MaineC/orgs","repos_url":"https://api.github.com/users/MaineC/repos","events_url":"https://api.github.com/users/MaineC/events{/privacy}","received_events_url":"https://api.github.com/users/MaineC/received_events","type":"User","site_admin":false},"created_at":"2016-06-30T18:33:28Z","updated_at":"2016-06-30T18:33:28Z","body":"Closing in favor of collaborating on this code in branch https://github.com/elastic/elasticsearch/tree/feature/rank-eval"}}
{"actor":{"display_login":"MaineC"}}