{"action":"created","issue":{"url":"https://api.github.com/repos/apache/flink/issues/2838","repository_url":"https://api.github.com/repos/apache/flink","labels_url":"https://api.github.com/repos/apache/flink/issues/2838/labels{/name}","comments_url":"https://api.github.com/repos/apache/flink/issues/2838/comments","events_url":"https://api.github.com/repos/apache/flink/issues/2838/events","html_url":"https://github.com/apache/flink/pull/2838","id":190678660,"number":2838,"title":"[FLINK-4712] [FLINK-4713] [ml] Ranking recommendation & evaluation (WIP)","user":{"login":"gaborhermann","id":6912263,"avatar_url":"https://avatars.githubusercontent.com/u/6912263?v=3","gravatar_id":"","url":"https://api.github.com/users/gaborhermann","html_url":"https://github.com/gaborhermann","followers_url":"https://api.github.com/users/gaborhermann/followers","following_url":"https://api.github.com/users/gaborhermann/following{/other_user}","gists_url":"https://api.github.com/users/gaborhermann/gists{/gist_id}","starred_url":"https://api.github.com/users/gaborhermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gaborhermann/subscriptions","organizations_url":"https://api.github.com/users/gaborhermann/orgs","repos_url":"https://api.github.com/users/gaborhermann/repos","events_url":"https://api.github.com/users/gaborhermann/events{/privacy}","received_events_url":"https://api.github.com/users/gaborhermann/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-11-21T10:40:53Z","updated_at":"2016-11-21T13:28:39Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/flink/pulls/2838","html_url":"https://github.com/apache/flink/pull/2838","diff_url":"https://github.com/apache/flink/pull/2838.diff","patch_url":"https://github.com/apache/flink/pull/2838.patch"},"body":"Please note that this is a work-in-progress PR for discussing API design decisions. We propose here a class hierarchy for fitting the ranking evaluations into the proposed evaluation framework (see [PR](https://github.com/apache/flink/pull/1849)).\r\nThe features are mostly working, but documentation is missing and minor refactoring is needed. The evaluations currently work with top 100 rankings (burnt-in), and we still need to fix that. We need feedback for two main solutions, so we can go on with the PR. Thanks for any comment!\r\n\r\n### `RankingPredictor`\r\n\r\nWe have managed to rework the evaluation framework proposed by @thvasilo, so that ranking predictions would fit in. Our approach is to use separate `RankingPredictor` and `Predictor` traits. One main problem however remains: there is no common superclass for `RankingPredictor` and `Predictor` so the pipelining mechanism might not work. A `Predictor` can only be at the and of the pipeline, so this should not really be a problem, but I do not know for sure. An alternative solution would be to have different objects `ALS` and `RankingALS` that give different predictions, but both extends only a `Predictor`. There could be implicit conversions between the two. I would prefer the current solution if it does not break the pipelining. @thvasilo what do you think about this?\r\n\r\n(This seems to be a problem similar to having a `predict_proba` function in scikit learn classification models, where the same model for the same input gives two different predictions: a `predict` for discrete predictions and `predict_proba` for giving a probability.)\r\n\r\n### Generalizing `EvalutateDataSetOperation`\r\n\r\nOn the other hand, we seem to have solved the scoring issue. The users can evaluate a recommendation algorithm such as ALS by using a score operating on rankings (e.g. nDCG), or a score operating on ratings (e.g. RMSE). They only need to modify the `Score` they use in their code, and nothing else.\r\n\r\nThe main problem was that the evaluate method and `EvaluateDataSetOperation` were not general enough. They prepare the evaluation to `(trueValue, predictedValue)` pairs (i.e. a `DataSet[(PredictionType, PredictionType)]`), while ranking evaluations needed a more general input with the true ratings (`DataSet[(Int,Int,Double)]`) and the predicted rankings (`DataSet[(Int,Int,Int)]`).\r\n\r\nInstead of using `EvaluateDataSetOperation` we use a more general `PrepareOperation`. We rename the `Score` in the original evaluation framework to `PairwiseScore`. `RankingScore` and `PairwiseScore` has a common trait `Score`. This way the user can use both a `RankingScore` and a `PairwiseScore` for a certain model, and only need to alter the score used in the code.\r\n\r\nIn case of pairwise scores (that only need true and predicted value pairs for evaluation) `EvaluateDataSetOperation` is used as a `PrepareOperation`. It prepares the evaluation by creating `(trueValue, predicitedValue)` pairs from the test dataset. Thus, the result of preparing and the input of `PairwiseScore`s will be `DataSet[(PredictionType,PredictionType)]`. In case of rankings the `PrepareOperation` passes the test dataset and creates the rankings. The result of preparing and the input of `RankingScore`s will be `(DataSet[Int,Int,Double], DataSet[Int,Int,Int])`. I believe this is a fairly acceptable solution that avoids breaking the API."},"comment":{"url":"https://api.github.com/repos/apache/flink/issues/comments/261938553","html_url":"https://github.com/apache/flink/pull/2838#issuecomment-261938553","issue_url":"https://api.github.com/repos/apache/flink/issues/2838","id":261938553,"user":{"login":"gaborhermann","id":6912263,"avatar_url":"https://avatars.githubusercontent.com/u/6912263?v=3","gravatar_id":"","url":"https://api.github.com/users/gaborhermann","html_url":"https://github.com/gaborhermann","followers_url":"https://api.github.com/users/gaborhermann/followers","following_url":"https://api.github.com/users/gaborhermann/following{/other_user}","gists_url":"https://api.github.com/users/gaborhermann/gists{/gist_id}","starred_url":"https://api.github.com/users/gaborhermann/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gaborhermann/subscriptions","organizations_url":"https://api.github.com/users/gaborhermann/orgs","repos_url":"https://api.github.com/users/gaborhermann/repos","events_url":"https://api.github.com/users/gaborhermann/events{/privacy}","received_events_url":"https://api.github.com/users/gaborhermann/received_events","type":"User","site_admin":false},"created_at":"2016-11-21T13:28:39Z","updated_at":"2016-11-21T13:28:39Z","body":"Hi @thvasilo, of course, thanks for taking a look! :)"}}
{"actor":{"display_login":"gaborhermann"}}