{"action":"created","issue":{"url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/28","repository_url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor","labels_url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/28/comments","events_url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/28/events","html_url":"https://github.com/radanalyticsio/jiminy-predictor/pull/28","id":298039521,"number":28,"title":"Guard against invalid user id in top-k requests","user":{"login":"ruivieira","id":327909,"avatar_url":"https://avatars1.githubusercontent.com/u/327909?v=4","gravatar_id":"","url":"https://api.github.com/users/ruivieira","html_url":"https://github.com/ruivieira","followers_url":"https://api.github.com/users/ruivieira/followers","following_url":"https://api.github.com/users/ruivieira/following{/other_user}","gists_url":"https://api.github.com/users/ruivieira/gists{/gist_id}","starred_url":"https://api.github.com/users/ruivieira/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ruivieira/subscriptions","organizations_url":"https://api.github.com/users/ruivieira/orgs","repos_url":"https://api.github.com/users/ruivieira/repos","events_url":"https://api.github.com/users/ruivieira/events{/privacy}","received_events_url":"https://api.github.com/users/ruivieira/received_events","type":"User","site_admin":false},"labels":[{"id":699337796,"url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/labels/bug","name":"bug","color":"ee0701","default":true},{"id":699337798,"url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-02-17T21:51:31Z","updated_at":"2018-02-19T20:42:33Z","closed_at":"2018-02-19T20:42:26Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/pulls/28","html_url":"https://github.com/radanalyticsio/jiminy-predictor/pull/28","diff_url":"https://github.com/radanalyticsio/jiminy-predictor/pull/28.diff","patch_url":"https://github.com/radanalyticsio/jiminy-predictor/pull/28.patch"},"body":"When requesting top-k predictions for an invalid user `id` (see #27), `MatrixFactorizationModel` crashes and Spark ALS stops providing predictions.\r\n\r\nSince Spark's ALS model has no built-in facility to catch this, a safeguard must be built at the application logic level. The solution proposed in this PR consists of:\r\n\r\n - At the model loading stage extract an `RDD` of user `id`s and another of product `id`s\r\n - This is used by the `Model` class to implement two methods (`valid_user(id)` and `valid_product(id)`) to check if the provided `id`s are within the model\r\n - At the request loop, prior to performing prediction, `id`s are checked for validity and if they are not valid, the predicted top-k product list will be empty, bypassing prediction.\r\n\r\nThis prevents crashing the ALS model at the cost of one call to an `RDD` of `id`s.\r\n\r\nExample (after patching):\r\n\r\nUsing user `id=900000` (invalid) we issue\r\n\r\n```\r\ncurl --request POST \\\r\n   --url http://0.0.0.0:8080/predictions/ranks \\\r\n   --header 'content-type: application/json' \\\r\n   --data '{\"user\": 900000, \"topk\": 25}'\r\n```\r\n\r\nwe get the server response\r\n\r\n```\r\n{\r\n  \"prediction\": {\r\n    \"id\": \"ea632652c61c4508bed702bfecad32c7\",\r\n    \"products\": [],\r\n    \"topk\": 25,\r\n    \"user\": 900000\r\n  }\r\n}\r\n```\r\n\r\nthe log alerts us:\r\n\r\n```\r\n2018-02-17 21:46:23,947 - jiminy-predictor - ERROR - Requesting rankings for invalid user id=900000\r\n```\r\n\r\nFetching the prediction with\r\n\r\n```\r\ncurl --request GET --url http://0.0.0.0:8080/predictions/ranks/ea632652c61c4508bed702bfecad32c7\r\n```\r\n\r\nReturning the response:\r\n```\r\n{\r\n  \"id\": \"ea632652c61c4508bed702bfecad32c7\",\r\n  \"products\": [],\r\n  \"topk\": 25,\r\n  \"user\": 900000\r\n}\r\n```\r\n\r\nbut without crashing the Spark model."},"comment":{"url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/comments/366798191","html_url":"https://github.com/radanalyticsio/jiminy-predictor/pull/28#issuecomment-366798191","issue_url":"https://api.github.com/repos/radanalyticsio/jiminy-predictor/issues/28","id":366798191,"user":{"login":"elmiko","id":190649,"avatar_url":"https://avatars3.githubusercontent.com/u/190649?v=4","gravatar_id":"","url":"https://api.github.com/users/elmiko","html_url":"https://github.com/elmiko","followers_url":"https://api.github.com/users/elmiko/followers","following_url":"https://api.github.com/users/elmiko/following{/other_user}","gists_url":"https://api.github.com/users/elmiko/gists{/gist_id}","starred_url":"https://api.github.com/users/elmiko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elmiko/subscriptions","organizations_url":"https://api.github.com/users/elmiko/orgs","repos_url":"https://api.github.com/users/elmiko/repos","events_url":"https://api.github.com/users/elmiko/events{/privacy}","received_events_url":"https://api.github.com/users/elmiko/received_events","type":"User","site_admin":false},"created_at":"2018-02-19T20:42:33Z","updated_at":"2018-02-19T20:42:33Z","author_association":"COLLABORATOR","body":"closes #27 "}}
{"actor":{"display_login":"elmiko"}}