{"action":"created","issue":{"url":"https://api.github.com/repos/ucbrise/clipper/issues/401","repository_url":"https://api.github.com/repos/ucbrise/clipper","labels_url":"https://api.github.com/repos/ucbrise/clipper/issues/401/labels{/name}","comments_url":"https://api.github.com/repos/ucbrise/clipper/issues/401/comments","events_url":"https://api.github.com/repos/ucbrise/clipper/issues/401/events","html_url":"https://github.com/ucbrise/clipper/issues/401","id":294144226,"number":401,"title":"[Metrics] Enable user-defined metrics","user":{"login":"simon-mo","id":21118851,"avatar_url":"https://avatars3.githubusercontent.com/u/21118851?v=4","gravatar_id":"","url":"https://api.github.com/users/simon-mo","html_url":"https://github.com/simon-mo","followers_url":"https://api.github.com/users/simon-mo/followers","following_url":"https://api.github.com/users/simon-mo/following{/other_user}","gists_url":"https://api.github.com/users/simon-mo/gists{/gist_id}","starred_url":"https://api.github.com/users/simon-mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simon-mo/subscriptions","organizations_url":"https://api.github.com/users/simon-mo/orgs","repos_url":"https://api.github.com/users/simon-mo/repos","events_url":"https://api.github.com/users/simon-mo/events{/privacy}","received_events_url":"https://api.github.com/users/simon-mo/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"simon-mo","id":21118851,"avatar_url":"https://avatars3.githubusercontent.com/u/21118851?v=4","gravatar_id":"","url":"https://api.github.com/users/simon-mo","html_url":"https://github.com/simon-mo","followers_url":"https://api.github.com/users/simon-mo/followers","following_url":"https://api.github.com/users/simon-mo/following{/other_user}","gists_url":"https://api.github.com/users/simon-mo/gists{/gist_id}","starred_url":"https://api.github.com/users/simon-mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simon-mo/subscriptions","organizations_url":"https://api.github.com/users/simon-mo/orgs","repos_url":"https://api.github.com/users/simon-mo/repos","events_url":"https://api.github.com/users/simon-mo/events{/privacy}","received_events_url":"https://api.github.com/users/simon-mo/received_events","type":"User","site_admin":false},"assignees":[{"login":"simon-mo","id":21118851,"avatar_url":"https://avatars3.githubusercontent.com/u/21118851?v=4","gravatar_id":"","url":"https://api.github.com/users/simon-mo","html_url":"https://github.com/simon-mo","followers_url":"https://api.github.com/users/simon-mo/followers","following_url":"https://api.github.com/users/simon-mo/following{/other_user}","gists_url":"https://api.github.com/users/simon-mo/gists{/gist_id}","starred_url":"https://api.github.com/users/simon-mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simon-mo/subscriptions","organizations_url":"https://api.github.com/users/simon-mo/orgs","repos_url":"https://api.github.com/users/simon-mo/repos","events_url":"https://api.github.com/users/simon-mo/events{/privacy}","received_events_url":"https://api.github.com/users/simon-mo/received_events","type":"User","site_admin":false}],"milestone":null,"comments":3,"created_at":"2018-02-03T20:54:15Z","updated_at":"2018-04-05T23:32:15Z","closed_at":"2018-04-05T23:32:15Z","author_association":"CONTRIBUTOR","body":"We shouldn’t just let user define how the metric is computed via a configuration file. \r\n\r\nFollowing that main idea, here are three approaches:\r\n* Pass in a queue/pipe entry point to the specific model function. For example:\r\n    *  def sum_model(xs, metric_queue=None)\r\n* Let the model return prediction + metric.\r\n* (My vote): expose the metric “queue” through http endpoint, let the model container post this data to the metric end point. This does not require any code changes in RPC logic. \r\n    * e.g. http://localhost: 1391\r\n    * in the model, the user can post any metric they want. (for example, the covariance, the categorical result histogram, …)\r\n    * in the rpc.py metric server loop, we have another thread to setup a simple http server, check the posted result and move it inside the metric pipe/queue. \r\n    * The idea is: don’t let the user mess with the metric pipe/queue. We handle the parsing and metric forming for the user. \r\n    * Caveat: this might require user to both update their model and the metric_config.yaml. \r\n\r\nFor performance: we might be able to change http server to \r\n1. zeromq (since they just need to expose tcp port)"},"comment":{"url":"https://api.github.com/repos/ucbrise/clipper/issues/comments/379106740","html_url":"https://github.com/ucbrise/clipper/issues/401#issuecomment-379106740","issue_url":"https://api.github.com/repos/ucbrise/clipper/issues/401","id":379106740,"user":{"login":"simon-mo","id":21118851,"avatar_url":"https://avatars3.githubusercontent.com/u/21118851?v=4","gravatar_id":"","url":"https://api.github.com/users/simon-mo","html_url":"https://github.com/simon-mo","followers_url":"https://api.github.com/users/simon-mo/followers","following_url":"https://api.github.com/users/simon-mo/following{/other_user}","gists_url":"https://api.github.com/users/simon-mo/gists{/gist_id}","starred_url":"https://api.github.com/users/simon-mo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simon-mo/subscriptions","organizations_url":"https://api.github.com/users/simon-mo/orgs","repos_url":"https://api.github.com/users/simon-mo/repos","events_url":"https://api.github.com/users/simon-mo/events{/privacy}","received_events_url":"https://api.github.com/users/simon-mo/received_events","type":"User","site_admin":false},"created_at":"2018-04-05T23:32:15Z","updated_at":"2018-04-05T23:32:15Z","author_association":"CONTRIBUTOR","body":"Closed via #440 "}}
{"actor":{"display_login":"simon-mo"}}