{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16466","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16466/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16466/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16466/events","html_url":"https://github.com/tensorflow/tensorflow/issues/16466","id":291995940,"node_id":"MDU6SXNzdWUyOTE5OTU5NDA=","number":16466,"title":"[Feature request] Adding a PR curves to canned estimators for (binary) classifiers","user":{"login":"eisenjulian","id":7776575,"node_id":"MDQ6VXNlcjc3NzY1NzU=","avatar_url":"https://avatars0.githubusercontent.com/u/7776575?v=4","gravatar_id":"","url":"https://api.github.com/users/eisenjulian","html_url":"https://github.com/eisenjulian","followers_url":"https://api.github.com/users/eisenjulian/followers","following_url":"https://api.github.com/users/eisenjulian/following{/other_user}","gists_url":"https://api.github.com/users/eisenjulian/gists{/gist_id}","starred_url":"https://api.github.com/users/eisenjulian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eisenjulian/subscriptions","organizations_url":"https://api.github.com/users/eisenjulian/orgs","repos_url":"https://api.github.com/users/eisenjulian/repos","events_url":"https://api.github.com/users/eisenjulian/events{/privacy}","received_events_url":"https://api.github.com/users/eisenjulian/received_events","type":"User","site_admin":false},"labels":[{"id":473173272,"node_id":"MDU6TGFiZWw0NzMxNzMyNzI=","url":"https://api.github.com/repos/tensorflow/tensorflow/labels/type:feature","name":"type:feature","color":"c2e0c6","default":false}],"state":"open","locked":false,"assignee":{"login":"chihuahua","id":4221553,"node_id":"MDQ6VXNlcjQyMjE1NTM=","avatar_url":"https://avatars0.githubusercontent.com/u/4221553?v=4","gravatar_id":"","url":"https://api.github.com/users/chihuahua","html_url":"https://github.com/chihuahua","followers_url":"https://api.github.com/users/chihuahua/followers","following_url":"https://api.github.com/users/chihuahua/following{/other_user}","gists_url":"https://api.github.com/users/chihuahua/gists{/gist_id}","starred_url":"https://api.github.com/users/chihuahua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chihuahua/subscriptions","organizations_url":"https://api.github.com/users/chihuahua/orgs","repos_url":"https://api.github.com/users/chihuahua/repos","events_url":"https://api.github.com/users/chihuahua/events{/privacy}","received_events_url":"https://api.github.com/users/chihuahua/received_events","type":"User","site_admin":false},"assignees":[{"login":"chihuahua","id":4221553,"node_id":"MDQ6VXNlcjQyMjE1NTM=","avatar_url":"https://avatars0.githubusercontent.com/u/4221553?v=4","gravatar_id":"","url":"https://api.github.com/users/chihuahua","html_url":"https://github.com/chihuahua","followers_url":"https://api.github.com/users/chihuahua/followers","following_url":"https://api.github.com/users/chihuahua/following{/other_user}","gists_url":"https://api.github.com/users/chihuahua/gists{/gist_id}","starred_url":"https://api.github.com/users/chihuahua/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chihuahua/subscriptions","organizations_url":"https://api.github.com/users/chihuahua/orgs","repos_url":"https://api.github.com/users/chihuahua/repos","events_url":"https://api.github.com/users/chihuahua/events{/privacy}","received_events_url":"https://api.github.com/users/chihuahua/received_events","type":"User","site_admin":false}],"milestone":null,"comments":16,"created_at":"2018-01-26T18:49:27Z","updated_at":"2018-07-16T22:07:08Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: No\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Windows 10\r\n- **TensorFlow installed from (source or binary)**: binary\r\n- **TensorFlow version (use command below)**: 1.4\r\n- **Python version**: 3.6.3\r\n- **Bazel version (if compiling from source)**:\r\n- **GCC/Compiler version (if compiling from source)**:\r\n- **CUDA/cuDNN version**:\r\n- **GPU model and memory**:\r\n- **Exact command to reproduce**:\r\n\r\n### Describe the problem\r\nThis is a feature request, and I'd be happy to **contribute** if you think it's a valuable addition. \r\nI have been using estimators both pre-made and custom for classification tasks. I like that sharing the use of a `head`  as defined in [tensorflow/python/estimator/canned/head.py](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/estimator/canned/head.py) allows both the canned models and the custom ones to share prediction and evaluation metrics for comparison, however currently it feels that some key metrics are missing, mainly PR curves which are fully supported by tensorboard.  Currently, the `head` constructor allows a list of thresholds, although they are not used by default. The problem is that when used, it creates scalar summaries for the precision and recall at each threshold, which is not that useful as in general one wants to compare how different models compare precision wise while fixing recall and the other way round.\r\n\r\nAdding the PR summary op from [here](https://github.com/tensorflow/tensorboard/tree/master/tensorboard/plugins/pr_curve) would make the eval metrics more informative IMO.\r\n\r\nThanks for taking the time to read this!\r\n\r\n### Source code / logs\r\nAny code that uses pre-made estimator classifiers relies on the same head. [This](https://github.com/tensorflow/models/blob/master/official/wide_deep/wide_deep.py) is one example."},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/405396725","html_url":"https://github.com/tensorflow/tensorflow/issues/16466#issuecomment-405396725","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/16466","id":405396725,"node_id":"MDEyOklzc3VlQ29tbWVudDQwNTM5NjcyNQ==","user":{"login":"tensorflowbutler","id":28546240,"node_id":"MDQ6VXNlcjI4NTQ2MjQw","avatar_url":"https://avatars2.githubusercontent.com/u/28546240?v=4","gravatar_id":"","url":"https://api.github.com/users/tensorflowbutler","html_url":"https://github.com/tensorflowbutler","followers_url":"https://api.github.com/users/tensorflowbutler/followers","following_url":"https://api.github.com/users/tensorflowbutler/following{/other_user}","gists_url":"https://api.github.com/users/tensorflowbutler/gists{/gist_id}","starred_url":"https://api.github.com/users/tensorflowbutler/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tensorflowbutler/subscriptions","organizations_url":"https://api.github.com/users/tensorflowbutler/orgs","repos_url":"https://api.github.com/users/tensorflowbutler/repos","events_url":"https://api.github.com/users/tensorflowbutler/events{/privacy}","received_events_url":"https://api.github.com/users/tensorflowbutler/received_events","type":"User","site_admin":false},"created_at":"2018-07-16T22:07:08Z","updated_at":"2018-07-16T22:07:08Z","author_association":"MEMBER","body":"Nagging Assignee @chihuahua: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly."}}
{"actor":{"display_login":"tensorflowbutler"}}