{"action":"created","issue":{"url":"https://api.github.com/repos/rhiever/tpot/issues/199","repository_url":"https://api.github.com/repos/rhiever/tpot","labels_url":"https://api.github.com/repos/rhiever/tpot/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/rhiever/tpot/issues/199/comments","events_url":"https://api.github.com/repos/rhiever/tpot/issues/199/events","html_url":"https://github.com/rhiever/tpot/pull/199","id":166014432,"number":199,"title":"[WIP] Custom scoring","user":{"login":"bartleyn","id":3230070,"avatar_url":"https://avatars.githubusercontent.com/u/3230070?v=3","gravatar_id":"","url":"https://api.github.com/users/bartleyn","html_url":"https://github.com/bartleyn","followers_url":"https://api.github.com/users/bartleyn/followers","following_url":"https://api.github.com/users/bartleyn/following{/other_user}","gists_url":"https://api.github.com/users/bartleyn/gists{/gist_id}","starred_url":"https://api.github.com/users/bartleyn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartleyn/subscriptions","organizations_url":"https://api.github.com/users/bartleyn/orgs","repos_url":"https://api.github.com/users/bartleyn/repos","events_url":"https://api.github.com/users/bartleyn/events{/privacy}","received_events_url":"https://api.github.com/users/bartleyn/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rhiever/tpot/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-07-18T03:43:17Z","updated_at":"2016-08-02T21:51:47Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/rhiever/tpot/pulls/199","html_url":"https://github.com/rhiever/tpot/pull/199","diff_url":"https://github.com/rhiever/tpot/pull/199.diff","patch_url":"https://github.com/rhiever/tpot/pull/199.patch"},"body":"## What does this PR do?\r\n Addresses #156 to allow for classification metrics from scikit-learn's metrics module to be passed to TPOT as scoring functions, along with keyword arguments for the function. Currently  lumps together scoring functions that require decision functions and prediction probabilities together. \r\n\r\n- [x] actual implementation \r\n- [x] unit-tests for different metrics\r\n- [x] documentation change\r\n\r\n## Where should the reviewer start?\r\n\r\nfit(), _evaluate_individual(), and _train_model_and_predict()\r\n\r\n## How should this PR be tested?\r\n\r\nWith classification scoring functions from sklearn.metrics\r\n\r\n## Any background context you want to provide?\r\n\r\nSome of the functions require a classifier's _decision-function_ output in place of a list of predicted classes, and some of the functions require a classifier's _prediction-probabilities_ output. These functions are currently lumped together in consideration -- it is difficult to inspect the input scoring function to see if it requires one or the other (without explicit listing of all the scoring functions, which would become bulky when we start considering regression metrics, etc), so scoring functions requiring decision functions _may_ mis-represent an individual's fitness.  \r\n\r\nIt was decided to lump decision function scoring functions into the prediction probability ones (and not vice versa), since decision functions can return negative numbers, and probabilities not. \r\n\r\n## What are the relevant issues?\r\n\r\n#156 \r\n\r\n## Screenshots (if appropriate)\r\n\r\n\r\n\r\n## Questions:\r\n\r\n- Do the docs need to be updated?\r\nYes, at the very least to reflect the caveats discussed above about the decision_function. \r\n- Does this PR add new (Python) dependencies?\r\nNo. \r\n- How much should TPOT handle wrt type checking, etc, and how much should we rely on the users? "},"comment":{"url":"https://api.github.com/repos/rhiever/tpot/issues/comments/237056909","html_url":"https://github.com/rhiever/tpot/pull/199#issuecomment-237056909","issue_url":"https://api.github.com/repos/rhiever/tpot/issues/199","id":237056909,"user":{"login":"coveralls","id":2354108,"avatar_url":"https://avatars.githubusercontent.com/u/2354108?v=3","gravatar_id":"","url":"https://api.github.com/users/coveralls","html_url":"https://github.com/coveralls","followers_url":"https://api.github.com/users/coveralls/followers","following_url":"https://api.github.com/users/coveralls/following{/other_user}","gists_url":"https://api.github.com/users/coveralls/gists{/gist_id}","starred_url":"https://api.github.com/users/coveralls/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/coveralls/subscriptions","organizations_url":"https://api.github.com/users/coveralls/orgs","repos_url":"https://api.github.com/users/coveralls/repos","events_url":"https://api.github.com/users/coveralls/events{/privacy}","received_events_url":"https://api.github.com/users/coveralls/received_events","type":"User","site_admin":false},"created_at":"2016-08-02T21:51:47Z","updated_at":"2016-08-02T21:51:47Z","body":"\n[![Coverage Status](https://coveralls.io/builds/7277687/badge)](https://coveralls.io/builds/7277687)\n\nCoverage increased (+6.4%) to 79.158% when pulling **d879dc2e1725845e739f11f858bee48fe6b85d0d on bartleyn:custom_scoring** into **c4c9930808b6a7826e40ebd8c86d10b14dc9fd0a on rhiever:development**.\n"}}
{"actor":{"display_login":"coveralls"}}