{"action":"closed","issue":{"url":"https://api.github.com/repos/rhiever/tpot/issues/108","repository_url":"https://api.github.com/repos/rhiever/tpot","labels_url":"https://api.github.com/repos/rhiever/tpot/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/rhiever/tpot/issues/108/comments","events_url":"https://api.github.com/repos/rhiever/tpot/issues/108/events","html_url":"https://github.com/rhiever/tpot/issues/108","id":139361742,"number":108,"title":"Replace balanced_accuracy with macro-averaged recall from sklearn","user":{"login":"rhiever","id":1719223,"avatar_url":"https://avatars.githubusercontent.com/u/1719223?v=3","gravatar_id":"","url":"https://api.github.com/users/rhiever","html_url":"https://github.com/rhiever","followers_url":"https://api.github.com/users/rhiever/followers","following_url":"https://api.github.com/users/rhiever/following{/other_user}","gists_url":"https://api.github.com/users/rhiever/gists{/gist_id}","starred_url":"https://api.github.com/users/rhiever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhiever/subscriptions","organizations_url":"https://api.github.com/users/rhiever/orgs","repos_url":"https://api.github.com/users/rhiever/repos","events_url":"https://api.github.com/users/rhiever/events{/privacy}","received_events_url":"https://api.github.com/users/rhiever/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rhiever/tpot/labels/easy","name":"easy","color":"bfe5bf"},{"url":"https://api.github.com/repos/rhiever/tpot/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/rhiever/tpot/milestones/1","html_url":"https://github.com/rhiever/tpot/milestones/0.4%20release","labels_url":"https://api.github.com/repos/rhiever/tpot/milestones/1/labels","id":1798985,"number":1,"title":"0.4 release","description":"","creator":{"login":"rhiever","id":1719223,"avatar_url":"https://avatars.githubusercontent.com/u/1719223?v=3","gravatar_id":"","url":"https://api.github.com/users/rhiever","html_url":"https://github.com/rhiever","followers_url":"https://api.github.com/users/rhiever/followers","following_url":"https://api.github.com/users/rhiever/following{/other_user}","gists_url":"https://api.github.com/users/rhiever/gists{/gist_id}","starred_url":"https://api.github.com/users/rhiever/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rhiever/subscriptions","organizations_url":"https://api.github.com/users/rhiever/orgs","repos_url":"https://api.github.com/users/rhiever/repos","events_url":"https://api.github.com/users/rhiever/events{/privacy}","received_events_url":"https://api.github.com/users/rhiever/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":8,"state":"open","created_at":"2016-05-31T14:42:16Z","updated_at":"2016-06-17T14:50:00Z","due_on":"2016-06-17T04:00:00Z","closed_at":null},"comments":4,"created_at":"2016-03-08T18:54:19Z","updated_at":"2016-06-17T14:50:00Z","closed_at":"2016-06-17T14:50:00Z","body":"From conversations with @amueller, we discovered that \"balanced accuracy\" (as we've called it) is also known as \"macro-averaged recall\" as implemented in sklearn. As such, we don't need our own custom implementation of `balanced_accuracy` in TPOT. Let's refactor TPOT to replace `balanced_accuracy` with [recall_score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html#sklearn.metrics.recall_score).\r\n\r\nThe correct call is:\r\n\r\n```python\r\nrecall_score(y_test, predictions, average='macro')\r\n```\r\n\r\nwhere `y_test` is `class` and `predictions` is `guess` in our case.\r\n\r\nHere's some code that compares the two and confirms that they're the same:\r\n\r\n```python\r\nfrom sklearn.datasets import load_digits\r\nfrom sklearn.ensemble import RandomForestClassifier\r\nfrom sklearn.cross_validation import train_test_split\r\nfrom sklearn.metrics import recall_score\r\nimport numpy as np\r\nimport pandas as pd\r\n\r\ndigits = load_digits(10)\r\nfeatures, labels = digits['data'], digits['target']\r\n\r\nX_train, X_test, y_train, y_test = train_test_split(features, labels, train_size=0.75, test_size=0.25)\r\n\r\nclf = RandomForestClassifier(n_estimators=100, n_jobs=-1)\r\nclf.fit(X_train, y_train)\r\n\r\ndef balanced_accuracy(result):\r\n    all_classes = list(set(result['class'].values))\r\n    all_class_accuracies = []\r\n    for this_class in all_classes:\r\n        this_class_accuracy = len(result[(result['guess'] == this_class) & (result['class'] == this_class)])\\\r\n            / float(len(result[result['class'] == this_class]))\r\n        all_class_accuracies.append(this_class_accuracy)\r\n\r\n    balanced_accuracy = np.mean(all_class_accuracies)\r\n\r\n    return balanced_accuracy\r\n\r\npredictions = clf.predict(X_test)\r\n\r\nprint('Macro-averaged recall:\\t', recall_score(y_test, predictions, average='macro'))\r\n\r\ndata = pd.DataFrame({'class': y_test,\r\n                     'guess': predictions})\r\n\r\nprint('Balanced accuracy:\\t', balanced_accuracy(data))\r\n```"}}
{"actor":{"display_login":"rhiever"}}