{"action":"created","issue":{"url":"https://api.github.com/repos/autonomio/talos/issues/56","repository_url":"https://api.github.com/repos/autonomio/talos","labels_url":"https://api.github.com/repos/autonomio/talos/issues/56/labels{/name}","comments_url":"https://api.github.com/repos/autonomio/talos/issues/56/comments","events_url":"https://api.github.com/repos/autonomio/talos/issues/56/events","html_url":"https://github.com/autonomio/talos/issues/56","id":350486889,"node_id":"MDU6SXNzdWUzNTA0ODY4ODk=","number":56,"title":"How to use f1 measure for \"best\" model?","user":{"login":"bml1g12","id":2605767,"node_id":"MDQ6VXNlcjI2MDU3Njc=","avatar_url":"https://avatars2.githubusercontent.com/u/2605767?v=4","gravatar_id":"","url":"https://api.github.com/users/bml1g12","html_url":"https://github.com/bml1g12","followers_url":"https://api.github.com/users/bml1g12/followers","following_url":"https://api.github.com/users/bml1g12/following{/other_user}","gists_url":"https://api.github.com/users/bml1g12/gists{/gist_id}","starred_url":"https://api.github.com/users/bml1g12/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bml1g12/subscriptions","organizations_url":"https://api.github.com/users/bml1g12/orgs","repos_url":"https://api.github.com/users/bml1g12/repos","events_url":"https://api.github.com/users/bml1g12/events{/privacy}","received_events_url":"https://api.github.com/users/bml1g12/received_events","type":"User","site_admin":false},"labels":[{"id":923120512,"node_id":"MDU6TGFiZWw5MjMxMjA1MTI=","url":"https://api.github.com/repos/autonomio/talos/labels/high%20priority%20fix","name":"high priority fix","color":"b60205","default":false},{"id":994489876,"node_id":"MDU6TGFiZWw5OTQ0ODk4NzY=","url":"https://api.github.com/repos/autonomio/talos/labels/investigation","name":"investigation","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2018-08-14T15:44:21Z","updated_at":"2018-08-17T02:43:21Z","closed_at":null,"author_association":"NONE","body":"When I import `from talos.metrics.keras_metrics import fbeta_score` and compile the model with this metric, then run talos with the parameter `reduction_metric=\"fbeta_score\"` the output csv seems to list the the val_acc of the best epoch for val_acc, but only the first epoch's value for fbeta_score. This seems like something is going wrong, as if anything it should be producing the corresponding fbeta_score for that epoch I would have though.  \r\n\r\nI am not interested  in accuracy due to class imbalance in my system, and the accuracy saturates after a few epochs, so I need to talos to store for each parameter combination either:\r\n\r\na) The result of the last epoch \r\nb) Ideally the result with the best fbeta_score \r\n\r\nGiven that fbeta_score has been implemented, I assume this must be possible but I don't see how. \r\n\r\nI am using the latest dev branch  v0.2 (as I have augmented data, I needed the functionality to supply x_val and y_val as parameters). In order to run this code without bugs, I needed to change;\r\n`talos/metrics/score_model.py` line 17 \r\nfrom `y_pred = self.keras_model.predict_classes(self.x_val)`\r\nto ` y_pred = self.keras_model.predict(self.x_val)` \r\n\r\nWhich might be related to my problem. "},"comment":{"url":"https://api.github.com/repos/autonomio/talos/issues/comments/413741569","html_url":"https://github.com/autonomio/talos/issues/56#issuecomment-413741569","issue_url":"https://api.github.com/repos/autonomio/talos/issues/56","id":413741569,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzc0MTU2OQ==","user":{"login":"x94carbone","id":26401020,"node_id":"MDQ6VXNlcjI2NDAxMDIw","avatar_url":"https://avatars3.githubusercontent.com/u/26401020?v=4","gravatar_id":"","url":"https://api.github.com/users/x94carbone","html_url":"https://github.com/x94carbone","followers_url":"https://api.github.com/users/x94carbone/followers","following_url":"https://api.github.com/users/x94carbone/following{/other_user}","gists_url":"https://api.github.com/users/x94carbone/gists{/gist_id}","starred_url":"https://api.github.com/users/x94carbone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/x94carbone/subscriptions","organizations_url":"https://api.github.com/users/x94carbone/orgs","repos_url":"https://api.github.com/users/x94carbone/repos","events_url":"https://api.github.com/users/x94carbone/events{/privacy}","received_events_url":"https://api.github.com/users/x94carbone/received_events","type":"User","site_admin":false},"created_at":"2018-08-17T02:43:21Z","updated_at":"2018-08-17T02:43:21Z","author_association":"COLLABORATOR","body":"Ah I'm sorry. You're totally right. The lack of sleep is getting to me! I'll get on this sometime soon when I have more energy.\r\n\r\nI'm not sure if Talos is actually using the Keras metric to generate the F1 score you see in the pandas output or if its using its own. This needs to be consistent in the future.\r\n\r\nAnyway, I will look into this at some point. I appreciate you bringing this to our attention since it is a very real problem that needs fixing."}}
{"actor":{"display_login":"x94carbone"}}