{"action":"created","issue":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10840","repository_url":"https://api.github.com/repos/scikit-learn/scikit-learn","labels_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10840/labels{/name}","comments_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10840/comments","events_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10840/events","html_url":"https://github.com/scikit-learn/scikit-learn/issues/10840","id":307000035,"number":10840,"title":"label_ranking_average_precision_score averages over samples, not labels","user":{"login":"dpwe","id":552145,"avatar_url":"https://avatars3.githubusercontent.com/u/552145?v=4","gravatar_id":"","url":"https://api.github.com/users/dpwe","html_url":"https://github.com/dpwe","followers_url":"https://api.github.com/users/dpwe/followers","following_url":"https://api.github.com/users/dpwe/following{/other_user}","gists_url":"https://api.github.com/users/dpwe/gists{/gist_id}","starred_url":"https://api.github.com/users/dpwe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dpwe/subscriptions","organizations_url":"https://api.github.com/users/dpwe/orgs","repos_url":"https://api.github.com/users/dpwe/repos","events_url":"https://api.github.com/users/dpwe/events{/privacy}","received_events_url":"https://api.github.com/users/dpwe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-20T19:07:26Z","updated_at":"2018-03-21T06:33:25Z","closed_at":null,"author_association":"NONE","body":"label_ranking_average_precision_score is a measure of multi-class multi-label classifier performance, defined as the average precision over all the truth labels in an eval set of samples.  The value being averaged is the proportion of correct labels in a per-sample ranked list of labels (classes) that is truncated at each true label.\r\n\r\nAs presently calculated, the metric is\r\n\r\n(1/num_samples) * (sum of per-sample average precisions)\r\n\r\nwhere\r\n\r\nper-sample average precision = 1/num_true_labels * sum_(L in true labels for this sample) (proportion of true labels in ranked label list for this sample truncated at true label L)\r\n\r\nNote that label_ranking_average_precision_score is only interesting if your eval samples bear multiple labels, otherwise it's the same as mean reciprocal rank.\r\n\r\nIf, however, your samples have *differing* numbers of true labels, the contribution of each individual true label is diluted by the total number of true labels for the samples in which it occurs.  \r\n\r\nI would prefer that the average was calculated across each individual ground-truth label, meaning a single sample would contribute to the overall metric in proportion to the number of true labels it bears.\r\n\r\nAlthough I can't make a strong argument to prefer the values obtained averaging-across-samples or averaging-across-true-labels, there is one interesting advantage to averaging-across-true-labels: You can calculate a *per-class* label ranking average precision by taking all the instances of the ground truth label c in your eval set (by definition, at most one per sample), and averaging the precisions of all those truncated ranked lists.  If you then form a weighted average of the per-class average precisions weighted by the total count of that class in the ground truth, you get the averaged-by-label (unconditional) value for the whole data set.  \r\n\r\nIf you normalize each sample's precision by the number of ground-truth labels in that sample (current behavior), it's not possible to get a meaningful per-class precision that you can average up in this way - each individual precision value has to be discounted by total number of labels in that sample, so the per-class average is strongly confounded by the label co-occurrence statistics for that class.\r\n\r\nI'm wondering how hard it would be to change the definition of label ranking average precision to average over individual labels instead of samples as I'm proposing, or whether perhaps it's worth introducing another new metric?"},"comment":{"url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/comments/374844625","html_url":"https://github.com/scikit-learn/scikit-learn/issues/10840#issuecomment-374844625","issue_url":"https://api.github.com/repos/scikit-learn/scikit-learn/issues/10840","id":374844625,"user":{"login":"jnothman","id":78827,"avatar_url":"https://avatars2.githubusercontent.com/u/78827?v=4","gravatar_id":"","url":"https://api.github.com/users/jnothman","html_url":"https://github.com/jnothman","followers_url":"https://api.github.com/users/jnothman/followers","following_url":"https://api.github.com/users/jnothman/following{/other_user}","gists_url":"https://api.github.com/users/jnothman/gists{/gist_id}","starred_url":"https://api.github.com/users/jnothman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jnothman/subscriptions","organizations_url":"https://api.github.com/users/jnothman/orgs","repos_url":"https://api.github.com/users/jnothman/repos","events_url":"https://api.github.com/users/jnothman/events{/privacy}","received_events_url":"https://api.github.com/users/jnothman/received_events","type":"User","site_admin":false},"created_at":"2018-03-21T06:33:25Z","updated_at":"2018-03-21T06:33:25Z","author_association":"OWNER","body":"Okay. I get you now. This corresponds to micro- vs macro- averaging\nalthough we usually mean that over classes.\n\n What does the literature say? Do people do both?\n\nI would think that ordinarily one does want a measure that decomposes over\nsamples, such that you don't overemphasise the samples with lots of labels.\n\nI think it is also equivalent to weight samples by the number of labels. So\nimplementing sample_weight would be sufficient to make this calculable.\n"}}
{"actor":{"display_login":"jnothman"}}