{"action":"created","issue":{"url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/51","repository_url":"https://api.github.com/repos/ctlearn-project/ctlearn","labels_url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/51/labels{/name}","comments_url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/51/comments","events_url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/51/events","html_url":"https://github.com/ctlearn-project/ctlearn/issues/51","id":352924190,"node_id":"MDU6SXNzdWUzNTI5MjQxOTA=","number":51,"title":"Reproduction of benchmark 0.2.0 results","user":{"login":"Jsevillamol","id":5825805,"node_id":"MDQ6VXNlcjU4MjU4MDU=","avatar_url":"https://avatars0.githubusercontent.com/u/5825805?v=4","gravatar_id":"","url":"https://api.github.com/users/Jsevillamol","html_url":"https://github.com/Jsevillamol","followers_url":"https://api.github.com/users/Jsevillamol/followers","following_url":"https://api.github.com/users/Jsevillamol/following{/other_user}","gists_url":"https://api.github.com/users/Jsevillamol/gists{/gist_id}","starred_url":"https://api.github.com/users/Jsevillamol/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Jsevillamol/subscriptions","organizations_url":"https://api.github.com/users/Jsevillamol/orgs","repos_url":"https://api.github.com/users/Jsevillamol/repos","events_url":"https://api.github.com/users/Jsevillamol/events{/privacy}","received_events_url":"https://api.github.com/users/Jsevillamol/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-22T12:07:03Z","updated_at":"2018-09-21T19:37:20Z","closed_at":null,"author_association":"COLLABORATOR","body":"I have reproduced the benchmark 0.2.0 results in the UCM server (unsure of the NVIDIA GPU model @nietootein ?).\r\n\r\n|  Input | Telescope Type | Train Events | Val Events | Train steps | Batch size | Train time |   Val Acc  | Val Gamma Acc | Val Proton Acc |   Val AUC  |\r\n|:------:|:--------------:|:------------:|:----------:|:-----------:|:----------:|:----------:|:----------:|:-------------:|:--------------:|:----------:|\r\n| Single |       LST      |    161631    |    17960   |    37500    |     16     |  1h 0m 55s |  0.7034521 |   0.63914883  |    0.7676981   |  0.7905172 |\r\n| Single |      MSTF      |    666288    |    74032   |    37500    |     16     | 1h 25m 39s |  0.7445564 |   0.8048835   |   0.68821025   |  0.8311684 |\r\n| Single |      MSTN      |    772385    |    85821   |    37500    |     16     | 1h 29m 53s |  0.7803801 |   0.82118773  |    0.742127    |  0.8679488 |\r\n| Single |      MSTS      |    541990    |    60222   |    37500    |     16     | 1h 14m 38s | 0.78469664 |   0.8445853   |    0.7228224   | 0.86962146 |\r\n| Single |      SST1      |    379611    |    42180   |    37500    |     16     |   55m 59s  |  0.7793741 |   0.8133864   |   0.74446315   |  0.8592271 |\r\n| Single |      SSTA      |    404866    |    44986   |    37500    |     16     |  1h 7m 45s | 0.72549236 |   0.66570693  |   0.78389066   |  0.8107811 |\r\n| Single |      SSTC      |    392626    |    43626   |    37500    |     16     |  1h 5m 31s |  0.7493009 |   0.75805485  |    0.7405917   |  0.8215633 |\r\n|  Array |       LST      |     76860    |    8541    |    37500    |     16     |   37m 27s  |  0.7280178 |   0.8020253   |    0.6643433   | 0.82055163 |\r\n|  Array |      MSTF      |    224831    |    24982   |    37500    |     16     |  2h 7m 11s | 0.80393887 |   0.8210466   |     0.78773    |  0.895706  |\r\n|  Array |      MSTN      |    242425    |    26937   |    37500    |     16     | 2h 11m 18s |  0.8277462 |   0.87108314  |    0.7881501   |  0.9178416 |\r\n|  Array |      MSTS      |    200745    |    22306   |    37500    |     16     | 2h 31m 13s |  0.8198691 |   0.8415895   |    0.7982652   |  0.9099872 |\r\n|  Array |      SST1      |    178090    |    19788   |    37500    |     16     | 3h 13m 27s |  0.7991712 |   0.7722142   |    0.8288231   | 0.89470106 |\r\n|  Array |      SSTA      |    165302    |    18367   |    37500    |     16     |  2h 9m 23s | 0.76920563 |   0.7152889   |   0.82571363   | 0.86622685 |\r\n|  Array |      SSTC      |    171574    |    19064   |    37500    |     16     |  1h 51m 9s |  0.8208665 |   0.81669563  |    0.8252668   | 0.90955627 |\r\n\r\nThe results are comparable to those reported in `config/v_0_2_0_benchmark/readme.md`.\r\n\r\nThe models where the difference in accuracy was more than 1% are: SSTC single telescope, SST1 array, SSTA array, SSTC array. In all those cases the difference in accuracy was below 2%.\r\n\r\nThe difference in AUROC was below 0.01 in magnitude for all models except SSTA array, where it was 0.01062685.\r\n\r\nThe difference in train times is more significant, but this can be easily explained by the use of different machines."},"comment":{"url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/comments/423649016","html_url":"https://github.com/ctlearn-project/ctlearn/issues/51#issuecomment-423649016","issue_url":"https://api.github.com/repos/ctlearn-project/ctlearn/issues/51","id":423649016,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzY0OTAxNg==","user":{"login":"aribrill","id":19699101,"node_id":"MDQ6VXNlcjE5Njk5MTAx","avatar_url":"https://avatars1.githubusercontent.com/u/19699101?v=4","gravatar_id":"","url":"https://api.github.com/users/aribrill","html_url":"https://github.com/aribrill","followers_url":"https://api.github.com/users/aribrill/followers","following_url":"https://api.github.com/users/aribrill/following{/other_user}","gists_url":"https://api.github.com/users/aribrill/gists{/gist_id}","starred_url":"https://api.github.com/users/aribrill/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aribrill/subscriptions","organizations_url":"https://api.github.com/users/aribrill/orgs","repos_url":"https://api.github.com/users/aribrill/repos","events_url":"https://api.github.com/users/aribrill/events{/privacy}","received_events_url":"https://api.github.com/users/aribrill/received_events","type":"User","site_admin":false},"created_at":"2018-09-21T19:37:20Z","updated_at":"2018-09-21T19:37:20Z","author_association":"COLLABORATOR","body":"Differences of 1% could be significant. To identify the cause, the next step should be to rerun the benchmarks with settings to eliminate any stochasticity: turn off shuffling of the TensorFlow dataset and use a specified seed to initialize the network. If significant differences are still observed, there could be a problem, otherwise it's just normal variance."}}
{"actor":{"display_login":"aribrill"}}