{"action":"created","issue":{"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/17","repository_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules","labels_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/17/labels{/name}","comments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/17/comments","events_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/17/events","html_url":"https://github.com/Benjamin-Lee/deep-rules/issues/17","id":378431686,"node_id":"MDU6SXNzdWUzNzg0MzE2ODY=","number":17,"title":"If you don't have enough data, don't even start","user":{"login":"hugoaerts","id":13385526,"node_id":"MDQ6VXNlcjEzMzg1NTI2","avatar_url":"https://avatars1.githubusercontent.com/u/13385526?v=4","gravatar_id":"","url":"https://api.github.com/users/hugoaerts","html_url":"https://github.com/hugoaerts","followers_url":"https://api.github.com/users/hugoaerts/followers","following_url":"https://api.github.com/users/hugoaerts/following{/other_user}","gists_url":"https://api.github.com/users/hugoaerts/gists{/gist_id}","starred_url":"https://api.github.com/users/hugoaerts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hugoaerts/subscriptions","organizations_url":"https://api.github.com/users/hugoaerts/orgs","repos_url":"https://api.github.com/users/hugoaerts/repos","events_url":"https://api.github.com/users/hugoaerts/events{/privacy}","received_events_url":"https://api.github.com/users/hugoaerts/received_events","type":"User","site_admin":false},"labels":[{"id":1103338043,"node_id":"MDU6TGFiZWwxMTAzMzM4MDQz","url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/labels/rule","name":"rule","color":"6af7a7","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-11-07T19:18:41Z","updated_at":"2018-11-10T16:53:18Z","closed_at":null,"author_association":"COLLABORATOR","body":"This is, surprisingly, a very reoccurring problem. People spend a lot of time on analyzing small datasets, just as a proof-of-principle. However, the data are too small, no independent validation, etc to derive any meaningful results. Not enough data -> do something else  "},"comment":{"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/comments/437597936","html_url":"https://github.com/Benjamin-Lee/deep-rules/issues/17#issuecomment-437597936","issue_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/17","id":437597936,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNzU5NzkzNg==","user":{"login":"hugoaerts","id":13385526,"node_id":"MDQ6VXNlcjEzMzg1NTI2","avatar_url":"https://avatars1.githubusercontent.com/u/13385526?v=4","gravatar_id":"","url":"https://api.github.com/users/hugoaerts","html_url":"https://github.com/hugoaerts","followers_url":"https://api.github.com/users/hugoaerts/followers","following_url":"https://api.github.com/users/hugoaerts/following{/other_user}","gists_url":"https://api.github.com/users/hugoaerts/gists{/gist_id}","starred_url":"https://api.github.com/users/hugoaerts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hugoaerts/subscriptions","organizations_url":"https://api.github.com/users/hugoaerts/orgs","repos_url":"https://api.github.com/users/hugoaerts/repos","events_url":"https://api.github.com/users/hugoaerts/events{/privacy}","received_events_url":"https://api.github.com/users/hugoaerts/received_events","type":"User","site_admin":false},"created_at":"2018-11-10T16:53:18Z","updated_at":"2018-11-10T16:53:18Z","author_association":"COLLABORATOR","body":"Yes, I agree. But it is almost impossible to specify exactly what is 'enough' as this depends on several factors and will be different for almost every case. For example for training it depends on several aspects: complexity of network, which layers are retrained (transfer learning or not), your tuning set performance, difficulty of the classification/prediction problem, etc. For the testing dataset(s) this often comes down to statistics and power calculations: what is the expected effect size, event ratio of all classes, balance between training/tuning/test datasets, etc. \r\n\r\nAs we can't give a hard number, I think we can address the following: a lot of investigators know before they even start that their datasets are too small. They then say this is an \"exploratory analysis\" or \"proof-of-principle\" and even get these published in decent journals. However, these investigations are often worthless and don't add anything useful to the field. When confronted with such a problem, I often advise people to continue to look for another problem/dataset. "}}
{"actor":{"display_login":"hugoaerts"}}