{"action":"created","comment":{"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/comments/242335616","pull_request_review_id":185820179,"id":242335616,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDI0MjMzNTYxNg==","diff_hunk":"@@ -1 +1,18 @@\n ## Rule 4: Know your data and your question\n+\n+In the era of easily accessible datasets, one sometimes starts analyzing data without a good understanding of the study design, namely why the data were collected and how. \n+Having appropriate meta-data, a comprehensive data dictionary, and even the actual data collection protocol is essential for any analysis, including one that involves deep learning. \n+It's even better to also have access to a subject matter expert who has either collected or analyzed this type of data before! \n+For example, if the main reason why the data were collected was to test the impact of an intervention, it may be the case that a randomized controlled trial was performed.\n+However, ethical or other study considerations may make this impossible or impractical, in which case the design may have been an observational one, either prospective or retrospective.\n+These designs may also incorporate some amount of matching - for example, cases and controls may be selected so that the age range or weight distribution is similar.\n+All of these different designs have different assumptions and caveats, which cannot be ignored during a data analysis.\n+Many datasets are now passively collected or do not have a specific design, but even in this case it is important to know how individuals or samples were treated (for example, if all samples are from the same study site, if certain ethnic groups or zip codes are oversampled, if there are differences in processing dates or techniques.)\n+Systematic biases can lead to artifacts or \"batch effects,\" which mean that instead of finding correlates with an outcome or grouping of interest, the investigator may find correlates with variables that are not of interest and obtain misleading results [@doi:10.1038/nrg2825].\n+Other study design considerations that should not be overlooked include knowing whether a study involves biological or technical replicates or both.\n+For example, are some samples collected from the same individuals at different time points? Are those time points before and after some treatment?\n+If one assumes that all the samples are independent but that is in fact not the case, a variety of issues may arise, including having a lower effective sample size than expected.\n+In general, deep learning has an increased tendency for overfitting, compared to classical methods, due to the large number of parameters being estimated, making issues of adequate sample size even more important (see [Rule 7](https://github.com/Benjamin-Lee/deep-rules/blob/master/content/09.overfitting.md)).","path":"content/06.know-your-problem.md","position":15,"original_position":15,"commit_id":"edb86a7f9719192c9517b23f4e5556353e25dd17","original_commit_id":"edb86a7f9719192c9517b23f4e5556353e25dd17","user":{"login":"agitter","id":3532898,"node_id":"MDQ6VXNlcjM1MzI4OTg=","avatar_url":"https://avatars0.githubusercontent.com/u/3532898?v=4","gravatar_id":"","url":"https://api.github.com/users/agitter","html_url":"https://github.com/agitter","followers_url":"https://api.github.com/users/agitter/followers","following_url":"https://api.github.com/users/agitter/following{/other_user}","gists_url":"https://api.github.com/users/agitter/gists{/gist_id}","starred_url":"https://api.github.com/users/agitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/agitter/subscriptions","organizations_url":"https://api.github.com/users/agitter/orgs","repos_url":"https://api.github.com/users/agitter/repos","events_url":"https://api.github.com/users/agitter/events{/privacy}","received_events_url":"https://api.github.com/users/agitter/received_events","type":"User","site_admin":false},"body":"@dhimmel any ideas on how to do automatic numbering here?  Each rule is currently in it's own md file, but we would want to link to the manuscript section, not the md file.  [Section labels](http://lierdakil.github.io/pandoc-crossref/#section-labels) may be close to what we want.","created_at":"2018-12-17T22:11:22Z","updated_at":"2018-12-17T22:15:04Z","html_url":"https://github.com/Benjamin-Lee/deep-rules/pull/90#discussion_r242335616","pull_request_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/comments/242335616"},"html":{"href":"https://github.com/Benjamin-Lee/deep-rules/pull/90#discussion_r242335616"},"pull_request":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90"}}},"pull_request":{"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90","id":237547847,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM3NTQ3ODQ3","html_url":"https://github.com/Benjamin-Lee/deep-rules/pull/90","diff_url":"https://github.com/Benjamin-Lee/deep-rules/pull/90.diff","patch_url":"https://github.com/Benjamin-Lee/deep-rules/pull/90.patch","issue_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/90","number":90,"state":"open","locked":false,"title":"Started initial draft of Rule 4","user":{"login":"SiminaB","id":1932768,"node_id":"MDQ6VXNlcjE5MzI3Njg=","avatar_url":"https://avatars3.githubusercontent.com/u/1932768?v=4","gravatar_id":"","url":"https://api.github.com/users/SiminaB","html_url":"https://github.com/SiminaB","followers_url":"https://api.github.com/users/SiminaB/followers","following_url":"https://api.github.com/users/SiminaB/following{/other_user}","gists_url":"https://api.github.com/users/SiminaB/gists{/gist_id}","starred_url":"https://api.github.com/users/SiminaB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SiminaB/subscriptions","organizations_url":"https://api.github.com/users/SiminaB/orgs","repos_url":"https://api.github.com/users/SiminaB/repos","events_url":"https://api.github.com/users/SiminaB/events{/privacy}","received_events_url":"https://api.github.com/users/SiminaB/received_events","type":"User","site_admin":false},"body":"To escape the tyranny of the blank page","created_at":"2018-12-11T04:31:58Z","updated_at":"2018-12-17T22:15:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bfc2a4a2d57c8136c720ca40c5807a12378cb03c","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90/commits","review_comments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90/comments","review_comment_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/90/comments","statuses_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/statuses/edb86a7f9719192c9517b23f4e5556353e25dd17","head":{"label":"Benjamin-Lee:SiminaB-rule4","ref":"SiminaB-rule4","sha":"edb86a7f9719192c9517b23f4e5556353e25dd17","user":{"login":"Benjamin-Lee","id":13618205,"node_id":"MDQ6VXNlcjEzNjE4MjA1","avatar_url":"https://avatars2.githubusercontent.com/u/13618205?v=4","gravatar_id":"","url":"https://api.github.com/users/Benjamin-Lee","html_url":"https://github.com/Benjamin-Lee","followers_url":"https://api.github.com/users/Benjamin-Lee/followers","following_url":"https://api.github.com/users/Benjamin-Lee/following{/other_user}","gists_url":"https://api.github.com/users/Benjamin-Lee/gists{/gist_id}","starred_url":"https://api.github.com/users/Benjamin-Lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Benjamin-Lee/subscriptions","organizations_url":"https://api.github.com/users/Benjamin-Lee/orgs","repos_url":"https://api.github.com/users/Benjamin-Lee/repos","events_url":"https://api.github.com/users/Benjamin-Lee/events{/privacy}","received_events_url":"https://api.github.com/users/Benjamin-Lee/received_events","type":"User","site_admin":false},"repo":{"id":152173421,"node_id":"MDEwOlJlcG9zaXRvcnkxNTIxNzM0MjE=","name":"deep-rules","full_name":"Benjamin-Lee/deep-rules","private":false,"owner":{"login":"Benjamin-Lee","id":13618205,"node_id":"MDQ6VXNlcjEzNjE4MjA1","avatar_url":"https://avatars2.githubusercontent.com/u/13618205?v=4","gravatar_id":"","url":"https://api.github.com/users/Benjamin-Lee","html_url":"https://github.com/Benjamin-Lee","followers_url":"https://api.github.com/users/Benjamin-Lee/followers","following_url":"https://api.github.com/users/Benjamin-Lee/following{/other_user}","gists_url":"https://api.github.com/users/Benjamin-Lee/gists{/gist_id}","starred_url":"https://api.github.com/users/Benjamin-Lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Benjamin-Lee/subscriptions","organizations_url":"https://api.github.com/users/Benjamin-Lee/orgs","repos_url":"https://api.github.com/users/Benjamin-Lee/repos","events_url":"https://api.github.com/users/Benjamin-Lee/events{/privacy}","received_events_url":"https://api.github.com/users/Benjamin-Lee/received_events","type":"User","site_admin":false},"html_url":"https://github.com/Benjamin-Lee/deep-rules","description":"Ten Simple Rules for Deep Learning in Biology","fork":false,"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules","forks_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/forks","keys_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/teams","hooks_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/hooks","issue_events_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/events{/number}","events_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/events","assignees_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/assignees{/user}","branches_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/branches{/branch}","tags_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/tags","blobs_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/refs{/sha}","trees_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/statuses/{sha}","languages_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/languages","stargazers_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/stargazers","contributors_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/contributors","subscribers_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/subscribers","subscription_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/subscription","commits_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/commits{/sha}","git_commits_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/commits{/sha}","comments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/comments{/number}","issue_comment_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/comments{/number}","contents_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/contents/{+path}","compare_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/merges","archive_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/downloads","issues_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues{/number}","pulls_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls{/number}","milestones_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/milestones{/number}","notifications_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/labels{/name}","releases_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/releases{/id}","deployments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/deployments","created_at":"2018-10-09T02:10:24Z","updated_at":"2018-12-17T17:14:39Z","pushed_at":"2018-12-15T17:47:20Z","git_url":"git://github.com/Benjamin-Lee/deep-rules.git","ssh_url":"git@github.com:Benjamin-Lee/deep-rules.git","clone_url":"https://github.com/Benjamin-Lee/deep-rules.git","svn_url":"https://github.com/Benjamin-Lee/deep-rules","homepage":"https://benjamin-lee.github.io/deep-rules/","size":2011,"stargazers_count":126,"watchers_count":126,"language":"CSS","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":15,"mirror_url":null,"archived":false,"open_issues_count":51,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":15,"open_issues":51,"watchers":126,"default_branch":"master"}},"base":{"label":"Benjamin-Lee:master","ref":"master","sha":"a899a0d53a4604a4377fdde30f52bf3f2aaaa8dd","user":{"login":"Benjamin-Lee","id":13618205,"node_id":"MDQ6VXNlcjEzNjE4MjA1","avatar_url":"https://avatars2.githubusercontent.com/u/13618205?v=4","gravatar_id":"","url":"https://api.github.com/users/Benjamin-Lee","html_url":"https://github.com/Benjamin-Lee","followers_url":"https://api.github.com/users/Benjamin-Lee/followers","following_url":"https://api.github.com/users/Benjamin-Lee/following{/other_user}","gists_url":"https://api.github.com/users/Benjamin-Lee/gists{/gist_id}","starred_url":"https://api.github.com/users/Benjamin-Lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Benjamin-Lee/subscriptions","organizations_url":"https://api.github.com/users/Benjamin-Lee/orgs","repos_url":"https://api.github.com/users/Benjamin-Lee/repos","events_url":"https://api.github.com/users/Benjamin-Lee/events{/privacy}","received_events_url":"https://api.github.com/users/Benjamin-Lee/received_events","type":"User","site_admin":false},"repo":{"id":152173421,"node_id":"MDEwOlJlcG9zaXRvcnkxNTIxNzM0MjE=","name":"deep-rules","full_name":"Benjamin-Lee/deep-rules","private":false,"owner":{"login":"Benjamin-Lee","id":13618205,"node_id":"MDQ6VXNlcjEzNjE4MjA1","avatar_url":"https://avatars2.githubusercontent.com/u/13618205?v=4","gravatar_id":"","url":"https://api.github.com/users/Benjamin-Lee","html_url":"https://github.com/Benjamin-Lee","followers_url":"https://api.github.com/users/Benjamin-Lee/followers","following_url":"https://api.github.com/users/Benjamin-Lee/following{/other_user}","gists_url":"https://api.github.com/users/Benjamin-Lee/gists{/gist_id}","starred_url":"https://api.github.com/users/Benjamin-Lee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Benjamin-Lee/subscriptions","organizations_url":"https://api.github.com/users/Benjamin-Lee/orgs","repos_url":"https://api.github.com/users/Benjamin-Lee/repos","events_url":"https://api.github.com/users/Benjamin-Lee/events{/privacy}","received_events_url":"https://api.github.com/users/Benjamin-Lee/received_events","type":"User","site_admin":false},"html_url":"https://github.com/Benjamin-Lee/deep-rules","description":"Ten Simple Rules for Deep Learning in Biology","fork":false,"url":"https://api.github.com/repos/Benjamin-Lee/deep-rules","forks_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/forks","keys_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/teams","hooks_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/hooks","issue_events_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/events{/number}","events_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/events","assignees_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/assignees{/user}","branches_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/branches{/branch}","tags_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/tags","blobs_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/refs{/sha}","trees_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/statuses/{sha}","languages_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/languages","stargazers_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/stargazers","contributors_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/contributors","subscribers_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/subscribers","subscription_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/subscription","commits_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/commits{/sha}","git_commits_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/git/commits{/sha}","comments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/comments{/number}","issue_comment_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/comments{/number}","contents_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/contents/{+path}","compare_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/merges","archive_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/downloads","issues_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues{/number}","pulls_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls{/number}","milestones_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/milestones{/number}","notifications_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/labels{/name}","releases_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/releases{/id}","deployments_url":"https://api.github.com/repos/Benjamin-Lee/deep-rules/deployments","created_at":"2018-10-09T02:10:24Z","updated_at":"2018-12-17T17:14:39Z","pushed_at":"2018-12-15T17:47:20Z","git_url":"git://github.com/Benjamin-Lee/deep-rules.git","ssh_url":"git@github.com:Benjamin-Lee/deep-rules.git","clone_url":"https://github.com/Benjamin-Lee/deep-rules.git","svn_url":"https://github.com/Benjamin-Lee/deep-rules","homepage":"https://benjamin-lee.github.io/deep-rules/","size":2011,"stargazers_count":126,"watchers_count":126,"language":"CSS","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":true,"forks_count":15,"mirror_url":null,"archived":false,"open_issues_count":51,"license":{"key":"other","name":"Other","spdx_id":"NOASSERTION","url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":15,"open_issues":51,"watchers":126,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90"},"html":{"href":"https://github.com/Benjamin-Lee/deep-rules/pull/90"},"issue":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/90"},"comments":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/issues/90/comments"},"review_comments":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90/comments"},"review_comment":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/pulls/90/commits"},"statuses":{"href":"https://api.github.com/repos/Benjamin-Lee/deep-rules/statuses/edb86a7f9719192c9517b23f4e5556353e25dd17"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"agitter"}}