{"action":"created","issue":{"url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/4","repository_url":"https://api.github.com/repos/DDTBOX/DDTBOX","labels_url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/4/comments","events_url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/4/events","html_url":"https://github.com/DDTBOX/DDTBOX/issues/4","id":143785535,"number":4,"title":"Multiple Comparisons Corrections","user":{"login":"Feugs","id":17695493,"avatar_url":"https://avatars.githubusercontent.com/u/17695493?v=3","gravatar_id":"","url":"https://api.github.com/users/Feugs","html_url":"https://github.com/Feugs","followers_url":"https://api.github.com/users/Feugs/followers","following_url":"https://api.github.com/users/Feugs/following{/other_user}","gists_url":"https://api.github.com/users/Feugs/gists{/gist_id}","starred_url":"https://api.github.com/users/Feugs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Feugs/subscriptions","organizations_url":"https://api.github.com/users/Feugs/orgs","repos_url":"https://api.github.com/users/Feugs/repos","events_url":"https://api.github.com/users/Feugs/events{/privacy}","received_events_url":"https://api.github.com/users/Feugs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-03-27T07:42:08Z","updated_at":"2016-04-02T18:54:21Z","closed_at":null,"body":"We could implement the following corrections for multiple comparisons in the toolbox:\r\n- Bonferroni\r\n- Bonferroni-Holm method \r\n- False Discovery Rate correction (Benjamini-Hochberg procedure)\r\n- Cluster-based correction based on permutation testing or bootstrapping\r\n\r\n**Cluster-based correction method**\r\nFor an introduction to the rationale behind cluster-based corrections see Maris & Oostenveld (2007) doi:10.1016/j.jneumeth.2007.03.024\r\n\r\nFor the cluster-based corrections I am thinking of implementing the approach below, and then using simulations to check if it controls the Type 1 error rate correctly. Please let me know if you have suggestions for improvements.\r\n\r\n1. Using the classifier performance data run t tests at each step and identify steps at which the classifier performance statistically significantly differs from zero. \r\n\r\n2. Clusters are defined as groups of adjacent steps that are statistically significant. For example, three consecutive steps that are statistically significant are defined as one cluster. Calculate the sum of the test statistics (e.g. absolute t values) for each cluster separately. This is known as the cluster mass.\r\n\r\n3. Using the permutation test data (mean permutation test classifier accuracy for each participant) generate a bootstrap sample for each step. Find the steps which are significantly different from zero and calculate the cluster mass for each cluster. Record the test statistic value of the largest cluster mass.\r\n\r\n4. Repeat step 3 B times (B = number of bootstrap samples) to generate a distribution of the maximum cluster mass statistic under the null hypothesis. \r\n\r\n5. Check which clusters derived from step 1 have cluster masses above the 95th percentile of the null hypothesis distribution (when using an alpha level of 0.05). These clusters are defined as statistically significant effects. Clusters that do not have a mass larger than the 95th percentile are not declared as statistically significant.\r\n\r\nWe can test the type 1 error rate using simulations in a similar way to Pernet et al. (2015)        doi:10.1016/j.jneumeth.2014.08.003 . We could also include these simulations in the publication."},"comment":{"url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/comments/204779148","html_url":"https://github.com/DDTBOX/DDTBOX/issues/4#issuecomment-204779148","issue_url":"https://api.github.com/repos/DDTBOX/DDTBOX/issues/4","id":204779148,"user":{"login":"palday","id":1677783,"avatar_url":"https://avatars.githubusercontent.com/u/1677783?v=3","gravatar_id":"","url":"https://api.github.com/users/palday","html_url":"https://github.com/palday","followers_url":"https://api.github.com/users/palday/followers","following_url":"https://api.github.com/users/palday/following{/other_user}","gists_url":"https://api.github.com/users/palday/gists{/gist_id}","starred_url":"https://api.github.com/users/palday/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/palday/subscriptions","organizations_url":"https://api.github.com/users/palday/orgs","repos_url":"https://api.github.com/users/palday/repos","events_url":"https://api.github.com/users/palday/events{/privacy}","received_events_url":"https://api.github.com/users/palday/received_events","type":"User","site_admin":false},"created_at":"2016-04-02T18:54:21Z","updated_at":"2016-04-02T18:54:21Z","body":"Another place to look for inspiration on these things is the Mass Univariate stuff ([Groppe et al 2011, Psychophysiology](dx.doi.org/10.1111/j.1469-8986.2011.01273.x)).  Although it initially seems to have little in common with our work, *Mass* univariate suffers from the same multiple testing and computational issues as MVPA -- you could almost view some types of MU as MVPA applied to a single sensor. If I recall correctly, Groppe had implemented his ideas in a similar way to our current MVPA setup -- in many ways dependent on EEGLAB but not a proper plugin, so we may be able to borrow adapt some of his code for our purposes. His software provides the following multiple-comparisons corrections (copied nearly directly from the abstract for the first paper in the two-part series): \r\n\r\n*  permutation-based control of the family-wise error rate (FWER)\r\n* weak control of FWER via cluster-based permutation tests\r\n* permutation-based control of the generalized FWER\r\n* three false discovery rate control procedures)"}}