bonferroni quantiles tibshirani voxels boole quantile positives resampling estimator anova tukey manova multivariate biometrical westfall neuroimage m_ 9445 fmri experimentale grbd bayesian 10.1198 plackett randomization ancova scheffé factorial 00001648 nonparametric
when do you need a multiple comparison test
Multiple comparisons problem - wikipedia Multiple comparisons problem Jump to : navigation , search This article needs additional citations for verification . Please help improve this article by adding citations to reliable sources . Unsourced material may be challenged and removed . ( June 2016 ) ( Learn how and when to remove this template message ) An example of data produced by data dredging , apparently showing a close link between the letters in the winning word used in a spelling bee competition and the number of people in the United States killed by venomous spiders . The clear similarity in trends is a coincidence . If many data series are compared , similarly convincing but coincidental data may be obtained . In statistics , the multiple comparisons , multiplicity or multiple testing problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values . In certain fields it is known as the look - elsewhere effect . The more inferences are made , the more likely erroneous inferences are to occur . Several statistical techniques have been developed to prevent this from happening , allowing significance levels for single and multiple comparisons to be directly compared . These techniques generally require a stricter significance threshold for individual comparisons , so as to compensate for the number of inferences being made . Contents ( hide ) 1 History 2 Definition 2.1 Classification of multiple hypothesis tests 3 Controlling procedures 4 Large - scale multiple testing 4.1 Assessing whether any alternative hypotheses are true 5 See also 6 References 7 Further reading History ( edit ) The interest in the problem of multiple comparisons began in the 1950s with the work of Tukey and Scheffé . Other methods , such as the closed testing procedure ( Marcus et al. , 1976 ) and the Holm -- Bonferroni method ( 1979 ) , later emerged . In 1995 , work on the false discovery rate began . In 1996 , the first conference on multiple comparisons took place in Israel . This was followed by conferences around the world , usually taking place about every two years . Definition ( edit ) This section does not cite any sources . Please help improve this section by adding citations to reliable sources . Unsourced material may be challenged and removed . ( June 2016 ) ( Learn how and when to remove this template message ) Multiple comparisons arise when a statistical analysis involves multiple statistical tests , each of which has a potential to produce a `` discovery . '' Failure to compensate for multiple comparisons can have important real - world consequences , as illustrated by the following examples : Suppose the treatment is a new way of teaching writing to students , and the control is the standard way of teaching writing . Students in the two groups can be compared in terms of grammar , spelling , organization , content , and so on . As more attributes are compared , it becomes increasingly likely that the treatment and control groups will appear to differ on at least one attribute due to random sampling error alone . Suppose we consider the efficacy of a drug in terms of the reduction of any one of a number of disease symptoms . As more symptoms are considered , it becomes increasingly likely that the drug will appear to be an improvement over existing drugs in terms of at least one symptom . In both examples , as the number of comparisons increases , it becomes more likely that the groups being compared will appear to differ in terms of at least one attribute . Our confidence that a result will generalize to independent data should generally be weaker if it is observed as part of an analysis that involves multiple comparisons , rather than an analysis that involves only a single comparison . For example , if one test is performed at the 5 % level and the corresponding null hypothesis is true , there is only a 5 % chance of incorrectly rejecting the null hypothesis . However , if 100 tests are conducted and all corresponding null hypotheses are true , the expected number of incorrect rejections ( also known as false positives or Type I errors ) is 5 . If the tests are statistically independent from each other , the probability of at least one incorrect rejection is 99.4 % . The multiple comparisons problem also applies to confidence intervals . A single confidence interval with a 95 % coverage probability level will contain the population parameter in 95 % of experiments . However , if one considers 100 confidence intervals simultaneously , each with 95 % coverage probability , the expected number of non-covering intervals is 5 . If the intervals are statistically independent from each other , the probability that at least one interval does not contain the population parameter is 99.4 % . Techniques have been developed to prevent the inflation of false positive rates and non-coverage rates that occur with multiple statistical tests . Classification of multiple hypothesis tests ( edit ) The following table defines the possible outcomes when testing multiple null hypotheses . Suppose we have a number m of null hypotheses , denoted by : H , H , ... , H. Using a statistical test , we reject the null hypothesis if the test is declared significant . We do not reject the null hypothesis if the test is non-significant . Summing each type of outcome over all H yields the following random variables : Null hypothesis is true ( H ) Alternative hypothesis is true ( H ) Total Test is declared significant V ( \ displaystyle V ) S ( \ displaystyle S ) R ( \ displaystyle R ) Test is declared non-significant U ( \ displaystyle U ) T ( \ displaystyle T ) m − R ( \ displaystyle m-R ) Total m 0 ( \ displaystyle m_ ( 0 ) ) m − m 0 ( \ displaystyle m - m_ ( 0 ) ) m ( \ displaystyle m ) m ( \ displaystyle m ) is the total number hypotheses tested m 0 ( \ displaystyle m_ ( 0 ) ) is the number of true null hypotheses , an unknown parameter m − m 0 ( \ displaystyle m - m_ ( 0 ) ) is the number of true alternative hypotheses V ( \ displaystyle V ) is the number of false positives ( Type I error ) ( also called `` false discoveries '' ) S ( \ displaystyle S ) is the number of true positives ( also called `` true discoveries '' ) T ( \ displaystyle T ) is the number of false negatives ( Type II error ) U ( \ displaystyle U ) is the number of true negatives R = V + S ( \ displaystyle R = V + S ) is the number of rejected null hypotheses ( also called `` discoveries '' , either true or false ) In m ( \ displaystyle m ) hypothesis tests of which m 0 ( \ displaystyle m_ ( 0 ) ) are true null hypotheses , R ( \ displaystyle R ) is an observable random variable , and S ( \ displaystyle S ) , T ( \ displaystyle T ) , U ( \ displaystyle U ) , and V ( \ displaystyle V ) are unobservable random variables . Controlling procedures ( edit ) Further information : Family-wise error rate § Controlling procedures See also : False coverage rate § Controlling procedures , and False discovery rate § Controlling procedures If m independent comparisons are performed , the family-wise error rate ( FWER ) , is given by α _̄ = 1 − ( 1 − α ( per comparison ) ) m . ( \ displaystyle ( \ bar ( \ alpha ) ) = 1 - \ left ( 1 - \ alpha _ ( \ ( ( \ text ( per comparison ) ) \ ) ) \ right ) ^ ( m ) . ) Hence , unless the tests are perfectly positively dependent ( i.e. , identical ) , α _̄ ( \ displaystyle ( \ bar ( \ alpha ) ) ) increases as the number of comparisons increases . If we do not assume that the comparisons are independent , then we can still say : α _̄ ≤ m ⋅ α ( per comparison ) , ( \ displaystyle ( \ bar ( \ alpha ) ) \ leq m \ cdot \ alpha _ ( \ ( ( \ text ( per comparison ) ) \ ) ) , ) which follows from Boole 's inequality . Example : 0.2649 = 1 − ( 1 − . 05 ) 6 ≤ . 05 × 6 = 0.3 ( \ displaystyle 0.2649 = 1 - ( 1 -. 05 ) ^ ( 6 ) \ leq . 05 \ times 6 = 0.3 ) There are different ways to assure that the family-wise error rate is at most α _̄ ( \ displaystyle ( \ bar ( \ alpha ) ) ) . The most conservative method , which is free of dependence and distributional assumptions , is the Bonferroni correction α ( p e r c o m p a r i s o n ) = α _̄ / m ( \ displaystyle \ alpha _ ( \ mathrm ( \ ( per \ comparison \ ) ) ) = ( \ bar ( \ alpha ) ) / m ) . A marginally less conservative correction can be obtained by solving the equation for the family-wise error rate of m ( \ displaystyle m ) independent comparisons for α ( p e r c o m p a r i s o n ) ( \ displaystyle \ alpha _ ( \ mathrm ( \ ( per \ comparison \ ) ) ) ) . This yields α ( per comparison ) = 1 − ( 1 − α _̄ ) 1 / m ( \ displaystyle \ alpha _ ( \ ( ( \ text ( per comparison ) ) \ ) ) = 1 - ( ( 1 - ( \ bar ( \ alpha ) ) ) ) ^ ( 1 / m ) ) , which is known as the Šidák correction . Another procedure is the Holm -- Bonferroni method , which uniformly delivers more power than the simple Bonferroni correction , by testing only the lowest p - value ( i = 1 ( \ displaystyle i = 1 ) ) against the strictest criterion , and the higher p - values ( i > 1 ( \ displaystyle i > 1 ) ) against progressively less strict criteria . α ( p e r c o m p a r i s o n ) = α _̄ / ( m − i + 1 ) ( \ displaystyle \ alpha _ ( \ mathrm ( \ ( per \ comparison \ ) ) ) = ( \ bar ( \ alpha ) ) / ( m - i + 1 ) ) . This article or section may need to be cleaned up . It has been merged from Multiple testing correction . Multiple testing correction refers to re-calculating probabilities obtained from a statistical test which was repeated multiple times . In order to retain a prescribed family-wise error rate α in an analysis involving more than one comparison , the error rate for each comparison must be more stringent than α . Boole 's inequality implies that if each of m tests is performed to have type I error rate α / m , the total error rate will not exceed α . This is called the Bonferroni correction , and is one of the most commonly used approaches for multiple comparisons . In some situations , the Bonferroni correction is substantially conservative , i.e. , the actual family-wise error rate is much less than the prescribed level α . This occurs when the test statistics are highly dependent ( in the extreme case where the tests are perfectly dependent , the family-wise error rate with no multiple comparisons adjustment and the per - test error rates are identical ) . For example , in fMRI analysis , tests are done on over 100,000 voxels in the brain . The Bonferroni method would require p - values to be smaller than . 05 / 100000 to declare significance . Since adjacent voxels tend to be highly correlated , this threshold is generally too stringent . Because simple techniques such as the Bonferroni method can be conservative , there has been a great deal of attention paid to developing better techniques , such that the overall rate of false positives can be maintained without excessively inflating the rate of false negatives . Such methods can be divided into general categories : Methods where total alpha can be proved to never exceed 0.05 ( or some other chosen value ) under any conditions . These methods provide `` strong '' control against Type I error , in all conditions including a partially correct null hypothesis . Methods where total alpha can be proved not to exceed 0.05 except under certain defined conditions . Methods which rely on an omnibus test before proceeding to multiple comparisons . Typically these methods require a significant ANOVA , MANOVA , or Tukey 's range test . These methods generally provide only `` weak '' control of Type I error , except for certain numbers of hypotheses . Empirical methods , which control the proportion of Type I errors adaptively , utilizing correlation and distribution characteristics of the observed data . The advent of computerized resampling methods , such as bootstrapping and Monte Carlo simulations , has given rise to many techniques in the latter category . In some cases where exhaustive permutation resampling is performed , these tests provide exact , strong control of Type I error rates ; in other cases , such as bootstrap sampling , they provide only approximate control . Large - scale multiple testing ( edit ) Traditional methods for multiple comparisons adjustments focus on correcting for modest numbers of comparisons , often in an analysis of variance . A different set of techniques have been developed for `` large - scale multiple testing '' , in which thousands or even greater numbers of tests are performed . For example , in genomics , when using technologies such as microarrays , expression levels of tens of thousands of genes can be measured , and genotypes for millions of genetic markers can be measured . Particularly in the field of genetic association studies , there has been a serious problem with non-replication -- a result being strongly statistically significant in one study but failing to be replicated in a follow - up study . Such non-replication can have many causes , but it is widely considered that failure to fully account for the consequences of making multiple comparisons is one of the causes . In different branches of science , multiple testing is handled in different ways . It has been argued that if statistical tests are only performed when there is a strong basis for expecting the result to be true , multiple comparisons adjustments are not necessary . It has also been argued that use of multiple testing corrections is an inefficient way to perform empirical research , since multiple testing adjustments control false positives at the potential expense of many more false negatives . On the other hand , it has been argued that advances in measurement and information technology have made it far easier to generate large datasets for exploratory analysis , often leading to the testing of large numbers of hypotheses with no prior basis for expecting many of the hypotheses to be true . In this situation , very high false positive rates are expected unless multiple comparisons adjustments are made . For large - scale testing problems where the goal is to provide definitive results , the familywise error rate remains the most accepted parameter for ascribing significance levels to statistical tests . Alternatively , if a study is viewed as exploratory , or if significant results can be easily re-tested in an independent study , control of the false discovery rate ( FDR ) is often preferred . The FDR , loosely defined as the expected proportion of false positives among all significant tests , allows researchers to identify a set of `` candidate positives '' that can be more rigorously evaluated in a follow - up study . The practice of trying many unadjusted comparisons in the hope of finding a significant one is a known problem , whether applied unintentionally or deliberately , is sometimes called `` p - hacking . '' Assessing whether any alternative hypotheses are true ( edit ) A normal quantile plot for a simulated set of test statistics that have been standardized to be Z - scores under the null hypothesis . The departure of the upper tail of the distribution from the expected trend along the diagonal is due to the presence of substantially more large test statistic values than would be expected if all null hypotheses were true . The red point corresponds to the fourth largest observed test statistic , which is 3.13 , versus an expected value of 2.06 . The blue point corresponds to the fifth smallest test statistic , which is - 1.75 , versus an expected value of - 1.96 . The graph suggests that it is unlikely that all the null hypotheses are true , and that most or all instances of a true alternative hypothesis result from deviations in the positive direction . A basic question faced at the outset of analyzing a large set of testing results is whether there is evidence that any of the alternative hypotheses are true . One simple meta - test that can be applied when it is assumed that the tests are independent of each other is to use the Poisson distribution as a model for the number of significant results at a given level α that would be found when all null hypotheses are true . If the observed number of positives is substantially greater than what should be expected , this suggests that there are likely to be some true positives among the significant results . For example , if 1000 independent tests are performed , each at level α = 0.05 , we expect 50 significant tests to occur when all null hypotheses are true . Based on the Poisson distribution with mean 50 , the probability of observing more than 61 significant tests is less than 0.05 , so if more than 61 significant results are observed , it is very likely that some of them correspond to situations where the alternative hypothesis holds . A drawback of this approach is that it over-states the evidence that some of the alternative hypotheses are true when the test statistics are positively correlated , which commonly occurs in practice. . On the other hand , the approach remains valid even in the presence of correlation among the test statistics , as long as the Poisson distribution can be shown to provide a good approximation for the number of significant results . This scenario arises , for instance , when mining significant frequent itemsets from transactional datasets . Furthermore , a careful two stage analysis can bound the FDR at a pre-specified level . Another common approach that can be used in situations where the test statistics can be standardized to Z - scores is to make a normal quantile plot of the test statistics . If the observed quantiles are markedly more dispersed than the normal quantiles , this suggests that some of the significant results may be true positives . See also ( edit ) Key concepts Familywise error rate False positive rate False discovery rate ( FDR ) False coverage rate ( FCR ) Interval estimation Post-hoc analysis Experimentwise error rate General methods of alpha adjustment for multiple comparisons Closed testing procedure Bonferroni correction Boole -- Bonferroni bound Holm -- Bonferroni method Related concepts Testing hypotheses suggested by the data Texas sharpshooter fallacy Model selection References ( edit ) Jump up ^ Miller , R.G. ( 1981 ) . Simultaneous Statistical Inference 2nd Ed . Springer Verlag New York . ISBN 0 - 387 - 90548 - 0 . Jump up ^ Benjamini , Y. ( 2010 ) . `` Simultaneous and selective inference : Current successes and future challenges '' . Biometrical Journal . 52 ( 6 ) : 708 -- 721 . PMID 21154895 . doi : 10.1002 / bimj. 200900299 . Jump up ^ ( 1 ) Jump up ^ Aickin , M ; Gensler , H ( May 1996 ) . `` Adjusting for multiple testing when reporting research results : the Bonferroni vs Holm methods '' . Am J Public Health . 86 ( 5 ) : 726 -- 728 . PMC 1380484 . PMID 8629727 . doi : 10.2105 / ajph. 86.5. 726 . Jump up ^ Logan , B.R. ; Rowe , D.B. ( 2004 ) . `` An evaluation of thresholding techniques in fMRI analysis '' . NeuroImage. 22 ( 1 ) : 95 -- 108 . PMID 15110000 . doi : 10.1016 / j. neuroimage. 2003.12. 047 . Jump up ^ Logan , B.R. ; Geliazkova , M.P. ; Rowe , D.B. ( 2008 ) . `` An evaluation of spatial thresholding techniques in fMRI analysis '' . Human Brain Mapping. 29 ( 12 ) : 1379 -- 1389 . PMID 18064589 . doi : 10.1002 / hbm. 20471 . Jump up ^ Qu , Hui - Qi ; Tien , Matthew ; Polychronakos , Constantin ( 2010 - 10 - 01 ) . `` Statistical significance in genetic association studies '' . Clinical and Investigative Medicine . Medecine Clinique et Experimentale. 33 ( 5 ) : E266 -- E270 . ISSN 0147 - 958X . PMC 3270946 . PMID 20926032 . Jump up ^ Rothman , Kenneth J. ( 1990 ) . `` No Adjustments Are Needed for Multiple Comparisons '' . Epidemiology . Lippincott Williams & Wilkins. 1 ( 1 ) : 43 -- 46 . JSTOR 20065622 . PMID 2081237 . doi : 10.1097 / 00001648 - 199001000 - 00010 . Jump up ^ Benjamini , Yoav ; Hochberg , Yosef ( 1995 ) . `` Controlling the false discovery rate : a practical and powerful approach to multiple testing '' . Journal of the Royal Statistical Society , Series B. 57 ( 1 ) : 125 -- 133 . JSTOR 2346101 . Jump up ^ Storey , JD ; Tibshirani , Robert ( 2003 ) . `` Statistical significance for genome - wide studies '' . PNAS. 100 ( 16 ) : 9440 -- 9445 . JSTOR 3144228 . PMC 170937 . PMID 12883005 . doi : 10.1073 / pnas. 1530509100 . Jump up ^ Efron , Bradley ; Tibshirani , Robert ; Storey , John D. ; Tusher , Virginia ( 2001 ) . `` Empirical Bayes analysis of a microarray experiment '' . Journal of the American Statistical Association . 96 ( 456 ) : 1151 -- 1160 . JSTOR 3085878 . doi : 10.1198 / 016214501753382129 . Jump up ^ Noble , William S. ( 2009 - 12 - 01 ) . `` How does multiple testing correction work ? '' . Nature Biotechnology . 27 ( 12 ) : 1135 -- 1137 . ISSN 1087 - 0156 . PMC 2907892 . PMID 20010596 . doi : 10.1038 / nbt1209 - 1135 . Jump up ^ Young , S.S. , Karr , A. ( 2011 ) . `` Deming , data and observational studies '' ( PDF ) . Significance. 8 ( 3 ) . CS1 maint : Multiple names : authors list ( link ) Jump up ^ Smith , G.D. , Shah , E. ( 2002 ) . `` Data dredging , bias , or confounding '' . BMJ. 325 ( 7378 ) : 1437 -- 1438 . PMC 1124898 . PMID 12493654 . doi : 10.1136 / bmj. 325.7378. 1437 . CS1 maint : Multiple names : authors list ( link ) Jump up ^ Kirsch , A ; Mitzenmacher , M ; Pietracaprina , A ; Pucci , G ; Upfal , E ; Vandin , F ( June 2012 ) . `` An Efficient Rigorous Approach for Identifying Statistically Significant Frequent Itemsets '' . Journal of the ACM. 59 ( 3 ) : 12 : 1 -- 12 : 22 . doi : 10.1145 / 2220357.2220359 . Further reading ( edit ) F. Betz , T. Hothorn , P. Westfall ( 2010 ) , Multiple Comparisons Using R , CRC Press S. Dudoit and M.J. van der Laan ( 2008 ) , Multiple Testing Procedures with Application to Genomics , Springer B. Phipson and G.K. Smyth ( 2010 ) , Permutation P - values Should Never Be Zero : Calculating Exact P - values when Permutations are Randomly Drawn , Statistical Applications in Genetics and Molecular Biology Vol ... 9 Iss. 1 , Article 39 , doi : 10.2202 / 1544 - 6155.1585 P.H. Westfall and S.S. Young ( 1993 ) , Resampling - based Multiple Testing : Examples and Methods for p - Value Adjustment , Wiley P. Westfall , R. Tobias , R. Wolfinger ( 2011 ) Multiple comparisons and multiple testing using SAS , 2nd edn , SAS Institute A gallery of examples of implausible correlations sourced by data dredging <Th_colspan="2"> Design of experiments Scientific method Scientific experiment Statistical design Control Internal and external validity Experimental unit Blinding Optimal design : Bayesian Random assignment Randomization Restricted randomization Replication versus subsampling Sample size Treatment and blocking Treatment Effect size Contrast Interaction Confounding Orthogonality Blocking Covariate Nuisance variable Models and inference Linear regression Ordinary least squares Bayesian Random effect Mixed model Hierarchical model : Bayesian Analysis of variance ( Anova ) Cochran 's theorem Manova ( multivariate ) Ancova ( covariance ) Compare means Multiple comparison Designs Completely randomized Factorial Fractional factorial Plackett - Burman Taguchi Response surface methodology Polynomial and rational modeling Box - Behnken Central composite Block Generalized randomized block design ( GRBD ) Latin square Graeco - Latin square Orthogonal array Latin hypercube Repeated measures design Crossover study Randomized controlled trial Sequential analysis Sequential probability ratio test <Td_colspan="2"> Glossary Category Statistics portal Statistical outline Statistical topics <Th_colspan="2"> ( hide ) Statistics <Td_colspan="2"> Outline Index <Td_colspan="2"> <Th_colspan="2"> Descriptive statistics <Td_colspan="2"> Continuous data Center Mean arithmetic geometric harmonic Median Mode Dispersion Variance Standard deviation Coefficient of variation Percentile Range Interquartile range Shape Central limit theorem Moments Skewness Kurtosis L - moments Count data Index of dispersion Summary tables Grouped data Frequency distribution Contingency table Dependence Pearson product - moment correlation Rank correlation Spearman 's rho Kendall 's tau Partial correlation Scatter plot Graphics Bar chart Biplot Box plot Control chart Correlogram Fan chart Forest plot Histogram Pie chart Q -- Q plot Run chart Scatter plot Stem - and - leaf display Radar chart <Td_colspan="2"> <Th_colspan="2"> Data collection <Td_colspan="2"> Study design Population Statistic Effect size Statistical power Sample size determination Missing data Survey methodology Sampling stratified cluster Standard error Opinion poll Questionnaire Controlled experiments Design control optimal Controlled trial Randomized Random assignment Replication Blocking Interaction Factorial experiment Uncontrolled studies Observational study Natural experiment Quasi-experiment <Td_colspan="2"> <Th_colspan="2"> Statistical inference <Td_colspan="2"> Statistical theory Population Statistic Probability distribution Sampling distribution Order statistic Empirical distribution Density estimation Statistical model L space Parameter location scale shape Parametric family Likelihood ( monotone ) Location -- scale family Exponential family Completeness Sufficiency Statistical functional Bootstrap U V Optimal decision loss function Efficiency Statistical distance divergence Asymptotics Robustness Frequentist inference Point estimation Estimating equations Maximum likelihood Method of moments M - estimator Minimum distance Unbiased estimators Mean - unbiased minimum - variance Rao -- Blackwellization Lehmann -- Scheffé theorem Median unbiased Plug - in Interval estimation Confidence interval Pivot Likelihood interval Prediction interval Tolerance interval Resampling Bootstrap Jackknife Testing hypotheses 1 - & 2 - tails Power Uniformly most powerful test Permutation test Randomization test Multiple comparisons Parametric tests Likelihood - ratio Wald Score Specific tests <Td_colspan="2"> Z ( normal ) Student 's t - test Goodness of fit Chi - squared Kolmogorov -- Smirnov Anderson -- Darling Lilliefors Jarque -- Bera Normality ( Shapiro -- Wilk ) Likelihood - ratio test Model selection Cross validation AIC BIC Rank statistics Sign Sample median Signed rank ( Wilcoxon ) Hodges -- Lehmann estimator Rank sum ( Mann -- Whitney ) Nonparametric anova 1 - way ( Kruskal -- Wallis ) 2 - way ( Friedman ) Ordered alternative ( Jonckheere -- Terpstra ) Bayesian inference Bayesian probability prior posterior Credible interval Bayes factor Bayesian estimator Maximum posterior estimator <Td_colspan="2"> <Th_colspan="2"> Correlation Regression analysis <Td_colspan="2"> Correlation Pearson product - moment Partial correlation Confounding variable Coefficient of determination Regression analysis Errors and residuals Regression model validation Mixed effects models Simultaneous equations models Multivariate adaptive regression splines ( MARS ) Linear regression Simple linear regression Ordinary least squares General linear model Bayesian regression Non-standard predictors Nonlinear regression Nonparametric Semiparametric Isotonic Robust Heteroscedasticity Homoscedasticity Generalized linear model Exponential families Logistic ( Bernoulli ) / Binomial / Poisson regressions Partition of variance Analysis of variance ( ANOVA , anova ) Analysis of covariance Multivariate ANOVA Degrees of freedom <Td_colspan="2"> <Th_colspan="2"> Categorical / Multivariate / Time - series / Survival analysis <Td_colspan="2"> Categorical Cohen 's kappa Contingency table Graphical model Log - linear model McNemar 's test Multivariate Regression Manova Principal components Canonical correlation Discriminant analysis Cluster analysis Classification Structural equation model Factor analysis Multivariate distributions Elliptical distributions Normal Time - series General Decomposition Trend Stationarity Seasonal adjustment Exponential smoothing Cointegration Structural break Granger causality Specific tests Dickey -- Fuller Johansen Q - statistic ( Ljung -- Box ) Durbin -- Watson Breusch -- Godfrey Time domain Autocorrelation ( ACF ) partial ( PACF ) Cross-correlation ( XCF ) ARMA model ARIMA model ( Box -- Jenkins ) Autoregressive conditional heteroskedasticity ( ARCH ) Vector autoregression ( VAR ) Frequency domain Spectral density estimation Fourier analysis Wavelet Survival Survival function Kaplan -- Meier estimator ( product limit ) Proportional hazards models Accelerated failure time ( AFT ) model First hitting time Hazard function Nelson -- Aalen estimator Test Log - rank test <Td_colspan="2"> <Th_colspan="2"> Applications <Td_colspan="2"> Biostatistics Bioinformatics Clinical trials / studies Epidemiology Medical statistics Engineering statistics Chemometrics Methods engineering Probabilistic design Process / quality control Reliability System identification Social statistics Actuarial science Census Crime statistics Demography Econometrics National accounts Official statistics Population statistics Psychometrics Spatial statistics Cartography Environmental statistics Geographic information system Geostatistics Kriging <Td_colspan="2"> Category Portal Commons WikiProject Retrieved from `` https://en.wikipedia.org/w/index.php?title=Multiple_comparisons_problem&oldid=799705209 '' Categories : Statistical hypothesis testing Multiple comparisons Hidden categories : CS1 maint : Multiple names : authors list Articles needing additional references from June 2016 All articles needing additional references All articles with unsourced statements Articles with unsourced statements from June 2016 Articles with unsourced statements from August 2012 Articles with unsourced statements from January 2012 Talk Contents Interaction About Wikipedia Deutsch 한국어 עברית Nederlands Polski Русский Svenska Edit links This page was last edited on 9 September 2017 , at 10 : 12 . Text is available under the Creative Commons Attribution - ShareAlike License ; additional terms may apply . By using this site , you agree to the Terms of Use and Privacy Policy . Wikipedia ® is a registered trademark of the Wikimedia Foundation , Inc. , a non-profit organization . About Wikipedia