secondary data analysis is analysis of data that was collected by someone else for another primary purpose .
increasingly , generalist researchers start their careers conducting analyses of existing datasets , and some continue to make this the focus of their career .
using secondary data enables one to conduct studies of high - impact research questions with dramatically less time and resources than required for most studies involving primary data collection .
for fellows and junior faculty who need to demonstrate productivity by completing and publishing research in a timely manner , secondary data analysis can be a key foundation to successfully starting a research career .
successful completion demonstrates content and methodological expertise , and may yield useful data for future grants . despite these attributes , conducting high quality secondary data research requires a distinct skill set and substantial effort .
however , few frameworks are available to guide new investigators as they conduct secondary data analysies.13 in this article we describe key principles and skills needed to conduct successful analysis of secondary data and provide a brief description of high - value datasets and online resources .
the primary target audience of the article is investigators with an interest but limited prior experience in secondary data analysis , as well as mentors of these investigators , who may find this article a useful reference and teaching tool .
while we focus on analysis of large , publicly available datasets , many of the concepts we cover are applicable to secondary analysis of proprietary datasets .
datasets we feature in this manuscript encompass a wide range of measures , and thus can be useful to evaluate not only one disease in isolation , but also its intersection with other clinical , demographic , and psychosocial characteristics of patients .
many worthwhile studies simply can not be done in a reasonable timeframe and cost with primary data collection .
for example , if you wanted to examine racial and ethnic differences in health services utilization over the last 10  years of life , you could enroll a diverse cohort of subjects with chronic illness and wait a decade ( or longer ) for them to die , or you could find a dataset that includes a diverse sample of decedents . even for less dramatic examples , primary data collection can be difficult without incurring substantial costs , including time and money  scarce resources for junior researchers in particular .
secondary datasets , in contrast , can provide access to large sample sizes , relevant measures , and longitudinal data , allowing junior investigators to formulate a generalizable answer to a high impact question .
for those interested in conducting primary data collection , beginning with a secondary data analysis may provide a 
bird s eye view  of epidemiologic trends that future primary data studies examine in greater detail .
secondary data analyses , however , have disadvantages that are important to consider . in a study focused on primary data , you can tightly control the desired study population , specify the exact measures that you would like to assess , and examine causal relationships ( e.g. , through a randomized controlled design ) . in secondary data analyses ,
the study population and measures collected are often not exactly what you might have chosen to collect , and the observational nature of most secondary data makes it difficult to assess causality ( although some quasi - experimental methods , such as instrumental variable or regression discontinuity analysis , can partially address this issue ) . while not unique to secondary data analysis , another disadvantage to publicly available datasets is the potential to be 
scooped ,  meaning that someone else publishes a similar study from the same data set before you do .
on the other hand , intentional replication of a study in a different dataset can be important in that it either supports or refutes the generalizability of the original findings . if you do find that someone has published the same study using the same dataset , try to find a unique angle to your study that builds on their findings .
the same basic research principles that apply to studies using primary data apply to secondary data analysis , including the development of a clear research question , study sample , appropriate measures , and a thoughtful analytic approach . for purposes of secondary data analysis ,
these principles can be conceived as a series of four key steps , described in table  1 and the sections below .
table  2 provides a glossary of terms used in secondary analysis including dataset types and common sampling terminology . 

table  1a practical approach to successful research with large datasetsstepspractical advice(1 ) define your research topic and question(1 ) start with a thorough literature review(2 ) ensure that the research question has clinical or policy relevance and is based on sound a priori reasoning .
a good question is what makes a study good , not a large sample size(3 ) be flexible to adapt your question to the strengths and limitations of the potential datasets(2 ) select a dataset(1 ) use a resource such as the society of general internal medicine s online compendium ( www.sgim.org/go/datasets ) ( table  3)(2 ) to increase the novelty of your work , consider selecting a dataset that has not been widely used in your field or link datasets together to gain a fresh perspective(3 ) factor in complexity of the dataset(4 ) factor in dataset cost and time to acquire the actual dataset(5 ) consider selecting a dataset your mentor has used previously(3 ) get to know your dataset(1 ) learn the answers to the following questions:why does the database exist?who reports the data?what are the incentives for accurate reporting?how are the data audited , if at all?can you link your dataset to other large datasets?(2 ) read everything you can about the database(3 ) check to see if your measures have been validated against other sources(4 ) get a close feel for the data by analyzing it yourself or closely reviewing outputs if someone else is doing the programming(4 ) structure your analysis and presentation of findings in a way that is clinically meaningful(1 ) think carefully about the clinical implications of your findings(2 ) be cautious when interpreting statistical significance ( i.e. , p - values ) .
large sample sizes can yield associations that are highly statistically significant but not clinically meaningful(3 ) consult with a statistician for complex datasets and analyses(4 ) think carefully about how you portray the data .
a nice figure sometimes tells the story better than rows of datatable  2glossary of terms used in secondary dataset analysis researchtermmeaningtypes of datasets ( not mutually exclusive)administrative or claims datadatasets generated from reimbursement claims , such as icd-9 codes used to bill for clinical encounters , or discharge data such as discharge diagnoseslongitudinal datadatasets that measure factors of interest within the same subjects over timeclinical registriesdatasets generated from registries of specific clinical conditions , such as regional cancer registries used to create the surveillance epidemiology and end results program ( seer ) datasetpopulation - based surveya target population is available and well - defined , and a systematic approach is used to select members of that population to take part in the study .
for example , seer is a population - based survey because it aims to include data on all individuals with cancer cared for in the included regionsnationally representative surveysurvey sample that is designed to be representative of the target population on a national level .
often uses a complex sampling scheme . the health and retirement study ( hrs ) , for example , is nationally representative of community - dwelling adults over age 50panel surveya longitudinal survey in which data are collected in the same panel of subjects over time . as
one panel is at the middle or end of its participation , a panel of new participants is enrolled . in the medical expenditures panel survey ( meps ) , for example , individuals in the same household are surveyed several times over the course of 2 yearsstatistical sampling termsclusteringeven simple random samples can be prohibitively expensive for practical reasons such as geographic distance between selected subjects . identifying subjects within defined clusters , such as geographic regions or subjects treated by the same physicians , reduces cost and improves the feasibility of the study but may decrease the precision of the estimated variance ( e.g. , wider confidence intervals)complex survey designa survey design that is not a simple random selection of subjects . surveys that incorporate stratification , clustering and oversampling ( with patient weights ) are examples of complex data .
statistical software is available that can account for complex survey designs and is often needed to generate accurate findingsoversamplingintentionally sampling a greater proportion of a subgroup , increasing the precision of estimates for that subgroup .
for example , in the hrs , african - americans , latinos , and residents of florida are oversampled ( see also survey weights)stratificationin stratification , the target population is divided into relatively homogeneous groups , and a pre - specified number of subjects is sampled from within each stratum .
for example , in the national ambulatory medical care survey physicians are divided by specialty within each geographic area targeted for the survey , and a certain number of each type of physician is then identified to participate and provide data about their patientssurvey weightsweights are used to account for the unequal probability of subject selection due to purposeful over- or under - sampling of certain types of subjects and non - response bias .
the survey weight is the inverse probability of being selected . by applying survey weights , the effects of over- and under - sampling of certain types of patients
can be corrected such that the data are representative of the entire target population a practical approach to successful research with large datasets glossary of terms used in secondary dataset analysis research case a fellow in general medicine has a strong interest in studying palliative and end - of - life care .
building on his interest in racial and ethnic disparities , he wants to examine disparities in use of health services at the end of life .
he is leaning toward conducting a secondary data analysis and is not sure if he should begin with a more focused research question or a search for a dataset.investigators new to secondary data research are frequently challenged by the question  which comes first , the question or the dataset ?  in general , we advocate that researchers begin by defining their research topic or question .
a good question is essential  an uninteresting study with a huge sample size or extensively validated measures is still uninteresting .
the answer to a research question should have implications for patient care or public policy .
if possible , select a question that will be interesting regardless of the direction of the findings : positive or negative . also , determine a target audience who would find your work interesting and useful.it is often useful to start with a thorough literature review of the question or topic of interest .
this effort both avoids duplicating others work and develops ways to build upon the literature .
once the question is established , identify datasets that are the best fit , in terms of the patient population , sample size , and measures of the variables of interest ( including predictors , outcomes , and potential confounders ) .
once a candidate dataset has been identified , we recommend being flexible and adapting the research question to the strengths and limitations of the dataset , as long as the question remains interesting and specific and the methods to answer it are scientifically sound .
some measures of interest may not have been ascertained directly , but data may be available to construct a suitable proxy . in some cases , you may find a dataset that initially looked promising lacks the necessary data ( or data quality ) to answer research questions in your area of interest reliably . in that case , you should be prepared to search for an alternative dataset.a specific research question is essential to good research
. however , many researchers have a general area of interest but find it difficult to identify specific research questions without knowing the specific data available . in that case , combing research documentation for unexamined yet interesting measures in your area of interest can be fruitful .
beginning with the dataset and no focused area of interest may lead to data dredging  simply creating cross tabulations of unexplored variables in search of significant associations is bad science . yet , in our experience , many good studies have resulted from a researcher with a general topic area of interest finding a clinically meaningful yet underutilized measure and having the insight to frame a research question that uses that measure to answer a novel and clinically compelling question ( see references for examples).48 dr .
warren browner once exhorted ,  just because you were not smart enough to think of a research question in advance does nt mean it s not important ! 
case continued after a review of available datasets that fit his topic area of interest , the fellow decides to use data from the surveillance epidemiology and end results program linked to medicare claims ( seer - medicare).the range and intricacy of large datasets can be daunting to a junior researcher .
fortunately , several online compendia are available to guide researchers ( table  3 ) , including one recently developed by this manuscript s authors for the society of general internal medicine ( sgim ) ( www.sgim.org/go/datasets ) .
the sgim research dataset compendium was developed and is maintained by members of the sgim research committee .
sgim compendium developers consulted with experts to identify and profile high - value datasets for generalist researchers .
the compendium includes a description of and links to over 40 high - value datasets used for health services , clinical epidemiology , and medical education research .
the sgim compendium provides detailed information of use in selecting a dataset , including sample sizes and characteristics , available measures and how data was measured , comments from expert users , links to the dataset , and example publications ( see box  for example ) .
sgim members can request a one - time telephone consultation with an expert user of a large dataset ( see details on the compendium website ) . 

table  3online compendia of secondary datasetscompendiumweb addressdescriptionsociety of general internal medicine ( sgim ) research dataset compendiumwww.sgim.org/go/datasetsdesigned to assist investigators conducting research on existing datasets , with a particular emphasis on health services research , clinical epidemiology , and research on medical education .
includes information on strengths and weaknesses of datasets and the insights of experienced users about making best use of the datanational information center on health services research and health care technology ( nichsr)http://www.nlm.nih.gov / nichsr / index.htmlthis group of sites provides links to a wide variety of data tools and statistics , including research datasets , data repositories , health statistics , survey instruments , and more .
it is sponsored by the national library of medicineinter - university consortium for political and social research ( icpsr)https://www.icpsr.umich.eduworld s largest archive of digital social science data , including many datasets with extensive information on health and health care .
icpsr includes many sub - archives on specific topic areas , including minority health , international data , substance abuse , and mental health , and morepartners in information access for the public health workforcehttp://phpartners.org / health_stats.htmlprovides links to a variety of national , state , and local health and public health datasets . also provides links to sites providing a wide variety of health statistics , information on health information technology and standards , and other resources .
sponsored by a collaboration of us government agencies , public health organizations , and health sciences librariescanadian research data centreshttp://www.statcan.gc.ca / rdc - cdr / index - eng.htmlinks to datasets available for analysis through canada s research data centres ( rdc ) programdirectory of health and human services data resources ( us dept . of health and human services)http://aspe.hhs.gov / datacncl / datadir / index.shtmlthis site
provides brief information and links to almost all datasets from national institutes of health ( nih ) , centers for disease control and prevention ( cdc ) , centers for medicare and medicaid services ( cms ) , agency for healthcare research and quality ( ahrq ) , food and drug administration ( fda ) , and other agencies of the us department of health and human servicesnational center for health statistics ( nchs)http://www.cdc.gov / nchs / index.htmthis site links to a variety of datasets from the national center for health statistics , several of which are profiled in table  4 .
these datasets are available for downloading at no costmedicare research data assistance center ( resdac ) ; and centers for medicare and medicaid services ( cms ) research , statistics , data & systemswww.resdac.umn.edu/available_cms_data.aspthese sites link to a variety of datasets from the centers for medicare and medicaid services ( cms)veterans affairs ( va ) datawww.virec.research.va.gov/index.htma series of datasets using administrative and computerized clinical data to describe care provided in the va health care system , including information on outpatient visits , pharmacy data , inpatient data , cost data , and more . with some exceptions ,
use is generally restricted to researchers with va affiliations ( this can include a co - investigator with a va affiliation)table  4examples of high value datasetscost , availability , and complexitydatasetdescriptionsample publicationsfree . readily available .
. does not require special statistical techniques to address complex samplingsurveillance , epidemiology and end results program ( seer ) http://www.seer.cancer.gov/population-based multi - regional cancer registry database .
can be linked to medicare claims and files ( see medicare below)trends in breast - conserving surgery among asian americans and pacific islanders , 1992200012treatment and outcomes of gastric cancer among us - born and foreign - born asians and pacific islanders13free . readily available .
requires statistical considerations to account for complex sampling design and use of survey weightsnational ambulatory medical care survey ( namcs ) & national hospital ambulatory care survey ( nhamcs ) http://www.cdc.gov/nchs/ahcd.htmnationally-representative serial cross - sectional surveys of outpatient and emergency department visits .
can combine survey years to increase sample sizes ( e.g. , for uncommon conditions ) or evaluate temporal trends .
provides national estimatespreventive health examinations and preventive gynecological examinations in the us14the namcs and nhamcs are conducted annually . do not link to other datasetsprimary care physician office visits for depression by older americans15national health interview survey ( nhis ) http://www.cdc.gov/nchs/nhis.htmnationally-representative serial cross - sectional survey of individuals and families including information on health status , injuries , health insurance , access and utilization information .
can combine survey years to look at rare conditionspsychological distress in long - term survivors of adult - onset cancer : results from a national survey16can be linked to national center for health statistics mortality data ; medicare enrollment and claims data ; social security benefit history data ; medical expenditure panel survey ( meps ) data ; and national immunization provider records check survey ( niprcs ) data from 19971999diabetes and cardiovascular disease among asian indians in the us17behavioral risk factor surveillance system ( brfss ) http://www.cdc.gov/brfss/serial cross - sectional nationally - representative survey of health risk behaviors , preventative health practices , and health care access .
provides national and state estimates . since 2002 , the selected metropolitan / micropolitan area risk trends ( smart ) project has also used brfss data to identify trends in selected metropolitan and micropolitan statistical areas ( mmsas ) with 500 or more respondents .
does not link to other datasetsperceived discrimination in health care and use of preventive health services18use of recommended ambulatory care services : is the veterans affairs quality gap narrowing?19free or minimal cost . readily available
accounting for complex sampling design and use of survey weights can be more complex when using multiple waves 
seek support from a statistician . or can restrict sample to single waves for ease of usenationwide inpatient sample ( nis ) http://www.hcup-us.ahrq.gov/databases.jspthe largest us database of inpatient hospital stays that incorporates data from all payers , containing data from approximately 20% of us community hospitals .
sampling frame includes approximately 90% of discharges from us hospitalsfactors associated with patients who leave acute - care hospitals against medical advice20nis data is collected annually . for most states ,
the nis includes hospital identifiers that permit linkages to the american hospital association ( aha ) annual survey database and county identifiers that permit linkages to the area resource file ( arf)impact of hospital volume on racial disparities in cardiovascular procedure mortality21national health and nutrition examination survey ( nhanes ) http://www.cdc.gov / nchs / nhanes.htmnationally- representative series of studies combining data from interviews , physical examination , and laboratory testsdemographic differences and trends of vitamin d insufficiency in the us population,1988 - 200422nhanes data are collected annually .
can be linked to national death index ( ndi ) mortality data ; medicare enrollment and claims data ; social security benefit history data ; and medical expenditure panel survey ( meps ) data ; and dual energy x - ray absorptiometry ( dxa ) multiple imputation data files from 19992004association of hypertension , diabetes , dyslipidemia , and metabolic syndrome with obesity : findings from the national health and nutrition examination survey , 1999 to 200423the health and retirement study ( hrs ) http://hrsonline.isr.umich.edu/index.phpa nationally - representative longitudinal survey of adults older than 50 designed to assess health status , employment decisions , and economic security during retirementchronic conditions and mortality among the oldest old24hrs data is collected every 2 years .
can be linked to social security administration data ; internal revenue service data ; medicare claims data ( see medicare below ) ; and minimum data set ( mds ) dataadvance directed and surrogate decision making before death25medical expenditure panel survey ( meps ) http://www.meps.ahrq.gov/mepsweb/serial nationally - representative panel survey of individuals , families , health care providers , and employers covering a variety of topics .
meps data are collected annuallyloss of health insurance among non - elderly adults in medicaid26can be linked by request to the agency for healthcare research and quality to numerous datasets including the nhis , medicare data , and social security datainfluence of patient - provider communication on colorectal cancer screening27data costs are in the thousands to tens of thousands of dollars . requires an extensive application and time to acquire data is on the order of months at a minimum .
require additional statistical considerations to account for complex sampling design , use of survey weights , or longitudinal analysis .
complex database structure requires a higher degree of analytic and programming skill to create a study dataset efficiently.medicare claims data ( alone ) , seer - medicare , and hrs - medicare http://www.resdac.org/medicare/data_available.aspclaims data on medicare beneficiaries including demographics and resource utilization in a wide variety of inpatient and outpatient settings .
medicare claims data are collected continually and made available annually . can be linked to other medicare datasets that use the same unique identifier numbers for patients , providers , and institutions , for example , the medicare current beneficiary survey , the long - term care minimum data set , the american hospital association annual survey , and others .
seer and the hrs offer linkages to medicare data as well ( as described above)long - term outcomes and costs of ventricular assist devices among medicare beneficiaries28association between the medicare modernization act of 2003 and patient wait times and travel distance for chemotherapy29medicare current beneficiary survey ( mcbs ) http://www.cms.gov/mcbs/panel survey of a nationally - representative sample of medicare beneficiaries including health status , health care use , health insurance , socioeconomic and demographic characteristics , and health expenditures .
can be linked to other medicare datacost - related medication nonadherence and spending on basic needs following implementation of medicare part d30medicare beneficiaries and free prescription drug samples : a national survey31dataset complexity , cost , and time to acquire the data and obtain institutional review board ( irb ) approval are critical considerations for junior researchers , who are new to secondary analysis , have few financial resources , and limited time to demonstrate productivity .
table  4 illustrates the complexity and cost of large datasets across a range of high value datasets used by generalist researchers . dataset complexity increases by number of subjects , file structure ( e.g. , single versus multiple records per individual ) , and complexity of the survey design .
many publicly available datasets are free , and others can cost tens of thousands of dollars to obtain .
some datasets can be downloaded from the web , others require multiple layers of permission and security , and in some cases data must be analyzed in a central data processing center .
if the project requires linking new data to an existing database , this linkage will add to the time needed to complete the project and probably require enhanced data security .
one advantage of most secondary studies using publicly available datasets is the rapid time to irb approval .
many publicly available large datasets contain de - identified data and are therefore eligible for expedited review or exempt status .
if you can download the dataset from the web , it is probably exempt , but your local irb must make this determination.linking datasets can be a powerful method for examining an issue by providing multiple perspectives of patient experience .
many datasets , including seer , for example , can be linked to the area resource file to examine regional variation in practice patterns . however , linking datasets together increases the complexity and cost of data management .
a new researcher might consider first conducting a study only on the initial database , and then conducting their next study using the linked database .
for some new investigators , this approach can progressively advance programming skills and build confidence while demonstrating productivity .
online compendia of secondary datasets examples of high value datasets case continued the fellow s primary mentor encourages him to closely examine the accuracy of the primary predictor for his study  race and ethnicity  as reported in seer - medicare .
the fellow has a breakthrough when he finds an entire issue of the journal medical care dedicated to seer - medicare , including a whole chapter on the accuracy of coding of sociodemographic factors.9 in an analysis of primary data you select the patients to be studied and choose the study measures .
this process gives you a close familiarity with study subjects , and how and what data were collected , that is invaluable in assessing the validity of their measures , the potential bias in measuring associations between predictors and outcome variables ( internal validity ) , and the generalizability of their findings to target populations ( external validity ) .
the importance of this familiarity with the strengths and weaknesses of the dataset can not be overemphasized .
secondary data research requires considerable effort to obtain the same level of familiarity with the data .
practically , this objective requires scouring online documentation and technical survey manuals , searching pubmed for validation studies , and closely reading previous studies using your dataset , to answer the following types of questions : who collected the data , and for what purpose ?
do your measures capture what you think they capture?we strongly recommend taking advantage of help offered by the dataset managers , typically described on the dataset s website .
for example , the research data assistance center ( resdac ) is a dedicated resource for researchers using data from the centers for medicare and medicaid services ( cms).assessing the validity of your measures is one of the central challenges of large dataset research . for large survey datasets , a good first step in assessing
the validity of your measures is to read the questions as they were asked in the survey .
others , unfortunately , were collected in a way that makes the measure meaningless , problematic , or open to a range of interpretations .
these ambiguities can occur in how the question was asked or in how the data were recorded into response categories.another essential step is to search the online documentation and published literature for previous validation studies . a pubmed search using the dataset name or measure name / type and the publication type  validation studies  is a good starting point .
the key question for a validity study relates to how and why the question was asked and data were collected ( e.g. , self - report , chart abstraction , physical measurements , billing claims ) in relationship to a gold standard . for example , if you are using claims data you should recognize that the primary purpose of those data was not for research , but for reimbursement .
consequently , claims data are limited by the scope of services that are reimbursable and the accuracy of coding by clinicians completing encounter forms for billing or by coders in the claims departments of hospitals and clinics .
some clinical measures can be assessed by asking subjects if they have the condition of interest , such as self reported diagnosis of hypertension .
self - reported data may be adequate for some research questions ( e.g. , does a diagnosis of hypertension lead people to exercise more ? ) , but inadequate for others ( e.g. , the prevalence of hypertension among people with diabetes ) .
even measured data , such as blood pressure , have limitations in that methods of measurement for a study may differ from methods used to diagnose a disorder in the clinician s office . in the national health and nutrition examination survey , for example , subject s blood pressure is based on the average of several measures in a single visit .
this differs from the standard clinical practice of measuring blood pressure at separate office visits before diagnosing hypertension .
rarely do available measures capture exactly what you are trying to study . in our experience measures in existing datasets
are often good enough to answer the research question , with proper interpretation to account for what the measures actually assesses and how they differ from the underlying constructs.finally , we suggest paying close attention to the completeness of measures , and evaluating whether missing data are random or non - random ( the latter might result in bias , whereas the former is generally acceptable ) . statistical approaches to
missing data are beyond the scope of this paper , and most statisticians can help you address this problem appropriately . however , pay close attention to 
skip patterns  ; some data are missing simply because the survey item is only asked of a subset for which it applies .
for example , in the health and retirement study the question about need for assistance with toileting is only asked of subjects who respond that they have difficulty using the toilet .
if you were unaware of this skip pattern and attempted to study assistance with toileting , you would be distressed to find over three - quarters of respondents had missing responses for this question ( because they reported no difficulty using the toilet).fellows and other trainees usually do their own computer programming .
although this may be daunting , we encourage this practice so fellows can get a close feel for the data and become more skilled in statistical analysis .
datasets , however , range in complexity ( table  4 ) . in our experience ,
fellows who have completed introductory training in sas , stata , spss , or other similar statistical software have been highly successful analyzing datasets of moderate complexity without the on - going assistance of a statistical programmer
. however , if you do have a programmer who will do much of the coding , be closely involved and review all data cleaning and statistical output as if you had programmed it yourself .
close attention can reveal all sorts of patterns , problems , and opportunities with the data that are obscured by focusing only on the final outputs prepared by a statistical programmer .
programmers and statisticians are not clinicians ; they will often not recognize when the values of variables or patterns of missingness do nt make sense .
if estimates seem implausible or do not match previously published estimates , then the analytic plan , statistical code , and measures should be carefully rechecked.keep in mind that  the perfect may be the enemy of the good . 
no one expects perfect measures ( this is also true for primary data collection ) .
the closer you are to the data , the more you see the warts  dont be discouraged by this .
the measures need to pass the sniff test , in other words have clinical validity based primarily on judgement that they make sense clinically or scientifically , but also supported where possible by validation procedures , reference to auditing procedures , or in other studies that have independently validated the measures of interest .
case continued the fellow finds that blacks are less likely to receive chemotherapy in the last 2 weeks of life ( blacks 4% , whites 6% , p < 0.001 ) .
he debates the meaning of this statistically significant 2% absolute difference.often , the main challenge for investigators who are new to secondary data analysis is carefully structuring the analysis and presentation of findings in a way that tells a meaningful story .
based on what you ve found , what is the story that you want your target audience to understand ?
when appropriate , it can be useful to conduct carefully planned sensitivity analysis to evaluate the robustness of your primary findings .
a sensitivity analysis assesses the effect of variation in assumptions on the outcome of interest .
for example , if 10% of subjects did not answer a  yes  or  no  question , you could conduct sensitivity analyses to estimate the effects of excluding missing responses , or categorizing them as all  yes  or all  no .
 because large datasets may contain multiple measures of interests , co - variates , and outcomes , a frequent temptation is to present huge tables with multiple rows and columns .
these tables can be challenging to sort through , and the clinical importance of the story resulting from the analysis can be lost . in our experience , a thoughtful figure often captures the take - home message in a way that is more interpretable and memorable to readers than rows of data tables.you should keep careful track of subjects you decide to exclude from the analysis and why .
the best way to keep track is to construct a flow diagram from the original denominator to the final sample.dont confuse statistical significance with clinical importance in large datasets . due to large sample sizes
one concern that frequently arises at this stage in large database research is the acceptability of  exploratory  analyses , or the practice of examining associations between multiple factors of interest .
on the one hand , exploratory analyses risk finding a significant association by chance alone from testing multiple associations ( a false - positive result ) . on the other hand ,
the critical issue is not a statistical one , but rather whether the issue is important.10 exploratory analyses are acceptable if done in a thoughtful way that serves an a priori hypothesis , but not if merely data dredging looking for associations.we recommend consulting with a statistician when using data from a complex survey design ( see table  2 ) or developing a conceptually advanced study design , for example , using longitudinal data , multilevel modeling with clustered data , or surivival analysis . the value of input ( even if informal ) from a statistician or other advisor with substantial methodological expertise can not be overstated .
case a fellow in general medicine has a strong interest in studying palliative and end - of - life care . building on his interest in racial and ethnic disparities , he wants to examine disparities in use of health services at the end of life .
he is leaning toward conducting a secondary data analysis and is not sure if he should begin with a more focused research question or a search for a dataset.investigators new to secondary data research are frequently challenged by the question  which comes first , the question or the dataset ?  in general , we advocate that researchers begin by defining their research topic or question .
a good question is essential  an uninteresting study with a huge sample size or extensively validated measures is still uninteresting .
the answer to a research question should have implications for patient care or public policy .
if possible , select a question that will be interesting regardless of the direction of the findings : positive or negative . also , determine a target audience who would find your work interesting and useful.it is often useful to start with a thorough literature review of the question or topic of interest .
this effort both avoids duplicating others work and develops ways to build upon the literature .
once the question is established , identify datasets that are the best fit , in terms of the patient population , sample size , and measures of the variables of interest ( including predictors , outcomes , and potential confounders ) .
once a candidate dataset has been identified , we recommend being flexible and adapting the research question to the strengths and limitations of the dataset , as long as the question remains interesting and specific and the methods to answer it are scientifically sound .
some measures of interest may not have been ascertained directly , but data may be available to construct a suitable proxy . in some cases , you may find a dataset that initially looked promising lacks the necessary data ( or data quality ) to answer research questions in your area of interest reliably . in that case , you should be prepared to search for an alternative dataset.a specific research question is essential to good research
. however , many researchers have a general area of interest but find it difficult to identify specific research questions without knowing the specific data available . in that case , combing research documentation for unexamined yet interesting measures in your area of interest can be fruitful .
beginning with the dataset and no focused area of interest may lead to data dredging  simply creating cross tabulations of unexplored variables in search of significant associations is bad science . yet , in our experience , many good studies have resulted from a researcher with a general topic area of interest finding a clinically meaningful yet underutilized measure and having the insight to frame a research question that uses that measure to answer a novel and clinically compelling question ( see references for examples).48 dr .
warren browner once exhorted ,  just because you were not smart enough to think of a research question in advance does nt mean it s not important ! 
case continued after a review of available datasets that fit his topic area of interest , the fellow decides to use data from the surveillance epidemiology and end results program linked to medicare claims ( seer - medicare).the range and intricacy of large datasets can be daunting to a junior researcher .
fortunately , several online compendia are available to guide researchers ( table  3 ) , including one recently developed by this manuscript s authors for the society of general internal medicine ( sgim ) ( www.sgim.org/go/datasets ) .
the sgim research dataset compendium was developed and is maintained by members of the sgim research committee .
sgim compendium developers consulted with experts to identify and profile high - value datasets for generalist researchers .
the compendium includes a description of and links to over 40 high - value datasets used for health services , clinical epidemiology , and medical education research .
the sgim compendium provides detailed information of use in selecting a dataset , including sample sizes and characteristics , available measures and how data was measured , comments from expert users , links to the dataset , and example publications ( see box  for example ) .
sgim members can request a one - time telephone consultation with an expert user of a large dataset ( see details on the compendium website ) . 

table  3online compendia of secondary datasetscompendiumweb addressdescriptionsociety of general internal medicine ( sgim ) research dataset compendiumwww.sgim.org/go/datasetsdesigned to assist investigators conducting research on existing datasets , with a particular emphasis on health services research , clinical epidemiology , and research on medical education .
includes information on strengths and weaknesses of datasets and the insights of experienced users about making best use of the datanational information center on health services research and health care technology ( nichsr)http://www.nlm.nih.gov / nichsr / index.htmlthis group of sites provides links to a wide variety of data tools and statistics , including research datasets , data repositories , health statistics , survey instruments , and more .
it is sponsored by the national library of medicineinter - university consortium for political and social research ( icpsr)https://www.icpsr.umich.eduworld s largest archive of digital social science data , including many datasets with extensive information on health and health care .
icpsr includes many sub - archives on specific topic areas , including minority health , international data , substance abuse , and mental health , and morepartners in information access for the public health workforcehttp://phpartners.org / health_stats.htmlprovides links to a variety of national , state , and local health and public health datasets . also provides links to sites providing a wide variety of health statistics , information on health information technology and standards , and other resources .
sponsored by a collaboration of us government agencies , public health organizations , and health sciences librariescanadian research data centreshttp://www.statcan.gc.ca / rdc - cdr / index - eng.htmlinks to datasets available for analysis through canada s research data centres ( rdc ) programdirectory of health and human services data resources ( us dept . of health and human services)http://aspe.hhs.gov / datacncl / datadir / index.shtmlthis site
provides brief information and links to almost all datasets from national institutes of health ( nih ) , centers for disease control and prevention ( cdc ) , centers for medicare and medicaid services ( cms ) , agency for healthcare research and quality ( ahrq ) , food and drug administration ( fda ) , and other agencies of the us department of health and human servicesnational center for health statistics ( nchs)http://www.cdc.gov / nchs / index.htmthis site links to a variety of datasets from the national center for health statistics , several of which are profiled in table  4 .
these datasets are available for downloading at no costmedicare research data assistance center ( resdac ) ; and centers for medicare and medicaid services ( cms ) research , statistics , data & systemswww.resdac.umn.edu/available_cms_data.aspthese sites link to a variety of datasets from the centers for medicare and medicaid services ( cms)veterans affairs ( va ) datawww.virec.research.va.gov/index.htma series of datasets using administrative and computerized clinical data to describe care provided in the va health care system , including information on outpatient visits , pharmacy data , inpatient data , cost data , and more . with some exceptions , use is generally restricted to researchers with va affiliations ( this can include a co - investigator with a va affiliation)table  4examples of high value datasetscost , availability , and complexitydatasetdescriptionsample publicationsfree . readily available .
population - based survey with cross - sectional design . does not require special statistical techniques to address complex samplingsurveillance , epidemiology and end results program ( seer ) http://www.seer.cancer.gov/population-based multi - regional cancer registry database .
can be linked to medicare claims and files ( see medicare below)trends in breast - conserving surgery among asian americans and pacific islanders , 1992200012treatment and outcomes of gastric cancer among us - born and foreign - born asians and pacific islanders13free . readily available .
requires statistical considerations to account for complex sampling design and use of survey weightsnational ambulatory medical care survey ( namcs ) & national hospital ambulatory care survey ( nhamcs ) http://www.cdc.gov/nchs/ahcd.htmnationally-representative serial cross - sectional surveys of outpatient and emergency department visits .
can combine survey years to increase sample sizes ( e.g. , for uncommon conditions ) or evaluate temporal trends .
provides national estimatespreventive health examinations and preventive gynecological examinations in the us14the namcs and nhamcs are conducted annually .
do not link to other datasetsprimary care physician office visits for depression by older americans15national health interview survey ( nhis ) http://www.cdc.gov/nchs/nhis.htmnationally-representative serial cross - sectional survey of individuals and families including information on health status , injuries , health insurance , access and utilization information .
can combine survey years to look at rare conditionspsychological distress in long - term survivors of adult - onset cancer : results from a national survey16can be linked to national center for health statistics mortality data ; medicare enrollment and claims data ; social security benefit history data ; medical expenditure panel survey ( meps ) data ; and national immunization provider records check survey ( niprcs ) data from 19971999diabetes and cardiovascular disease among asian indians in the us17behavioral risk factor surveillance system ( brfss ) http://www.cdc.gov/brfss/serial cross - sectional nationally - representative survey of health risk behaviors , preventative health practices , and health care access .
provides national and state estimates . since 2002 , the selected metropolitan / micropolitan area risk trends ( smart ) project has also used brfss data to identify trends in selected metropolitan and micropolitan statistical areas ( mmsas ) with 500 or more respondents .
does not link to other datasetsperceived discrimination in health care and use of preventive health services18use of recommended ambulatory care services : is the veterans affairs quality gap narrowing?19free or minimal cost . readily available
accounting for complex sampling design and use of survey weights can be more complex when using multiple waves 
seek support from a statistician . or can restrict sample to single waves for ease of usenationwide inpatient sample ( nis ) http://www.hcup-us.ahrq.gov/databases.jspthe largest us database of inpatient hospital stays that incorporates data from all payers , containing data from approximately 20% of us community hospitals .
sampling frame includes approximately 90% of discharges from us hospitalsfactors associated with patients who leave acute - care hospitals against medical advice20nis data is collected annually . for most states ,
the nis includes hospital identifiers that permit linkages to the american hospital association ( aha ) annual survey database and county identifiers that permit linkages to the area resource file ( arf)impact of hospital volume on racial disparities in cardiovascular procedure mortality21national health and nutrition examination survey ( nhanes ) http://www.cdc.gov / nchs / nhanes.htmnationally- representative series of studies combining data from interviews , physical examination , and laboratory testsdemographic differences and trends of vitamin d insufficiency in the us population,1988 - 200422nhanes data are collected annually .
can be linked to national death index ( ndi ) mortality data ; medicare enrollment and claims data ; social security benefit history data ; and medical expenditure panel survey ( meps ) data ; and dual energy x - ray absorptiometry ( dxa ) multiple imputation data files from 19992004association of hypertension , diabetes , dyslipidemia , and metabolic syndrome with obesity : findings from the national health and nutrition examination survey , 1999 to 200423the health and retirement study ( hrs ) http://hrsonline.isr.umich.edu/index.phpa nationally - representative longitudinal survey of adults older than 50 designed to assess health status , employment decisions , and economic security during retirementchronic conditions and mortality among the oldest old24hrs data is collected every 2 years . can be linked to social security administration data ; internal revenue service data ; medicare claims data ( see medicare below ) ; and minimum data set ( mds ) dataadvance directed and surrogate decision making before death25medical expenditure panel survey ( meps ) http://www.meps.ahrq.gov/mepsweb/serial nationally - representative panel survey of individuals , families , health care providers , and employers covering a variety of topics .
meps data are collected annuallyloss of health insurance among non - elderly adults in medicaid26can be linked by request to the agency for healthcare research and quality to numerous datasets including the nhis , medicare data , and social security datainfluence of patient - provider communication on colorectal cancer screening27data costs are in the thousands to tens of thousands of dollars .
requires an extensive application and time to acquire data is on the order of months at a minimum
require additional statistical considerations to account for complex sampling design , use of survey weights , or longitudinal analysis .
complex database structure requires a higher degree of analytic and programming skill to create a study dataset efficiently.medicare claims data ( alone ) , seer - medicare , and hrs - medicare http://www.resdac.org/medicare/data_available.aspclaims data on medicare beneficiaries including demographics and resource utilization in a wide variety of inpatient and outpatient settings .
medicare claims data are collected continually and made available annually . can be linked to other medicare datasets that use the same unique identifier numbers for patients , providers , and institutions , for example , the medicare current beneficiary survey , the long - term care minimum data set , the american hospital association annual survey , and others .
seer and the hrs offer linkages to medicare data as well ( as described above)long - term outcomes and costs of ventricular assist devices among medicare beneficiaries28association between the medicare modernization act of 2003 and patient wait times and travel distance for chemotherapy29medicare current beneficiary survey ( mcbs ) http://www.cms.gov/mcbs/panel survey of a nationally - representative sample of medicare beneficiaries including health status , health care use , health insurance , socioeconomic and demographic characteristics , and health expenditures .
can be linked to other medicare datacost - related medication nonadherence and spending on basic needs following implementation of medicare part d30medicare beneficiaries and free prescription drug samples : a national survey31dataset complexity , cost , and time to acquire the data and obtain institutional review board ( irb ) approval are critical considerations for junior researchers , who are new to secondary analysis , have few financial resources , and limited time to demonstrate productivity .
table  4 illustrates the complexity and cost of large datasets across a range of high value datasets used by generalist researchers .
dataset complexity increases by number of subjects , file structure ( e.g. , single versus multiple records per individual ) , and complexity of the survey design .
many publicly available datasets are free , and others can cost tens of thousands of dollars to obtain .
some datasets can be downloaded from the web , others require multiple layers of permission and security , and in some cases data must be analyzed in a central data processing center .
if the project requires linking new data to an existing database , this linkage will add to the time needed to complete the project and probably require enhanced data security .
one advantage of most secondary studies using publicly available datasets is the rapid time to irb approval .
many publicly available large datasets contain de - identified data and are therefore eligible for expedited review or exempt status .
if you can download the dataset from the web , it is probably exempt , but your local irb must make this determination.linking datasets can be a powerful method for examining an issue by providing multiple perspectives of patient experience .
many datasets , including seer , for example , can be linked to the area resource file to examine regional variation in practice patterns . however , linking datasets together increases the complexity and cost of data management .
a new researcher might consider first conducting a study only on the initial database , and then conducting their next study using the linked database .
for some new investigators , this approach can progressively advance programming skills and build confidence while demonstrating productivity .
case continued the fellow s primary mentor encourages him to closely examine the accuracy of the primary predictor for his study  race and ethnicity  as reported in seer - medicare .
the fellow has a breakthrough when he finds an entire issue of the journal medical care dedicated to seer - medicare , including a whole chapter on the accuracy of coding of sociodemographic factors.9 in an analysis of primary data you select the patients to be studied and choose the study measures .
this process gives you a close familiarity with study subjects , and how and what data were collected , that is invaluable in assessing the validity of their measures , the potential bias in measuring associations between predictors and outcome variables ( internal validity ) , and the generalizability of their findings to target populations ( external validity ) .
the importance of this familiarity with the strengths and weaknesses of the dataset can not be overemphasized .
secondary data research requires considerable effort to obtain the same level of familiarity with the data . therefore , knowing your data in detail is critical .
practically , this objective requires scouring online documentation and technical survey manuals , searching pubmed for validation studies , and closely reading previous studies using your dataset , to answer the following types of questions : who collected the data , and for what purpose ?
do your measures capture what you think they capture?we strongly recommend taking advantage of help offered by the dataset managers , typically described on the dataset s website .
for example , the research data assistance center ( resdac ) is a dedicated resource for researchers using data from the centers for medicare and medicaid services ( cms).assessing the validity of your measures is one of the central challenges of large dataset research . for large survey datasets , a good first step in assessing
the validity of your measures is to read the questions as they were asked in the survey .
others , unfortunately , were collected in a way that makes the measure meaningless , problematic , or open to a range of interpretations .
these ambiguities can occur in how the question was asked or in how the data were recorded into response categories.another essential step is to search the online documentation and published literature for previous validation studies .
a pubmed search using the dataset name or measure name / type and the publication type  validation studies  is a good starting point .
the key question for a validity study relates to how and why the question was asked and data were collected ( e.g. , self - report , chart abstraction , physical measurements , billing claims ) in relationship to a gold standard . for example , if you are using claims data you should recognize that the primary purpose of those data was not for research , but for reimbursement .
consequently , claims data are limited by the scope of services that are reimbursable and the accuracy of coding by clinicians completing encounter forms for billing or by coders in the claims departments of hospitals and clinics .
some clinical measures can be assessed by asking subjects if they have the condition of interest , such as self reported diagnosis of hypertension .
self - reported data may be adequate for some research questions ( e.g. , does a diagnosis of hypertension lead people to exercise more ? ) , but inadequate for others ( e.g. , the prevalence of hypertension among people with diabetes ) . even measured data , such as blood pressure , have limitations in that methods of measurement for a study may differ from methods used to diagnose a disorder in the clinician s office . in the national health and nutrition examination survey ,
for example , subject s blood pressure is based on the average of several measures in a single visit .
this differs from the standard clinical practice of measuring blood pressure at separate office visits before diagnosing hypertension .
rarely do available measures capture exactly what you are trying to study . in our experience measures in existing datasets
are often good enough to answer the research question , with proper interpretation to account for what the measures actually assesses and how they differ from the underlying constructs.finally , we suggest paying close attention to the completeness of measures , and evaluating whether missing data are random or non - random ( the latter might result in bias , whereas the former is generally acceptable ) . statistical approaches to missing data
are beyond the scope of this paper , and most statisticians can help you address this problem appropriately .
skip patterns  ; some data are missing simply because the survey item is only asked of a subset for which it applies .
for example , in the health and retirement study the question about need for assistance with toileting is only asked of subjects who respond that they have difficulty using the toilet .
if you were unaware of this skip pattern and attempted to study assistance with toileting , you would be distressed to find over three - quarters of respondents had missing responses for this question ( because they reported no difficulty using the toilet).fellows and other trainees usually do their own computer programming .
although this may be daunting , we encourage this practice so fellows can get a close feel for the data and become more skilled in statistical analysis .
datasets , however , range in complexity ( table  4 ) . in our experience ,
fellows who have completed introductory training in sas , stata , spss , or other similar statistical software have been highly successful analyzing datasets of moderate complexity without the on - going assistance of a statistical programmer
. however , if you do have a programmer who will do much of the coding , be closely involved and review all data cleaning and statistical output as if you had programmed it yourself .
close attention can reveal all sorts of patterns , problems , and opportunities with the data that are obscured by focusing only on the final outputs prepared by a statistical programmer .
programmers and statisticians are not clinicians ; they will often not recognize when the values of variables or patterns of missingness do nt make sense .
if estimates seem implausible or do not match previously published estimates , then the analytic plan , statistical code , and measures should be carefully rechecked.keep in mind that  the perfect may be the enemy of the good . 
no one expects perfect measures ( this is also true for primary data collection ) .
the closer you are to the data , the more you see the warts  dont be discouraged by this .
the measures need to pass the sniff test , in other words have clinical validity based primarily on judgement that they make sense clinically or scientifically , but also supported where possible by validation procedures , reference to auditing procedures , or in other studies that have independently validated the measures of interest .
case continued the fellow finds that blacks are less likely to receive chemotherapy in the last 2 weeks of life ( blacks 4% , whites 6% , p < 0.001 ) .
he debates the meaning of this statistically significant 2% absolute difference.often , the main challenge for investigators who are new to secondary data analysis is carefully structuring the analysis and presentation of findings in a way that tells a meaningful story .
based on what you ve found , what is the story that you want your target audience to understand ?
when appropriate , it can be useful to conduct carefully planned sensitivity analysis to evaluate the robustness of your primary findings .
a sensitivity analysis assesses the effect of variation in assumptions on the outcome of interest .
for example , if 10% of subjects did not answer a  yes  or  no  question , you could conduct sensitivity analyses to estimate the effects of excluding missing responses , or categorizing them as all  yes  or all  no .  because large datasets may contain multiple measures of interests , co - variates , and outcomes , a frequent temptation is to present huge tables with multiple rows and columns .
these tables can be challenging to sort through , and the clinical importance of the story resulting from the analysis can be lost . in our experience , a thoughtful figure often captures the take - home message in a way that is more interpretable and memorable to readers than rows of data tables.you should keep careful track of subjects you decide to exclude from the analysis and why .
the best way to keep track is to construct a flow diagram from the original denominator to the final sample.dont confuse statistical significance with clinical importance in large datasets . due to large sample sizes , associations may be statistically significant but not clinically meaningful .
one concern that frequently arises at this stage in large database research is the acceptability of  exploratory  analyses , or the practice of examining associations between multiple factors of interest .
on the one hand , exploratory analyses risk finding a significant association by chance alone from testing multiple associations ( a false - positive result ) . on the other hand ,
the critical issue is not a statistical one , but rather whether the issue is important.10 exploratory analyses are acceptable if done in a thoughtful way that serves an a priori hypothesis , but not if merely data dredging looking for associations.we recommend consulting with a statistician when using data from a complex survey design ( see table  2 ) or developing a conceptually advanced study design , for example , using longitudinal data , multilevel modeling with clustered data , or surivival analysis .
the value of input ( even if informal ) from a statistician or other advisor with substantial methodological expertise can not be overstated .
case conclusion two years after he began the project the fellow completes the analysis and publishes the paper in a peer - reviewed journal.11a 2-year timeline from inception to publication is typical for large database research .
academic potential is commonly assessed by the ability to see a study through to publication in a peer - reviewed journal .
this timeline allows a fellow who began a secondary analysis at the start of a 2-year training program to search for a job with an article under review or in press.in conclusion , secondary dataset research has tremendous advantages , including the ability to assess outcomes that would be difficult or impossible to study using primary data collection , such as those involving exceptionally long follow - up times or rare outcomes . for junior investigators ,
the potential for a shorter time to publication may help secure a job or career development funding .
some of the time  saved  by not collecting data yourself , however , needs to be  spent  becoming familiar with the dataset in intimate detail .
ultimately , the same factors that apply to successful primary data analysis apply to secondary data analysis , including the development of a clear research question , study sample , appropriate measures , and a thoughtful analytic approach .
 abstractsecondary analyses of large datasets provide a mechanism for researchers to address high impact questions that would otherwise be prohibitively expensive and time - consuming to study . 
 this paper presents a guide to assist investigators interested in conducting secondary data analysis , including advice on the process of successful secondary data analysis as well as a brief summary of high - value datasets and online resources for researchers , including the sgim dataset compendium ( www.sgim.org/go/datasets ) . the same basic research principles that apply to primary data analysis apply to secondary data analysis , including the development of a clear and clinically relevant research question , study sample , appropriate measures , and a thoughtful analytic approach . 
 a real - world case description illustrates key steps : ( 1 ) define your research topic and question ; ( 2 ) select a dataset ; ( 3 ) get to know your dataset ; and ( 4 ) structure your analysis and presentation of findings in a way that is clinically meaningful . 
 secondary dataset analysis is a well - established methodology . 
 secondary analysis is particularly valuable for junior investigators , who have limited time and resources to demonstrate expertise and productivity . 
INTRODUCTION
REASONS TO CONDUCT OR TO AVOID A SECONDARY DATA ANALYSIS
STEPS TO CONDUCTING A SUCCESSFUL SECONDARY DATA ANALYSIS
Define your Research Topic and Question
Select a Dataset
Get to Know your Dataset
Structure your Analysis and Presentation of Findings in a Way that Is Clinically Meaningful
CONCLUSIONS