 model 3a). Model 4, a hybrid model incorporating the diabetes variable based on Form 2728 and the CVD variable based on the clinical data, demonstrates that the risk of dying associated with CVD is adjusted only minimally from 3.22 to 3.12: <25% of the full adjustment effect. Furthermore, the OR of mortality associated with diabetes is greatly attenuated in model 4 and not statistically significant when Form 2728 is used to define diabetes (OR 1.15; P = 0.34).
Discussion
This cross-sectional study of 1005 dialysis patients compares data for 17 comorbid conditions on Form 2728 to clinical data contained in the outpatient dialysis clinic record. The study sample was national and representative of ESRD patients in the United States.
The prevalence of all conditions except cardiac arrest was lower for Form 2728 compared to the clinical data, and the difference in prevalence for all conditions except HIV and AIDS was statistically significant. The specificity of Form 2728 was greater than 0.91 for all 17 conditions. The sensitivity of Form 2728 ranged from 0.15 to 0.83, being highest for HIV disease, hypertension, and diabetes and lowest for dysrhythmia, ambulation status, pericarditis, COPD, and smoking. The condition-specific sensitivity did not change significantly even after adjustment for eight independent variables.
The average sensitivity of Form 2728 across all 17 conditions was 0.59. Regression analysis of average sensitivity demonstrated significant differences in sensitivity by modality (PD higher than HD), diabetes status (diabetic patients higher than nondiabetic patients), and number of comorbid conditions (lower with higher number of conditions). These results suggest that comparing the Form 2728 comorbid conditions of PD to HD patients or diabetic patients to nondiabetic patients may produce spurious results simply because these conditions are even more underreported in HD than in PD, and in nondiabetic patients than diabetic patients. The strongest association that emerged in the model was the lower sensitivity with more comorbid conditions. One could speculate that the extra effort required to fill out the form for patients with more comorbid conditions discourages completion of the form. This type of underreporting may result in differential bias by selectively misclassifying those with the greatest comorbidity and mortality risk. Assuming continuity of care by a single physician, one might expect that the longer the relationship of the physician with the patient, the better the reporting of comorbid illness. However, models incorporating the above independent variables and time from first nephrology consultation to date of Form 2728 completion found no effect of time on Form 2728 sensitivity. In addition, time from first nephrology consultation to enrollment, first dialysis to enrollment, or Form 2728 completion to enrollment also had no effect on Form 2728 sensitivity.
The average sensitivity did not improve over the 3 years of the study, a time frame corresponding to the first 3 years the form was in use. This lack of improvement over time suggests that lack of familiarity with the form may not be the primary cause of the low sensitivity, and that efforts should be made to improve the coding of Form 2728. Wider availability of electronic patient records may contribute to an improved sensitivity of Form 2728 in the future.
Our findings are similar to validation studies of other administrative databases (17,18,19,20,21,22,23,24). Most studies evaluating administrative data sources report that they underestimate the prevalence of comorbid conditions (17, 18, 22, 23). Although the specificity of administrative data tends to be very good (>0.90), the sensitivity of these data for some conditions can be 0.50 or even lower (18, 22). These studies suggest that while diagnoses reported in administrative databases do not generally attribute diagnoses to those without the condition, many diagnoses are often significantly underreported.
The low sensitivity of Form 2728 raises concern regarding two types of statistical bias. First, an underreporting bias would result in lower estimates of disease burden. For example, an estimate of the prevalence of CHF (sensitivity = 0.52) would be biased downward to 52% of its true prevalence if Form 2728 data were used. Second, when sensitivity is as low as seen in this study, potential exists for a differential bias that is important when comparing two groups (e.g., HD versus PD). If underreporting occurs to a greater degree in one group compared with another, associations with other comorbid conditions may be masked (as shown with the OR estimates of CVD associated with modality in Table 6), or spurious relationships may be created.
To illustrate these biases, we tested the association of diabetes and dialysis modality with prevalent CVD and found that these associations were underestimated when Form 2728 CVD data were used (Table 6). When comparing PD to HD, use of Form 2728 CVD data created a bias that produced a point estimate for the OR for CVD in the opposite direction as that generated by the clinical data and eliminated the significant association between modality and CVD seen with the clinical data.
We also tested the association of diabetes and prevalent CVD with simulated 1-yr mortality and found similar biases (Table 7). Use of Form 2728 data for diabetes and prevalent CVD significantly underestimated the OR of mortality compared with the clinical data, and resulted in <25% adjustment for confounding. The degree of underestimation was similar to that found by Romano et al., who compared coronary artery bypass graft hospital discharge data from two populations with clinical data from two other populations (17). In that study, the administrative data underestimated the relative risk of 30-d mortality by 40 to 80% for various conditions, including previous MI, angina, hypertension, hyperlipidemia, cardiomegaly, bundle branch block, and premature ventricular contractions. It was not significantly different, however, for tobacco use, diabetes, CHF, or PVD (17).
Implications for Use of Form 2728 Data
The low sensitivity of Form 2728 has implications for its potential uses. For example, the prevalence and economic burden of comorbidity in ESRD patients may be underestimated when determined using Form 2728. Reimbursement for patient care may be insufficient if rates of provider payment were to be linked to comorbidity defined by Form 2728. However, the results of this study could be used to adjust prevalence and cost estimates. More importantly, bias may be introduced if Form 2728 data are used to study associations between comorbid conditions, treatment, and mortality or other outcomes. Use of Form 2728 data will bias such associations toward null findings and will result in incomplete adjustment for confounding, as seen in Tables 6 and 7. Differential bias (described above) could bias estimates in either direction. Furthermore, biased sampling could occur if Form 2728 were used to identify specific subgroups of patients for study.
Limitations
There are limitations to our study. The criteria used to identify conditions in both data sets are crucial when comparing diagnoses. Overall, the CHOICE criteria were as restrictive or more restrictive than the definitions presented on Form 2728. For example, to avoid including CAD diagnoses having lower specificity in the ESRD population, we excluded from the definition of CAD those with only angina (n = 3), ischemia diagnosed by clinical tests such as electrocardiogram or stress test (n = 1), or electrocardiogram evidence of old MI (n = 2) without other physician documentation of CAD. Such diagnoses may be reported on Form 2728; thus, our definition of CAD provides a conservative estimate of Form 2728 sensitivity for CAD. There are two important exceptions. First, for most conditions CHOICE identified any history of a condition as sufficient for inclusion. In contrast, Form 2728 requires a history of the condition within the past 10 yr. Thus, the CHOICE data may include conditions that occurred before the Form 2728 10-yr window. We suspect the effect of this difference is probably minimal.
Second, the sensitivity presented for dysrhythmia and COPD may be biased downward because our definitions included conduction problems and asthma, respectively, which were not explicitly included in the Form 2728 definitions. Although these effects would spuriously lower the calculated sensitivity for these conditions, it is improbable that they would introduce errors on the order of magnitude seen in this study.
While deriving the clinical data directly from medical records is a primary strength of this study, an independent validation of these records using other sources—such as the records or reports of primary care physicians—was not performed. However, the high specificity of Form 2728 in the face of the low sensitivity provides an internal validation of the data. To produce spuriously low Form 2728 sensitivity values of the magnitude found by this study, the clinical data itself would need to have an extraordinarily high rate of false positives. Such high false positive rates are extremely unlikely; other studies have shown that medical record data generally have very low false positive rates (between 0 and 8%) (22, 25). We therefore conclude that the low sensitivity of Form 2728 is more likely due to errors on Form 2728 than in the clinical data.
Summary
This study found generally low sensitivity but high specificity for 17 prevalent comorbid conditions on Form 2728 when compared with the medical records of 1005 dialysis patients. The sensitivity was higher in PD compared with HD, in diabetic patients compared with nondiabetic patients, and in those with fewer comorbid conditions. This differential reporting of comorbid conditions could result in either under- or overestimation of risk estimates when Form 2728 data are used. Incomplete adjustment for confounding variables also results when defined using Form 2728. Scientific research and policy initiatives that use Form 2728 data should take into account the inherent limitations of these data. Considerable effort should be expended to improve Form 2728 coding if it is to be used to provide accurate estimates of comorbid diseases in ESRD patients.
Acknowledgments
Acknowledgments
CHOICE is supported by R01-HS-08365 (Agency for Health Care Policy and Research). Dr. Powe is supported in part by K24-DK-02643 (National Institute of Diabetes Digestive and Kidney Disorders, National Institutes of Health). Dr. Longenecker is supported by K08-HL-03896 (National