Claims-Based ICU Research: Learning From Imperfect Data*
When one digs deeper into this validation study, however, there is reason to be concerned. The 2004 study compared claims to careful chart review in three Canadian hospitals (3). The study sought to assess the validity of procedural codes in general—and only 43 (3.6%) of the 1,200 hospitalizations reviewed had an ICD-9-CM code for MV (3). So, data from a limited number of mechanically ventilated patients in another country have been extrapolated to gauge the quality of U.S. administrative coding for MV.
In this issue of Critical Care Medicine, Wunsch et al (4) provide much needed information on the validity of MV codes in U.S. Medicare beneficiaries, as well as the validity of ICU codes. Although most studies validating ICD-9-CM codes compare claims to chart review, Wunsch et al (4) take a different approach. They linked hospital claims for elderly U.S. Medicare beneficiaries to clinical data in the Acute Physiology and Chronic Health Evaluation (APACHE) outcomes database. This novel data linkage facilitated the multicenter study of over 50,000 hospitalizations—far more than would be feasible using chart review.
APACHE outcomes is a voluntary subscription service from Cerner (5). After providing patient-level data, subscribing ICUs receive risk-adjusted data on their unit’s outcomes relative to national benchmarks and comparable institutions (5). As ICUs pay to participate, they are presumably motivated to provide accurate data to the APACHE outcomes database. However, hospitals enrolled in APACHE tend to be larger teaching institutions, so the quality of administrative coding may not be perfectly reflective of all hospitals (6, 7).
The data linkage worked well: 80% of older patients in the APACHE outcomes database were matched to Medicare claims. The 20% nonmatch rate is easily explained by the 23–27% of Medicare population enrolled in Advantage Plans (managed care plans administered by private health insurance companies) during 2009–2012, for whom Medicare does not require claims.
The study found that MV codes were highly specific, but not sensitive. Of mechanically ventilated patients in APACHE, over 40% were missing a MV code in their hospital claim. Rates of missingness for MV codes ranged from 7% to 78% across individual hospitals.
Patients with missing MV codes were younger, with lower severity of illness, shorter duration of MV, and lower mortality. They were also more likely to be in the ICU because of an elective surgical hospitalization. Thus, claims identified a sicker subset of the mechanically ventilated population than the clinical database, concordant with another recent validation (8).
Of patients in APACHE outcomes (and therefore physically bedded within an ICU), 97.3% had an ICU (or cardiac care unit) bed charge on their hospital claim. The other 2.7% were presumably either “boarding” in an ICU bed or were inadvertently not billed for their ICU-level care.
Length of ICU stay was more difficult to validate. Median duration differed by 2 days—but much of this discrepancy can be explained by incomplete capture of ICU days in APACHE outcomes.