External Validation of Kidney Donor Risk Index Does Not Mitigate Its Basic Limitations
Because the KDRI is used to predict graft survival (without considering treatment effect), in calculations of expected outcomes and for allocation purposes, KDRI for all intents and purposes is being used as a “prognostic” biomarker. A prognostic biomarker is a marker that can predict a clinical outcome regardless of treatment.
The study by Peters-Sengers et al3 in this month's Transplantation has provided a thoughtful and well-done external validation of the original KDRI in the Dutch population. They performed validation using 3201 adult kidney recipients between years 2002 and 2012 in Netherlands. KDRIdonor-only included 9 donor factors as opposed to 10 donor factors used in original KDRI excluding ethnicity. KDRIfull included the 4 recipient factors as well.
The calibration slope for the Dutch KDRI was 0.96, indicating good correlation. The survival probabilities at 1, 3, 5, and 7.5 years for the different KDRI quintiles were comparable between the Dutch and the US cohort. Graft survival was lower at 1 year but better at 5 and 7.5 years in the Dutch population. Regression coefficients for donor age, weight, and cold ischemia time were different from the US study.
There were discrepancies in the predicted versus observed graft survival especially in the higher quintiles with KDRI of 1.45 or greater in the Dutch population, which in turn might influence donor acceptance and transplant activities.
The discriminatory ability of KDRIfull and KDRIdonor-only for graft survival at 5 years in Dutch population as measured by Harrell C statistic was 0.63 (95% confidence interval, 0.62-0.64) and 0.62 (95% confidence interval, 0.61-0.63), respectively, and was comparable to the discriminative ability for graft survival as identified in the original KDRI.4 Death-censored graft survival also had similar C statistic value. Increase in recipient age decreased the effect of KDRI on graft survival.
Minor differences aside, the Dutch study offers an excellent external validation of KDRI, and the authors should be applauded for this effort. However, a far more important and worrisome message lurks beneath the surface. Though validated, is KDRI a valid prognostic biomarker that should be used as a tool for regulatory/allocation purpose and individual patient use?
The above studies have examined and used KDRI as a prognostic biomarker for graft survival at 5 years or so. The authors report C statistic of around 0.6 which may be statistically significant but in real life has little clinical significance in predicting the clinical outcome of graft survival.
The receiver operating characteristic curve is an important tool to assess the utility of a test at predicting an outcome, which is a binary variable. It is a plot of true positive rate/sensitivity against false-positive rate/specificity of a diagnostic test.5 The utility of a test is measured using area under the receiver operating characteristic curve (AUC). The AUC corresponds to the C statistic or the ability of the test to predict outcome. C statistic of 1 implies perfect correlation between the test and outcome whereas AUC of 0.