Risk prediction and model comparison

 Risk prediction and model comparison
 
The potential approaches of prediction and comparison:

• Relative risk/hazard ratio/odds ratio, P-value.
• Sensitivity/specificity
• Area under ROC (receiver operating characteristics) curve (AUC)/Harrell's c statistic
• Somers' D (Kendall's Tau): the mathmatic conversion of Somers'D and c statistic are: [c statistic = D/2+0.5] or [D = (c - 0.5) x 2]. (SAS: PROC FREQ)
• NRI (net reclassification improvement)
• IDI (integrated discrimination improvement).
• K-fold cross-validation - Wikipedia
• ...

References:

• Pencina (2011). Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers (Comment).
• Pencina (2013). Understanding increments in model performance metrics.
• Pencina (2008). Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond.
• Measuring the Accuracy of Prediction Models - An International Symposium (2008): IDI, NRI and different c statistics (pdf)
• Hilden (2013). A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index.
• Liu (2012). Evaluating a New Risk Marker’s Predictive Contribution in Survival
• Mühlenbruch (2013). Assessing improvement in disease prediction using net reclassification improvement: impact of risk cut-offs and number of risk categories (Commentary).
• Cassell (2007). Don't be loopy: re-Sampling and simulation the SAS way (pdf).
• Kerr (2011). Evaluating the incremental value of new biomarkers with integrated discrimination improvement.

Software:

• Pepe Lab has some homemade software for biomarker/risk evaluation using Stata, SAS, R, SPSS, and even FORTRAN.
• Stata: to install 'Risk Prediction Package' (predcurve and incrisk), in a Stata session type: ".net from http://labs.fhcrc.org/pepe/stata/" and follow the instruction.To update the risk_prediction package at a later time, in Stata type: ".adoupdate risk_prediction, update".
• UCR posted SAS, Stata, and R codes for NRI & IDI on the website.
• Stata: How can I get a Somers' D after logistic regression in Stata?