This paper presents an approach to the evaluation and validation of the diagnostic potential of mass spectrometry data in an application on the construction of an "early warning" diagnostic procedure. Our approach is based on a full implementation and application of double cross-validatory calibration and evaluation. It is a key feature of this methodology that we can jointly optimize the classifiers for prediction while simultaneously calculating validated error rates. The methodology leaves the size of the training data nearly intact. We present application to data from a designed experiment in a colon-cancer study. Subsequent to presentation of results from the double cross-validatory analysis, we explore a post-hoc analysis of the calibrated classifiers to identify the markers that drive the classification.