Survival analysis · Validation
Testing a survival model on a hospital it has never seen
Centre-grouped CV: 0.725 ± 0.021 · Public leaderboard (new hospital): 0.754 · Clinical score IPSS-R: 0.671
In the QRT × Institut Gustave Roussy challenge, the goal is to predict how long patients with myeloid leukemia will survive, using their blood counts, karyotype (a text description of their chromosomes) and gene mutations. The twist is that the test patients come from a hospital that does not appear in the training data. A model that only works at familiar hospitals is not much use.
What the score measures
The challenge uses the concordance index, or C-index. Take two patients where we know who died first. If the model gave the higher risk score to the patient who died earlier, that pair counts as correct. The C-index is the share of such pairs the model gets right. A score of 0.5 is a coin flip; 1.0 is a perfect ranking.
Survival data has a complication: some patients are still alive when the study ends, so we do not know their full survival time. This is called censoring. The IPCW version of the C-index reweights pairs to correct for patients who leave the data early, so the score is not biased by who happened to be followed for longer. I derived both Kaplan–Meier and the IPCW C-index by hand and checked that my numbers matched the library's before trusting it.
Why ordinary cross-validation would mislead
Standard CV shuffles patients into random folds. Patients from the same hospital then sit on both sides of the split. Hospitals differ in their lab equipment, their patient mix and how they record data, so the model can learn hospital quirks and still look good on validation. That is a quiet form of leakage.
Instead, I grouped the folds by centre: every patient from a given hospital is either in training or in validation, never both. Each fold then asks the real question, "how well does this work at a hospital you have not seen?"
Choosing models without fooling myself
I compared penalized Cox regression, a random survival forest and an XGBoost survival model, then blended them. Tuning and choosing an ensemble on the same folds you report would inflate the score, so I used nested cross-validation: an inner loop made the choices and an outer loop only measured them, across 15 folds in total.
| Measure | Result |
|---|---|
| IPSS-R clinical risk score (baseline) | 0.671 |
| Ensemble, centre-grouped nested CV | 0.725 ± 0.021 |
| Folds where the ensemble beat the best single model | 15 of 15 |
| Public leaderboard (unseen hospital) | 0.754 |
Because the CV was strict, the leaderboard score came in above it rather than below, which is the direction you want. Permutation importance showed that mutation burden, the number of mutations a patient carries, was the strongest single predictor.