van der Schaar Lab

Machine learning tackles the problem of dynamic disease progression in prostate cancer patients

We present the first machine learning application to dynamic patient risk prediction and clustering, with full utilization of time-varying data, to improve personalized follow-up in early prostate cancer patients, and illustrate its utility in a clinical demonstrator web app

Watch this short video to quickly get an overview of this work.

Predicting risk in a dynamic setting

The human body is a dynamic system, with innumerable biochemical reactions occurring continually, as well as high-level changes like growth and development, and of course individual actions, behaviors, life events and so on unfolding over time.  It follows that a longitudinal (time-varying) perspective is therefore required to accurately model risks we face – especially so for a complex disease like cancer.

Despite this, many models and guidelines for disease progression and risk prediction in clinical use today rely primarily on baseline (fixed) characteristics of a patient [1], [2].  It would be of great value to patients and clinicians to see progress in development of more powerful and flexible medical models, which incorporate time-varying patient data, especially since more such high-quality data is being collected.  With this in mind, we introduce a novel, machine learning-based model for risk prediction and patient clustering in the setting of prostate cancer active surveillance.

Modelling prostate cancer progression

Prostate cancer is the most common male malignancy in the Western world [3]; while its prevalence is high, mortality is relatively low.  However, it is a diverse disease known to take varied clinical trajectories in different patients [4].  Around one in five men newly diagnosed are now placed under active surveillance (AS) [5], that is, the patient’s condition is closely monitored but no treatment is given unless tests indicate the condition is worsening.

We applied our model to a dataset of 585 men placed on AS with early prostate cancer, whose condition was categorized as Cambridge Prognostic Group (CPG) 1 or 2.  Patient data consisted of baseline characteristics (age, ethnicity, family history etc.), but also, crucially, of the longitudinal measurements, such as repeat Prostate Specific Antigen (PSA) tests, multi-parametric Magnetic Resonance Imaging (MRI), as well as repeat prostate biopsies.

The goal of our model was twofold:

  1. To dynamically (that is, over time) estimate the risk of an adverse event: the progression of the patient’s condition to the more severe CPG3 category (or higher).
  2. To dynamically allocate each patient to clusters, corresponding to different levels of risk.

The data flow and an illustration of the prediction and clustering set-up is clarified in Figure 1.

Illustration of data flow and model overview

Figure 1: (Top) Illustration of data collection and its use in prediction in our set-up.  (Bottom) A schematic illustrating the risk prediction and temporal clustering aspects of the model.  The green and yellow lines represent longitudinal data.  Cluster 4 in this illustration contains higher risk patients than cluster 3, hence an “earlier” time-to-event profile is shown.

Why and how is this useful?  First, dynamic risk prediction allows for a “real-time” view of the patient’s condition, as whenever a new measurement is added to the patient history, the prediction is updated to incorporate this information.  Second, the patient’s risk cluster assignment, which too, changes over time, provides a high-level indication of the disease trajectory.  In short, such a model significantly enriches the consultation and planning management experience for both the patient and the clinician.

Interactive clinical demonstrator

To illustrate the potential utility of our work, we provide an interactive web app, demonstrating a possible future clinical platform, which we encourage the reader to try out.  Two illustrative patients “Case A” (lower risk) and “Case B” (higher risk) are presented.  The app consists of three tabs:

  • “New Observation”, where the user can enter new measurements, such as PSA or biopsy information, and observe the impact on the risk prediction curve and cluster assignments.
  • “Historic Risk”, where the estimated risk and cluster assignments, computed after each new measurement is taken, are shown.
  • “Cluster Space”, where the patient’s trajectory across the risk clusters over time can be seen.

Figure 2 gives an overview of this demonstrator.

Interactive demonstrator web app

Figure 2: Interactive demonstrator web app at  Yellow box: patient baseline data.  Green box: patient longitudinal data.  Red box: a form for adding a new observation, which updates predicted risk score (red curve, top chart) and cluster assignment (bottom chart).

How does it work?

Our model is powered by cutting-edge machine learning (deep learning), a data-driven pattern recognition methodology.  As such, it isn’t restricted by assumptions and constraints of traditional statistical methods.  In particular, we term our predictive model Dynamic-DeepHit-Lite (DDHL), as it is derived from Dynamic-DeepHit [6].  This makes use of a recurrent neural network, with the risk estimate computed using the methods from survival analysis.  In turn, the temporal cluster assignment component, which “wraps around” this predictor, uses the Actor-Critic approach, and is thus termed Actor-Critic Temporal Predictive Clustering (AC-TPC).  This powerful method of clustering is first described and then further enhanced in these innovative papers: [7], [8].  For those visually oriented, and for the machine learners, Figure 3 shows a block diagram of the model.

Block diagram of model

Figure 3: (Top) Block diagram of the Dynamic-DeepHit-Light (DDHL) model, which predicts patient risk score.  (Bottom) Block diagram of the AC-TPC model which assigns patients to clusters stratifying the level of risk.


We compare the prediction performance of our model with two commonly used methods: Cox proportional hazards with the baseline data, and a landmarking version of the Cox model at three- and five-year time points from the start of active surveillance.  We use the concordance index (C-index) to evaluate the discriminative power of the model – the ability to correctly rank the individual risk scores.  We find that the performance of DDHL is comparable with both Cox and landmarking Cox when using the baseline data alone, but as more longitudinal data is incorporated, DDHL significantly improves over the others: at the five-year time point, the C-index was 0.82 (± 0.08) for DDHL, 0.75 (± 0.08) for Cox and 0.73 (± 0.09) for landmarking Cox.  Model calibration was good across all models tested.

We then compare the discriminative performance of the four clusters discovered by AC-TPC with four clusters of the Canary-PASS risk stratification method [9], on our dataset.  Here again, we find that AC-TPC performance is superior, with the C-index of 0.92 vs 0.79. Another advantage of our temporal clustering approach is that it provides insights into the disease trajectory on the patient and population levels, both unpacked in Figure 4.  In the left panel, two example patients’ trajectories over time are illustrated in the “cluster space” (PCA projection is used to show this in two dimensions), with clusters labelled 1 to 4 from lowest to highest risk.  We observe that Patient A deteriorates over time, moving from cluster 3 to 4, while Patient B improves, transitioning from cluster 2 to 1.  The right panel, in turn, displays the population-level dynamics in a form of a transition diagram: the probability of any patient staying in a particular cluster vs moving to an adjacent cluster are made clear.

Temporal clustering illustration

Figure 4: (Left) Trajectory of two illustrative patients through the space of temporal clusters, with clusters 1-4 corresponding to lowest-to-highest risk groups.  (Right) A population-level cluster transition diagram for the same four clusters, showing the probabilities of any patient staying in the current cluster, or moving to an adjacent one.

In summary

We present the first machine learning application to dynamic risk prediction and temporal clustering on continuous data to inform personalized follow-up in prostate cancer active surveillance patients.  We demonstrate that the algorithm outperforms standard statistical techniques and improves its predictive power over time.  Finally, we illustrate how it can be developed into a clinical tool that can be used in practice.

Developing machine learning algorithms for dynamic estimation of progression during active surveillance for prostate cancer

Changhee Lee, Alexander Light, Evgeny Saveliev, Mihaela van der Schaar, Vincent Gnanapragasam



[1] Clinically Localized Prostate Cancer: AUA/ASTRO Guideline 2022. (2022, January). American Urological Association.

[2] Drost, F. J. H., Nieboer, D., Morgan, T. M., Carroll, P. R., Roobol, M. J., Trock, B., … & Helleman, J. (2019). Predicting biopsy outcomes during active surveillance for prostate cancer: external validation of the canary prostate active surveillance study risk calculators in five large active surveillance cohorts. European Urology, 76(5), 693-702.

[3] Cai, Q., Chen, Y., Zhang, D., Pan, J., Xie, Z., Xu, C., … & Wang, Y. (2020). Estimates of over-time trends in incidence and mortality of prostate cancer from 1990 to 2030. Translational Andrology and Urology, 9(2), 196.

[4] Prostate cancer: diagnosis and management. (2019, May 9). The National Institute for Health and Care Excellence. Retrieved August 3, 2022, from

[5] Liu, Y., Hall, I. J., Filson, C., & Howard, D. H. (2021, July). Trends in the use of active surveillance and treatments in Medicare beneficiaries diagnosed with localized prostate cancer. In Urologic Oncology: Seminars and Original Investigations (Vol. 39, No. 7, pp. 432-e1). Elsevier.

[6] Lee, C., Yoon, J., & Van Der Schaar, M. (2019). Dynamic-deephit: A deep learning approach for dynamic survival analysis with competing risks based on longitudinal data. IEEE Transactions on Biomedical Engineering, 67(1), 122-133.

[7] Lee, C., & Van Der Schaar, M. (2020, November). Temporal phenotyping using deep predictive clustering of disease progression. In International Conference on Machine Learning (pp. 5767-5777). PMLR.

[8] Lee, C., Rashbass, J., & Van der Schaar, M. (2020). Outcome-oriented deep temporal phenotyping of disease progression. IEEE Transactions on Biomedical Engineering, 68(8), 2423-2434.

[9] Cooperberg, M. R., Zheng, Y., Faino, A. V., Newcomb, L. F., Zhu, K., Cowan, J. E., … & Lin, D. W. (2020). Tailoring intensity of active surveillance for low-risk prostate cancer based on individualized prediction of risk stability. JAMA oncology, 6(10), e203187-e203187.

Evgeny Saveliev

Evgeny is one of the lab’s research engineers, and has been a part-time Ph.D. student since 2021. His educational background is Natural Sciences at the University of Cambridge, followed by postgraduate study in Computer Science at University of Southampton.

Evgeny was an AI Resident at Microsoft Research Cambridge before joining the lab, where he worked on projects covering meta-learning and reinforcement learning as applied to recommender systems. He also has experience in computational finance, having worked in a fintech start-up and commodities trading.

Evgeny facilitates turning the lab’s research code into robust production quality code, making it more scalable, applying software engineering best practices; he also collaborates with our PhD students on some research topics.

He is particularly interested in working on AutoML, ML for time-series, synthetic data, and federated learning.