Early detection and diagnosis: how machine learning can bring healthcare’s holy grail within reach

This page was first published on January 25, 2022, and is authored and maintained by Mihaela van der Schaar and Nick Maxfield.

The importance of early detection and diagnosis

The area of early detection and diagnosis (ED&D) stands alone in terms of the dramatic life-saving potential that can be unlocked through even incremental advances.

Consider cancer survival: according to Cancer Research UK, over 90 percent of patients diagnosed with stage 1 bowel cancer will survive for 5 or more years; conversely, this figure drops to 10 percent for patients diagnosed at stage 4. The same trend holds true for all other prevalent cancer types.

Despite impressive progress in increasing cancer survival rates in recent decades, late-stage diagnosis rates have remained stubbornly high. In England, for example, almost half of all cancer patients are diagnosed at a late stage; similarly, the late-stage breast cancer diagnosis rate in the U.S. has only fallen from 45 to 43 percent since 2004. In this context, ambitions to increase early stage diagnosis rates (such as the NHS’s target of 75% by 2028) will require a fundamentally new approach if they are to be considered achievable.

Meanwhile, global population and demographic trends suggest strongly that overall cancer incidence will continue to rise over the coming decades, with authoritative forecasts predicting over 500,000 cases per year (versus roughly 375,000 at present) in the U.K. by 2035, and a global increase from roughly 19 million cases per year at present to over 28 million by 2040.

A great deal of pioneering work has been done in the area of ED&D by healthcare leaders such as Cancer Research UK’s Early Detection Programme and the Canary Center at Stanford for Cancer Early Detection. On this page, however, you will find something rather different: an early vision for how machine learning and AI can help transform ED&D. We will attempt to touch on practically every aspect of ED&D, rather than focusing on a single application such as imaging or genetic risk (though we have provided references for further reading on each of the areas featured on this page).

As mentioned above, this page only offers an early vision; this vision itself will continue to evolve as we discuss ED&D with our clinical partners and reflect their thoughts.

It is also worth noting that—though we are using cancer as our primary example throughout this page—the impact of progress in ED&D would likely be no less pronounced for almost any chronic or progressive disease (for example, cardiovascular diseases, respiratory diseases, and many others).

Pathways to diagnosis

A patient who has not yet been diagnosed with a disease is still likely to be monitored (either on a routine basis or for a specific purpose) as their “hidden” actual state progresses from healthy towards the development of a disease. Monitoring a patient over time involves a variety of different observations and types of information. When the patient’s underlying state is healthy, the most relevant information is likely to be risk-related: genomic/biomarker data, EHR data, social and demographic data, behavioral and personal activity data, and so forth. As the underlying patient state progresses, the emergence of disease may manifest itself in the form of early signs and indicators and non-specific symptoms that will need to be observed if an early-stage diagnosis is to be made. However, as symptoms become increasingly specific, a late-stage diagnosis becomes more likely—as does a less desirable outcome. This is shown in a highly simplified and generalized manner by the figure below.

Generally speaking, there are two main pathways to detection and diagnosis for a disease like cancer: in one, an asymptomatic patient is screened based on risk stratification (usually on the basis of age and sex) and a condition is detected; in the other, a patient presents with signs or symptoms, and a diagnosis is made.

The first pathway is relatively orderly: asymptomatic individuals are selected for screening on the basis of risk scores determined by genetic information, family history, exposures, and so forth. For certain types of cancer (such as breast, colon, or cervical cancer), the screening process is well-defined. This is not to say that this pathway is in any way simple, since producing a comprehensive array of ever-evolving risk scores is an incredibly complex undertaking; rather, the point being made is that the path through screening and referral for specific types of cancer is relatively clear.

By contrast, the second pathway presents a difficult challenge: individuals frequently display very vague symptoms and signs with unclear and potentially conflicting diagnostic relevance. These must be interpreted and managed using clinical practice guidelines by primary care clinicians tasked with mapping out the pathway leading from their own investigations to (and beyond) referral—including working out the appropriate degree of urgency depending on the suspected type of cancer. Some of these guidelines can be so Byzantine in nature that even understanding and explaining them in the first place can be a considerable ordeal—as attested in this account from the BMJ information designer who valiantly (and remarkably successfully) tasked with making sense of updated NICE guidelines in 2015. As explained in detail below, machine learning can leverage a wealth of data sources to provide elegant solutions for both pathways—ranging from dynamic personalized risk scores for multiple diseases to recommendations for personalized screening and monitoring programs, or offering valuable insights into disease patterns and pathways—and much more.

Early detection and diagnosis: the need for a unified vision

Within the healthcare ecosystem there are many stakeholder groups, each of which has a specific set of needs with regard to ED&D. These groups include patients, primary and secondary care clinicians, radiologists, epidemiologists, geneticists, clinical researchers, and policymakers—as depicted in the figure below.

Expertise is somewhat fragmented across the area of ED&D, and an integrated view is needed across disciplines. Each group of stakeholders shown in the figure above will inevitably see the problem of ED&D from a different perspective.

Each of these stakeholders must act based on different information and knowledge, and different objectives; however, these views must be integrated in order to build a shared approach to making early detection and diagnosis as powerful and efficient as possible. What is needed is an overarching ecosystem capable of pulling together the broad range of available data, needs, and perspectives—a task perfectly suited to machine learning. By contextualizing and presenting information in a manner specific to each user and situation, machine learning can extract actionable intelligence across this range of different datasets, and tailor insights to specific stakeholders.

This is not something that can be done at the standard machine level (for example, where a model is given information, and makes and displays a prediction on the basis of that information). Rather, this degree of integration would require machine learning to work on what might be called a “meta-level”—integrating the outputs of multiple models and other data sources. This is a topic we plan to explore through our upcoming Revolutionizing Healthcare sessions for clinicians, through a double-header specifically on the topic of ED&D in February and March.

How machine learning can help transform early detection and diagnosis

For the last two decades, medicine has been reliant on statistics-based tools and techniques to identify individuals who may have a clinical condition before they present clinically; one example of a successful statistical score might be the Tyrer-Cuzick (IBIS) model for risk stratification. However, while such methods exist for certain diseases, they do not exist for many others. Additionally, they work at a population-level, and generally predict risks on the basis of limited factors and static data.

While these techniques may have led to meaningful increases in early diagnosis rates, it is clear that they have very significant limitations because they do not address the needs of the individual patient in a complex environment (as they simply take population-level estimates). The value of machine learning for ED&D lies in its ability to go beyond population-based risk statistics, while addressing a whole range of additional questions with a degree of certainty and resolution that have never before been achieved.

On this page, we will introduce a range of machine learning methods and approaches that can catalyze transformational progress across practically every area of early detection and diagnosis.

These methods and approaches fall into one or more of the four broad categories within our lab’s framework defining the impact of machine learning for healthcare: bespoke medicine; empowering clinicians; population health and public health policy; and systems, pathways, and processes. It is worth noting, however, that few methods and approaches will fit neatly under just one category within this framework.

Our summary on this page will loosely reflect how we see machine learning methods driving transformation at each step of the patient diagnostic journey. This is mapped out in the figure below (which also shows how each method or approach fits within our framework for machine learning for healthcare).

The table below shows how each of the potential applications of machine learning in ED&D, as introduced in the figure above, would stand to benefit different groups of stakeholders throughout healthcare.

To our knowledge, this page represents one of the very first attempts to articulate a comprehensive perspective and broad vision for machine learning in ED&D in and beyond cancer, rather than focusing on one particular disease or one specific problem within the diagnostic journey (such as machine learning for risk prediction or imaging).

This summary does not aim to provide a comprehensive list of all areas of ED&D where machine learning can have an impact, since there are practically limitless possibilities. Rather, our focus will be on the areas in which our lab has been conducting meaningful and impactful research and upon which we can share an informed view. We should also point out that this page will focus primarily on machine learning methods and approaches that have compellingly demonstrated potential to transform ED&D—but most of these have not yet been translated into practice or validated. Translation and validation are extremely important and currently lacking (as pointed out very clearly by the authors of this JMIR review). Part of our lab’s work does involve building methods to support validation of new machine learning tools (including clinical trials)—and additionally, this is one of the key aims of our ongoing Revolutionizing Healthcare engagement series for clinicians.

Risk prediction

Current statistical risk scoring models may be considered personalized, but they only make use of a handful of factors that have been identified as potentially important, such as genetic risks. Machine learning can help us evolve from personalized medicine to bespoke medicine, which includes many other factors that may be just as important as (or more important than) factors such as genetic risks in determining an individual’s risks for diseases such as cancer—such as clinical information (including imaging, notes, and medications), family history, demographic, exposures and socio-economic data, and more. Many of these factors are now becoming available through new data sources—such the pioneering work of UK Biobank, CPRD, and the Project Baseline Health Study.

Machine learning can offer bespoke risk predictions for asymptomatic patients—leveraging information such as clinical information (including imaging, notes, and medications), genomics, family history, demographic, and socio-economic data, and more. These predictions can also be provided with confidence intervals—giving patients and clinicians a good indication of their degree of trustworthiness. A range of related approaches and methods can be found in the pages linked below.

During training, machine learning models can draw from cross-sectional multi-modal datasets across a wide range of sources and identify which factors and which data are pertinent to specific classes of patients. At test time, the models will restrict themselves to focusing only on information that has been identified as valuable; this is important, since our aim should be to collect only the minimal predictive information necessary to yield an accurate early diagnosis.

Dynamic risk prediction

Just as no individual’s health status remains unchanging over time, disease risks continue to evolve as the mix of relevant patient features continues to change. Accordingly, risk scores must be dynamically updated over time as additional relevant information about the particular patient becomes available.

One key limitation of current widely used survival models is that they utilize only a small fraction of the available longitudinal (repeated) measurements of biomarkers and other risk factors. In particular, even though biomarkers and other risk factors are measured repeatedly over time, survival analysis is typically based on the last available measurement. This represents a severe limitation since the evolution of biomarkers and risk factors has been shown to be informative in predicting the onset of disease and various risks.

Armed with a fully quantitative and scientific understanding of the progression of multiple diseases over time, we can unlock the full capabilities of machine learning to create long-term comprehensive patient management programs that evolve with each individual’s changing context and history, thereby considering the constantly changing risks for diseases such as cancer over time.

Several relevant approaches are outlined within the page linked directly below.

Prediction of competing risks

The co-occurrence of multiple diseases among the general population is an important problem, increasing the risk of complications and representing a large share of health care expenditure.

Learning to predict competing risks over time (for example, multiple types of related cancer such as breast and ovarian cancer) is a challenging problem because the risks of developing such diseases are correlated.

This is a particularly important but complicated aspect of our ongoing drive to make bespoke medicine a reality, fully leveraging the power of machine learning to create a view of the individual that is both holistic (i.e., spanning all available medical history and complementary data, and generating comprehensive arrays of competing risk scores and predictions) and dynamic (i.e., factoring in time-series data and evolving over time to provide lifetime care).

Conventional risk scores and survival analysis approaches are not equipped to handle competing risks effectively. This is why our lab has devoted significant research effort to developing machine learning models for time-to-event analysis, estimating competing risks as well as evolution of competing risks over time as more data from becomes available, which can provide predictions that offer superior accuracy and quality of insight.

Impact of comorbidities on risk

Many illnesses arise not just from individual causes for a specific disease, but as a complex interaction between other diseases or conditions that a patient may already have had. For example, long-term diabetes increases the risk of cardiovascular and renal disease, making high blood pressure and its complications (such as heart attacks) more likely.

Comorbid diseases co-occur and progress via complex temporal patterns that vary among individuals. Accordingly, identifying and understanding the contribution of comorbidities to disease progression and outcomes is of paramount importance for ED&D. Having learned these temporal patterns, early morbidities could be treated to reduce the risk of related comorbidities: for example, managing diabetes lowers the risk of cardiovascular and renal disease.

The page linked directly below offers a range of machine learning approaches and methods designed to help predict the impact (including the impact over time) of comorbidities on individual risks.

Personalized screening

Screening among high-risk individuals within an asymptomatic population is important for the timely detection and diagnosis of a wide variety of diseases. This is in contrast to what we refer to here as monitoring, which refers to monitoring a patient who is exhibiting symptoms or signs that may be indicative of disease.

Screening can be very diverse – ranging from radiological screening, genetic screening, blood biopsies, and beyond. Various clinical decisions are needed to manage the screening process: selecting initial screening tests, interpreting test results, and deciding if further diagnostic tests are required. A good screening policy should be personalized to the features of the patient, and to their dynamic risk history (including prior history of screening). The best screening policies should also take into account that the trade-off between benefit and cost should be different for different diseases – and also for different patients.

Despite this, a “one-size-fits-all” approach to screening remains prevalent within healthcare: a particular screening procedure is carried out at fixed intervals for every patient in the relevant population. This approach is designed to work well (on average) for a population, and can only offer coarse expert-based patient stratification that is not rigorously validated through data. In breast cancer, for instance, younger women are unlikely to be screened because current screening policies are guided by clinical practice guidelines designed to work well on average for an entire population (though there are some additional genetic risk factors like BRCA and Li Fraumeni that alter screening).

Additionally, different countries have different standards for the age at which they start screening, the frequency of screening, the groups that are screened and the modalities of screening. This is due to factors including setup of the healthcare system (such as single payer vs. private) and differences in risks for patients in different countries.

An example of variation in clinical practice guidelines (related to breast cancer)

Since the risks and benefits of screening tests are functions of each patient’s features and characteristics, personalized screening policies tailored to the features of individuals would yield a significantly more accurate and efficient result. By a “personalized screening policy,” we mean a clustering of patient features, and a set of customized screening guidelines for each cluster.

Machine learning is particularly well-positioned to deliver this by integrating a wide variety of patient features with the patient’s history, and creating individualized screening policies that are better for the patient—while also making better use of expensive and scarce resources. On one hand, clinical practice guidelines prescribe a set of tests based on population-level approaches and do not provide confidence guarantees. By contrast, machine learning methods are able to recommend a sequence of screening tests based on each individual patient’s features and risk assessment—and can also provide personalized confidence estimates while doing so.

Several machine learning approaches to this problem have been developed by our own lab. Some (such as DPScreen, linked below) can identify the best time to screen patients. Others (such as ConfidentCare—described below in more detail) can recommend the best screening modalities.

ConfidentCare: A Clinical Decision Support System for Personalized Breast Cancer Screening

Breast cancer screening policies attempt to achieve timely diagnosis by regularly screening healthy women via various imaging tests.

Since the risks and benefits of screening tests are functions of each patient’s features, personalized screening policies tailored to the features of individuals are desirable. To address this, our lab developed ConfidentCare, a computer-aided clinical decision support system that learns a personalized screening policy from electronic health record (EHR) data.

ConfidentCare operates by computing clusters of patients with similar features, then learning the “best” screening procedure for each cluster using a supervised learning algorithm. The algorithm ensures that the learned screening policy satisfies a predefined accuracy requirement with a high level of confidence for every cluster.

By applying ConfidentCare to real-world data, we show that it outperforms the current CPGs in terms of cost efficiency and false positive rates: a reduction of 31% in the false positive rate can be achieved.

Under a machine learning approach such as ConfidentCare, a personalized screening policy can be seen as a forest of decision trees—one for every subgroup of similar patients.

If you’d like to learn more about ConfidentCare, please take a look at the full paper published in IEEE Transactions on Multimedia.

Optimizing the referral process

Once a patient exhibits disease symptoms, their primary care clinician will need to decide whether to proceed with in-house testing, or whether to refer the patient to a specialist—for example, to determine whether they have additional genetic risks. The decision facing the clinician is complex: where should they send that patient next, and what kind of specialist would be the most appropriate?

While referral processes are in place within many healthcare systems, they are less prevalent in less developed regions, among underserved patient groups, or for rare symptoms/diseases. Furthermore, even well-established processes could stand to benefit from approaches that are truly tailored to specific patients.

Machine learning has enabled the development of recommender systems to optimize clinical workflows by personalizing the matching of new patient cases with the appropriate diagnostic expertise—whether in the form of a clinical decision support system (CDSS), a domain expert who specializes in similar types of cases, or another institution. One example of such an approach is Discover the Expert; for more details, see the paper linked directly below.

Efficiency of screening and testing workflow

The volume of screening examinations conducted globally continues to increase. While this is essentially a welcome trend, it does carry a significant burden on human and financial resources—and potentially on outcomes, if these resources become too thinly stretched.

Breast cancer offers a compelling example: as interpretation time increases with newer screening technologies such as digital breast tomosynthesis, clinicians will be forced to read more patients in less time. There is, therefore, a distinct need for technologies that can reduce the resource burden of screening and testing by improving workflow efficiency. Machine learning can provide multiple avenues for such improvements.

One such approach, for example, could recognize different subgroups of patients, learn the policy that fits each subgroup, and prompt recommendations for which screening tests to adopt (for instance: for breast cancer, given the unique characteristics of the patient, should a mammogram be used, should an ultrasound be used, should an MRI be used, or a combination of these screening modalities?) and clinical decisions that if followed, will lead to a desired accuracy requirement with a desired level of confidence. This is the approach adopted by ConfidentCare (linked directly below).

Other approaches may involve triaging workflow to prioritize and balance resources between human experts and autonomous diagnostic tools. Again, this can be perfectly illustrated through the example of breast cancer: since a large majority of mammograms examined by radiologists are negative, it is possible to significantly reduce the daily interpretive workload of radiologists using machine learning methods that can triage a subset of examinations as negative with extremely high accuracy, and refer the rest to a human radiologist—thereby ensuring that the radiologist can focus on more suspicious examinations and diagnostic workups. This approach is explored in an award-winning paper on workflow efficiency for mammography (linked directly below).

Personalized monitoring

It is important to determine which symptomatic (or potentially symptomatic) but as-yet not diagnosed patients may require monitoring, what type of monitoring they may require, and at what intervals.

Machine learning can play a vital role in this by creating systems capable of integrating data from a wide range of sources, and alerting clinicians and patients regarding the need for (and timing of) further consultation. Such systems, when implemented in a patient-facing manner (for example, through wearables) can also empower and encourage patients to take a more active role in their own healthcare by providing them with feedback about the (positive or negative) effects of their own actions and behaviors. They can also increase the efficiency of the patient/clinician relationship by keeping both informed about the necessary level of interaction.

Value of information

To provide a better understanding of disease progression, it is essential to incorporate longitudinal measurements of biomarkers and risk factors into a model. Rather than discarding valuable information recorded over time, this allows us to make better risk assessments on the clinical events.

At the same time, deciding what and when to observe is critical when making observations is costly. In a medical setting where observations can be made sequentially, making these observations (or not) should be an active choice. Performing lab tests on a patient often incurs a financial cost as well as causing fatigue to the patient. The decision, therefore, involves a trade-off between the value of the information obtained from the observation and the cost of making the observation.

This problem presents itself when the data can be observed sequentially, so that we can observe a particular measurement before deciding which other measurements to observe. The problem of deciding what to observe in the future based on the measurements observed so far is present in many healthcare applications; we refer to it as active sensing.

Active sensing approaches powered by machine learning can perform a critical role in clinical decision support by providing advice regarding which observations should be measured, and when. More detail can be found in the pages linked below.

Temporal phenotyping

Phenotyping and subgroup identification—whether for personalized ED&D or for development of post-diagnosis treatment plans—is an important challenge that becomes particularly complicated in a dynamic setting where longitudinal datasets are in use.

The conventional notion of clustering seeks to group patients together in an unsupervised manner, based on their static or longitudinal features (covariates). However, unsupervised clustering does not account for patients’ observed outcomes (such as cancer onset or onset of comorbidities), and thus often leads to heterogeneous outcomes within a given cluster. Therefore, this type of clustering yields information that is of relatively limited use to clinicians and patients—after all, chronic diseases such as cancer are heterogeneous in nature, with widely differing outcomes, even when patients’ features seem relatively similar.

What clinicians and patients actually need to know is what types of events will likely occur in the future for a given patient, given the observations (features) they have obtained so far. We are, therefore, interested in a type of clustering or phenotyping in which patients are grouped based on similarity of future outcomes (for example, incidence of a particular stage of cancer), rather than solely on similarity of observations. Machine learning is perfectly suited to this kind of complex but highly informative approach – as outlined in detail in the page linked below.

Estimation of preventative treatment and intervention effects at the individual level

A major challenge in healthcare is ascertaining whether a given treatment influences or determines an outcome. While it is natural to primarily associate this problem with post-diagnosis treatment (for example, planning a personalized treatment course for a cancer patient), it is also very relevant from a preventative standpoint. For instance, we may need insight regarding whether or not to preventatively prescribe a chemoprevention drug (such as tamoxifen or raloxifene) to lower an individual’s risk of cancer. In such a case, knowing the individual’s risk level and expected survival benefit would help determine whether or not such a prescription would constitute overmedication.

Current treatment guidelines have been developed with the “average” patient in mind (on the basis of randomized control trials), but there is ample evidence that different treatments result in different effects and outcomes from one individual to another: for any given treatment, it is quite likely that only a small proportion of people will actually respond in a manner that resembles the “average” patient. Rather than making treatment decisions based on blanket assumptions about “average” patients, the goal of clinical decision-makers is now to determine the optimal treatment course for any given patient at any given time. Our own lab has been engaged in developing methods for doing so in a quantitative fashion based on insights from machine learning (details are covered extensively on the page linked below)—this is what we refer to as individualized treatment effect inference.

Understanding and empowering clinician learning and decision-making

In many areas within healthcare, it is common to find variation in practice and decision-making. ED&D is no exception: for example, there can be considerable variation in the timing and types of cancer tests ordered (or not ordered) for patients, and in pathways for diagnosis—which may lead to late diagnosis. It is important to understand the reasons for such differences, with the aim of identifying mistakes, addressing biases, and (on a larger scale) minimizing unwanted variation in practice.

This is the motivation behind our lab’s creation of quantitative epistemology, a new and transformationally significant research pillar that uses machine learning to understand decision-making and empower decision-makers.

Quantitative epistemology involves studying human decision-making, identifying potential suboptimalities in beliefs and decision processes (such as cognitive biases, selective attention, imperfect retention of past experience, etc.), and understanding risk attitudes and their implications for learning and decision-making. This allows us to construct decision support systems that provide humans with information pertinent to their intended actions, their possible alternatives and counterfactual outcomes, and other evidence to empower better decision-making.

Individual clinicians can use the information gained from quantitative epistemology to detect both one-time and systematic errors, to examine and validate their intended decisions, to gain awareness of (and account for) their beliefs and biases; and to discover new ways to learn and train.

Beyond individual clinicians, quantitative epistemology can also benefit a wide range of healthcare stakeholders. It can enable patients, for example, to understand the reasoning behind decisions and ensure that the right decisions are made. Supervisors and managers may be able to detect systematic errors, analyze variation in practice, monitor behavior, and recommend training. For policymakers, the insights gained may be applied to assessing current practices and measuring the impact of new policies.

Learning across populations

There are significant opportunities throughout healthcare to use machine learning to identify and understand the differences between countries or populations in terms of risk factors (e.g., socio-economic or demographic features), clinical practice (e.g., risk stratification approaches and screening programs), or standards of care (e.g., types of screening and testing technologies).

We can do this using information related to two main types of “mismatch” between populations in such cases: feature mismatch and distribution mismatch. The former can arise, for example, when the practices in different hospitals lead to different features being measured. The latter might be found when hospitals located in wealthier areas serve wealthier patients, who are likely to have received a different standard of care in the past and may therefore appear “healthier.”

If we were trying to validate risk scores from one population to another, we would need to try to address this mismatch to minimize bias. However, if our aim is to compare populations, we may in fact use machine learning to make meaningful observations from informative sources of mismatch.

Clinical trials of ED&D technologies

There is no shortage of potentially promising new technologies (including machine learning technologies and biotechnologies) being developed for ED&D.

Before receiving approval for use, however, these technologies will need to undergo clinical trials to determine their safety and comparative effectiveness relative to existing technologies or approaches.

While randomized controlled trials (RCTs) are the gold standard for evaluating new treatments or technologies, they are costly and time-consuming to implement, and they do not always recruit representative patients. This last point makes external validity an issue, as findings sometimes fail to generalize beyond the study population. This may be due to narrow inclusion criteria in RCTs compared with the real world, where historically, population restrictions with respect to disease severity, comorbidities, elderly patients, and ethnic minorities can be under-represented. By contrast, when technologies or treatments are approved by regulators such as the U.S. FDA after the clinical trials stage, they start being administered to a much larger and varied population of patients.

In recent years, there has been a growing trend to leverage machine learning approaches, especially tools from reinforcement learning (RL) such as Markov decision processes and multi-armed bandits, to improve and expedite clinical trial designs. This one of our own lab’s key research priorities, and is an area in which we have already developed an array of novel approaches.

Adapting to new diagnosis tests and integration of new data (continuous learning)

Since diseases such as cancer represent such an important frontier for healthcare, new tests (such as liquid biopsies) are frequently introduced, and new types of data collected. Additionally, the characteristics of populations themselves change over time (e.g., lifestyle, demographics, etc.) which may change the incidence of diseases.

As a result, a machine learning model optimized at any given time will likely come to underperform if it is not continually re-optimized to reflect new norms.

We can use advances in automated machine learning (AutoML) to determine when a model’s algorithms and hyperparameter configurations need to be re-optimized based on changes or “drifts” caused by newly acquired data, and to conduct this optimization as needed.

In 2019, our lab developed an AutoML technique to perform ongoing re-optimization of models: lifelong Bayesian optimization (LBO). By automating the model optimization process based on new data acquisition, LBO not only speeds up the learning process for newly arriving datasets, but also outperforms the results achieved under the standard approach of repeatedly optimizing a model by hand.

A dynamic health and disease atlas for ED&D

We have so far provided an array of areas where machine learning can realistically help transform ED&D—all of which are related to our own lab’s ongoing research.

In addition to these, a further strength of machine learning lies in its ability to tie together a multitude of sources of information (produced by machine learning methods and non-machine learning methods alike) into an adaptive and integrated view of an individual’s health. These can be brought to bear comprehensively in a manner that spans the entire scope of our lab’s framework for machine learning for healthcare (as introduced above).

This is what we refer to as a dynamic health and disease atlas—could be adapted to meet the needs of specific individuals or situations. Drawing on many of the machine learning approaches to ED&D outlined above, the dynamic health and disease atlas could provide an “at-a-glance” summary while still allowing users to dig down into details and make their own choices on the basis of this information; it could provide a “snapshot” of current health while also predicting the evolution of risks over time; it could provide explanations, interpretations, and types of confidence guarantee tailored to each user.

Based on a holistic view of each individual’s history and features, the dynamic health and disease atlas could offer patients access to truly bespoke medicine in the form of tailor-made reports and recommendations (for example, lifestyle changes to reduce risks, or the need for future checkups or tests). Writ large, this can be aggregated across entire populations, revolutionizing population health and public health policy.

Such a system could also support and empower clinicians by helping them make and explain decisions regarding testing and referrals, while making healthcare systems, pathways, and processes more effective and efficient through the optimal allocation of scarce resources.

Acknowledgements

The van der Schaar Lab would like to express our sincere gratitude to Dr. Hari Trivedi (Emory University School of Medicine, U.S.), Dr. Jem Rashbass (formerly NHS Digital and PHE, U.K.), Prof. Tony Ng (King’s College London, U.K.), and Prof. Willie Hamilton (University of Exeter, U.K.) for their guidance, ideas, and feedback during the process of creating and reviewing the content and this page.

Learn more and get involved

Early detection and diagnosis will be the subject of another upcoming clinician roundtable on March 10, 2022, held as part of the lab’s ongoing Revolutionizing Healthcare engagement series. Revolutionizing Healthcare is a forum for members of the clinical community to share ideas and discuss topics that will define the future of machine learning in healthcare (no machine learning experience required). If you are a practicing clinician, you can sign up for Revolutionizing Healthcare here.

Our research related to early detection and diagnosis is closely linked to a number of our lab’s other core areas of focus. If you’re interested in branching out from clustering, we’d recommend reviewing our summaries on time series in healthcare and survival analysis, competing risks, and comorbidities—as well as a page explaining how machine learning can transform care at every stage of the cancer patient’s pathway.

A full list of our papers can be found here.