Charité study reveals the limits of large language models in precision medicine
Berlin, 20 November 2023
The treatment of cancer is becoming increasingly complex, but also offers more and more possibilities. The better we understand the biology of a tumour, the more approaches there are for treatment. In order to be able to offer patients a personalised therapy tailored to their disease, a complex analysis and interpretation of various data is required. Researchers at Charité – Universitätsmedizin Berlin and Humboldt-Universität zu Berlin have now investigated whether generative artificial intelligence (AI) such as ChatGPT can help with this. It is one of many projects at Charité in which the opportunities of AI for patient care are being analysed.
If certain gene mutations can no longer be repaired by the body itself, this can lead to uncontrolled cell growth – a tumour develops. The decisive factor here is an imbalance of growth-promoting and growth-inhibiting factors, for example through changes in oncogenes. Precision oncology, a specialist field of personalised medicine, makes use of this knowledge: overactive oncogenes are specifically switched off with the help of certain drugs such as small molecule inhibitors or antibodies.
In order to know which gene mutations can be targeted for treatment, the tumour tissue is first genetically analysed. The molecular variants of the tumour DNA that are necessary for a precise diagnosis and therapy are determined. The doctors then use this information to derive individualised treatment recommendations. In particularly complex cases, this requires knowledge from various medical fields. At the Charité, the so-called molecular tumour board (MTB) then comes together: Experts from pathology, molecular pathology, oncology, human genetics and bioinformatics jointly analyse which therapies promise the greatest success based on the current study situation. This is a very complex process that results in a personalised therapy recommendation.
Can artificial intelligence help with treatment decisions?
Dr Damian Rieke, a physician at Charité, Prof Dr Ulf Leser and Xing David Wang from Humboldt-Universität zu Berlin and Dr Manuela Benary, a bioinformatician at Charité, wondered whether artificial intelligence could provide support in this area. In a study now published in the journal JAMA Network Open*, they and other researchers investigated the opportunities and limitations of large language models such as ChatGPT in the automated review of scientific literature for the selection of a personalised therapy.
“We set these models the task of identifying personalised treatment options for fictitious cancer patients and compared this with the recommendations of experts,” explains Damian Rieke. His conclusion: “In principle, artificial intelligence was able to identify personalised treatment options – but it could not match the ability of human experts.”
For the experiment, the team created ten molecular tumour profiles of fictitious patients. A specialist doctor and four large language models were then tasked with determining a personalised treatment option. These results were presented to the members of the molecular tumour board for evaluation – without them knowing where a recommendation came from.
Improved AI models give hope for future applications
“In some cases, there were surprisingly good treatment options that were identified by artificial intelligence,” reports Manuela Benary. “However, the performance of large language models is significantly worse than that of human experts.” Furthermore, data protection and reproducibility pose particular challenges when applying artificial intelligence to real patients, says Benary.
Nevertheless, Damian Rieke is fundamentally optimistic about the potential applications of AI in medicine: “We were also able to show in the study that the performance of the AI models continues to improve with newer models. This could mean that AI will also be able to provide more support in complex diagnostic and therapeutic processes in the future – as long as humans monitor the results of the AI and ultimately decide on therapies.”
AI projects at the Charité aim to improve patient care
Prof Dr Dr Felix Balzer, Director of the Institute of Medical Informatics, is also certain that medicine will benefit from AI. As Chief Medical Information Officer (CMIO) at Charité’s IT division, he works at the interface between medicine and information technology. “In terms of more efficient patient care, there is a particular focus on digitalisation and therefore also on the use of automation and artificial intelligence,” says Balzer.
For example, his institute is working on AI models for fall prevention in the care sector. But other areas of Charité are also intensively involved in research into artificial intelligence: the Charité Lab for Artificial Intelligence in Medicine is working on the development of tools for AI-based prognosis after strokes, and the TEF-Health project, led by Prof Petra Ritter from the Berlin Institute of Health at Charité (BIH), is pursuing the goal of facilitating the validation and certification of AI and robotics in medical devices.
*Benary W, Wang XD, Schmidt M et al. Leveraging Large Language Models for Decision Support in Personalised Oncology. JAMA Netw Open. 2023;6(11). doi:10.1001/jamanetworkopen.2023.43689
About the study
The study was conducted under the leadership of researchers from Charité and Humboldt University. Dr Damian Rieke (Department of Medicine with a focus on haematology, oncology and tumour immunology and Charité Comprehensive Cancer Center), Prof. Dr Ulf Leser (Deputy Director of the Institute of Informatics at Humboldt-Universität zu Berlin), Dr Manuela Benary (bioinformatician at the Charité Comprehensive Cancer Center and the Berlin Institute of Health at Charité (BIH)) and Xing Wang (Humboldt-Universität zu Berlin) contributed equally. The study was mainly funded by the German Research Foundation (DFG), German Cancer Aid and the Innovation Fund of the Joint Federal Committee.
Image: Organoid model of a tumour. In such models, uncontrolled cell growth and targeted treatments can be simulated. Ana Cristina Afonseca Pestana