Can you trust ChatGPT with your health? Why LLMs with doctor oversight are a safer bet

The rise of large language models (LLMs)

Truth be told, NLP has been around for quite some time, but it was only recently that people other than Data Scientists, and tech aficionados, became aware of NLP applications such as ChatGPT. 

In fact, the early beginnings of NLP date back to the late 1940s, when there was a growing interest in computer-based machine translation. However, that early form of NLP vastly differs from what we have today. While there was more accelerated progress over the past few decades, the adoption amongst the user base had been staggered until recently. 

Now, with the increased processing power of computers, and the gigantic volumes of data available, we are experiencing firsthand a new era in technological development: The mass adoption of LLMs by the general public. From specialists to lay people, there has been an ever-growing interest in the field of generative AI.

A brief introduction to ChatGPT

ChatGPT runs on a sophisticated pattern recognition technology and belongs to the latest generation of LLMs. It is a class of AI that is programmed to be able to perform deep learning, i.e., to improve itself with each user interaction and as online content continues to grow.

Informally called a chatbot, this generative pre-trained transformer is capable of mimicking a human conversation to such a degree that users are having a hard time identifying content that was written by AI as opposed to a human. The surge of users within the first few months since its public release at the end of November 2022 has been overwhelming: it had 1 million users within five days of its launch, and two months later, the active user base was estimated to be upwards of a 100 million users.

While a lot of people are using it for fun and out of curiosity, some users are also discovering the possibilities it holds for professional settings. As job demands and time constraints grow throughout a variety of different industries, saving time and working more efficiently becomes ever more important. It can even help with coding, content creation, or planning and scheduling, just to name a few. In fact, to some, ChatGPT can even serve as a chat partner to whom they can open up and ask for personal advice.

Use cases of ChatGPT in healthcare & medical research

Generative AI has the potential to revolutionize several industries, including healthcare. Use cases of ChatGPT in healthcare include:

• Medical recordkeeping & writing: Automatically generate summaries of patient interactions and medical histories, real-time assistance for writing and documenting medical reports, including clinical notes and discharge summaries.
• Medical translation: Real-time translation services to promote effective communication between healthcare providers and patients.
• Medical education: Students and healthcare professionals can have immediate access to pertinent medical information and resources, which can aid in their continual growth and development.

Overall, ChatGPT can potentially support healthcare professionals with repetitive and routine administrative tasks. However, some of its users might even ask the chatbot for medical advice. A recent investigation of researchers from the Johns Hopkins University and the University of California San Diego revealed that people preferred chatbot AI responses over replies given by physicians when it comes to both quality and empathy. 

And herein lies one of the many pitfalls of LLMs such as ChatGPT in the context of healthcare: Sometimes, it can appear so convincingly human and knowledgeable that people trust its advice as much as they would a medical professional. Given the current state of the technology, this could lead to serious risks and harm.

How trustworthy is information from a black box?

The latest version of ChatGPT was trained on billions of pages of material from all across the internet, including social media platforms. The sheer size of GPT-4 and OpenAI’s lack of openness regarding the data sources it was trained on effectively make it an impenetrable black box of information. Although much of the machine learning itself is supervised, the countless responses it gives cannot all be scrutinized beyond user feedback, users who are knowledge seekers, not knowledge holders to begin with. And that user feedback is yet another new input into the black box.

It is not only difficult to discern where ChatGPT’s data is being sourced from but also the validity and trustworthiness of that data. So although it is capable of generating content in a variety of different fields, all of this comes with an unverifiable level of factuality, along with no way to filter out truth bias from its sources. There is even the risk of ChatGPT leaking parts of chat histories, or coming up with “hallucinations”, i.e., making false statements and even citing false or non-existent articles.

Knowing the pitfalls of this technology, LLMs are still a work in progress. Experts are not even sure about the limitations of AI-based LLMs, and there is both excitement and fear about their capabilities and their potential for misuse. There is a strong call from inside and outside the industry for some type of regulation, but the timeframe for this happening is already severely lagging behind the technology itself.

The race to establish a regulatory framework for Artificial Intelligence by authorities worldwide reflects how urgent of a matter it is. The EU's Artificial Intelligence Act is among those legislative proposals that aim to establish regulations for products and services utilizing AI systems. An essential part of this AI Act is a classification system specifically designed to evaluate the potential risks posed to the health, fundamental rights, and safety of individuals by AI technologies. The EU's AI Act is expected to undergo its final approval in late 2023 or early 2024, making the European Union a forerunner in AI regulation.

ChatGPT vs. XUND's Medical API

The differences between ChatGPT and a solution such as XUND's, a Medical API that was created in collaboration with a team of doctors, are quite significant.

Key aspects to consider when using AI in healthcare

The successful application of Natural Language Processing (NLP) in healthcare depends on three crucial principles: accuracy, transparency, and reproducibility. These principles are important not only for us at XUND but in every application of NLP in the healthcare industry.

• Accuracy

• Transparency

• Reproducibility

Validation and quality assurance

The GPT model is pulling answers from countless numbers of unverified sources across the world wide web, all without testing, verification, or expert corroboration. 

We at XUND, however, have developed an AI-based healthcare assistant that is specialized in finding connections between different risk factors and diseases. Our in-house team of medical experts regularly conducts medical reviews and clinical trials in order to validate the data, while our AI-powered solution is constantly being improved by our data science team. The Medical API is certified as a class IIa medical device under European Medical Device Regulation (MDR), and our quality management system (QMS) is ISO 13485-compliant and regularly audited by a Notified Body, in our case, the TÜV SÜD.

Costs and resources
Additionally, the cost of both computing resources and human resources is a significant factor to consider. Using ChatGPT to process medical text would require a significant investment. Processing one million medical records would need roughly 500 million tokens, which at current prices would cost upwards of $2 to $4 million, depending on usage.

A more specifically focused solution like XUND's achieves more accurate results by focusing the company's resources on validation, as opposed to just retrofitting a general LLM such as ChatGPT, which lacks specificity and accuracy.

Large language models with doctor oversight are a safer bet

Modern LLM-based NLP models attempt to address various problems through language, but their flexibility poses challenges in terms of validation and trustworthiness. In medicine, validation and trustworthiness are of utmost importance. Even an LLM with doctor oversight can provide incorrect information as no system is 100% foolproof, but it is much less likely to have serious inconsistencies as compared to ChatGPT. 

Given the text-heavy nature of healthcare in general, there are numerous areas where LLMs can be applied, but finding a balance will be crucial going forward. The capability to generate convincing, coherent text does not eliminate the need for oversight and medical validation. How LLMs are trained and what sources they are trained on greatly affect what will then be given as output by the model.

Overall, while ChatGPT is a highly convenient tool for seeking out medical information and advice, the lack of openness, transparency, and the potential for serious errors make it a very risky option in healthcare. For something as important as medical decisions, no LLM is at a level where it can be trusted and used as a sole source of advice.

That being said, AI-powered solutions such as XUND’s certified Medical API, where each input is carefully chosen and validated by medical professionals, are not only a much safer bet, but they also offer great opportunities in the field of healthcare. Using the technology of XUND alongside current healthcare systems and professionals can improve efficiency and accuracy, and therefore lead to better patient outcomes.