ChatGPT-5.5 nearly matches oncologists in judging immunotherapy videos; Gemini and MedGemma approve too much

By Bioengineer.org · November 8, 2026.

A study published in Frontiers in Public Health has measured whether large language models (LLMs) can screen the quality of online videos about cancer immunotherapy. According to the authors (Yaslikaya, Ataş İpek and Erdem Palaz), ChatGPT-5.5 came closest to the specialists' judgment. Other systems frequently approved material that oncologists deemed inadequate.

The design was as follows. The researchers analyzed 90 transcripts of English-language YouTube videos. Three medical oncologists scored each one, and their majority served as the reference. Four models did the same in three independent sessions: ChatGPT-5.5, Claude Opus 4.8, Gemini 3.1 Pro and MedGemma 27B, an open-weights model adapted for medicine. They used four validated instruments: DISCERN, the Global Quality Score and the two dimensions (understandability and actionability) of PEMAT, a tool for evaluating patient education materials.

First finding: quality is scarce. According to the oncologists' consensus, only 35 of the 90 videos (38.9%) met at least three of the four adequacy criteria, and only 24 (26.7%) met all four. Moreover, a video's popularity did not predict its quality.

Second finding: ChatGPT-5.5 was closest to expert judgment on all four instruments. In classifying a video as adequate or not, it achieved a sensitivity of 94.3% (95% CI: 81.4–98.4) and a specificity of 89.1% (95% CI: 78.2–94.9). Its Cohen's kappa, a measure of agreement that corrects for chance, was 0.82 (95% CI: 0.69–0.93). The intervals are wide because the sample is small.

Third finding: the rest were more lenient. Gemini produced 43 false positives, that is, videos it rated as adequate and the oncologists did not. MedGemma produced 26. The source does not detail Claude Opus 4.8's figures beyond noting that agreement between models was limited overall, so it is best not to draw conclusions about it.

The authors themselves urge caution. They state that the results do not establish clinical validity or an operational threshold for autonomous screening, and that specialist review remains necessary.

Our take. The thesis: the headline "AI evaluates medical content" is misleading, because what matters is which model, with which prompt and with which error bias. Here the dangerous failure is not imprecision but systematic leniency. A filter that lets poor videos through to patients who need to recognize the side effects of cancer treatment is worse than no filter, because it gives a false sense of control.

There is a second uncomfortable message for the industry. The specialized medical model, MedGemma, did not outperform the general-purpose models on this task. In general, domain specialization does not guarantee better judgment in evaluation tasks; it should be measured case by case, not taken for granted. And a single study, with 90 transcripts and a single specialty, does not allow us to say that ChatGPT-5.5 "wins" in general: a different prompt or a different set of videos could change the ranking.

The practical value lies in the division of labor. Oncologists are a scarce resource and cannot review everything circulating on YouTube. A high-sensitivity model that prioritizes which materials to look at first, with a human making the decision, is a reasonable and already realistic use. It is a small example of the underlying thesis: in the short term, AI amplifies the specialist's capacity rather than replacing them. If that oversight is maintained and measured, the long term points toward any patient having access to reliable information without depending on the lottery of the recommendation algorithm.