REFINe-ing AI’s Role in Nephrology Diagnosis

Welcome to AudioScholar. Today we're summarizing "REFINe-ing AI’s Role in Nephrology Diagnosis", from NephJC. Artificial intelligence, particularly large language models, has shown remarkable ability in passing standardized medical examinations. However, multiple-choice questions do not reflect the complex, open-ended nature of real-world clinical practice. In nephrology, making a diagnosis requires integrating highly diverse data, including history, laboratory values, urine microscopy, and pathology images. To address this gap, Bentegeac and colleagues conducted the REFINe trial, which stands for Reasoning Enhancement With Feedback From a Generative Artificial Intelligence in Nephrology. Instead of testing whether an artificial intelligence model could independently diagnose patients, this study asked a more clinically relevant question: Can access to a high-reasoning large language model improve a physician's diagnostic accuracy when facing complex, open-ended nephrology cases?

The REFINe trial was a prospective, randomized, open-label, parallel-group superiority trial conducted online. The researchers recruited ninety-seven participants, including residents and board-certified physicians, who were randomized to evaluate ten complex nephrology cases either with or without assistance from the GPT-5 model. To prevent the artificial intelligence from simply recognizing cases it had already seen during its training, two nephrologists meticulously rewrote two hundred and forty-five clinical cases from a published journal series. They altered every sentence and number while preserving the underlying clinical logic and final diagnoses, ultimately selecting ten cases at random for the trial. These cases spanned glomerular diseases, tubular disorders, electrolyte disturbances, acute kidney injury, and transplant medicine, with the majority of cases including pathology or microscopy.

Participants in the control group formulated up to three diagnoses and a confidence score without any assistance. In the intervention group, participants did the same initial step, but were then shown a pre-generated, static response from GPT-5 containing three ranked diagnoses and brief reasoning. The participants could then revise their answers before submitting.

The primary analysis revealed that diagnostic accuracy improved significantly with artificial intelligence assistance, with both top-one and top-three diagnostic accuracy increasing by approximately twenty percentage points. Interestingly, the study highlighted a significant human-to-AI gap. Out of one hundred and seventy-eight instances where a physician's initial diagnosis was incorrect but the artificial intelligence provided the correct answer, only forty-seven percent of those cases were corrected by the physician. More than half of the participants chose to ignore the correct advice. On the other hand, the risk of being misled was very low, as only one and a half percent of initially correct diagnoses were changed to incorrect ones after viewing a wrong suggestion from the model. These findings suggest that the bottleneck in artificial intelligence integration may not be the accuracy of the models themselves, but rather how clinicians interact with and trust these tools. The fact that more than half of the correct suggestions were rejected highlights the need for formal training in artificial intelligence literacy, helping physicians understand when to accept or challenge machine-generated recommendations.

However, several caveats must be considered. This was an online study with a relatively small sample of ninety-seven participants, and the control group did not have access to standard clinical resources like search engines or reference databases, which does not reflect typical clinical practice. Furthermore, the artificial intelligence interaction was static, meaning participants could not engage in a dialogue with the model to interrogate its reasoning. Clinicians should consult the primary literature and consider these workflow limitations before changing how they integrate these tools into their practice. That was a summary of REFINe-ing AI’s Role in Nephrology Diagnosis, from NephJC. For the full piece, visit the original source. Until next time.

This is an automated summary generated by artificial intelligence, which can make mistakes. Always review the original source materials.