CheXthought

A global multimodal dataset of clinical chain-of-thought reasoning and visual attention for chest X-ray interpretation.
Sonali Sharma1,2 · Jin Long3 · George Shih4 · Sarah Eid5 · Christian Bluethgen1,2 · Francine L. Jacobson5 · Emily B. Tsai2 · · Ahmed M. Alaa6,7 · Curtis P. Langlotz1,2,8
1Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Stanford, CA, USA
2Department of Radiology, Stanford University, Stanford, CA, USA
3Department of Pediatrics, Stanford University School of Medicine, Stanford, California, USA
4Department of Radiology, Weill Cornell Medicine, New York, NY, USA
5Department of Radiology, Brigham and Women’s Hospital, Boston, MA, USA
6University of California, Berkeley, CA, USA
7University of California, San Francisco, CA, USA
8Department of Biomedical Data Science, Stanford University, Palo Alto, CA, USA
Read the paper
103,592reasoning traces
6.61Mattention coordinates
50,312chest X-rays
501radiologists
71countries

Abstract

Chest X-ray interpretation is a primary target for medical AI, yet most vision–language models are trained on images paired with labels or reports rather than the cognitive process and visual attention that underlie interpretation. CheXthought contains 103,592 chain-of-thought reasoning traces and 6,609,082 synchronized spatial annotations across 50,312 multi-read chest X-rays from 501 radiologists in 71 countries. The dataset captures how clinicians search an image, integrate context, express uncertainty, compare prior studies, and connect findings to spatial evidence. CheXthought is designed as a resource for studying multimodal clinical reasoning and for developing vision–language models that are more transparent, visually grounded, and uncertainty-aware.
Dataset Construction
Data Source

CheXpert Plus

50,312 chest X-rays were sourced from the CheXpert Plus dataset.

Data Extraction

Linked clinical context

Patient information and radiology reports were extracted and linked to each image.

Annotation Platform

Web-based environment

Images, reports, and patient data were hosted on a web-based platform that displayed DICOM and captured dictation and spatial annotations.

Global Recruitment

Radiologist annotators

Annotators from PGY-2 through attending radiologists were recruited globally.

Annotation Process
1

Access Annotation Platform

Annotators accessed the web-based annotation platform from any location and chose their dictation language.

2

Dictate Chain-of-Thought

Annotators dictated Chain-of-Thought reasoning via speech-to-text, covering all structured reporting elements.

3

Spatial Annotations

Annotators traced their eye path, outlining where they were looking, synchronized with their dictation.

4

Review and Submit

Annotators reviewed and edited their Chain-of-Thought transcription, then submitted it.

Explore Differences in Human Reasoning vs Model Reasoning

Choose one of seven CheXpert chest-X rays and compare human radiologist reasoning to model reasoning side by side.

CheXpert case image
Loading CheXpert JPG…
Case 1

Map of CheXthought Contributors

Synchronized Chain-of-Thought and Spatial Annotations

Chain-of-Thought Elements Captured

Click an element to highlight the corresponding part of the example reasoning below.
CheXthought chest radiograph example
CheXthought chain-of-thought
57-year-old male · acute hypoxia · follow-up chest radiograph
First I check the clinical information, 57 year old male, the history says 56 years old though with acute hypoxia. The ID is the same as the last patient so this is a follow up. Then I check for radiograph markers [#1] [#2] it is upright. Then I check lines and tubes. There is no ET tube this time [#3]–[#15] Right PICC line, tubular structure that goes from the tissues in the right arm, curves around in the upper right hemithorax and into the cavoatrial junction. [#16]–[#25] Chest drain in the right [#26]–[#35] I see there are some lucencies in the subcutaneous tissues of the right lateral chest wall, subcutaneous emphysema. I am unsure what the tubular foreign body is that overlies the left neck and left chest. Then I look at the lungs, starting at the apices looking for pneumothorax. There is none. The pleural line extends all the way out to the periphery of the lung. Then I examine the rest of the lung parenchyma [#36]–[#78] I look at the CP angles. In the right lower zone, there is increased opacity that I can see the bones through, the right costophrenic angle is blunt. So I think this is a pleural effusion. I compare to the left side and I can see the lower zone much clearer. [#79]–[#96] Then I look at the mediastinum, normal heart size and mediastinum, as the stripe beside the trachea is clear and thin. Last I look at the bones and soft tissues [#97]–[#141] I do not see a fracture on this image. I am uncertain what the tubular foreign body is that overlies the left neck and left chest, I am assuming it is something that is external to the patient. Right pleural effusion. Right chest drain with subcutaneous emphysema. This looks worse than priors as there is a new pleural effusion. CT thorax can be helpful to evaluate the lung and pleural fluid. Check for cardiac failure which can be a cause of pleural effusion.