Multimodal Representation Learning for Pneumonia Detection in Chest Radiographs

Abstract

Accurate interpretation of chest radiographs is essential for supporting the early identification of pulmonary diseases. However, variations in image acquisition, patient characteristics, and institutional imaging protocols can limit the generalization of automated diagnostic systems. This study introduces a multimodal representation learning framework that combines chest radiographs with structured clinical information for pneumonia detection. The proposed architecture employs a vision transformer for image feature extraction and a feature-fusion network to integrate demographic and clinical indicators. The evaluation is designed around 74,600 chest radiographs from 11 medical institutions, with patient-level separation between training and testing datasets. Under the illustrative experimental setting, the proposed method achieves an AUROC of 0.944, a sensitivity of 0.902, and a specificity of 0.858. Compared with an image-only convolutional baseline, the model improves AUROC by 4.2 percentage points. External validation on data from previously unseen institutions produces an AUROC of 0.914, indicating a degree of cross-institutional robustness. These results illustrate the potential of multimodal learning to support more consistent radiographic screening across heterogeneous clinical environments.