The team of Hongen Liao and Hui Zhang from the School of Biomedical Engineering, Tsinghua University, has published an online research paper titled “Rethinking modality fusion: CT-assisted PET network for efficient 3D tumor segmentation” in Information Fusion, a leading journal in the field of information fusion. The first author is Fengyi Zhang, a direct-entry Ph.D. student at the School of Biomedical Engineering, Tsinghua University, and the corresponding authors are Research Professor Hui Zhang and Professor Hongen Liao of the School of Biomedical Engineering, Tsinghua University.

[Article online publication page]

Information Fusion (five-year average impact factor: 19.9) is a Category 1 TOP journal in Computer Science under the Chinese Academy of Sciences classification. It has long focused on original research on multi-sensor/multimodal fusion algorithms, fusion architectures, and their applications in medical imaging, remote sensing, and other directions.

In clinical PET/CT tumor segmentation, PET provides tumor metabolic information, while CT mainly provides anatomical reference. However, most existing networks treat the two modalities equally and ignore their inherent modality asymmetry. Indiscriminate feature fusion not only causes parameter redundancy but also easily dilutes PET tumor features with the large amount of background information in CT, which is highly unfavorable for segmenting tiny tumors that account for an extremely low proportion. At the same time, this task suffers from severe foreground-background class imbalance; small lesions are easily overwhelmed by massive background, restricting the model’s clinical segmentation performance.

[Information differences between PET and CT]

To address these challenges, this study proposes a new CT-assisted PET network, CAP-Net, for efficient 3D PET/CT tumor segmentation. The network consists of a shallow CNN branch using asymmetric feature fusion SCAF and a CAP-RWKV branch, performing modality-asymmetric fusion at local and global levels, respectively, establishing PET as the dominant modality while CT plays an anatomical auxiliary role. In the CAP-RWKV module, the study designs an LVS-TAA (Local Variance Similarity Token Region Allocation) strategy, which directly localizes high-uptake lesion regions based on PET voxel features and allocates fine-grained tokens to them, while merging large background areas into large-size tokens, avoiding tumor features being masked by massive background information. Based on this adaptive token allocation mechanism, the study further designs a KT-CE key token cross-entropy loss to specifically alleviate the severe class imbalance in PET/CT segmentation tasks. This study is the first to explicitly use PET as the dominant modality to achieve asymmetric PET/CT feature fusion in tumor segmentation, while innovatively using PET prior information to guide token region partitioning.

[Architecture of the proposed CAP-Net]

The study systematically validates on three public PET/CT datasets: AutoPET, Hecktor, and PSMA, covering different clinical scenarios such as whole-body tumors, head and neck tumors, and prostate tumors. Two model scales, small and huge, are set up, and comparisons are conducted with representative segmentation algorithms from CNN, ViT, Mamba, RWKV, and multimodal fusion categories. Results show that CAP-Net achieves optimal or suboptimal segmentation performance on the vast majority of evaluation metrics, with particularly prominent advantages on small-tumor subsets where tumor volume proportion is extremely low; among them, the lightweight version CAP-Net (small) reduces parameter count by 57.4% compared with the state-of-the-art efficient model MobileU-ViT, balancing segmentation accuracy and model efficiency and demonstrating excellent effectiveness and generalization ability.

[Comparison results on the three datasets. Blue represents false negatives (FN), and red represents false positives (FP).]

Notably, compared with entropy-based token allocation strategies, the LVS-TAA proposed in this study can rely on PET voxel features to directly localize high-uptake lesion regions and assign fine-grained tokens to them, while simultaneously merging large background areas, effectively reducing computational overhead while preserving small-tumor features.

[Visualization of adaptive token allocation by the proposed LVS-TAA method]

This work is grounded in the clinical diagnostic logic of PET/CT, breaks away from traditional symmetric multimodal fusion approaches, starts from modality-asymmetric priors, and combines adaptive token allocation with a customized loss function. It significantly reduces the number of parameters while improving the segmentation accuracy of 3D tumors, especially tiny tumors.

Original paper link: https://doi.org/10.1016/j.inffus.2026.104690