Leveraging vision-language models to select trustworthy super-resolution samples generated by diffusion models

Abstract

Super-resolution (SR) is an ill-posed inverse problem with many feasible solutions that are consistent with a given low-resolution image. On one hand, regressive SR models aim to balance fidelity and perceptual quality to yield a single solution; but this trade-off often leads to artifacts that introduce ambiguity in information-critical applications such as identifying digits or letters. On the other hand, diffusion models generate a diverse set of SR images; but now selecting the most trustworthy solution out of this set becomes a challenge. This paper introduces a robust, automated framework for identifying the most trustworthy SR sample from a diffusion-generated set by leveraging the semantic reasoning capabilities of vision-language models (VLMs). Specifically, VLMs such as BLIP-2, GPT-4o, and their variants are prompted with structured queries to evaluate semantic correctness, visual quality, and the presence of artifacts. The top-ranked SR candidates are then ensembled to yield a single trustworthy output in a cost-effective manner. To rigorously assess the validity of VLM-selected samples, we propose a novel Trustworthiness Score (TWS)-a hybrid metric that quantifies SR reliability based on three complementary components: semantic similarity using CLIP embeddings, structural integrity via SSIM on edge maps, and artifact sensitivity measured through a multi-level wavelet decomposition. We empirically demonstrate that TWS correlates strongly with human preference in both ambiguous and natural images, and that VLM-guided selections consistently yield high TWS values. Compared to conventional metrics like PSNR, LPIPS, and DISTS-which fail to reflect information fidelity-our approach offers a principled, scalable, and generalizable solution for navigating the uncertainty of the diffusion SR space. By aligning model outputs with human expectations and semantic correctness, this work sets a new benchmark for trustworthiness in generative SR tasks.