Effect of training and test datasets on image restoration and super-resolution by deep learning

Abstract

Many papers have recently been published on image restoration and single-image super-resolution (SISR) using different deep neural network architectures, training methodology, and datasets. The standard approach for performance evaluation in these papers is to provide a single "average" mean-square error (MSE) and/or structural similarity index (SSIM) value over a test dataset. Since deep learning is data-driven, performance of the proposed methods depends on the size of the training and test sets as well as the variety and complexity of images in them. Furthermore, the performance varies across different images within the same test set. Hence, comparison of different architectures and training methods using a single average performance measure is difficult, especially when they are not using the same training and test sets. We propose new measures to characterize the variety and complexity of images in the training and test sets, and show that our proposed dataset complexity measures correlate well with the mean PSNR and SSIM values obtained on different test data sets. Hence, better characterization of performance of different methods is possible if the mean and variance of the MSE or SSIM over the test set as well as the size, resolution and complexity measures of the training and test sets are specified.