Denoising and restoration of digital images

a) Image denoising.

In any digital image, the measurement of the three observed color values at each pixel is subject to some perturbations. These perturbations are due to the random nature of the photon counting process in each sensor. The noise can be amplified by digital corrections of the camera or by any image processing software. For example, tools removing blur from images or increasing the contrast enhance the noise.

The search for efficient image denoising methods still is a valid challenge, at the crossing of functional analysis and statistics. The principle of the first denoising methods was quite simple: Replacing the color of a pixel with an average of the colors of nearby pixels. The NL-means algorithm (see [1] for the original paper) chooses for each pixel a different average configuration adapted to the image. For a given pixel i, we take into account the similarity between the neighborhood configuration of i and all the pixels of the image. The similarity between pixels is measured as a decreasing function of the Euclidean distance of the similarity windows. Due to the fast decay of the exponential kernel, large Euclidean distances lead to nearly zero weights, acting as an automatic threshold.

In the NL-means algorithm, each patch is replaced by a weighted mean of the most similar patches present in a neighborhood. Images being mostly self-similar, such instances of similar patches are generally found, and averaging them increases the SNR. NL-means created a paradigm that has been ever since been followed by most contributions in image processing in the past seven years. Best paper CVPR 2005, quoted more than 2400 times in Google scholar, this algorithm has an IPOL (www.ipol.im) archive containing more than 5000 online experiments on original images uploaded by the journal’s readers. In 2015, this paper obtain the Longuet-Higgins Prize, an award recognizes CVPR papers from ten years ago that have made a significant impact on computer vision research. In [2], [3] and [2], we can find an improvement of the original algorithm based on the non-local assumptions.

More recently, the NL-Bayes strategy improves on NL-means by evaluating for each group of similar patches a Gaussian vector model, see [5]. To each patch is therefore associated a mean (which would be the result of NL-me ans), but also a covariance matrix estimating the variability of the patch group. This permits to compute an optimal (in the sense of Bayesian minimal mean square error) estimate of each noisy patch in the group, by a simple matrix inversion.

From a complementary point of view, in [6] a non-parametric method estimating an intensity and frequency dependent noise from a single image is presented. The noise model is estimated on image patches and can be used consequently in all patch-based denoising methods

Finally, the work made in [7] is an example of the restoration problem by studying both problems: denoising and deblurring.

Figure 1. Comparison between Gauss filter, Anisotropic filter and NL-means. Denoising experience on a natural image. Top original noisy image with standard deviation 20. In the middle and from left to right: Gauss, Neighborhood and NL-means filtering. In the bottom and from left to right: difference between the noisy image and Gauss filter, difference between the noisy image and Anisotropic filter and difference between the noisy image and NL-means.
Figure 2. Comparison between NL-means and NL-Bayes. Top-Left: Original image. Top-Right: Noisy original image with standard deviation 20. Bottom- Left: Denoised image by applying NL-means. Bottom-right: Denoised image by applying NL-Bayes.

b) Image demosaicking.

The goal of the demosaicking process is to reconstruct a full colour image from the spatially under-sampled colour channels given by the Bayer CFA configuration. From such a matrix of values, the two missing color values per pixel must be interpolated. In [8] and the related patent [Pat2], we proposed a new model based on the principle of self-similarity of the image. In the papers [9] and [10], we improved the underlying self-similarity idea.

In both problems, the algorithms are implemented in the DxO company software.

Figure 3. Original image (left) and demosaicking results of Self Similarity Driven algorithm [8] (center) and Self-Similarity and Spectral Correlation Adaptive algorithm [9] (right).

Main References:

[1] A. Buades, B. Coll, J.M Morel. “A review of image denoising methods, with a new one”, SIAM Multiscale Modeling and Simulation, Vol. 4(2): 490-530, 2005.

[2] A.Buades, B. Coll, J.M. Morel, “Image Denoising Methods. A new Nonlocal Principle”, SIAM Rev. Volume 52, Issue 1, pp. 113-147 (2010).

[3] A. Buades, B. Coll, J.M Morel “Local Smoothing Neighborhood Filters”, Handbook of Mathematical Methods in Imaging, Springer, 2010.

[4] A. Buades, B. Coll, J.M Morel, “Self-similarity based image denoising”, Communications of the ACM, Vol. 54 (5), pp. 109-117, 2011.

[5] M. Lebrun, A. Buades, J.M Morel, “A Non-local Bayesian image denoising algorithm “, SIAM J. Imaging Sci., Vol. 6(3), 1665–1688, 2013.

[6] M. Colom, M. Lebrun, A. Buades, J.M Morel “Nonparametric Multiscale Blind Estimation of Intensity-Frequency Dependent Noise”, IEEE Transactions on Image Processing, Accepted, 2015.

[7] B. Coll, J. Duran, C. Sbert, “Half-linear Regularization for Nonconvex Image Restoration Models”, AIMS Inverse Problems and Imaging, Vol. 9(2), pp.337-370, 2015.

[8] A. Buades, B. Coll, J.M Morel, C. Sbert “Self similarity driven demosaicking”, IEEE Transactions on Image Processing, Vol. 18(6), pp:1192-1202, 2009.

[9] L. Zhang, X. Wu, A. Buades, X. Li, “Color Demosaicking by Local Directional Interpolation and Non-local Enhancement with Adaptive Sparse Representation”, Journal of Electronic Imaging, Vol. 20 (2), 2011.

[10] J. Duran. A. Buades, “Self-Similarity and Spectral Correlation Adaptive Algorithm for Color Demosaicking”, IEEE Transactions on Image Processing, Vol. 23(9), pp. 4031 – 4040, 2014

[Pat2] F. Cao, F. Guichard, N. Azzabou, A. Buades, B. Coll, J.M. Morel, “Procedé de traitement d’objet numerique et systeme associé”, DxO company, Ref. EP2174289 A2, FR2919943-A1; WO2009022083-A2.