Simultaneous inpainting and denoising by directional global three-part decomposition: connecting variational and Fourier domain-based image processing

We consider the very challenging task of restoring images (i) that have a large number of missing pixels, (ii) whose existing pixels are corrupted by noise, and (iii) that ideally contain both cartoon and texture elements. The combination of these three properties makes this inverse problem a very difficult one. The solution proposed in this manuscript is based on directional global three-part decomposition (DG3PD) (Thai, Gottschlich. 2016 EURASIP. J. Image Video Process. 2016, 1–20 (doi:10.1186/s13640-015-0097-y)) with a directional total variation norm, directional G-norm and ℓ∞-norm in the curvelet domain as key ingredients of the model. Image decomposition by DG3PD enables a decoupled inpainting and denoising of the cartoon and texture components. A comparison with existing approaches for inpainting and denoising shows the advantages of the proposed method. Moreover, we regard the image restoration problem from the viewpoint of a Bayesian framework and we discuss the connections between the proposed solution by function space and related image representation by harmonic analysis and pyramid decomposition.


Introduction and related work
Image enhancement and image restoration are two superordinate concepts in image processing which encompass a plethora of methods to solve a multitude of important real-world problems [1,2]. Image enhancement has the goal of improving an input image for a specific application, e.g. in areas such as medical image processing, biometric recognition, computer vision, optical character recognition, texture recognition or machine inspection of surfaces [3][4][5]. Methods for image enhancement can be grouped by the domain in which they perform their operations: images are processed in the spatial domain or Fourier domain, or modified, e.g. in the wavelet or curvelet domain [6]. The types of enhancement methods include contextual filtering, e.g. for fingerprint image enhancement [7-9], contrast enhancement, e.g. by histogram equalization [10], and image super-resolution [11]. Image restoration is connected to the notion that a given input image suffers from degradation and the goal is restore an ideal version of it. Degradations are caused by various types of noise, missing pixels or blurring and their countermeasures are denoising, inpainting and deblurring. In general, one has to solve a linear or nonlinear inverse problem to reconstruct the ideal image from its given degraded version. Denoising aims to remove noise from an image and denoising methods include total variation (TV) minimization-based approaches [12], the application of non-local means (NL means) [13] or other dictionaries of image patches for smoothing, and adaptive thresholding in the wavelet domain [14]. Inpainting [15] is the filling-in of missing pixels from the available information in the image, and it is applied for scratch removal from scanned photographs, for occlusion filling, for removing objects or persons from images (in image forgery [16] or for special effects), and for filling-in of pixels which were lost during the transmission of an image or left out on purpose for image compression [17]. Deblurring [18] addresses the removal of blurring artefacts and is not the focus of this paper.
Rudin et al. [19] pioneered two-part image decomposition by TV regularization for denoising. Shen & Chan [20] applied TV regularization to image inpainting, called the TV inpainting model, and they also suggested image inpainting by curvature-driven diffusions [21]. Starck et al. [22] defined a model for twopart decomposition based on the dictionary approach. Then, Elad et al. [23] applied this decomposition idea for image inpainting by introducing the indicator function in the 2 norm of the residual; see eqn (6) in [23]. Esedoglu & Shen [24] introduced two inpainting models based on the Mumford-Shah model [25] and its higher order correction-the Mumford-Shah-Euler image model. They also presented numerical computation based on the Γ -convergence approximations [26,27]. Shen et al. [28] proposed image inpainting based on bounded variation and elastica models for non-textured images.
Image inpainting can be an easy or difficult problem depending on the amount of missing pixels [21], the complexity of the image content and whether prior knowledge about the image content is available. Methods have been proposed which perform only cartoon inpainting (also referred to as structure inpainting) [20,28,29] or only texture inpainting [30]. Images which consist of both cartoon (structure) and texture components are more challenging to inpaint. Bertalmio et al. [31], Elad et al. [23] and Cai et al. [32] have proposed methods for inpainting which can handle images with both cartoon (structure) and texture components.
In this paper, we tackle an even more challenging problem. Consider an input image f which has the following three properties: (i) a large percentage of pixels in f are missing and shall be inpainted, (ii) the known pixels in f are corrupted by noise, (iii) f contains both cartoon and texture elements.
The co-occurrence of noise and missing pixels in an image with cartoon and texture components increases the difficulty of both the inpainting problem and the denoising problem. A multitude of methods have been proposed for inpainting and denoising. Existing inpainting methods in the literature typically assume that the non-missing pixels in a given image contain only a small amount of noise or are noise free, and existing methods for denoising typically assume that all pixels of the noisy image are known. The proposed method for solving this challenging problem is inspired by the works of Efros & Leung [30], Bertalmio et al. [31], Vese & Osher [33], Aujol & Chambolle [34], Buades et al. [13] and Elad et al. [23], and is based on the directional global three-part decomposition (DG3PD) [35]. The DG3PD method decomposes an image into three parts: a cartoon image, a texture image and a residual image. Advantages of the DG3PD model lie in the properties which are enforced on the cartoon and texture images. The geometric objects in the cartoon image have a very smooth surface and sharp edges. The   texture image yields oscillating patterns on a defined scale which is both smooth and sparse. Recently, the texture images have been applied as a very useful feature for fingerprint segmentation [35][36][37].
We address the challenging task of simultaneous inpainting and denoising in the following way. The advanced DG3PD model introduced in the next section decomposes a noisy input image f (with missing regions D) into cartoon u, texture v and residual components. At the same time, the missing regions D of the cartoon component u are interpolated and the available regions of u are denoised by the advantage of multi-directional bounded variation. This effect benefits from the help of the indicator function in the measurement of the residual, i.e. C{χ c D · × } ∞ in (2.1). However, texture v is not interpolated due to the 'cancelling' effect of this supremum norm for residual in unknown regions. Therefore, the texture component v is inpainted and denoised by a dictionary-based approach instead. The DG3PD decomposition drives noise into the residual component , which is discarded. The reconstruction of the ideal version of f is obtained by summation of the inpainted and denoised cartoon and texture components (see figure 1 for a visual overview). Moreover, we uncover the link between the calculus of variations [38][39][40] and filtering in the Fourier domain [41] by analysing the solution of the convex minimization in equation (2.1), i.e. roughly speaking the solution of the DG3PD inpainting model which can be understood as the response of the lowpass filter LP(ω), highpass filter HP(ω) and bandpass filter BP(ω), and the unity condition is satisfied, i.e.
where ω ∈ [−π , π ] 2 is a coordinator in the Fourier domain. We observe that this decomposition is similar to the wavelet or pyramidal decomposition scheme [42][43][44]. However, the basis elements obtaining the decomposition, i.e. the scaling function and frame (or wavelet-like) function, are constructed by discrete differential operators (due to the discrete setting in minimizing (2.1)), which are referred to as waveletlike operators in [45]. In particular, -the scaling function and wavelet-like function for the cartoon u are from the effect of the multidirectional TV norm, -the scaling function and wavelet-like function to extract the texture v are reconstructed by the effect of the multi-directional G-norm, -the effect of the ∞ norm C{χ c D · × } ∞ is to remove the remaining signal in the known regions of the residual (due to the duality property of ∞ ).
We also describe flowcharts to show that the method of variational calculus (or the DG3PD inpainting) is a closed loop pyramidal decomposition, which is different from an open loop one, e.g. wavelet [46], curvelet [47], see §7. By numerics, we observe that the closed loop filter design by the calculus of variation will result in lowpass, highpass and bandpass filters which are 'unique' for different images ( figure 17). We also analyse the DG3PD inpainting model from the perspective of a Bayesian framework and then define a discrete innovation model for this inverse problem. This paper is organized as follows. In §2, we describe the DG3PD model for image inpainting and denoising. In §3, we show how to compute the solution of the convex minimization in the DG3PD inpainting problem by the augmented Lagrangian method. In §4, we describe the proposed method for texture inpainting and denoising. In §5, we compare the proposed method with existing ones (TVL2 inpainting [48,49]). Moreover, in order to enhance our evaluation in terms of simultaneous inpainting and denoising effects, we also perform a comparison with the NL-means filter for denoising, namely block-matching and three-dimensional filtering (BM3D), in [50]. In §6, we consider our inverse problem from a statistical point of view, i.e. the Bayesian framework, to describe how to select priors for cartoon u and texture v. We analyse the relation between the calculus of variations and the traditional pyramid decomposition scheme, e.g. Gaussian pyramid, in §7. We conclude the study with §8. For more detailed notation and mathematical preliminaries, we refer the reader to [36,51].

Inpainting by DG3PD
We define a method for restoration of the original noisy/non-noisy image f with a set of known missing regions D. The proposed model is a generalized version of DG3PD [35] for the inpainting and denoising problem. This modification for our DG3PD inpainting is inspired by Elad et al. [23].
In particular, the discrete image f (size m × n) with missing regions D is simultaneously decomposed into cartoon u, texture v and noise ; in particular, cartoon u is interpolated on the missing regions due to the indicator function χ D in the modified model.
A set of missing regions D on a bounded domain Ω is defined by the indicator function χ D whose complement is Instead of putting χ c D on the 2 norm of the residual (see eqns (4) and (6) in [23]), we introduce χ c D for the residual in the DG3PD model with the point-wise multiplication operator · × , i.e. C{χ c D · × } ∞ . We propose the DG3PD inpainting as follows: where C is a curvelet transform [47], g s is an unknown variable in the G-norm [52] and a finite difference matrix is

Solution of the DG3PD inpainting model
In this section, we present a numerical algorithm for obtaining the solution of the DG3PD-inpainting model stated in (2.1). To simplify the calculation, we introduce three new variables and denote G * (e/ν) as the indicator function on the feasible convex set A(ν) of (2.1), i.e.
Owing to the constrained minimization problem in (2.1), the augmented Lagrangian method is applied to turn it into an unconstrained one as where the Lagrange function is Similar to [35], the alternating directional method of multipliers is applied to solve the unconstrained minimization problem (3.1). Its minimizer is numerically computed through iterations, t = 1, 2, . . ., and the Lagrange multipliers are updated after every step t. We also initialize In each iteration, we first solve the seven subproblems in the listed order: , 'e-problem', ' -problem', and then we update the five Lagrange multipliers, namely [λ 1l ] L−1 l=0 , [λ 2a ] S−1 a=0 , λ 3 , λ 4 , λ 5 (see algorithm 1). In this section, we only present the solution of the 'e-problem' and ' -problem' as well as the updated Lagrange multiplier λ 5 . We refer readers to [35] for a detailed explanation of the other subproblems and the other Lagrange multipliers.

Choice of parameters
The solution of (3.3) is defined as a projection operator on a convex set The solution of (3.5) with the point-wise division operator and 1 mn is an m × n matrix of ones. Updated Lagrange multiplier λ 5 ∈ X:

Choice of parameters
Owing to an adaptability to specific images and a balance between the smoothing terms and updated terms for the solution of the above subproblems, the selection of parameters (μ 1 , μ 2 , β 1 , β 2 , β 3 ) is described in [35] and β 5 is defined as  Figure 2d illustrates the estimated texture before the substraction in equation (3.4) as where ROI is the region of interest obtained by the morphological operator for the texture image in [37]. In order to analyse the effects of the proposed model in terms of interpolation (for u) and decomposition, we consider a one-dimensional signal (which is extracted along the red line in figure    Because of the directional TV norm, the DG3PD inpainting produces 'sharp' edges without a 'staircase' effect in a cartoon u (e) and the mean values of u remain almost unchanged. Owing to noise and the shrinkage soft-thresholding operator by v 1 in (2.1), texture v is reconstructed with a 'sharp' transition on areas where texture presents in the original signal (c). However, due to 'heavy' noise on the known domains in (e), the DG3PD fails to reconstruct a 'full' texture. In general, this is a challenging problem to recover an oscillating (or weak) signal corrupted by heavy noise. (h) A reconstructed image (without the inpainting texture step) and (i) a quantized signal, i.e. truncated to [0, 255] and quantized. Moreover, small-scale objects, e.g. noise, are removed from v.
However, the term C{χ c D · × } ∞ causes a cancelling effect which removes the interpolated texture in an unknown area during the decomposition process. Therefore, texture inpainting and denoising are tackled separately, as described in the next section, by a generalized version of the dictionary learning for texture synthesis [30].

Texture inpainting and denoising
The proposed method for reconstructing the texture component combines the following ideas in five subsequent steps: -texture extraction by DG3PD [35], -morphological processing [36,37], -texture inpainting inspired by the work of Efros & Leung [30] or [54], -texture denoising by NL means [13], -image synthesis by summation with the cartoon component u.
The DG3PD model enforces sparsity and smoothness on the obtained texture component v. Sparsity means that the vast majority of coefficients in v are equal to zero (the percentage of zero coefficients depends on how much texture a specific image contains). We make good use of this property to answer the following question: which pixels of those that are missing shall be inpainted with texture? First, the texture component v is segmented into zero (grey pixels in figure 4a) and non-zero coefficients (black and white pixels in figure 4a). Morphological processing (as described in [36,37] for fingerprint segmentation) obtains texture regions (figure 4b). The mask of missing pixels D shown in figure 4c is dilated (we used a circular structure element with a radius of 5 pixels to avoid border effects on the margin between existing and missing pixels) and then it is intersected with texture region segmentation to obtain the mask I shown in figure 4d, which shows the pixels to be inpainted with texture.
The proposed texture inpainting proceeds in two phases. First, we build a dictionary of texture patches and, second, we inpaint all pixels of mask I in a specific order. For the dictionary, we select all patches of size s × s pixels (we used s = 15 in our experiments) from texture component v which meet the following two criteria: (i) at least p 1 per cent of the pixels are known (also the central pixel of the patch has to exist) and (ii) at least p 2 per cent of the coefficients are non-zero (we used p 1 = 70% and p 2 = 60% in our experiments). The first criterion excludes patches which contain too many missing (unknown) pixels and the second criterion excludes patches without texture from the dictionary. Next, we iterate over all pixels of mask I and consider the pixel to be inpainted as the central pixel of an image patch with size s × s. We count the number of known pixels inside the patch and compute the percentage of known pixels. Pixels are inpainted in the following order. We start with a threshold percentage t = 90%; this is decreased in steps of 5% per iteration and, in each iteration, we inpaint all pixels of mask I for which at least t per cent are known. The rationale behind this ordering is that the more pixels are known (or already inpainted) in the neighbourhood around a missing pixel, the better it can inpainted, because this additional information improves the chances of finding a good match in the texture dictionary. A third constraint ensures that the overlap of known pixels between a dictionary patch and the patch around the missing pixels contains at least p 3 per cent of known pixels (we used p 3 = 30% in our experiments). Inpainting a pixel means that we find the best fitting patch in our texture dictionary and set the pixel value to the central pixel of that patch. Best fitting is defined as the minimum sum of squared differences per pixel, divided by the number of pixels which overlap.
After all missing texture pixels have been inpainted, the texture region is denoised by n iterations of NL means. In each iteration, we first construct a dictionary considering all patches in the texture region with known and previously inpainted pixels. Next, for each pixel to be denoised, we find the top k fitting patches in the dictionary using the same distance function (sum of squared pixel-wise differences) and set the denoised pixel to the average value of the top k central pixel values. In our experiments, we used k = 5 and we have observed that, after about n = 10 iterations, the image has reached a steady state. See    (e) ( f )  Furthermore, a noise-free, ideal version f 0 depicted in figure 7a is available which serves as the ground truth for evaluating the quality of inpainted and denoised images by different methods. The availability of a noise-free image is an advantage for comparing and evaluating different inpainting and denoising methods. By contrast, images taken by a digital camera contain a certain amount of noise and, for them, an ideal version is not available, see, for example, the Barbara image we used in previous comparisons [35]. The choice of the ideal image (shown in figure 7) enables us to clearly see and understand the advantages and limitations of the compared methods ( figure 6).
Owing to the fidelity term, the L 2 -norm, in the TVL2 inpainting model, noise is reduced and the homogeneous areas are well interpolated. However, the method is known to cause a 'staircase' effect on the homogeneous regions and texture, while small-scale objects tend to be eliminated. If the parameter β is chosen smaller (e.g. β 5 = 0.005), the resulting inpainted image is smoother, i.e. the 'staircase' effect is reduced, but also more texture parts are removed. DG3PD inpainting produces a smooth result without the staircase effect. In comparison with the other approaches, it is the only method which can reconstruct the texture regions in a satisfactory way.
In order to evaluate the proposed model in terms of image denoising, we compared it with BM3D in [50]. This NL-mean filter enhances the sparse representation of an image in a transform domain by grouping similar two-dimensional image blocks into three-dimensional 'groups'. Collaborative filtering is applied for these three-dimensional groups by using three successive steps: three-dimensional transformation of a group, shrinkage of the transform spectrum, and inverse three-dimensional transformation. Finally, aggregation (i.e. an averaging procedure) is applied for different estimates of each pixel due to several overlapped blocks.
BM3D is a state-of-the-art method for image denoising. However, it is not designed for inpainting and, in order to be able to run the BM3D code available at http://www.cs.tut.fi/~foi/GCF-BM3D/, we have to provide input values for the missing pixels; therefore, we In the end, we realized that, while BM3D can separate noise reasonably well from a corrupted image in figure 8a, the resultant image illustrated in figure 8b does not have a good performance. In particular, -from an inpainting aspect, BM3D cannot interpolate mixing areas on homogeneous or textured regions and -from a denoising aspect, reconstructed areas by BM3D are blurred without sharp edges and with a loss of contrast as well as with several artefacts on homogeneous regions.

From the Bayesian framework to the DG3PD inpainting model
Following [55], in this section, we describe the DG3PD inpainting model from the perspective of a Bayesian framework through a maximum a posteriori (MAP) estimator. We assume that u and v are independent random variables (r.v.).
Thus, the distribution of a r.v. U k is a multi-dimensional Laplace distribution with operator ∇ + L : The joint probability density function (p.d.f.) of a prior u is We choose γ = 1 2 . The potential function of u in matrix form is Note that the original u is not a Laplace distribution, but a transform of u under an operator ∇ + L has an independent multi-dimensional Laplace distribution. -Prior of texture image v: Under a transform operator T , the texture image v is decomposed in different orientations. Operator T is suitably chosen to capture the texture components in the original image f. As for the definition of a discrete multi-dimensional G-norm [35], the transform T (in a direction s, s = 0, . . . , S − 1) is defined by its inversion (note that T T −1 = Id): We assume that a texture v is sparse in a transform domain T and sparse in the spatial domain (non-zero coefficients of v are only due to texture). To satisfy these conditions, we assume that a r.v. V k has a mixture of a Laplace distribution in a spatial domain with a p.
To define the distribution of a r.v. V 1k , we define the distribution of the S-dimension r.v. G k = [G 0,k , . . . , G S−1,k ]. Similar to U k , we assume that r.v. G k has a multi-variate Laplace distribution, or Putting (6.2) and (6.3) into (6.1), we have the p.d.f. of r.v. V k as -Likelihood (or the joint p.d.f. of p F|U,V (f | u, v)): If we assume that the residual in an image f has a power exponential distribution [56,57], i.e. E k ∼ PE(μ, σ 2 , ξ ), e.g. Gaussian (ξ = 1) or Laplacian (ξ = 1 2 ), its density function at k ∈ Ω is Owing to the inpainting problem with a missing region D, the likelihood is defined on a known domain Ω\D as -A posteriori: Since u and v are independent of each other, a posteriori is written as However, in practice, we do not know which types of noise are observed in a signal. Instead of the 2ξ norm of the residual on a known domain Ω\D to characterize the properties of noise, we control the smoothness of the solution by the maximum of the curvelet coefficient of on a domain Ω\D by a constant ν, i.e. C{χ c D · × } ∞ ≤ ν.
In [59], if noise in image f is Gaussian, as in extreme value theory, the r.v. C{χ c D · × } ∞ has a Gumbel distribution. Thus, there is a condition to choose ν. However, in practice, if noise cannot be identified, ν is chosen by α-quantile. Note that the condition C{χ c D · × } ∞ ≤ ν is similar to the Dantzig selector [60]. Note that the DG3PD inpainting model (2.1) can be described as an inverse problem in image restoration (e.g. image inpainting and denoising); see figure 10 for the discrete innovation model.   Figure 9. The empirical density functions of the solution in (3.2) by an alternating direction method of multipliers: (a)-(g) are the empirical density functions of u, v, , r 0 , g 0 , w 0 and the QQ-plot of the residual , respectively. We assume that the prior of ∇ + L u has a multi-Laplacian distribution, i.e. a changed variable r = [r l ] L−1 l=0 = ∇ + L u has a multi-Laplace distribution, or r l has a univariate Laplace distribution. This effect causes the sparseness of r l due to the shrinkage operator; see (d). The sparsity for ∇ + L u causes the smoothness of the objects in cartoon u (a). The smooth and sparse texture v (due to the expansion of the whole range in intensity value of its non-zero coefficients caused by v 1 in (2.1)) is illustrated in (b). Because of an additive white Gaussian noise in a simulation, the distribution of in Ω\D is approximately normal; see plot (c) and its QQ-plot (g). The reasons for the differences near the boundary in (g) may be as follows. (1) A simulation of a Gaussian noise, i.e. the tail of a Gaussian noise does not go to infinity in a simulation. (2) The remaining texture in (due to a selection of ν). Changing variable w = g causes a non-sparsity in g and a sparsity in w (which is shrunk from g); c.f. (e) and (f ). Figure 10. Overview of the discrete innovation model for the DG3PD-based image inpainting and denoising. In this study, a blurring operator H is an identity matrix. Note that the L −1 does not exist, in general, but the reconstructed image is obtained by the function space of the inverse problem.

Relation between variational analysis and pyramid decomposition
In this section, we discuss connections and similarities between the proposed DG3PD inpainting model and existing work in the area of pyramid representations, multi-resolution analysis and scalespace representation. Historically, Gaussian pyramids and Laplacian pyramids have been developed for applications such as texture synthesis, image compression and denoising. Early works in this area include a paper by Burt & Adelson [42] applying Laplacian pyramids for image compression. Pyramidal decomposition schemes [42][43][44] can be grouped into two categories. Their main property is either lowpass or bandpass filtering. Closely related are transforms from multi-resolution analysis, including wavelet, curvelet [47,53], contourlet [61] and steerable wavelet [62]. A difference in the proposed DG3PD inpainting model is that its basis elements for obtaining the decomposition are constructed by discrete differential operators and the decomposition is solved in a nonlinear way which adapts to each image (enabled by update iterations, corresponding to the loop in figure 15). The discussion of commonalities and differences is made more precise and detailed in the rest of the section.
Let us denote F ←→ as a discrete Fourier transform pair. Given z 1 = e jω 1 and z 2 = e jω 2 and the Dirac delta function δ(·), the impulse responses of the discrete directional derivative operators (l = 0, . . . , L − 1) and their spectra are -the forward operator: -the backward operator:

The 'u-problem'
In order to describe the variational analysis for the u-problem from the point of view of filtering in the Fourier domain for a pyramid decomposition scheme, we define the discrete inverse Fourier transform of X (z) and Y(z) in the 'u-problem' in [35] as and Y(z) Note that the function X (z) satisfies and is similar to the autocorrelation function in [63], which follows the condition of the Riesz basis. We rewrite cartoon u in the Fourier and spatial domain in the scheme of the filter design as :=u update [k] :=FST ψ,ψ,c 1 (u,λ 1l /β 1 ,1/β 1 ):=u smooth [k] (frame soft thresholding) . (7.1) Given β 1 = c 1 β 4 , the spectra and impulse responses of the scaling and frame functions in (7.1) (l = 0, . . . , L − 1) are and Equation (7.1) is somewhat similar to the projection operator in the pyramid decomposition scheme (see curvelet [47,53], contourlet [61], steerable wavelet [62], etc.) with: -scaling function ϕ(·) and its dualφ(·) with the interpolant φ = ϕ * φ, -frame function ψ l (·) and its dualψ l (·), -proximity operator for frame elements, i.e. Shrink(·, ·).
Note that the frame soft-thresholding FST(·, ·, ·) in (7.1) consists of frame element ψ l , its dualψ l and shrinkage operator Shrink(·, ·) together with the updated Lagrange multiplier. Although FST(·, ·, ·) is similar to the wavelet/curvelet shrinkage operator, this operator is obtained from the calculus of variation.
It is easy to obtain the unity property of frame elements (φ, ψ l ,ψ l ) in the Fourier domain as which satisfies the condition of the perfect reconstruction in the scheme of the wavelet pyramid; see figure 11 for illustration with L = 4 directions. When the number of directions L increases, the bandwidth of the scaling function Φ(z) in the spectral domain reduces ( figure 12). Owing to the effect of this lowpass filter, the cartoon u is smoother. These frame elements (φ, ψ l ,ψ l ), which consist of partial differential operators, can be considered as a kind of a wavelet-like operator [45]. However, the procedure to obtain these elements is from the calculus of variation.

The '[g s ] S−1 s=0 -problem'
In analogy to the u-problem in the variational framework, we define the frame operators to extract the directional features of the texture v in the pyramid decomposition scheme. The discrete inverse Fourier transforms of A(z) and B(z) in the solution of the 'v-problem' [35] are defined as .  Figure 13. The spectrum of the frame functions in the g-problem with S = 8 directions.
Texture v is rewritten in the Fourier and spatial domains with a = 0, . . . , S − 1, as Given β 2 = c 2 β 3 , the spectra and impulse responses of the frame functions in (7.2) are  Figure 14. The effect of a noise measurement C{χ c D · × } ∞ on Ω\D in (2.1) by introducing a new variable e = χ c D · × . The first and second rows are the images e 1 , e 2 and e and their one-dimensional signals along the red lines at iteration 20, respectively. Similarly, the third and fourth rows are at iteration 100. We observe that the cartoon and texture still remain in e 1 and its curvelet smoothing term e 2 at iteration 20. However, these geometrical signals are reduced in their difference e as a result of a subtraction operator. At iteration 100, there is almost no cartoon in e 1 , e 2 and e. Note that there is some information in the missing domain D due to the updated λ 5 . and lowpass highpass system f f (u, v, , g) (u*, v*, *, g*) N-th the wavelet pyramid decomposition the variational calculus decomposition the updated scheme for u, v, and g b4 FST y,y (·, ·, ) Figure 15. The pyramid scheme for the DG3PD inpainting model. Similar to the u-problem, it is easy to obtain the unity property of frame elements (ξ , ψ a , ψ a ) with a ∈ [0, S − 1] in the Fourier domain:

The 'v-problem'
The solution to the v-problem is rewritten as  Figure 16. The three-dimensional filters for the fingerprint image in figure 17a by DG3PD with the same parameters as in figure 18. The linear filters are different and unique (in terms of computation) to obtain a 'good' approximated solution. The scheme of filtering design depends on the characteristics of the image. The variational method will automatically design a filter in a 'nonlinear way' (due to minimization via an iterative method) and adapt to the characteristics of each image. MSE, mean squared error.
Note that there is a shrinkage operator with two terms in texture v, namely a smoothing term and an updated term, balanced by a parameter θ.

The 'e-problem'
The noise term C{χ c D · × } ∞ in equation ( Figure 17. (a-h) Decomposition by DG3PD applied with the same filters as in figure 19 (in a linear convolution) for a similar fingerprint image to that in figure 18. We observe that applying the same linear filter for a 'similar' image cannot give a good result for the decomposition. Thus, the linear filters are 'unique' for every image to obtain a suitable decomposition. We apply DG3PD with an iterative method to obtain the decomposition in (i-o) with the same parameters as in figure 19, and MSE for the reconstructed image (i) is much smaller than that in (b); see figure 16 for its three-dimensional linear filters.

The ' -problem'
At iteration t, we denote the updated residual by the unity condition as   We see that there are two terms in the residual (t) , including the updated term   can be considered as a closed loop of the pyramid scheme with the true solution (u * , v * , * , g * ) in the minimization problem. The loop of the updated Lagrange multipliers can be seen as a refinement. If we remove this loop from the scheme in figure 15a, the minimization in DG3PD becomes the quadratic penalty method. The removal would cause an imperfect reconstruction for the constraints, e.g. u + v + = f; see fig. 7(p) and (q) in [35]. We note that the lowpass, bandpass and highpass filters for the cartoon, texture and residual components are 'custom-made' for each image, resulting in different filters for different images (figures 16 and 17). The directional properties of texture and their directional filter banks are illustrated in figures 18 and 19.

Conclusion
In this paper, we have addressed the very challenging task of restoring images with a large number of missing pixels, whose existing pixels are corrupted by noise and, importantly, the image to be restored contains both cartoon and texture elements. The proposed DG3PD model for inpainting and denoising can cope with this threefold difficult problem. The task of simultaneous inpainting and denoising for cartoon and texture components is solved by DG3PD decomposition, followed by inpainting and denoising both components separately and, finally, image restoration by synthesis of the restored components. More specifically, the DG3PD inpainting model is based on a regularization in the Banach space in a discrete setting which is solved by the augmented Lagrangian method and the alternating direction method of multipliers.
In summary, the decomposition step of the proposed method has two major advantages for tackling this challenging problem. Firstly, the decomposition step denoises simultaneously the cartoon and texture component. Secondly, it allows the cartoon and texture to be handled in different ways. The cartoon image is inpainted by DG3PD, as described in § §2 and 3. Separately, the texture component is inpainted followed by further denoising as described in §4. Therefore, the proposed DG3PD decomposition can also be understood as breaking the full problem down into 'smaller' subproblems (e.g. texture only inpainting) which are easier to solve.
Image restoration (or image denoising and inpainting) can be understood as an inverse problem and it can be described by a discrete innovation model ( figure 10). The assumption in this procedure is that signal has a sparse representation in suitable transform domains. It is known from probability theory that the sparsity (by 1 ) is connected to the Laplace distribution of the signal, which results in the heavy tail distribution. Note that the Laplace distribution with the 1 norm is one of the best approximations of the sparsity by the 0 norm.
Moreover, by choosing the priors and posterior according to the Bayesian framework and an MAP, we can understand the selection of parameters (see figure 9 for the results of the selection of the heavy-tailed distribution, e.g. the Laplace distribution). Note that the MAP requires an assumption on the noise, e.g. Gaussian or Laplacian, to establish a minimization. In our proposed model, there is no requirement for an assumption on noise distribution, e.g. independent identical distributed, or correlated and Gaussian or non-Gaussian (due to the measurement C{·} ∞ , which is similar to the Dantzig selector [60]).