# What does the $H$ operator exactly do in the deconvolution process and why is it needed?

According to Deblurring Dynamic Scenes via Spatially Varying Recurrent Neural Networks, given a 2D sharp image $$x(m, n)$$ and a blur kernel $$h(k, l)$$, the blurred image is obtained as

$$y(m, n) = (x*h)(m, n) + e(m, n) = \sum_{k=-\infty}^{\infty}\sum_{l=-\infty}^{\infty} h(k,l)x(m-k, n-l) + e(m,n),$$

where $$*$$ is the convolution operator, $$m$$ and $$n$$ are row and column indices of pixel elements, $$e(m, n)$$ denotes the additive noise, and it is assumed that the image extends infinitely in the positive and negative directions, therefore, the boundary effects are ignored.

The deblurred image $$\hat{x}$$ can be obtained by minimizing the loss index including quadratic penalties for the restoration error and for the excessive values in the restored image

$$\hat{x} = \arg \min_x \sum_{k=-\infty}^{\infty}\sum_{l=-\infty}^{\infty} \| y(k,l) - (H* x)(k,l)\|^2 + R(\hat{x}),$$

where $$R(\hat{x})$$ is a regularization function and $$H$$ is a two-dimensional discrete transform function of $$h(k,l)$$:

$$H(\lambda_1, \lambda_2)=\sum_{k=-\infty}^{\infty}\sum_{l=-\infty}^{\infty}h(k,l) \lambda_{1}^{k} \lambda_{2}^{l},$$

where $$\lambda_1$$ and $$\lambda_2$$ denote spatial one-pixel shift operators.

Since in some papers, (e.g. Fast Image Deconvolution using Hyper-Laplacian Priors) the $$H$$ operator is not used in the deconvolution process and instead, they use $$h$$ in the formula directly as the following, my question is: What does the $$H$$ operator exactly do and why is it needed?

$$\hat{x} = \arg \min_x \sum_{k=-\infty}^{\infty}\sum_{l=-\infty}^{\infty} \| y(k,l) - (h* x)(k,l)\|^2 + R(\hat{x}).$$

• Where it says H*x, it is supposed to say Hx, or h*x. Those two are equivalent. H would be a matrix that, when multiplied by the vectorized image, yields the result of the convolution. The matrix multiplication notation makes the inverse problem similar to other inverse problems, and thus easier to solve. Jul 10, 2023 at 15:01

This is all semantics.
At the end, for a linear degradation operation, the problem can be written as:

$$\arg \min_{\boldsymbol{x}} \frac{1}{2} {\left\| \boldsymbol{H} \boldsymbol{x} - \boldsymbol{y} \right\|}_{2}^{2} + \eta R \left( \boldsymbol{x} \right)$$

In case of a convolution we can replace, semantically, the linear operator with the convolution operator. So $$\boldsymbol{H} \boldsymbol{x} \to \boldsymbol{h} \ast \boldsymbol{x}$$.
If we choose t stay in the matrix form for the convolution, the matrix $$\boldsymbol{H}$$ has a special form of a Toeplitz Matrix (Block for 2D signals).

If we assume the operation is convolution with periodic boundary condition (Like Discrete Fourier Transform based convolution) then the matrix becomes a Circulant Matrix.
This matrix can be built using shift operations which is what you called "spatial one-pixel shift operators".

The whole idea is to solve the optimization problem while taking advantage of the structure of the operators.

• @CrisLuengo, is it better now?
– Royi
Jul 10, 2023 at 17:53
• It was already good, I just though some might not pick up on that detail. Wonderful answer! Jul 10, 2023 at 18:00
• @Royi Thanks. I am already familiar with the lexicographical notation, but in the paper the $H$ operator is convolved with the image $x$ and not multiplied with it, (i.e. $H * x$). Do you believe it is a typo? I suspected $H$ may be related to the spatial varying nature of the proposed network. Jul 13, 2023 at 6:24
• @user153245, It is maybe just to show the convolution is with a 2D signal which is a matrix.
– Royi
Jul 13, 2023 at 9:02