Significance of outer product in structure tensor / second moment matrix

Question

How does one interpret the outer product $\mathbf{M} = \nabla I \otimes \nabla I = \begin{bmatrix}\left(\frac{\partial I}{\partial x}(x)\right)^2 & \frac{\partial I}{\partial x}(x) \cdot \frac{\partial I}{\partial y}(y)\\\frac{\partial I}{\partial y}(y) \cdot \frac{\partial I}{\partial x}(x) & \left(\frac{\partial I}{\partial y}(y)\right)^2\end{bmatrix}$ where $\otimes$ is the outer/tensor product and $I : \mathbb{R}^2 \to \mathbb{R}$ is an image? This matrix is called the structure tensor / second moment matrix.

Background:

The structure tensor tells us how much the pixels change between two image patches (infinitesimal small change $\mathbf{v}$).

\begin{align*} f(v_1, v_2) &= \sum_{x,y \in N} \left(I(x, y) - I(x + v_1, y + v_2)\right)^2\\ \end{align*}

where $N$ is a neighborhood. The first order Taylor approximation is

\begin{align*} I(x + v_1, y + v_2) &= I(x, y) + \mathbf{v}^T\nabla I(x, y) \end{align*}

Then

\begin{align*} f(v_1, v_2) &\approx \sum_{x,y \in N} \left(\mathbf{v}^T\nabla I(x, y)\right)^2\\ &= \sum_{x,y \in N} \left(v_1\frac{\partial f}{\partial x}(x) + v_2\frac{\partial f}{\partial y}(y)\right)^2\\ &= \sum_{x,y \in N} \left(v_1\frac{\partial f}{\partial x}(x)\right)^2 + \left(v_2\frac{\partial f}{\partial y}(y)\right)^2 + 2v_1v_2\frac{\partial f}{\partial y}(y)\frac{\partial f}{\partial x}(x)\\ &= \sum_{x,y \in N} \mathbf{v}^T\mathbf{M}\mathbf{v} \end{align*}

I know this is an ellipsoid and there is a relationship to PCA and the covariance matrix. But I am trying to figure out the significance of computing the outer product of the same gradient. It looks a bit like the Hessian matrix $\mathbf{H}(I(\mathbf{x})) = \mathbf{J}(\nabla I(\mathbf{x})) = \begin{bmatrix}\frac{\partial^2 I}{\partial x^2} & \frac{\partial^2 I}{\partial x \partial y}\\\frac{\partial^2 I}{\partial y\partial x} & \frac{\partial^2 I}{\partial^2 y}\end{bmatrix}$ but without the 2nd derivative.

Answer 1

I have been looking again into this question. The tensor product doesn't really have such a special meaning here.

According to Nonlinear Structure Tensors:

"Although this tensor product contains no more information than the gradient itself, it has the advantage that it can be smoothed without cancellation effects in areas where gradients have opposite signs."

I think it is then just similar to a covariance matrix i.e. $M_{11}, M_{22}$ is the variance of the $x$ and $y$ direction. $M_{12}$ and $M_{21}$ is the covariance of $x,y$ and $y,x$. It is just important to see that we sum each component of the matrix (see Wikipedia). The outer product is then $E[X]E[X]^T$. So we consider an image $I : \Omega \to \mathbb{R}^n$ as a vector field and look for local variations $||dI||^2 = dI^TdI$. The outer product appears because all $4$ directions are compared.

Another explanation comes from diffusion equations (i.e. movement from a region of higher concentration to a region of lower concentration). Without the outer product, we would only look at the gradient at a point. The diffusion is the same in every direction (isotropic). This is called (nonlinear) isotropic diffusion:

$$\frac{\partial u}{\partial t} = \text{div}\left(g\left(|\nabla u|^2\right)\nabla u\right)$$

When $g(\cdot) = 1$ ("linear"), this is the regular heat equation because $\text{div}\left(\nabla u\right) = \nabla^2 u$. The solution would be a Gauss filter, which blurs everything. Wikipedia calls nonlinear isotropic diffusion anisotropic, even though the scalar product $|\nabla u|^2$ is applied (instead of tensor product).

The second possibility is anisotropic diffusion. The PDE is

$$\frac{\partial u}{\partial t} = \text{div}\left(g\left(\nabla u\nabla u^T\right)\nabla u\right)$$

$\nabla u\nabla u^T$ is the structure/diffusion tensor. The diffusion is different in every direction. Diffusion across edges is prevented. When we only look at the diagonal of the matrix, the diffusion is isotropic again. Some more information can be found here and here.