3
$\begingroup$

I have occasionally come upon the term "shrinkage", mostly in the context of denoising methods.

My rough understanding is that it refers to the part where the real distribution might not be exactly like the theoretical one that's being modeled while still being relatively similar (e.g. mostly Gaussian with some outliers), and to the fact that "shrinkage factors" are introduced/estimated in statistical estimators to counter that and have a "better" estimation (for a given definition of "better" that's probably not always the same depending on your goal).

The Wikipedia page gives out the example of the sample/population variances and hints at the fact that using $\frac{1}{n-1}$ or $\frac{1}{n}$ depends on the shape of the distribution, but it's hard to have a good idea of why I should choose one or the other.

How can I put that more intuitively? Can you point me towards other examples that may help me form a bigger picture of this concept?

$\endgroup$
2
$\begingroup$

In the image processing field shrinkage means we reduce the power of some components of the signal.

The idea is to have a basis which the signal can represented in in a sparse manner.
For instance, if the signal is harmonic, efficient basis would be the Fourier.
So if we look at harmonic signals and noise in Fourier we expect to see dominant component and other components are the noise, which will be shrinked in order to apply Denoising.

This is the idea behind what's known as Sparse Representations.

In the context of Image Denoising you may have a look at Image Denoising with Shrinkage and Redundant Representations.
In the general context you may have a look at the course Sparse Representations in Signal and Image Processing: Fundamentals.

$\endgroup$
9
  • $\begingroup$ If I read that correctly the image processing community and statistics community have two slightly different definitions of shrinkage then? $\endgroup$ – F.X. Dec 9 '20 at 8:30
  • $\begingroup$ You can build a connection between them. It is mostly the operation: We shrink the effect of some component. $\endgroup$ – Royi Dec 9 '20 at 8:32
  • $\begingroup$ That makes sense, a bit. What threw me off was that I was expecting a more concrete mathematical link instead of it being figurative like this ;) $\endgroup$ – F.X. Dec 9 '20 at 8:34
  • $\begingroup$ Many denoising algorithms actually involve an explicit Shrinkage step, maybe that's the mathematical link you're looking for? people.ee.duke.edu/~lcarin/figueiredo.pdf $\endgroup$ – sansuiso Dec 9 '20 at 11:36
  • $\begingroup$ Shrinkage often comes about as the result a gradient descent step when $|x|$ is used in the optimization function. If I recall correctly look at the Quadratic formulations. You have to allow a pseudo gradient, which essentially produces the shrinkage operator. Shrinkage also appears in Stein's Estimators, which are biased estimates but lower MSE. Note - sparse representations introduce a bias in their solutions. $\endgroup$ – David Dec 9 '20 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.