5
$\begingroup$

I solved the Basis Pursuit Denoising Problem looking for a sparse solution (I am in compressive sensing):
$$ {x}^{\ast} = \arg \min_{x} \left\{ \frac{1}{2} {\left\| A x - y \right\|}_{2}^{2} + \lambda {\left\| x \right\|}_{1} \right\} $$

for $ 100 $ different $ \lambda $ values with ADMM.

Now I have $100$ different $x^*$ and want to find "the best one" out of them. Moreover, I solved several optimization problems for CS under noise and noisefree cases (e.g. also Lasso, etc.)

What is the best measure to determine sparsity? Found the paper Comparing Measures of Sparsity by Niall Hurley and Scott Rickard, saying it's the Gini index. Do you agree? In my opinion sparsity in this case is not all, isn't it? It's also about solving the equation as good as possible, e.g. having a small norm on:

$$ {\left\| A {x}^{\ast} - y \right\|}_{1} $$

What do you recommend?

$\endgroup$
1
  • $\begingroup$ A very acute question but I believe it is still an open one. Why would be the meaning of the $\ell 1$ nom on the residuals? $\endgroup$ Commented Aug 20, 2019 at 16:18

2 Answers 2

6
$\begingroup$

I am sorry I cannot comment your answer due to my low reputation. Gini and your suggested sparsity ratio ($l_1(x)/l_2(x)$) both give me the same value for $\lambda$. But

  1. The problem I still see is that I cannot take into account how well the vector is solving the equation $Ax-y$. I would like to combine the residuum $l_1(A\hat{x}-y)$ and the sparsity $l_1(\hat{x})/l_2(\hat{x})$ into a new metric value. Do you know something from the literature?
  2. Gini only works with positive values, but my measurement vector has lots of negative values. In my eyes, this is a heavy disadvantage because many sparse vectors can have negative values.
$\endgroup$
1
  • $\begingroup$ A very acute question. I did some editing, but I believe your question is an open one. Why $\ell_1$ on the residuals? $\endgroup$ Commented May 13, 2017 at 19:03
3
$\begingroup$

Norms like $\ell_p$, $p \ge 1$, or quasi-norms ($0<p< 1$) are all $1$-homogeneous: $\ell_p(\lambda x) = |\lambda|\ell_p( x)$. Which is not the case for the $\ell_0$ count measure, which is scale invariant: a signal and its scaled versions possess the same sparsity index, as they have the same quantity of zero values.

I have been thinking a lot about Hurley's paper in the past years. A sparsity index should be (close to) scale invariant. From an more error perspective, Gini is excellent, but some $\ell_p( x)/\ell_q( x)$ norm ratios are almost as good. From an optimization perspective, Gini remains difficult as a penalty, while some $\ell_p( x)/\ell_q( x)$ ratios of norms are, slowly, becoming tractable. An example, and useful references, are given in Audrey Repetti et al., 2015, IEEE Signal Processing Letters, Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed ℓ1/ℓ2 Regularization.

So I would use both to try to measure sparsity alone.

Having the same $\lambda $ with both Gini and $\ell_1( x)/\ell_2( x)$ is by itself an interesting result. Sparsity of the recovered signal is one thing. The statistics of the residuals is another thing. I do not know of any standard metric combining both of them. Indeed, they don't have the same physical units: any loss quasi-norm

$$\ell_p(A\hat{x}−y)$$

will be $1$-homogeneous with respect to $y$, while the ratio $$\ell_1( \hat{x})/\ell_2( \hat{x})$$ has "no unit".

So for a linear combination, one needs a coupling weight $\omega$:

$$\ell_p(A\hat{x}−y) + \omega\ell_1( \hat{x})/\ell_2( \hat{x})$$

and like in the regression model, how do you chose $\omega$?

I believe that a 2D metric should be better, but then you lose the possibility to have a natural order to compare two outcomes. In a similar context, I am also struggling to find a nice quantitative comparison of scaled-, delayed- and warped-signals.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.