# Calculating Shannon-like entropy function of a 1D signal with random noise

I have been searching for a measure of Shannon's entropy $$\ H$$ or other entropy-like formulae that vary smoothly with noise for real 1D signals. MATLAB has built in functions for image entropy. The ultimate goal is to use that function for denoising with chi-square ($$\chi^2$$) as a constraint. While the objective function and minimization are secondary concerns, the primary challenge lies in dealing with negative values introduced by noise.

The key question that no denoising paper explicitly mentions is: How do researchers in this area deal with negative values due to noise? The reason is that many authors seem to use the raw values of noisy signals in their pseudo-"entropy" calculations for denoising, specifically employing the formula MathOverflow for the details, if interested:

$$H = - \sum_i x_i \log(x_i)$$

where $$\ x_i$$ represents the signal values. However, the actual Shannon entropy formula for a probability distribution $$P$$ is:

$$H(P) = - \sum_i p_i \log(p_i)$$

where $$p_i$$ are the probabilities that must satisfy $$\ p_i \geq 0$$ and $$\sum_i p_i = 1$$.

The true Shannon entropy formula deals with probabilities, and this is typically calculated using the histogram method. However, it seems that that the histogram method can show erratic behavior as a function of noise for discrete data, which raises further concerns about the reliability of entropy calculations under noisy conditions.

Given that noise can introduce negative values in $$x_i$$, how these negative values are handled in entropy-based denoising approaches specifically when people use "raw" values. Specifically, how do researchers ensure that the entropy calculation remains valid and meaningful when the signal values can be negative due to noise? Are there any standard preprocessing steps, transformations, or specific adaptations of the entropy formula used to address the issue of random negative values? How does the chi-square constraint come into play in such scenarios? Are there alternative methods to the histogram approach that mitigate erratic behavior?

• "The reason is that many authors seem to use the raw values of noisy signals in their pseudo-"entropy" calculations for denoising" I've seen that, but only when the distribution of values is actually uniform, and the probability looked at is $X \le x$ or equivalent, so that $x\propto p_x$ holds. (and then they either normalize the result or make it clear it's not actually a Shannon entropy) Are you sure you're not missing constraints made? Commented Jul 22 at 9:18
• "The true Shannon entropy formula deals with probabilities, and this is typically calculated using the histogram method." That's one way of estimating probabilities, and for anything that's not discrete in value and the number of discrete values much much smaller than your number of samples, and the number of histogram bins equal to the number of discrete, a pretty dangerous one, unless you can make extensive statements on the distribution a priori, in which case I'd wonder whether a more efficient PDF estimator exists. Commented Jul 22 at 9:22
• "However, it seems that that the histogram method can show erratic behavior as a function of noise for discrete data, which raises further concerns about the reliability of entropy calculations under noisy conditions." No, it actually becomes better, this is called dithering Commented Jul 22 at 9:22

You're a bit missing the point here: when you use the formula $$H = - \sum_i x_i \log(x_i)$$ instead of $$H(P) = - \sum_i p_i \log(p_i)$$ for anything where $$P(x_i) \ne x_i$$, then that's where your entropy estimate becomes wrong.

The noise (assuming it's stochastically independent from the signal) actually adds precisely its own entropy to the "clean" signal's entropy. That's exactly the motivation for Shannon's noise formula – independent sources contribute entropy linearly.

So, a correct estimator would correctly see the entropy being increased by noise, exactly by the amount of entropy in the noise.

Your proposed one doesn't do that, aside from very strange combinations of signal and noise probability density functions, so it doesn't seem fit for the purpose.

• I understand that (pseudo-)entropy part. It is not Shannon's entropy. This was clear in the question but this formulation is very common in signal processing papers related to spectroscopy. The authors either do not understand it, or they have something in their codes which they never supply in their papers. This is exactly what I am complaining about...same concern as yours.
– ACR
Commented Jul 22 at 13:42
• For example: SNR Enhancement and Deconvolution of Raman Spectra Using a Two-Point Entropy Regularization Method, opg.optica.org/as/abstract.cfm?uri=as-49-4-425 or Near-optimal smoothing using a maximum entropy criterion pubs.acs.org/doi/pdf/10.1021/ac00042a007...wrote to the first author but no response!
– ACR
Commented Jul 22 at 13:45
• as I can't access these articles: what does $x$ describe in these, and is there information on $P(x)$? because, and I fully agree with you there, we can look at a generally wrong formula all day, and notice how it's wrong in multiple ways, if we don't look at under which conditions it's getting used. "very common" isn't going to tell us that! Commented Jul 22 at 15:29
• because, and sorry if I overstress this: the tilte of your question is "calculating Shannon entropy", not "calculating some other function than Shannon entropy". So it's a reasonable assumption that the formula these papers are using is somewhat related to Shannon entropy – we need to figure that out. Commented Jul 22 at 16:08
• These papers are behind paywall, but they do not use probabilities. This is clear for sure. They simply use the raw data. Their $x$ is apparently nothing but raw data. However, it begs big questions, how did they deal with negative values if they were adding random Gaussian news. My best bet is that they used absolute values because the Shannon-like formula with absolute signal value shows the same trend as the histogram method (ignoring the blips) as I showed in the figure. Since this is an optimization, it may not care if it were true Shannon's entropy.
– ACR
Commented Jul 22 at 16:29