0
$\begingroup$

The spectral entropy (SE) of a signal is a measure of its spectral power distribution. The concept is based on the Shannon entropy, or information entropy, in information theory. The SE treats the signal's normalized power distribution in the frequency domain as a probability distribution, and calculates the Shannon entropy of it.

In general, Shannons' entropy of a signal is in between 0 and 1. 0 means no information content is present in the signal and 1 means otherwise. In communications, if I recall correctly we are uncertain of what the next symbol should be and so we prefer uncertainty as it conveys more information. My question is what if the entropy of the signal (spectral entropy) is say 0.12 i.e., quite small. For example, in a communication channel there is white noise but as soon as somebody speaks, the spectral entropy decreases (please see the example of detecting sine wave from white noise https://www.mathworks.com/help/signal/ref/pentropy.html)

Therefore, the information content of the signal now reduces during the time interval of the duration of the speech signal.

QUESTION: Does the reduced information content of the signal imply that the speech signal contains no information? How come a speech signal has no information or worth? Do we prefer high or low entropy (spectral and Shannon entropy)?

My confusions are:

High entropy --> more information content -- is that preferred or is useless?

Low entropy --> less information content -- Is that preferred?

Please correct me where wrong.

Improve this question
$\endgroup$
1
$\begingroup$

A signal's spectral entropy is not really a measure of information. Rather, it is a statistical measure of dispersion in the signal's power spectrum.

In other words, a random noise signal will tend to have large spectral entropy because its power spectrum is distributed among all frequencies. Speech signals tend to concentrate their power in a small number of bins, and their power spectrums are not flat. This results in lower spectral entropy.

A few comments about entropy:

  • Signals have information; sources have entropy. Signals never have entropy, and any reference to such concept is an abuse of terminology.

  • In communications, we prefer to have sources with large entropy, because then the messages that we transmit will convey a large amount of information. If you have a source with low entropy, then you should apply a source encoder to turn it into a large entropy source.

  • Shannon's entropy deals with uncorrelated sources. Many real-world sources produce correlated signals (such as text or speech) and their entropy is harder to define and calculate.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thank you for your answer & clarifications. I would request you to kindly elaborate how entropy of source is related to information content of the signal; high entropy of source implies high information content but in communication this will mean that we are highly uncertain of what the next symbol is going to be. Shouldn't we reduce uncertainty of the occurrence of next symbol as that would imply less error? This is just a thought which I am scared to ask professors for sounding stupid. $\endgroup$ – SKM Oct 24 '18 at 1:06
  • $\begingroup$ Imagine I say to you: "Russia is the largest country in the world!" You'll reply "Duh! everybody knows that". My message conveyed no information to you. But if I say: "They just discovered a 2,400 years old shipwreck 2 km down at the bottom of the Black Sea!" you'll say "Wow!" because this is totally unexpected: my message carried information (it told you something you didn't know). So, a low-entropy source generates messages that don't surprise anybody. A high-entropy source, on the other hand, generates interesting messages. $\endgroup$ – MBaz Oct 24 '18 at 1:41
  • $\begingroup$ Regarding errors, you're not wrong: the way we fight against errors is with error control codes, which introduce redundancy to the data. So, ECC do reduce the information content of a message, but they do it in a way that makes them very effective: for little cost in information content, you can correct many errors. $\endgroup$ – MBaz Oct 24 '18 at 1:43
  • $\begingroup$ Final comment: talk to your professor during office hours. I'm sure they'll be happy to help you, or at least point you in the right direction. Also, get good textbooks, such as MacKay's "Information theory, inference, and learning algorithms", which is available for free online. $\endgroup$ – MBaz Oct 24 '18 at 1:46
  • $\begingroup$ @MBaz could you please give an example of source encoder that "If you have a source with low entropy, then you should apply a source encoder to turn it into a large entropy source"? Is it kind of data compression e.g. Huffman code, but the entropy of source should not be changed by manipulation? Maybe I misunderstood the terminology. $\endgroup$ – AlexTP Oct 24 '18 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.