until today I knew that one bit is a variable, or a space in memory that can hold a value of either One (high) or Zero (low). This is the concept I learned from studying computer programming, microprocessor or DATA bus etc.
But after starting the course on information theory, I found out that bit is expressed as the information content of a symbol in message. This is calculated taking the logarithm (base 2) of the inverse of the probability of occurrence of the symbol.
Are these two concepts same ? On one hand one bit is a variable that can store either zero or one. On the other hand, one bit is the uncertainty associated with one of two symbols with probability of occurrence of 0.5. So, does 1 bit in computer programming or ASCII code mean 1 bit in information content of source or information theory?
A little edit: here is one thing I am finding trouble understanding this topic. See, in data transfer of English alphabets, if we use ASCII code, we basically represent each symbol with 8 bits. Suppose that's 00000000 for a, 00000001 for b etc. So we are essentially allocating 8 quantization levels for each symbol.
But when the information theory comes into play, we take the probability of each symbol into account. 'E' has the highest frequency, where 'Z' has the lowest. So average information content comes down to 3 or 4 bits, right ?
My book says, 'Entropy or average information content is the minimum average number of bits required to represent each sample without distortion'.So, in this case, for efficient data transfer, are we creating maximum four quantization levels for each symbols? Because, on an average they carry information worth 4 bits. If that's so, isn't bit in information theory the same as the one in computer programming, data transfer or ASCII code etc ?
You probably get that I am clearly a noob here :p