0
$\begingroup$

For a given digitised voltage trace, I want to avoid any quantization error introduced via an insufficient precision of the float data type – and at the same time reduce the memory cost. However, I am not sure that I have understood all the concepts correctly. The digitised signal is given as a 32-bit integer which needs to be multiplied by a constant ($12958 \times 10^{-12}$) to get the voltage value...

The signal originates from an ADC with an input range of ±10V and 24-bit resolution, the quantisation step size should be $\frac{20}{2^{24}} \approx 1.19e{-06}$. Therefore it should not be possible to store the resulting (int*const) float value as float32, since the float32 resolution on my machine (using numpy -> np.finfo(np.float32).resolution) is $1e{-06}$.

Does this make sense or am I confusing some concepts?

EDIT: I want to process the signal (e.g. filtering) therefore further use of the integers is not possible. And as I work in Python I will work with floating point integers.
I want to point out, that I do not want to map the integers to floats. As far as I understand the integers are stored for memory efficiency and after multiplication with a constant get the 'real' float signal of the sensor. My concern is, that the floating numbers does lose some information because the step size of the 24-bit integers multiplied with the constant is less than the minimal step size of the float (in that region, accounting for relative error of float). The scaling constant for the integer is $12958 \times 10^{-12}$ (for Volt)- However, since I want to work with µV I usually normalise with $12958 \times 10^{-6}$. The usual range of the signal is about ±200µV.

This is the machine epsilon of my machine for the relevant signal interval. absolute Machine Error in relevant signal range

Does this mean, that it should be 'save' to use the float32 since the step sizes in the relevant signal range are smaller than my scaling constant $12958 \times 10^{-6}$?

I would highly appreciate some literature suggestion to understand the concepts better.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ A 32-bit IEEE-754 float can represent every signed 25-bit integer exactly. If the output of the ADC has 24-bit resolution, that is every value is one of $2^{24}$ values, that can be represented exactly, with no loss of data, by a 32-bit float. $\endgroup$
    – robert bristow-johnson
    Commented Jun 5, 2023 at 18:11
  • $\begingroup$ Thanks! I am not sure if you mean this, but I see the point that it is possible to map any 24-bit integer to a 32-bit float without loss, since the latter has a higher cardinality. However, I now realise that my question is whether the 32-bit float can also hold the integer values transformed by the scaling constant without loss. I am afraid that in some regions the step size of the float is not sufficient to account for this. See the edit for more information. Does your argument still hold? And if so, can you explain/suggest some mathematical/computational intuition/literature? $\endgroup$
    – Helmut
    Commented Jun 6, 2023 at 5:56
  • $\begingroup$ C'mon, if you can save your 24-bit sample value as a 24-bit integer that can be represented exactly in a 32-bit IEEE float, then you can store it without loss. Now, if you choose to store the same fixed-point number, but scaled by a power of 2 so that the range of the number fits a specified range (usually it's from -1 to +1), then the mantissa is exactly the same bits. Nothing is lost. $\endgroup$
    – robert bristow-johnson
    Commented Jun 6, 2023 at 7:28
  • $\begingroup$ hmm ok, although this might disqualify myself. Why is it obvious that the 24bit can be lossless expressed by a float32? I read this at wikipedia as well but it is not that obvious for me. I see the point for everything in the exponent (7bit) and adding the mantisse another bit. How is it for bit 8 to 24? $\endgroup$
    – Helmut
    Commented Jun 6, 2023 at 14:57
  • $\begingroup$ Helmut, it's the number of bits in the mantissa that matter the most. There needs to be enough range in the exponent value (and 8 bits in the exponent easily give us enough range, 5 bits would be enough). But this comes simply from the explicit definition of the value of the floating-point number with 23 mantissa bits, one *"hidden 1" bit, and one sign bit. And from the definition of the 25-bit signed integer using the sign-magnitude format. Write the definitions down side-by-side. $\endgroup$
    – robert bristow-johnson
    Commented Jun 6, 2023 at 16:33

1 Answer 1

Reset to default
2
$\begingroup$

There is a BIG difference between fixed point and floating point format. Fixed point has a constant absolute error, floating point has constant relative error.

A 32-bit floating number has 1 sign bit, 23 mantissa bits and 8 bits of the exponent. For a full scale sine wave the quantization error will be the same as a 24-bit fixed point. Things get more interesting as you decrease the signal level: for fixed point formats the quantization error stays put so the signal to noise ratio decreases rapidly. For floating point formats, the quantization error decreases with the signal and the signal to noise ratio stays constant.

If you ONLY want to store the ADC signal as is, keeping in 24-bit fixed is the most memory efficient way. Perhaps you can add the calibration as a side info. If you want to do any type of processing, you are much better off using 32-bit floating point.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ You missed the "hidden 1-bit". A 32-bit IEEE float is as good as 25-bit fixed for a waveform that goes up to full-scale. And, even for that full-scale waveform, the average quantization error will be even less because of all of the samples that are below 1/2 full scale will have even less quantization error. It's a factor of 4/7 in quantization noise power, if you assume uniform distribution of the samples (which a sine wave isn't, so let's say it's a triangle wave brought up to full scale). $\endgroup$
    – robert bristow-johnson
    Commented Jun 5, 2023 at 20:50
  • $\begingroup$ Thank you for your answer! I edited my question to provide some more information. Can you give me some further explaination of the floating point format part? Why does the quantisation error decrease? And does the SNR stays the same because of the relative error? $\endgroup$
    – Helmut
    Commented Jun 6, 2023 at 6:11
  • $\begingroup$ @Helmut: let's look at a simple example. Let's say you have a full scale sine wave that you want to scaled down by 100dB and then up by 100dB again. If you do this in 16-bit fixed point, you end up with all zeros. The downscaling has wiped out all bits. Doing the same operation in floating point gives you the same signal you started with (as it should be). Does that help ? $\endgroup$
    – Hilmar
    Commented Jun 6, 2023 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.