# Proper rounding when converting between digitized values and floating point

I have some very large files that are unsigned 8 bit (0 to 255), real valued. I need to do some simple tuning and decimation. In the past, when I've dealt with low bit rate samples, I've run into issues with artifacts. I want to make sure I get the rounding correct.

The data will be processed in Matlab using all native double precision. The first question is should I subtract 127.5 or 128 when converting, or does it depend?

To keep things simple, I'm not doing anything fancy with the filter gain or multiple passes to make sure I don't clip. Instead I'll notify the user if there was a problem. The output also needs to be the same format, so I can't use any additional effective bits gained by decimation. The next thing I'm unsure about is do I round 255.3 down to 255 or do I consider this an overflow condition?

EDIT

I thought I could just process the data with the offset in there doing something like this at the output.

y = y * scale + (1-scale)/2*255


The trouble is the tuning operation may filter out the offset before I correct for it. So instead, I decided the simplest thing to do is subtract 127.5 and add it back in at the end. Then round as I described above.

I'm not sure how the different methods of adjusting for the offset and scaling effect the resultant artifacts, but as arnfinn said there is probably nothing I can do about in any case (unless I'm willing to use dither).

EDIT 2

This is actually more along the lines of what I was thinking would be an issue. In the distant past I have experienced this exact problem with requantization. Certain combinations of quantization parameters can really mess up quality (worse artifacts). My situation is a little different because I'm processing the data before requantization. But I believe it can still be an issue.

Signal Processing: Image Communication. Volume 21, Issue 1, January 2006, Pages 13–21. Jae Won Moona, Jong Seok Leeb, Nam Ik Cho.

• I think you should subtract 127.5... but why would you ever have 255.3? – CMDoolittle Dec 1 '16 at 22:26
• sometimes there is DC from the ADC that gets into these old 8-bit unsigned .wav files. i think that 0x80 should be subtracted and you have a maximum negative that is slightly bigger in amplitude than the maximum positive. everything else is just scaling. i'm also wondering where you're getting "255.3" out of an unsigned 8-bit value? converting to 8-bit unsigned, you should scale it to a maximum of $\pm$127, round-to-nearest, and add 0x80. – robert bristow-johnson Dec 1 '16 at 23:32
• or, for better audio quality, dither an/or noise-shape the conversion back to 8-bit. – robert bristow-johnson Dec 1 '16 at 23:34
• Don't assume audio. I am familiar with dithering, but I'm not sure it makes sense here. Maybe I'm just worrying about a non issue with the artifacts. We have one vote now for 127.5 and one for 128. Does anyone have a more detailed answer? The reason for potentially having values above 255 is that I won't know the proper scale value ahead of time. That only works if you can load the entire file into memory all at once. – Todd Dec 2 '16 at 1:45
• Why do you want to shift the data? Is it to avoid initial transients? Is it from a constant mean process? If you want to remove any transient due to a DC component, you probably want to remove the mean value... – Arnfinn Dec 2 '16 at 4:55

## 1 Answer

I think you don't have to worry too much about so-called 'transparency' (ability to represent accurately the integer value as a floating point value) unless you do normalization. See the libsndfile FAQ.

As quantization is a lossy operation (many-to-one, or surjective, mapping) you will always have artifacts when re-quantizing from floating point to integer values. The standard method for de-correlating the quantization error is to use dither. For small word lengths, such as 8 bit, the classic solution of adding noise with a triangular distribution might cause the dither noise to be unaccptably high, so you might have to look around for some semi-optimal (in terms of quantization error moments) dithering method that suits your application better. Or just re-quantize and accept the artifacts, they might be small, depending on your data and what you are planning to do with it.

• My question was maybe a little ill-conceived. I accepted your answer for providing useful information regarding the topic. – Todd Dec 5 '16 at 19:09