1
$\begingroup$

I have some very large files that are unsigned 8 bit (0 to 255), real valued. I need to do some simple tuning and decimation. In the past, when I've dealt with low bit rate samples, I've run into issues with artifacts. I want to make sure I get the rounding correct.

The data will be processed in Matlab using all native double precision. The first question is should I subtract 127.5 or 128 when converting, or does it depend?

To keep things simple, I'm not doing anything fancy with the filter gain or multiple passes to make sure I don't clip. Instead I'll notify the user if there was a problem. The output also needs to be the same format, so I can't use any additional effective bits gained by decimation. The next thing I'm unsure about is do I round 255.3 down to 255 or do I consider this an overflow condition?

EDIT

I thought I could just process the data with the offset in there doing something like this at the output.

y = y * scale + (1-scale)/2*255

The trouble is the tuning operation may filter out the offset before I correct for it. So instead, I decided the simplest thing to do is subtract 127.5 and add it back in at the end. Then round as I described above.

I'm not sure how the different methods of adjusting for the offset and scaling effect the resultant artifacts, but as arnfinn said there is probably nothing I can do about in any case (unless I'm willing to use dither).

EDIT 2

This is actually more along the lines of what I was thinking would be an issue. In the distant past I have experienced this exact problem with requantization. Certain combinations of quantization parameters can really mess up quality (worse artifacts). My situation is a little different because I'm processing the data before requantization. But I believe it can still be an issue.

A requantization algorithm for the transcoding of JPEG images

Signal Processing: Image Communication. Volume 21, Issue 1, January 2006, Pages 13–21. Jae Won Moona, Jong Seok Leeb, Nam Ik Cho.

Improve this question
$\endgroup$
  • $\begingroup$ I think you should subtract 127.5... but why would you ever have 255.3? $\endgroup$ – CMDoolittle Dec 1 '16 at 22:26
  • $\begingroup$ sometimes there is DC from the ADC that gets into these old 8-bit unsigned .wav files. i think that 0x80 should be subtracted and you have a maximum negative that is slightly bigger in amplitude than the maximum positive. everything else is just scaling. i'm also wondering where you're getting "255.3" out of an unsigned 8-bit value? converting to 8-bit unsigned, you should scale it to a maximum of $\pm$127, round-to-nearest, and add 0x80. $\endgroup$ – robert bristow-johnson Dec 1 '16 at 23:32
  • 1
    $\begingroup$ or, for better audio quality, dither an/or noise-shape the conversion back to 8-bit. $\endgroup$ – robert bristow-johnson Dec 1 '16 at 23:34
  • $\begingroup$ Don't assume audio. I am familiar with dithering, but I'm not sure it makes sense here. Maybe I'm just worrying about a non issue with the artifacts. We have one vote now for 127.5 and one for 128. Does anyone have a more detailed answer? The reason for potentially having values above 255 is that I won't know the proper scale value ahead of time. That only works if you can load the entire file into memory all at once. $\endgroup$ – Todd Dec 2 '16 at 1:45
  • $\begingroup$ Why do you want to shift the data? Is it to avoid initial transients? Is it from a constant mean process? If you want to remove any transient due to a DC component, you probably want to remove the mean value... $\endgroup$ – Arnfinn Dec 2 '16 at 4:55
1
$\begingroup$

I think you don't have to worry too much about so-called 'transparency' (ability to represent accurately the integer value as a floating point value) unless you do normalization. See the libsndfile FAQ.

As quantization is a lossy operation (many-to-one, or surjective, mapping) you will always have artifacts when re-quantizing from floating point to integer values. The standard method for de-correlating the quantization error is to use dither. For small word lengths, such as 8 bit, the classic solution of adding noise with a triangular distribution might cause the dither noise to be unaccptably high, so you might have to look around for some semi-optimal (in terms of quantization error moments) dithering method that suits your application better. Or just re-quantize and accept the artifacts, they might be small, depending on your data and what you are planning to do with it.

Improve this answer
$\endgroup$
  • $\begingroup$ My question was maybe a little ill-conceived. I accepted your answer for providing useful information regarding the topic. $\endgroup$ – Todd Dec 5 '16 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.