I have some very large files that are unsigned 8 bit (0 to 255), real valued. I need to do some simple tuning and decimation. In the past, when I've dealt with low bit rate samples, I've run into issues with artifacts. I want to make sure I get the rounding correct.
The data will be processed in Matlab using all native double precision. The first question is should I subtract 127.5 or 128 when converting, or does it depend?
To keep things simple, I'm not doing anything fancy with the filter gain or multiple passes to make sure I don't clip. Instead I'll notify the user if there was a problem. The output also needs to be the same format, so I can't use any additional effective bits gained by decimation. The next thing I'm unsure about is do I round 255.3 down to 255 or do I consider this an overflow condition?
EDIT
I thought I could just process the data with the offset in there doing something like this at the output.
y = y * scale + (1-scale)/2*255
The trouble is the tuning operation may filter out the offset before I correct for it. So instead, I decided the simplest thing to do is subtract 127.5 and add it back in at the end. Then round as I described above.
I'm not sure how the different methods of adjusting for the offset and scaling effect the resultant artifacts, but as arnfinn said there is probably nothing I can do about in any case (unless I'm willing to use dither).
EDIT 2
This is actually more along the lines of what I was thinking would be an issue. In the distant past I have experienced this exact problem with requantization. Certain combinations of quantization parameters can really mess up quality (worse artifacts). My situation is a little different because I'm processing the data before requantization. But I believe it can still be an issue.
A requantization algorithm for the transcoding of JPEG images
Signal Processing: Image Communication. Volume 21, Issue 1, January 2006, Pages 13–21. Jae Won Moona, Jong Seok Leeb, Nam Ik Cho.
0x80
should be subtracted and you have a maximum negative that is slightly bigger in amplitude than the maximum positive. everything else is just scaling. i'm also wondering where you're getting "255.3" out of an unsigned 8-bit value? converting to 8-bit unsigned, you should scale it to a maximum of $\pm$127, round-to-nearest, and add0x80
. $\endgroup$ – robert bristow-johnson Dec 1 '16 at 23:32