I'm implementing 3D DCT quantization as in Quantization of 3D-DCT Coefficients and Scan Order for Video Compression (sorry about the paywall). They propose the following equation for generating the quantization table

equation 16

For Ai=255, Ao=255, Bi=0.04, Bo=0.01 and C=10 that produces the following tables, for which they seem to obtain good quality and compression

enter image description here

This matches the table produced by my code:

import numpy as np

qtable = np.array([[[1.0, 10.9986925, 20.605322, 29.8353, 38.703323, 47.22365, 55.409904, 63.275146],
  [10.9986925, 29.8353, 47.22365, 63.275146, 78.092545, 28.835285, 33.31365, 37.70334],
  [20.605322, 47.22365, 70.831985, 28.835285, 35.51946, 42.006092, 48.301018, 54.40989],
  [29.8353, 63.275146, 28.835285, 37.70334, 46.22365, 54.40989, 62.275146, 69.832],
  [38.703323, 78.092545, 35.51946, 46.22365, 56.4058, 66.09135, 75.30453, 84.06839],
  [47.22365, 28.835285, 42.006092, 54.40989, 66.09135, 77.09253, 87.45305, 97.21023],
  [55.409904, 33.31365, 48.301018, 62.275146, 75.30454, 87.453064, 98.78027, 109.34169],
  [63.275146, 37.70334, 54.40989, 69.832, 84.06839, 97.21023, 109.34169, 120.54043]],
 [[10.9986925, 29.8353, 47.22365, 63.275146, 78.092545, 28.835285, 33.31365, 37.70334],
  [29.8353, 63.275146, 28.835285, 37.70334, 46.22365, 54.40989, 62.275146, 69.832],
  [47.22365, 28.835285, 42.006092, 54.40989, 66.09135, 77.09253, 87.45305, 97.21023],
  [63.275146, 37.70334, 54.40989, 69.832, 84.06839, 97.21023, 109.34169, 120.54043],
  [78.092545, 46.22365, 66.09135, 84.06839, 100.33468, 115.053024, 128.37073, 140.4211],
  [28.835285, 54.40989, 77.09253, 97.21023, 115.053024, 130.87817, 144.91382, 157.3623],
  [33.31365, 62.275146, 87.453064, 109.34169, 128.37074, 144.91383, 159.29565, 171.79866],
  [37.70334, 69.832, 97.21023, 120.54043, 140.4211, 157.3623, 171.79866, 184.10048]],
 [[20.605322, 47.22365, 70.831985, 28.835285, 35.51946, 42.006092, 48.301018, 54.40989],
  [47.22365, 28.835285, 42.006092, 54.40989, 66.09135, 77.09253, 87.45305, 97.21023],
  [70.831985, 42.006092, 60.338223, 77.09253, 92.40482, 106.399185, 119.18909, 130.87817],
  [28.835285, 54.40989, 77.09253, 97.21023, 115.053024, 130.87817, 144.91382, 157.3623],
  [35.51946, 66.09135, 92.40482, 115.053024, 134.54652, 151.32474, 165.76587, 178.19548],
  [42.006092, 77.09253, 106.399185, 130.87817, 151.32474, 168.40314, 182.66823, 194.58342],
  [48.301018, 87.45305, 119.18909, 144.91382, 165.76587, 182.66823, 196.36899, 207.47464],
  [54.40989, 97.21023, 130.87817, 157.3623, 178.19548, 194.58342, 207.47464, 217.61522]],
 [[29.8353, 63.275146, 28.835285, 37.70334, 46.22365, 54.40989, 62.275146, 69.832],
  [63.275146, 37.70334, 54.40989, 69.832, 84.06839, 97.21023, 109.34169, 120.54043],
  [28.835285, 54.40989, 77.09253, 97.21023, 115.053024, 130.87817, 144.91382, 157.3623],
  [37.70334, 69.832, 97.21023, 120.54043, 140.4211, 157.3623, 171.79866, 184.10048],
  [46.22365, 84.06839, 115.053024, 140.4211, 161.19073, 178.19548, 192.11777, 203.51637],
  [54.40989, 97.21023, 130.87817, 157.3623, 178.19548, 194.58342, 207.47464, 217.61522],
  [62.275146, 109.34169, 144.91383, 171.79866, 192.11777, 207.47464, 219.0811, 227.85309],
  [69.832, 120.54043, 157.3623, 184.10048, 203.51637, 217.61522, 227.85309, 235.2873]],
 [[38.703323, 78.092545, 35.51946, 46.22365, 56.4058, 66.09135, 75.30453, 84.06839],
  [78.092545, 46.22365, 66.09135, 84.06839, 100.33468, 115.053024, 128.37073, 140.4211],
  [35.51946, 66.09135, 92.40482, 115.053024, 134.54652, 151.32474, 165.76587, 178.19548],
  [46.22365, 84.06839, 115.053024, 140.4211, 161.19073, 178.19548, 192.11777, 203.51637],
  [56.4058, 100.33468, 134.54652, 161.19073, 181.94125, 198.1018, 210.68764, 220.48949],
  [66.09135, 115.053024, 151.32474, 178.19548, 198.1018, 212.84879, 223.7736, 231.86691],
  [75.30453, 128.37073, 165.76587, 192.11777, 210.68762, 223.7736, 232.99513, 239.49342],
  [84.06839, 140.4211, 178.19548, 203.51637, 220.48949, 231.86691, 239.49342, 244.60564]],
 [[47.22365, 28.835285, 42.006092, 54.40989, 66.09135, 77.09253, 87.45305, 97.21023],
  [28.835285, 54.40989, 77.09253, 97.21023, 115.053024, 130.87817, 144.91382, 157.3623],
  [42.006092, 77.09253, 106.399185, 130.87817, 151.32474, 168.40314, 182.66823, 194.58342],
  [54.40989, 97.21023, 130.87817, 157.3623, 178.19548, 194.58342, 207.47464, 217.61522],
  [66.09135, 115.053024, 151.32474, 178.19548, 198.1018, 212.84879, 223.7736, 231.86691],
  [77.09253, 130.87817, 168.40314, 194.58342, 212.84879, 225.59209, 234.4828, 240.68564],
  [87.45305, 144.91382, 182.66823, 207.47464, 223.7736, 234.4828, 241.51924, 246.1425],
  [97.21023, 157.3623, 194.58342, 217.61522, 231.86691, 240.68564, 246.1425, 249.51913]],
 [[55.409904, 33.31365, 48.301018, 62.275146, 75.30454, 87.453064, 98.78027, 109.34169],
  [33.31365, 62.275146, 87.453064, 109.34169, 128.37074, 144.91383, 159.29565, 171.79866],
  [48.301018, 87.453064, 119.18909, 144.91383, 165.76588, 182.66823, 196.36899, 207.47464],
  [62.275146, 109.34169, 144.91383, 171.79866, 192.11777, 207.47464, 219.0811, 227.85309],
  [75.30454, 128.37074, 165.76587, 192.11777, 210.68764, 223.7736, 232.99513, 239.49342],
  [87.453064, 144.91383, 182.66823, 207.47464, 223.7736, 234.4828, 241.51924, 246.1425],
  [98.78027, 159.29565, 196.36899, 219.0811, 232.99513, 241.51924, 246.74133, 249.94052],
  [109.34169, 171.79866, 207.47464, 227.85309, 239.49342, 246.1425, 249.94052, 252.10999]],
 [[63.275146, 37.70334, 54.40989, 69.832, 84.06839, 97.21023, 109.34169, 120.54043],
  [37.70334, 69.832, 97.21023, 120.54043, 140.4211, 157.3623, 171.79866, 184.10048],
  [54.40989, 97.21023, 130.87817, 157.3623, 178.19548, 194.58342, 207.47464, 217.61522],
  [69.832, 120.54043, 157.3623, 184.10048, 203.51637, 217.61522, 227.85309, 235.2873],
  [84.06839, 140.4211, 178.19548, 203.51637, 220.48949, 231.86691, 239.49342, 244.60564],
  [97.21023, 157.3623, 194.58342, 217.61522, 231.86691, 240.68564, 246.1425, 249.51913],
  [109.34169, 171.79866, 207.47464, 227.85309, 239.49342, 246.1425, 249.94052, 252.10999],
  [120.54043, 184.10048, 217.61522, 235.2873, 244.60564, 249.51913, 252.10999, 253.47612]]])

But applying this table to my DCT produces horrible results, depending on normalization. Between my DCT and inverse DCT, I inserted the following snippet, where cubes contain the DCT coefficients and qtable is as above.

        myqtable = (qtable*64)
        quantcubes = nunpy.round(cubes/myqtable)
        cubes = quantcubes*myqtable
        print(quantcubes.max(), quantcubes.min())

Since my DCT has different normalization than theirs, I'm playing a bit with the scaling of the table. Without the table, just rounding the numbers (15 bits) it looks perfect:

enter image description here

With mytable=512 the range of the coefficients is between 22.0 and -21.0 (6 bits) and the result still looks kind of reasonable, although a bit crummy:

enter image description here

But with myqtable = (qtable*64) the range of quantized coefficients is between 125.0 and -124.0 (8 bits) yet the result is horrible:

enter image description here

And even myqtable = (qtable*8), with a range between 999.0 and -990.0 (11 bits) looks worse than downscaling to 6 bits.

enter image description here

The authors of the paper seem to use 8 bits, which is what's shown in the worst picture above. So clearly something is horribly wrong. I'm kind of losing my mind at this point.

What seems especially odd is the horizontal lines at the 8px boundaries and the overflow to white (output is clipped at 0-255). Nothing about the code or the table suggests dependence about a particular axis. My DCT is implemented as a row-column-frame application of Loefflers algorithm.

Any idea or suggestion is greatly appreciated.

  • Have you done the scan order correctly? Raymond – Raymond K.W. Chan Sep 6 at 17:40

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.