Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
4 added 1414 characters in body
source | link

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman. Alternatively, go first from $\Re\{z\}+j\,\Im\{z\}$ to $|z|+e^{j\angle z}$ and compress $|z|$ based on a simple DPCM, and $\angle z$ with an elegantly wrapping linear predictor (reminds me a bit of the title of the '97 paper you cite).

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)


EDIT I remember why I didn't look into FLAC: zstd came up and distracted me.

So: I went ahead and generated a sum of sinusoids and noise: Sum of sines + noise

and saved everything as interleaved signed complex int16.

Then, I did a quick

flac --force-raw-format --channels 2 --bps 16 --sample-rate $((50*1000)) --endian little --sign signed -8 -o samples.flac samples.complex_sc16

(faking a 50 kHz sampling rate... but as far as I can tell, the FLAC coder doesn't care – the sampling rate info is just used to tell potential audio players what sampling rate to use).

The result was a 31% compression over the 191 MB of original data. Not that bad!

Zstd, on noise-ridden data, of course isn't that effective - a mere 5% compression was achieved (which is really meagre, considering that of the 16 bit per I and Q, most of the time, maybe 10 are used, and very few samples even came close to the maximum integer values, indicating that there should be significant runs of 0 and 1 in the bitstream. Anyway, as soon as I switched of noise, the 191 MB were immediately reduced to 12 MB – strength of a dictionary-based compressor on periodic data, especially since zstd uses a "modern PC-sized" dictionary size (and not the "oh, 16kB is soooo much" of gzip). Flac didn't do much better than on the noise-ridden signal.

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman. Alternatively, go first from $\Re\{z\}+j\,\Im\{z\}$ to $|z|+e^{j\angle z}$ and compress $|z|$ based on a simple DPCM, and $\angle z$ with an elegantly wrapping linear predictor (reminds me a bit of the title of the '97 paper you cite).

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman. Alternatively, go first from $\Re\{z\}+j\,\Im\{z\}$ to $|z|+e^{j\angle z}$ and compress $|z|$ based on a simple DPCM, and $\angle z$ with an elegantly wrapping linear predictor (reminds me a bit of the title of the '97 paper you cite).

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)


EDIT I remember why I didn't look into FLAC: zstd came up and distracted me.

So: I went ahead and generated a sum of sinusoids and noise: Sum of sines + noise

and saved everything as interleaved signed complex int16.

Then, I did a quick

flac --force-raw-format --channels 2 --bps 16 --sample-rate $((50*1000)) --endian little --sign signed -8 -o samples.flac samples.complex_sc16

(faking a 50 kHz sampling rate... but as far as I can tell, the FLAC coder doesn't care – the sampling rate info is just used to tell potential audio players what sampling rate to use).

The result was a 31% compression over the 191 MB of original data. Not that bad!

Zstd, on noise-ridden data, of course isn't that effective - a mere 5% compression was achieved (which is really meagre, considering that of the 16 bit per I and Q, most of the time, maybe 10 are used, and very few samples even came close to the maximum integer values, indicating that there should be significant runs of 0 and 1 in the bitstream. Anyway, as soon as I switched of noise, the 191 MB were immediately reduced to 12 MB – strength of a dictionary-based compressor on periodic data, especially since zstd uses a "modern PC-sized" dictionary size (and not the "oh, 16kB is soooo much" of gzip). Flac didn't do much better than on the noise-ridden signal.

3 deleted 2 characters in body
source | link

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman. Alternatively, go first from $\Re\{z\}+j\,\Im\{z\}$ to $|z|+e^{j\angle z}$ and compress $|z|$ based on a simple DPCM, and $\angle z$ with an elegantly wrapping linear predictor (reminds me a bit of the title of the '97 paper you cite).

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman.

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman. Alternatively, go first from $\Re\{z\}+j\,\Im\{z\}$ to $|z|+e^{j\angle z}$ and compress $|z|$ based on a simple DPCM, and $\angle z$ with an elegantly wrapping linear predictor (reminds me a bit of the title of the '97 paper you cite).

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

2 deleted 2 characters in body
source | link

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman.

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of float32int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman.

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of float32 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

Personally, I'd assume that coders that work well on real-valued data would work well on the separated imaginary and real parts too – and after those, another round of trying to compress by exploiting the fact that I and Q very often will have very similar derivatives.

Thus, my first action would be to try a DPCM coder on the separated real and imaginary parts. If that doesn't fit your needs, then do DPCM on the interleaved results, then do the huffman.

In fact, I'm trying to remember all the details, but we had an interesting discussion on the topic on the GNU Radio mailing list not so long ago (SDR users sometimes run into problems like having a hard time finding storage that can write 100MS/s of int16 complexes…). I remember looking at FLAC for the purpose, and then not writing any code. I wonder why that is. Anyway, I'd really try to convince flac to compress your samples as audio (maybe just simply stereo) and see how well linear prediction gets the most of your signal's entropy :)

1
source | link