There is a dataset of $N$ elements, each represented by $K$ bits. Now due to hardware limitations to reduce memory for storage, they have to be reprocessed into $K'$ bits each, and $K'<K$. What is the best coding strategy?
$\begingroup$
$\endgroup$
21
-
1$\begingroup$ Do you have any information on the purpose, the distribution of values, etc.? $\endgroup$– Laurent DuvalAug 23, 2022 at 8:58
-
$\begingroup$ To reduce the memory for storage. A distribution may be derived from the N elements of the dataset. $\endgroup$– c1119Aug 23, 2022 at 10:21
-
1$\begingroup$ "Best" induces a notion of "optimality". Do you have a quality criterion in mind? E.g. on the difference between the $K$ and the $K'$ version $\endgroup$– Laurent DuvalAug 23, 2022 at 10:35
-
1$\begingroup$ @c1119 in general, what you want to do is not possible: if you could just take any sequence of bits and compress it, even just a little bit, then you could do the same with the result of the compression, again and again. That would mean you could compress an arbitrary huge amount of data to $K'$ bits, in the end. And that's impossible, because there's infinitely much potential data, but only $2^{K'}$ possible $K'$ long sequences. So, you need to answer Laurent's questions, if you want us to be able to help you. $\endgroup$– Marcus MüllerAug 23, 2022 at 11:36
-
1$\begingroup$ @c1119 have you actually read my comment? $\endgroup$– Marcus MüllerAug 23, 2022 at 14:18
|
Show 16 more comments
1 Answer
$\begingroup$
$\endgroup$
6
"Best" requires you to provide some quality metrics and these are in most cases highly dependent on your specific application.
Generally there are two classes of compression algorithm "lossless" and "lossy". Lossless leverages redundancies in the data and allows full recovery of the original data. Whether this works or not for you, depends on how much redundancy is in the data in the first place and what amount of compression you need.
"lossy" compressor are application specific. Examples are MP3 for audio signals or JPG for pictures. These are trying to minimize the perceptual effects of the data compression. Since audio and visual perception are completely different, so are the algorithms.
-
-
$\begingroup$ In none of the comments you have provided any information about what the data is. The best strategy depends on the type of data and the requirement of your specific application. $\endgroup$– HilmarAug 26, 2022 at 14:12
-
$\begingroup$ Sorry. I added a comment just now. Please see if it is clear enough. $\endgroup$– c1119Aug 27, 2022 at 5:49
-
-
$\begingroup$ I'm sorry to make you confused. Please tell me which statement is not clear. $\endgroup$– c1119Aug 29, 2022 at 2:23