7
$\begingroup$

A client was trying to send me 250 GB worth of files. After attempting various ways of sharing the data, he sent me a zipped folder only 4 GB in size. That sounds like too much compression to me--I don't think when I've zipped things I've ever reduced more than 20% of the size.

What are some typical lossless compression ratios that one would see in practice? (Or alternatively, a range.)

UPDATE: I realise it's impossible to say without being able to guess at the actual information content, so perhaps this was an ill-worded question. I can't share the client's data. But looking at the XML files, there are a lot of repeated phrases e.g.

<thing>
    <property="1" value="2" />
    <property="3" value="4" />
    <property="5" value="6" />
    <property="7" value="8" />
    <property="9" value="10" />
    <property="11" value="12" />
    <property="13" value="14" />
</thing>

which does seem quite compressible.

$\endgroup$
  • 5
    $\begingroup$ I just made a 1 GB file of all null bytes, and it compressed to a 1 MB zip file, for a compression ratio of 900:1. (Compressing it to the phrase 1000000000 null bytes would be a compression ratio of 50,000,000:1.) $\endgroup$ – endolith Sep 27 '12 at 21:05
  • $\begingroup$ The ratio you mentioned 250:4 seems possible, considering the source (text file) you have uploaded here. $\endgroup$ – Fat32 Jan 11 '15 at 15:28
3
$\begingroup$

My survey paper on compression, "A Survey Of Architectural Approaches for Data Compression in Cache and Main Memory Systems", shows that most practical techniques on general benchmarks achieve compression ratio ~2X and some upto 4X, although higher potential (e.g. ~16X in some cases) exists (see Section 2.2). The reason for not achieving full potential is that techniques with higher compression ratio also have higher overhead (e.g. extra hardware which consumes power, lot of extra processing, etc.) or they may not be general enough (e.g. compressing file with all zeros).

$\endgroup$
8
$\begingroup$

It really depends on how much redundancy there is in the data. If all 250GB was just '0', then you could get fabulous levels of compression.

This page shows some results for compression of English text. It compresses 2,988,578 bytes worth of text using various techniques. The top three are: 330,571 (88.94%), 333,759 (88.83%), and 352,722 (88.20%).

Relating this back to your case: that means 250GB would compress (at best) to 27.65GB.

So perhaps there are lots of blank, repetitive patches in the data?


UPDATE:

Based on your edit, here is another paper that looks at XML compression rates. The upshot (Figure 8; screenshot below) is that they appear to see compression rates ranging from 1 (none) to about 50 at a maximum. That suggests that you are correct to be suspicious of your customer's 62.5 to 1 (250 to 4) compression rate.

enter image description here

$\endgroup$
  • $\begingroup$ Great answer Peter and @endolith. I've updated the question based on your responses. Very helpful. $\endgroup$ – isomorphismes Oct 1 '12 at 1:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.