0
$\begingroup$

This might not be quite the right forum, but with access to "consumer grade" memory storage and modern error correction algorithms, how long (order of magnitude) can data be preserved before it is rendered irrecoverable?

To further narrow the problem, assume you are starting with N perfect copies on separate but identical storage media (hard drives, ssd, tape, etc). This seems like a statistical error correction problem, I imagine the math is similar to recovering a signal from N redundant digital (binary) streams with noise of similar, but probably biased distributions (varying with storage medium).

What I'm really trying to get it is how many hard drives I would need, assuming none fail outright, to preserve my data for a lifetime or ten. I don't think its possible to store information indefinitely because entropy always wins in the end...

$\endgroup$
4
  • 1
    $\begingroup$ If your media work in the typical environments designed for them, check Annualized Failure Rate which is usually published by the manufacturers. Otherwise, you need to estimate the rate by yourself. Then, the derivation, which is not necessarily trivial, is straightforward. I will add the complete answer when I have time unless someone else would be eager to provide it. $\endgroup$
    – AlexTP
    Commented Aug 28, 2022 at 8:30
  • 1
    $\begingroup$ Just to be sure to understand your question, what do you mean by "entropy" in the last paragraph, and how relevant is it in this context? $\endgroup$
    – AlexTP
    Commented Aug 28, 2022 at 8:31
  • $\begingroup$ Are you constrained to "set and forget" storage or can you update and move location and storage medium once a decade ? $\endgroup$
    – Hilmar
    Commented Aug 28, 2022 at 15:02
  • $\begingroup$ If we're playing extrema, data radiated into outer space may last past the heat death of the Universe. $\endgroup$ Commented Aug 28, 2022 at 21:37

4 Answers 4

2
$\begingroup$

how long (order of magnitude)

In order of magnitudes: 100 years!

  • one and 10 years can be ruled out, because I've used plenty of CDs that are older. Are we larger than 50 years? Yeah, probably. There's certainly some still-partially-readable cassette tapes from 1972. Not to mention vinyl long play disks. Or microfiche. Or… the number of potential examples of machine-readable storage media is staggering.
  • 1000 years is unrealistic outside of an archeology context, as "reading out" will turn out to be an exercise in understanding primitive approaches to information processing

So, as Fermi estimate, 100 years is the safe bet. If we allow for archaeology-grade effort in retrieval, things look different:

but with access to "consumer grade" memory storage and modern error correction algorithms,

The most important error correction algorithm is still having multiple copies (repetition coding) and using the one without errors. Another state-of-the-art reliability-enhancing technology is using diversity, so that the things that affect multiple copies of the same information are different, so that a total loss of information becomes less likely.

So, put your info on $N$ archival-grade hard drives (these are consumer-grade), good USB flash memory, BluRays, good SSDs, and put them in $N$ different places. In 100 years, dig them up, and see what you can piece together.

This seems like a statistical error correction problem, I imagine the math is similar to recovering a signal from N redundant digital (binary) streams with noise of similar, but probably biased distributions (varying with storage medium).

It is the same problem, but it does incorporate the fact that the errors aren't uncorrelated. For example, if one hard drive fails often, then other hard drives of the same model are likely to have the same problem.

What I'm really trying to get it is how many hard drives I would need, assuming none fail outright, to preserve my data for a lifetime or ten.

Can't tell you. You need to read the hard drive spec sheets, they actually do state mean life times. And then do some modelling on that; for example, assume exponential failure rate (that's a simplistic, but relatively common model for failure rates) that lead to that mean life time, set a probability with which you want to be able to retrieve the data in ten lifetimes, define how long a life time is.

Also, look at non-consumer-grade storage. What sense does it make to buy $N$ cheap hard drives if, say, $\sqrt[5]{N}$ backup tapes and a reader for these would have done the same reliability? It's not like your ancestors will go and be like "a potentially partially damaged hard drive from 900 years ago! How handy, I can totally directly plug this into my PC, which for religious reasons still is a box under my desk, and for inexplicable reason still uses a SATA storage interface". Long-time archival is not in any way an optimization objective of consumer-grade storage, so it's unlikely consumer-grade storage is the cheapest way to achieve that.

I don't think its possible to store information indefinitely because entropy always wins in the end...

Indefinitely is also not relevant, seeing that the sun will end up swallowing the earth, and if civilization manages to leave this star system, then it will definitely not take a bunch of old hard drives with them. If anything, they will collect the data from these ancient hard drives and get rid of the original medium.

Here, the same problem as highlighted above with your "ten lifetimes" arises: Once you get into modelling of things with long timespans, low probabilities and exponents in their distributions, being handwavy about your requirements can quickly make orders of magnitude in difference. "A life time" might well be doable with a single copy on a long-term optimized storage medium. "Ten life times", is that 600 years or 1200 years, incorporating assumptions about the future of humanity and civilization? This might make the difference between, say, 20 copies and 1,000 copies, as well. So, sit down, do the hard, engineering part of writing down specs, collecting datasheet data from the storage media that you have access to. No shortcuts there.

$\endgroup$
2
$\begingroup$

I know it's not considered great to answer with just a link, but the Long Now Foundation is putting a LOT of energy and smart brains into answering the exact question that you're asking. Very much worth following their work (and contributing to it, if you're so inclined)!

$\endgroup$
1
$\begingroup$

Chisel it out in stone and/or incorporate it in the writings of a (successful) religion.

More seriously: short term you need to think about error rates, failure rates, redundancy. Long term you also need to consider what file formats and structures will be available in 200 and 1000 years. Will github still be up then? Will humanity regress to hunter-gatherers?

$\endgroup$
1
$\begingroup$

Print stuff with an OCR readable font on acid free paper. Some research libraries have paper documents nearly 1000 years old, thus providing a proof of longevity capability. Unlike stone cutting or metal etching, this can actually be done at reasonably high speed using cheaply and easily available computer printers.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.