Data Archiving

Data archiving moves information out of routine use while keeping it trustworthy, understandable, and retrievable for a defined period. The goal is not to keep every byte forever. The goal is to preserve the right information for as long as a business, legal, historical, or research need requires it.

An archive is therefore more than low-cost storage. Storage holds bits. An archive also keeps the context, controls, and evidence that let you find those bits and use them later.

Use this mental model:

select -> package -> protect -> monitor -> retrieve -> dispose or preserve

Each step answers a different question. What belongs in the archive? What context must travel with it? How will you detect damage? Can an authorized person retrieve and interpret it? What happens when retention ends?

Why organizations archive data

Active systems are optimized for current work. Their data changes often, and users expect fast access. Older information may still have value without needing the same performance.

Organizations archive data to meet several needs:

retain records for an approved period
preserve evidence of decisions and transactions
support audits, investigations, and legal discovery
keep scientific, cultural, or historical material usable
reduce the operational burden on active systems
make long-term datasets available for later analysis

These needs overlap, but they are not interchangeable. A retention rule may require eventual destruction. A preservation mission may require continuing access with no fixed end date. Write the purpose before choosing storage.

Archive, backup, and replication solve different problems

A backup is a recovery copy. It helps restore recent operational state after deletion, corruption, or failure. Backup sets usually rotate, and recovery speed often drives their design.

An archive is a retained body of information. It preserves selected content and its context across a longer lifecycle. Discovery, authenticity, retention, and future readability drive its design.

Replication maintains another copy of current state. It can improve availability, but it may also propagate an unwanted deletion or change.

One platform can provide several of these functions. That does not merge their purposes. An archive still needs retention and preservation controls even when the storage service also offers backup or replication.

Start with selection and retention

Archiving begins with a policy decision, not a transfer job. Identify the data owner, the reason for retention, the start event, the retention period, and the authorized disposition.

A retention schedule connects a class of records to a lifecycle rule. The rule may say to destroy temporary records after a period or transfer permanent records for continuing preservation. NARA uses disposition instructions to distinguish temporary records from permanent records and define what happens to each.

Retaining everything creates risk as well as cost. Unneeded data expands the material that must be secured, searched, migrated, and reviewed. Premature deletion creates a different risk. Apply approved schedules, holds, and policy exceptions before disposition.

A legal hold or other authorized hold suspends normal disposition for affected information. The archive must be able to identify the held set and prevent its deletion until the hold is released.

Package data with enough context

Bytes without context can become unusable even when storage remains healthy. An archive package should connect content to the information needed to manage and interpret it.

Useful metadata commonly includes:

a stable identifier
title, creator, owner, and creation time
record class and retention rule
format and software dependencies
access classification and rights information
checksum values and checksum algorithm
ingest, migration, access, and disposition events
relationships between files, datasets, schemas, and documentation

NARA requires descriptive, structural, and preservation metadata for permanent electronic records. The exact metadata set depends on the archive, but the principle is stable: preserve meaning and management evidence with the content.

For databases, exporting tables alone may be insufficient. You may also need schemas, code lists, data dictionaries, constraints, character encodings, query assumptions, and documentation of how related tables fit together.

Prove integrity with fixity

Fixity is assurance that a digital object has not changed unexpectedly. A checksum gives you a repeatable value derived from the object's bits. Recalculate it later and compare it with the recorded value.

A matching checksum supports the conclusion that the checked bits remain unchanged. A mismatch signals a change or error. A checksum does not explain where the change occurred, prove that the original content was correct, or repair damage by itself.

Establish fixity during ingest. Check it after transfer and at planned intervals. Keep logs of the results. If one copy is damaged, a verified independent copy can support repair.

Deliberate transformation changes the bits too. When you migrate a file to a new format, preserve the old checksum as provenance and establish a new checksum for the new object.

Design independent preservation storage

No storage medium is permanent. Devices fail. Media degrade. Administrators make mistakes. Credentials are compromised. Products and vendors disappear.

A sound archive uses multiple independent copies and separates failure domains. Copies may differ by location, storage technology, administrative authority, or online status. Independence matters more than a slogan about copy count.

The Digital Preservation Handbook recommends geographic redundancy, fixity monitoring, repair from another copy, documentation, and proactive migration away from obsolete storage. An offline copy can also limit the spread of online deletion or malicious change.

Catalog every copy. Record its location, owner, storage class, encryption state, and relationship to the archive object. NIST recommends inventory controls for storage media and protection for sensitive data at rest and in transit.

Protect confidentiality and authenticity

Archived data may remain sensitive long after daily use ends. Apply access control based on role and purpose. Separate archive administration from ordinary production access. Log privileged actions and retrievals where policy requires it.

Encryption can protect confidentiality, but encrypted archives create a long-lived dependency. You must preserve keys, algorithms, permissions, and recovery procedures for as long as the data must remain readable. Key loss can make intact data unusable.

Immutability can prevent alteration or deletion during an enforced period. It reduces some risks, including accidental deletion and malicious change. It does not replace identity controls, independent copies, integrity checks, or authorized disposition.

Preserve readability, not only bits

Bit preservation is necessary, but it is not sufficient. Future users also need software or specifications that can interpret the content.

The Library of Congress evaluates formats using factors such as disclosure, adoption, transparency, self-documentation, external dependencies, patents, and technical protection mechanisms. Open, well-documented formats often provide more preservation options, but format choice must also preserve the significant properties of the content.

Two migrations require different thinking:

storage migration moves unchanged digital objects to a new storage system or medium
format migration transforms content into a different encoding or container

Storage migration should preserve the object's bits and fixity. Format migration intentionally changes them, so validate the result and record the transformation. NARA preservation plans identify significant properties that should remain through format migration.

Sometimes migration is not the right strategy. Emulation can preserve access to software-dependent behavior by recreating an older technical environment. Keeping original software, specifications, and dependency information can widen future options.

Make retrieval a tested capability

An archive that cannot return understandable information has failed its purpose. Test retrieval before relying on it.

A retrieval test should confirm that you can:

find the object by identifier and metadata
authorize the request correctly
retrieve every required component
verify fixity
decrypt and decode the content
interpret it with retained documentation
record the access event

Test representative content, including large packages, complex datasets, encrypted objects, and material on slower storage tiers. Measure the actual time and cost. A vendor's retrieval promise does not prove your catalog, permissions, keys, and procedures work together.

Plan disposition and preservation actions

At the end of retention, do not let data drift into indefinite storage. Confirm that no hold or superseding requirement applies. Then follow the authorized disposition: destroy it, transfer it, or continue preservation under a new decision.

Destruction should cover all managed copies and leave an audit record appropriate to the policy. Transfer should include content, metadata, fixity information, and custody evidence. Continuing preservation should trigger a new risk review rather than an automatic extension.

Run periodic reviews before retention ends too. Check storage health, fixity results, access controls, format risks, vendor viability, key dependencies, retrieval tests, and migration plans.

Limits and decision points

Archiving does not guarantee legal compliance. Retention and privacy duties depend on jurisdiction, contract, record type, and organizational policy. Work with records, legal, privacy, security, and data owners when defining rules.

Archiving also does not replace backup, disaster recovery, or high availability. It protects a selected information history. Other capabilities protect operational continuity.

A mature archive can answer five questions for every object: why do we keep it, who controls it, how do we know it is intact, how will we read it, and what happens next?