Recovery methodsTechnical

Storage Metadata Generations and Consistency

Several copies of storage metadata can all be internally valid and still describe different moments in time — the hard part of recovery is often finding a mutually consistent generation, not finding valid metadata.

One concept, many storage systems

RAID event counters, LVM’s own metadata history (redundant copies across physical volumes plus historical records in its metadata areas), Btrfs generation numbers, ZFS transaction groups, an interrupted rebuild’s chronology, snapshot lineage, and application-level journals are all versions of the same underlying idea: recovery is often not “find valid metadata,” but find a mutually consistent metadata generation that describes the desired storage state. That single sentence generalizes across RAID, NAS pooling, copy-on-write file systems and SAN allocation alike.

Why “newest” is not automatically “correct”

Interrupted writes, a partially completed rebuild, and rollbacks can all leave behind a metadata generation that is technically the newest one present while still being the wrong one to trust. See Recovering After an Interrupted RAID Rebuild for a concrete case: the “newest” state on a replacement disk can be correct for one address range and simply absent for another. The same caution applies to RAID consistency, file-system consistency and application consistency — they are three separate questions, and a volume that merely mounts has only answered the first, weakest one.

Stronger validation crosses regions and crosses layers

A wrong reconstruction can still show a partition table, mount, display directory listings and open small files without error. Stronger validation checks file-system metadata coherence, checksums, parity consistency, large files that cross many stripes, recognizable database structures, multiple separate LBA regions, snapshot consistency, and known virtual-machine disk signatures — evidence that has to hold across the whole volume and across more than one layer of the stack, not just at the first mount attempt.

A practical checklist for comparing generations

Identify every candidate generation or metadata copy that actually exists, rather than assuming there is only one.
Note what timestamp, event counter or transaction ID each candidate carries.
Check whether the candidates agree at the specific layer where they overlap, rather than assuming agreement.
Treat the mutually consistent set as the working hypothesis — not simply the newest single candidate.
Re-validate at each layer above it before trusting the chosen generation any further.

Wrong-layer repair is a recurring destructive pattern

The same mistake shows up across very different systems: running fsck before RAID geometry is proven, running Btrfs repair on a wrongly assembled SHR pool, recreating a QNAP thin volume before the thin allocation metadata is recovered, importing a PERC foreign configuration before the full member set is verified, initializing a host-visible LUN because it merely appears blank, or creating a new VMFS datastore over a mispresented old LUN. In every case, a higher layer was repaired before the lower layer it depends on had actually been validated.

Central rule

Do not repair a higher layer until the lower layer it depends on has been validated. A mountable candidate is not proof of a correct reconstruction.