Storage Metadata Generations and Consistency

Several copies of storage metadata can all be internally valid and still describe different moments in time — the hard part of recovery is often finding a mutually consistent generation, not finding valid metadata.

One concept, many storage systems

RAID event counters, LVM metadata copies across physical volumes, together with any surviving host-side LVM backup or archive records, Btrfs generation numbers, ZFS transaction groups, an interrupted rebuild’s chronology, snapshot lineage, and application-level journals are all versions of the same underlying idea: recovery is often not “find valid metadata,” but find a mutually consistent metadata generation that describes the desired storage state. That single sentence generalizes across RAID, NAS pooling, copy-on-write file systems and SAN allocation alike — but the generalization is about the idea, not the numbers themselves. An md event count, an LVM sequence number, a Btrfs generation, a ZFS transaction group, a snapshot’s place in its own lineage, and a database journal’s position are each meaningful only within their own system, pool or device identity. A counter from one array, pool or device is not comparable to a counter from a different one as if both shared a universal clock, and none of them is a calendar date. What a higher number actually means, and exactly when it increments, depends on the specific domain — there is no single, universal commit semantics shared across md events, LVM sequences, Btrfs generations, ZFS TXGs and the rest. Within one identity's own counter domain, a higher value is reasonable evidence of more recent activity or ordering, but it is not by itself proof of a clean commit: a candidate with a higher counter can be a cleanly committed state or the product of a partial, interrupted operation, and telling those apart needs specific evidence about that candidate, not just its position in the count.

Why “newest” is not automatically “correct”

Interrupted writes, a partially completed rebuild, and rollbacks can all leave behind a metadata generation that is technically the newest one present while still being the wrong one to trust. See Recovering After an Interrupted RAID Rebuild for a concrete case: after a rebuild is interrupted partway through, a set’s old (pre-rebuild) and new (in-progress) member metadata are both real candidates. Assembling under each hypothesis in turn and then checking several separate LBA regions and a few large files that span many stripes typically leaves some ranges that read consistently under one hypothesis and others that remain genuinely uncertain — that is a map of confirmed and uncertain ranges, not a single yes/no verdict for the whole volume. A rebuild’s reported progress percentage reflects whatever unit that specific controller or software actually counts as progress — which unit that is is implementation-specific, and worth confirming from that vendor's own documentation rather than assumed. Either way, it is not a guarantee that every byte below that mark is now correct, and no single generation number should be assumed to describe every sector of a large volume equally well.

The same caution applies to RAID consistency, file-system consistency and application consistency — three separate questions, and a volume that merely mounts has not actually answered even the first one. Mounting means a file-system parser accepted a given representation of the data; per the Linux kernel’s own software-RAID (md) documentation, that representation is simply whatever block device the array currently presents once assembled — nothing about a later mount re-checks that the stripe geometry, member order or parity were actually correct when that device was built. A reconstruction with the wrong geometry can still mount if the resulting bytes happen to parse as a plausible file system, so a successful mount is evidence that the file-system layer accepted what it was given, not proof of correct RAID geometry, consistent parity, or the integrity of application data above it.

Stronger validation crosses regions and crosses layers

A wrong reconstruction can still show a partition table, mount, display directory listings and open small files without error. Stronger validation checks file-system metadata coherence, checksums, parity consistency, large files that cross many stripes, recognizable database structures, multiple separate LBA regions, snapshot consistency, and known virtual-machine disk signatures — evidence that has to hold across the whole volume and across more than one layer of the stack, not just at the first mount attempt. These checks point at different layers, and it helps not to conflate them: a bad file or a failed checksum shows something is wrong at or below that point, but it does not by itself prove the lower-layer mapping is wrong — the cause could just as easily be upper-layer damage or a single unread sector. In the other direction, parity being internally consistent across a stripe does not prove the historical correctness of the user data it protects: the kernel’s own documentation notes that a dirty, degraded array’s parity specifically cannot be trusted. Even on a clean array, a parity check only shows that the data and parity in the specific stripes examined are mathematically consistent with each other at the moment they were read — it says nothing about when or in what order that consistency was reached, and nothing about whether the data written was the data someone actually wanted. An old, correctly written stripe and a stripe that was later overwritten consistently but wrongly can both pass the identical check.

Scroll horizontally to see the full table →

A practical checklist for comparing generations

• Identify every candidate generation or metadata copy that actually exists, rather than assuming there is only one.
• Note what timestamp, event counter or transaction ID each candidate carries, recorded using the inventory fields above.
• Check whether the candidates agree at the specific layer where they overlap, rather than assuming agreement.
• Treat the mutually consistent set as the working hypothesis — not simply the newest single candidate.
• Re-validate at each layer above it before trusting the chosen generation any further.
• Preserve candidate metadata and any lower-level images rather than working only against the live source; build and test each hypothesis read-only, against copies.
• On unstable media, prioritize acquisition over broad validation — reading every parity stripe or every candidate in full can itself degrade a failing source faster than the check is worth.

None of the above is a recipe for forcing an assembly, import or repair operation to “see what happens.” Each step is meant to run read-only, against preserved copies, before anything is written back to the original candidates. Document whichever hypothesis is ultimately chosen together with the evidence that supports it, its coverage, and any ranges that remain unresolved. If evidence genuinely conflicts between candidates, that conflict is itself the finding — it should not be resolved by repairing one candidate “to see” which is correct, and the result should not be declared fully valid until the conflict is actually explained.

Wrong-layer repair is a recurring destructive pattern

The same mistake shows up across very different systems: running fsck before RAID geometry is proven, running Btrfs repair on a wrongly assembled SHR pool, recreating a QNAP thin volume before the thin allocation metadata is recovered, importing a PERC foreign configuration before the full member set is verified, initializing a host-visible LUN because it merely appears blank, or creating a new VMFS datastore over a mispresented old LUN. In every case, a higher layer was repaired before the lower layer it depends on had actually been validated.

Technical references: mdadm(8) manual page — event count; out-of-date member metadata · LVM: vgcfgrestore(8) — metadata backup and archive copies · OpenZFS: zpool-import(8) — -F and -T transaction-group rewind

Related: RAID Metadata, Member State and Foreign Configurations · Recovering After an Interrupted RAID Rebuild · LUN, Thin Provisioning and Snapshot Recovery · Understanding Storage Layers Before Recovery · Validating Recovered Data · Reconstructing NAS Storage Stacks · RAID Array Failure