Recovery methodsTechnical

Recovering After an Interrupted RAID Rebuild

A replacement disk that was rebuilding when the array failed again isn’t simply “partly done” — it can be genuinely correct in one address range and genuinely useless in another, and recovery has to treat those ranges separately.

What a rebuild actually writes

A RAID rebuild reconstructs a replacement member’s data from parity or a mirror across the surviving members, writing it progressively rather than all at once. If host writes continued against the array while the rebuild was in progress, the resulting picture can have several generations at once: the original failed disk holds an older state, the surviving members hold a newer state, the already-rebuilt prefix of the replacement disk holds a reconstructed newer state, and the not-yet-rebuilt suffix of the replacement disk holds neither.

Treat the replacement disk as two regions, not one

Picture the replacement disk in two parts: a rebuilt region reflecting the array’s reconstructed, current state, and an unreconstructed region that is old, blank or simply irrelevant. Recovery may need to choose the authoritative source by address range rather than by whole disk — a block from the rebuilt prefix and a block from the original failed member can both be correct, just for different addresses.

Rebuild consistency is not application consistency

A parity rebuild only answers one question: what should the missing member’s block contain, according to the surviving array state? It says nothing about whether the file system, database or virtual machine sitting on top was logically consistent at that moment. Database data pages and logs can be from different points in time, virtual-machine files can be merely crash-consistent, a file-system journal replay may still be required, and an older, stale member can occasionally preserve an older but internally coherent structure that the newest generation does not. Keep RAID consistency, file-system consistency and application consistency as three separate questions.

Parity is evidence, not a substitute for history

Parity is genuinely useful: it can reconstruct missing blocks, help test candidate disk order and stripe size, flag inconsistent regions, and validate a candidate layout. What it cannot do is establish chronology on its own after degraded writes, a partial rebuild, a stale member being reinserted, or multiple rebuild attempts — those require the member and metadata history covered in RAID Metadata, Member State and Foreign Configurations.

What to preserve before doing anything else

Record the rebuild percentage reached, the rebuild direction if it is known, the identity of the original failed member, the replacement or spare disk, every surviving member, relevant timestamps, controller logs, whether host writes continued during the rebuild, and the history of every subsequent rebuild attempt if there was more than one.

Why “just finish the rebuild” is not data recovery

Completing an interrupted rebuild commits the controller’s best current guess permanently. If that guess is wrong for some address range, finishing the rebuild removes the only chance to recover that range a different way.