Recovering After an Interrupted RAID Rebuild
After an interrupted rebuild, the replacement disk, the original failed member and the surviving members can each hold a different state of the same stripes, and the state can change from one address range to the next. Recovery starts by sorting regions into confirmed rebuilt, confirmed not rebuilt, and unknown, using preserved images and records. A reported progress percentage is not a map of the disk, and a region that was rebuilt has not yet been shown to be correct.
- A rebuild percentage cannot be converted mechanically into the exact address of the last correctly written block.
- Sort regions into confirmed rebuilt, confirmed not rebuilt, and unknown; unknown does not mean zeros, blank or garbage.
- A rebuilt region was calculated from the members the rebuild chose; it can still be wrong if that member set was wrong.
- Parity can reconstruct only as many untrusted blocks per stripe as the level allows, and an old failed member cannot always fill the gaps.
- Continuing, forcing or reinitializing can overwrite alternative evidence; images of every member keep a way back.
First establish which operation was running
“Rebuild” is used loosely for several different operations, and each one leaves members in a different state. Rebuild or recovery writes reconstructed data onto a replacement or spare member. A resync recalculates redundancy across members, for example after an unclean shutdown. A parity check reads and compares data with parity, and a repair-type check can also rewrite parity where they disagree. A reshape changes the geometry itself: the number of devices, the chunk size, the layout or the level. Linux md, for example, names these states separately (recover, resync, check, repair) and tracks a reshape with its own position and old and new geometry values (Linux md admin guide). Other controllers and appliances use their own terms, so confirm what the system actually reported.
If the geometry was changing when the process stopped, the array has two layouts to reconcile, and that is a different problem. Its records and evidence differ from those of a member replacement, so do not apply the replacement model on this page to it. For md, the mdadm documentation says an interrupted chunk-size or layout reshape needs its backup file to be made available again when the array is reassembled (mdadm(8)). If multiple operations overlapped, or a second rebuild started after the first failed, record each one separately.
What an interrupted rebuild can leave behind
A rebuild writes a replacement member’s data progressively, calculated from parity or a mirror across the members it treats as current. If host writes continued while the array was degraded or rebuilding, several generations can exist at once. The original failed member holds an older state. The surviving members hold a newer one. Ranges the rebuild reached hold a reconstructed state, calculated from whatever the other members contained at that moment. Ranges it never reached hold whatever was on the replacement before.
The simplest picture is a single rebuilt prefix followed by an unrebuilt suffix. That is a simplified example, not a universal map of the replacement disk. Several things can complicate it:
- Regions tracked as needing resync, such as those marked in a write-intent bitmap, may be recovered selectively rather than front to back.
- Unreadable areas on a surviving member can be skipped or retried, leaving holes inside an otherwise processed range.
- Host writes during the rebuild can land both ahead of and behind the rebuild position.
- Implementations differ in order, granularity and what they record, and a restarted rebuild may not resume where the first one stopped.
A progress percentage therefore cannot be converted mechanically into the exact address of the last correct block. Even where an implementation records a checkpoint, check its unit and scope. Linux md, for example, documents a per-device recovery_start value as a number of sectors from the start of that device known to be correct, and sync_completed as sectors completed out of the total the current sync action might need to process. A bitmap bit covers a configurable chunk of bytes, not one block. These are device sectors for one specific operation, not array logical addresses or file-system offsets, and they describe md only. A controller’s or NAS appliance’s percentage means whatever that product counts as progress, so confirm it from that model’s documentation.
Three region statuses, not two halves
Instead of deciding between “the replacement is good” and “the replacement is useless”, classify each address range of each member by what the evidence actually supports.
Scroll horizontally to see the full table →
| Status | What supports it | What it does not establish |
|---|---|---|
| Confirmed rebuilt | A checkpoint, bitmap or log with a known unit and scope, plus consistency between the replacement and the stripes recalculated from the other members on preserved images | That the members used for the calculation were the right generation, or that later host writes did not change the stripe again |
| Confirmed not rebuilt | Documented coverage, a checkpoint or a log with clear semantics showing the rebuild never reached the range, corroborated by other evidence | What the correct data for that range is; it has to come from other members, parity or an older source, if any of them can supply it |
| Unknown or conflicting | Missing or ambiguous records, unclear units, skipped or retried areas, or sources that disagree. The replacement’s previous contents also belong here on their own: old bytes can survive a skipped area, a failed write or a mixed state, so they support the idea that a range was not overwritten but do not establish “confirmed not rebuilt” without enough history | Anything about the content. Unknown is not the same as zeros, blank or garbage; the range may hold correct data, stale data or partly written data |
The same classification applies to the other members. A surviving member can have its own unreadable areas, and the original failed member is current only for stripes that were not written after it left the array.
Training scenario: a rebuild stops at a reported 60% and a second member disappears
A replacement member in a parity array reports 60% rebuild progress when a second member drops out. Host writes continued throughout the degraded period and the rebuild. The 60% figure is an illustrative example, not a threshold or a rule.
The 60% does not prove an exact boundary on the replacement, and it does not prove that the rebuilt range was calculated from the correct generation of members. Host writes may have changed stripes on both sides of wherever the rebuild actually was.
Stop further writes, image every member (including the original failed disk, if it is still available), preserve the controller or software logs and metadata, then build and compare candidate views on the images, classifying regions as confirmed rebuilt, confirmed not rebuilt or unknown.
Where a stripe no longer has enough independent, trustworthy blocks, it cannot be reconstructed from parity, whichever candidate is chosen. Some ranges may stay unknown, and the result has to say so.
Rebuilt is not the same as correct
A rebuild answers one question: what should the missing member’s block contain, according to the members the process treated as current at the time it wrote that block? If a stale member was treated as current, or the array was assembled with out-of-date metadata, the rebuilt range can be consistent with the wrong state. A region being rebuilt is a fact about coverage; whether its contents are correct is a separate fact that needs its own evidence.
There is a second distinction above that. Even a correct RAID-level result says nothing about whether the file system, database or virtual machine on top was logically consistent at that moment. Database pages and logs can come from different points in time, virtual-machine files can be merely crash-consistent, a file-system journal replay may still be needed, and an older, stale member can occasionally preserve an older but internally coherent structure that the newest generation does not. Keep RAID consistency, file-system consistency and application consistency as three separate questions; Storage Metadata Generations and Consistency covers how to compare them.
Parity is evidence, and it has limits
Parity is useful: it can reconstruct missing blocks, help test candidate member order and chunk size, flag inconsistent stripes, and check a candidate layout. It cannot establish chronology on its own after degraded writes, a partial rebuild, a stale member being reinserted, or several rebuild attempts. That needs the member and metadata history covered in RAID Metadata, Member State and Foreign Configurations.
Reconstruction is also limited per stripe. RAID 5 can reconstruct one unavailable or untrusted block per stripe, and RAID 6 up to two, but only when the remaining blocks and the parity are themselves correct for the same moment. The Linux md documentation notes, for its own RAID 5 and RAID 6 arrays, that an array which is both dirty and degraded could have undetectable data corruption, because a dirty array’s parity cannot be trusted. When a second member is lost during a rebuild, a stripe may have more unknown blocks than the level can recover. The old failed member does not automatically fill that gap: it is current only for stripes that were not written after it left, and it may have unreadable areas of its own. Treat it as one more candidate source, region by region. RAID Reconstruction Basics sets out what each level needs in general.
What to preserve, and what each record proves
Preserve these before any further rebuild, assembly or repair attempt. Every counter, offset and progress value needs to be recorded with its unit, its scope (which device, which array, which operation) and where it was read from.
Scroll horizontally to see the full table →
| Preserve | Helps establish | Does not prove |
|---|---|---|
| Identity of the original, replacement and spare disks | Which physical disk played which part, using serial numbers and slot or bay positions | Which logical role each disk held at a given moment; bay order can change |
| Member role and data offset | Each disk’s position in the layout and where its data area starts | That the member belongs to the current generation of the array |
| Event and generation metadata | Relative order of members within one array’s own counter domain | A clean commit, a calendar time, or anything comparable across different arrays |
| Rebuild checkpoint, direction or bitmap | Which ranges the process recorded as processed or still pending, in its own units | That every block below a checkpoint is correct, or that unrecorded ranges were never touched |
| Controller, appliance or system logs | The sequence of failures, insertions, rebuild starts and stops, and forced actions | Exact block ranges, unless the log states them; log clocks and wording vary by product |
| Host-write history | Whether data changed during the degraded period and the rebuild, and roughly where | Which stripes were written, unless the record is block-level |
| Member images and acquisition maps | A fixed copy of each member to test candidates on, and which ranges were actually read | That the content read is correct; a clean read can still hold stale data |
How to keep these images, maps and their dependencies apart from working copies is covered in Recovery Copies.
Work with candidate views on preserved images
Build every hypothesis virtually on member images, not on the original disks. The general procedure for assembling and testing a candidate is in RAID Reconstruction Basics. What an interrupted rebuild adds is that the choice of source can change by region:
- Confirm the layout and the state of each member, including which members were current at which point, before choosing any source.
- For each range, test which sources agree: the replacement, the surviving members, parity, and the original failed member where it is still current.
- Mark ranges that stay ambiguous as unknown in the candidate itself, rather than filling them silently.
- Do not splice older and newer blocks into one view without checking that they belong together. A stripe assembled from two generations can look valid to parity and still be semantically wrong for the file system above it.
Comparing several candidate generations, and recording each one consistently, is described in Storage Metadata Generations and Consistency.
Validate across the disputed boundary
Check layout, file-system mapping and important files on both sides of wherever the rebuild boundary is believed to be, and in several separate stripe regions across the volume, not just near the start. Large files that span many stripes are more informative than small ones. A single file that opens, or a run of stripes where parity matches, shows only that those specific areas are consistent; it does not show that the whole volume is recoverable.
Report each result with its scope: which candidate it applies to, which ranges were checked and how, which ranges remain unknown or conflicting, and which files overlap them. A partial result with honest uncertain regions is more useful than a single “recovered” verdict. File-level checks are described in How to Validate Recovered Data.
Before continuing, forcing or reinitializing
Continuing an interrupted rebuild writes calculated data to the replacement member. Force-assembling an array, re-adding a member or reinitializing can change what the members record about their own state. If the members have not been imaged, those writes can overwrite evidence that a different candidate would have needed. For md, the mdadm documentation describes its force-assembly option as assembling the array even if the metadata on some devices appears to be out-of-date, and warns that an array which needs it to start may contain data corruption.
None of this makes every rebuild a mistake. When the member set is confirmed, no other member has failed, there was no ambiguity about which disks were current, and verified backups exist, completing a rebuild can be the ordinary maintenance path. The point is to make that a decision, based on member state, rather than a reflex after an interruption. With verified images of every member, it is possible to return to the state before any continuation and try a different candidate if the first one proves wrong.
Technical references: Linux md admin guide — sync_action states (recover, resync, check, repair), recovery_start and sync_completed in device sectors, bitmap chunk size per bit, reshape_position, and the warning that a dirty, degraded RAID 5/6 array’s parity cannot be trusted · mdadm(8) — force assembly with out-of-date metadata and its corruption warning, bitmap-based re-add, and the backup file needed after an interrupted reshape. Both sources describe Linux md software RAID only; hardware RAID controllers and NAS appliances use their own terms, units and progress reporting, which have to be checked in that specific model’s documentation.
Related: RAID Metadata, Member State and Foreign Configurations · Storage Metadata Generations and Consistency · RAID Reconstruction Basics · Recovery Copies · How to Validate Recovered Data · Failed or Interrupted RAID Rebuild · RAID Array Failure