← Back to Bad Advice

RAID failure? Do not start a rebuild without checking

A NAS reports a degraded array or a controller marks a member as failed. The advice is immediate: replace a disk and rebuild.

That can be the correct maintenance procedure for a known single-member failure in an otherwise healthy redundant array. It is not a universal response to inaccessible RAID data.

What a rebuild actually does

A rebuild recreates redundancy from the members the system currently considers authoritative. Depending on the RAID layout, it copies mirrored data or reconstructs missing blocks using parity and surviving data.

It does not know which version of your project folder is correct, reverse a deletion or repair every layer above the array. A successful rebuild can leave a damaged file system or the wrong logical contents unchanged.

Degraded, offline and corrupted describe different problems

Scroll horizontally to see the full table →

Reported situationWhat a rebuild might accomplishWhat must be established first
One known member has failed in a redundant array, while the volume remains usableRestore redundancy onto the correct replacementCorrect failed member, suitable surviving members and protection of important data
Too many members are unavailable, or the array is offline after several eventsA normal rebuild may not be possibleWhich members contain the most useful and consistent data
The RAID is online, but a pool or file system is unreadableRebuilding may not address the failed layerWhether array geometry is correct and which higher-level structures are damaged
A previous rebuild stopped partway throughAnother attempt can change the current evidenceProgress, source errors, member history and what was already written

RAID 0 has no redundancy from which to rebuild a lost member. RAID 10 recovery depends on which mirror members are available, not just the total number of failed disks. A generic instruction to “replace the bad disk” ignores these layout-specific conditions.

A healthy disk can still be the wrong source

Consider an illustrative two-disk mirror. Disk A dropped out on Monday. Disk B kept receiving writes until it developed read errors on Friday. Disk A may pass a health check, but it does not contain the same logical state as B.

Treating A as the authoritative source can propagate an older state. Depending on the failure and layout, data from both members may be useful during recovery. A green health indicator does not decide which member is current, and the disk with the newest-looking timestamp is not automatically correct in every address range.

In parity arrays, member selection, stripe geometry and consistency likewise matter. A rebuild cannot infer the intended contents of an inconsistent stripe from its application-level meaning.

The assumptions can be wrong

Before rebuilding, establish which disk failed, whether another member has read errors, and whether the remaining members describe a consistent state. A previously removed disk can be stale even when it is physically healthy. Member state and metadata history help distinguish physical health from logical currency.

Interrupted rebuilds, multiple dropouts, incorrect member selection and controller resets complicate that history. Forcing members online or recreating an array to make the rebuild button available can write metadata or cause reconstruction from an unsuitable member set.

A rebuild also reads extensively from surviving members. A latent read problem may become critical while redundancy is reduced. The fact that the replacement disk is new does not make the surviving source healthy.

What to do instead

  1. Record the current array, pool and volume status. Preserve existing logs and note all replacements, dropouts and rebuild attempts in order.
  2. Identify every member by serial number and bay. Keep removed and failed members; do not erase or repurpose them.
  3. Verify available backups, including sample restores. If a degraded array is still stably readable, prioritize irreplaceable files to storage outside the affected pool.
  4. If access is unstable or there are multiple failures, read errors or an interrupted rebuild, pause ordinary maintenance and obtain a recovery assessment.
  5. Where member-level recovery is needed, preserve suitable images and reconstruct the array virtually. Check member order, geometry and consistency before trusting extracted files.
  6. Remember the layers above RAID: pools, encryption, thin provisioning, LUNs and file systems may need separate reconstruction.

Do not casually pull disks from a running appliance. Hot-swap capabilities and automatic rebuild behavior vary; plan member acquisition with the actual platform in mind.

Rebuild, resync, scrub and file-system repair are not interchangeable

Vendors use different labels, but these operations address different relationships. A rebuild reconstructs a member or missing redundancy. A resynchronization reconciles redundancy after certain inconsistencies or interruptions. A scrub or consistency check examines stored relationships and may repair them according to the platform and mode. File-system repair operates on the structures above that storage.

Do not assume a button labeled “check” is observational, or that an operation is safe because it does not say “format.” Read the platform's description of what it will read and write. Equally, routine scrubbing on healthy, protected storage is not the same situation as starting a new maintenance operation on an already failing array.

What to preserve before changing the configuration

Record the appliance or controller model, software version, RAID level, member serial numbers and bay positions. Preserve logs, configuration exports and encryption recovery material where available. Note whether the disks were moved, a controller was replaced, a foreign configuration was imported or any initialization was attempted.

Preserving history is particularly important after an interrupted rebuild: a replacement can contain reconstructed data in one region and incomplete or irrelevant content elsewhere. It should not be assumed to be a complete substitute for the original member.

Do not clear a foreign configuration, initialize a disk or recreate an array merely to clear an error message. Some procedures can be appropriate in a precisely understood platform-specific situation, but they must not be guessed on the only available member set.

If you already started or retried a rebuild

Record progress and errors without repeatedly refreshing the configuration through reboots. Keep every original and replacement member, with labels identifying its role and timing. Do not erase the “failed” member because the replacement is now installed.

If the volume is readable afterward, copy important data to independent storage and verify representative files, including large files. “Optimal” describes the controller's current view of the array; it does not certify that the recovered application data is correct.

When rebuilding is appropriate

A vendor-supported rebuild is normal when the failure is understood, the correct replacement and member set are identified, the remaining storage is suitable, and important data is protected. Follow the procedure for that platform rather than a generic command sequence.

If a rebuild is already running, neither “always let it finish” nor “cut the power immediately” is a sound universal rule. Record the situation and use platform-specific guidance before interrupting it.

The principle

Restore redundancy after establishing what must be preserved and which members can be trusted.

Sources and further reading

← Back to Bad Advice