Skip to main content

Why is RAID synchronization so slow in storage management

Normally, when adding or replacing a disk in RAID1, RAID5, and RAID10, a data synchronization process will be involved, and the process always seems to be slow. Generally, it takes about 3 hours to synchronize 1TB of data, but the synchronization speed may vary depending on system performance and load, and it is particularly strongly correlated with the performance of the disk itself.

Why does RAID5 perform synchronization right after it is created even when there is no data stored in it?

The "sync" or "initialization" performed on a newly created RAID essentially does not refer to "synchronizing data"(since there is no user data on the new hard drives yet), but rather to "establishing redundant check relationships" and "verifying hardware health status".

  1. This is the primary and fundamental reason for making the verification algorithm take effect.

Taking RAID 5 as an example, its data storage logic is as follows: A xor B xor C = Parity P

  • If the hard disk is completely empty, all its bytes are 0.

  • In binary logic, 0 xor 0 xor 0 = 0.

On the surface, the data appears to be correct even without synchronization. However:

The system must actually write the data once to logically establish the relationship of this "check stripe". Only by writing to the entire disk can it be guaranteed that any subsequent read operations can correctly use this redundancy algorithm. If this initialization is not performed, the system will temporarily calculate the parity when the first piece of data is written, which may lead to data corruption in the parity area in the event of a sudden power loss or abnormality.

  1. Iron out the differences between hard disks

Although the hard disk is brand new, there may be extremely minor discrepancies in it as early as the time of leaving the factory:

  • Sector latency: A certain sector on a hard disk reads and writes slightly slower than normal.

  • Bad blocks: Although the probability of bad sectors on a new drive is low, there may be factory defects (issues that have not yet been exposed in the P-list and G-list).

Through full write (synchronization), the RAID system will force access to every single sector of the drives. If a hard disk truly has a physical defect, it will be exposed and trigger a direct Fail error during the initialization phase. This is far better than having the system crash after several months of use when it is already filled with important data.

  1. Clear old "metadata"

If these hard drives were previously used in other devices (such as old computers or other NAS devices), old file system signatures or RAID metadata may remain in the header of the hard drives.

  • Performing synchronization/initialization for a new RAID will overwrite the entire disk with new data, completely erasing all old traces to prevent disk conflicts or recognition errors when the system restarts in the future.

Why is the synchronization process so slow?

  1. Full read and write operations are mandatory

This is the root cause. Take the most common RAID 5 as an example: when you replace a failed hard drive and start the rebuild process, the RAID controller or system is required to perform the following operations:

  • Read: Read the data and checksum of the corresponding stripe on all remaining normal hard disks.

  • Calculation: Infer the data that should be written to the new hard disk through the XOR (exclusive OR) logical operation.

  • Write: Write the calculated data to the new hard drive.

This means that to rebuild a 4TB hard drive, the system may need to read dozens of terabytes of data (depending on the number and configuration of hard drives in the array) and then write 4TB of data. This represents a huge I/O (Input/Output) throughput, which is limited by the read and write speeds of all the hard drives.

  1. The read-write speed bottleneck of hard disk

  • Mechanical Hard Disk Drive: For a traditional HDD, the seek time and sequential read/write speed are the biggest bottlenecks. Especially when the array is in operation, the read/write head needs to move back and forth between "serving user requests" and "serving RAID rebuild", which will further degrade the speed.

  • Performance of new hard drives: If a newly added hard drive is a large-capacity model, its write speed is usually around 150MB/s to 250MB/s. Simply writing 4TB of data takes several hours, and adding the read time will naturally make the whole process even longer.

  1. Resource consumption of parity calculation: Each rebuild of RAID 5 requires XOR parity calculation, which will consume a large amount of CPU resources