Online RAID5 expansion on a production server: the Background Initialization that outlived the weekend

A production server (running the SAP B1 database) was low on storage. Plan: add one drive to the existing RAID5 array online — a 2.4 TB 10K SAS drive into the free bay, absorbed into the existing virtual disk via the LSI RAID controller.

The operation itself is textbook. This post is about what the textbook leaves out.

Key operational points

1. The new drive appears as JBOD, showing up in Disk Management as an uninitialized disk — resist initializing it in Windows. It must join the RAID5 group through the controller (LSI Storage Authority), becoming part of the existing virtual disk, not a standalone drive;
2. The expansion was scheduled for Friday after hours (17:30) — a decision later proven very right, and still not cautious enough.

What the textbook leaves out: Background Initialization

What the “done over the weekend” plan actually looked like

Once the drive joins the array, the controller starts background initialization / restriping — redistributing data and parity across the new layout. This process:

• Runs for days. Started Friday evening; by Monday morning (68 hours) still not finished;
• Drags array performance down hard. Monday brought user reports: SAP B1 lagging. The database’s random IO and the background restripe were fighting over the same spindles;
• The saving grace: LSI supports pausing/resuming background initialization. Our operating mode became pause during working hours, resume after hours — the expansion turned into a multi-day tug-of-war.

In hindsight the duration was estimable: 2.4 TB drives, a RAID5 restripe, yielding to production IO — two to three days is the normal order of magnitude. The mental model going in was “done over the weekend.”

The four questions I’d ask first next time

1. How long will the restripe take? Estimate from capacity × the controller’s rebuild rate — it decides whether this is a weekend task or a cross-week campaign;
2. Is the performance impact manageable? Confirm the controller supports pausing BGI / tuning rebuild priority, and pre-plan the working-hours strategy instead of improvising when users complain;
3. Is online expansion even the best option? The alternatives — building a new array and migrating, or a one-shot rebuild onto bigger disks — can be faster and lower-risk overall if a downtime window exists;
4. What’s the rollback? For a mid-expansion failure, rollback essentially means restore-from-backup — so verify the backup restores before touching the array.

Lesson

Price storage changes as “restripe time × performance impact”, not “insert drive + a few clicks.”

Users don’t experience your slick console work. They experience every slow query across those three days.