batcave — ZFS Drive Replacement Runbook

Pool: tank (raidz3-0, 12 drives, da0–da11) Last used: 2026-08-12, replacing a dead da11 (Seagate ST2000NXCLAR2000, serial Z1X2MKXE) Scope: this is purely a pool/hardware procedure — no changes needed to zfs-snapshot-prune-fabric.sh/-vanilla.sh, forward-fabric-to-batcave.sh, or any dataset config for a routine drive swap.

Before anything else: raidz3 tolerates up to 3 simultaneous drive failures. One FAULTED drive is not an emergency — you have margin. Work through this calmly; there's no reason to rush a step with a live array.

Quick-reference: this array's layout

• 12 drives, 3 hot-swap trays × 4 bays each.
• No SES/expander on either mpr HBA. The only ses0 device on this box is the onboard AHCI SGPIO enclosure for the boot SSD (ada0) — unrelated to this array. sesutil map/locate will not help identify a bay here.
• The bay fault LED (blue, blinking) does work even without SES — it's driven directly off the backplane. Trust it, but always confirm by serial before pulling a drive (see Step 5).

Step 1 — Confirm the fault

zpool status -v tank

Gotcha: the pool/vdev summary line can show ONLINE even while a device underneath is FAULTED — this is a known OpenZFS 2.2.x display-lag bug, not a sign anything's misconfigured. If the state looks inconsistent with what's listed below it, run:

sudo zpool reopen tank

and check again.

Step 2 — Find which HBA/target the faulted device sits on

sudo camcontrol devlist -v

Cross-reference the faulted daN against the quick-reference table above.

Step 3 — Diagnose before condemning the drive

camcontrol devlist | grep da<N>

• Missing entirely from the list → drive/connection is genuinely gone, skip to Step 4.
• Shows only as (passN), no daN → the target still responds at some level but the disk driver never attached. Worth a closer look:

sudo camcontrol inquiry pass<N>       # confirm identity + get serial number (write it down)
sudo camcontrol tur pass<N> -v        # the real health check

Read the sense key from tur:

• HARDWARE ERROR (e.g. ASC 02/00 — No Seek Complete) = genuine mechanical/hardware fault. Stop here, don't bother with recovery, go to Step 4.
• NOT READY (04h family) = potentially transient. Worth trying:

  sudo camcontrol start pass<N>
  sudo camcontrol tur pass<N> -v
  

  If it comes back ready, try sudo camcontrol rescan <bus>:<target>:0 (scoped to just that target, not the whole bus) and check if daN reattaches. If so, sudo zpool online tank da<N> and watch for a resilver — you may have saved a drive.

Gotchas:

• camcontrol identify will fail on these drives — they're native SAS, not ATA/SATA. Use inquiry/tur instead.
• smartctl -a /dev/da<N> may report Unable to detect device type — this is a known FreeBSD/smartctl quirk with this HBA, not itself evidence of a bad drive. If you want SMART data, find the right -d flag by testing against a known-healthy drive first (e.g. smartctl -a -d sat /dev/da0, or try -d scsi), then apply the same flag to the suspect drive.
• dmesg only holds a limited ring buffer and may have already rolled past the original fault — check dmesg -a (fuller boot log) and grep -i da<N> /var/log/messages too.

Step 4 — Take it offline in the pool

sudo zpool offline tank da<N>

Safe to run even if the device is already gone from the bus.

Step 5 — Physically identify and pull the drive

Go to the machine and look for the blinking blue fault LED on one of the bays.

Before pulling anything, confirm the serial number printed on that bay's caddy matches the serial you got from camcontrol inquiry in Step 3. Don't pull on LED alone if you can't confirm the serial — a mismatch means you're about to pull a healthy drive out of a live array.

If there's no LED, or you can't confirm the serial: fall back to elimination. On each of the other known-good drives, in turn:

sudo dd if=/dev/da<X> of=/dev/null bs=1M count=2000

and watch which bay's activity light responds. Whichever bay never lights up during the full sweep is the faulted one, found without ever guessing.

Step 6 — Swap the drive

Pull the confirmed bad drive, seat a spare in the same bay.

Step 7 — Confirm the new drive

camcontrol devlist

It will very likely re-enumerate at the exact same HBA/target address (same physical bay = same wiring = same target number). Confirm it's genuinely new hardware, not a stale entry:

sudo camcontrol inquiry pass<N>

Check the serial does not match the dead drive's.

Step 8 — Replace it into the pool

sudo zpool replace tank da<N> da<N>

(Substitute the actual device name from Step 7 if it enumerated differently.) Both arguments are given explicitly — since the old device was already offlined/faulted rather than swapped live at the same path, zpool replace requires the new-device argument rather than defaulting it.

Step 9 — Monitor the resilver

zpool status -v tank

Immediately after starting, expect 0.00% done, no estimated completion time — that's normal, ZFS needs some real transfer time before it can compute an ETA. To watch it over time:

while true; do clear; zpool status -v tank; sleep 60; done

Reference data point: 122G resilvered in ~7h45m (May 2026). Resilver time scales with the pool's used space at the time, not drive size — check zpool list tank for current ALLOC before assuming that timing still applies; a pool with much more data will take proportionally longer.

Step 10 — Verify completion

Done when:

• The replacing-11-style entry collapses back to a plain da<N> ONLINE line
• Both the vdev and pool report ONLINE (not DEGRADED)
• The scan line shows resilvered <size> ... with 0 errors

Then get a fresh, non-stale final read:

sudo zpool reopen tank
zpool status -v tank

zpool clear tank if anything still shows stale error counts after that.

The short version, if you just need the command sequence

zpool status -v tank                          # confirm the fault
sudo camcontrol devlist -v                     # find the HBA/target
sudo camcontrol inquiry pass<N>                # get the serial, confirm identity
sudo camcontrol tur pass<N> -v                 # HARDWARE ERROR = real fault, proceed
sudo zpool offline tank da<N>
#  -- physically confirm serial, pull bad drive, seat spare --
camcontrol devlist                             # confirm new drive present
sudo camcontrol inquiry pass<N>                # confirm serial differs from dead drive
sudo zpool replace tank da<N> da<N>
zpool status -v tank                           # watch resilver to completion
sudo zpool reopen tank && zpool status -v tank # final clean check