Recovering From a Failed ZFS Root Mirror Disk

Hypothetical. Say, for example, you’ve got a ZFS mirror on disk0 and disk1, from which your Nexenta system boots. A year or two later, disk0 crashes, and starts spewing messages such as the following:

 ahci: [ID 860969 kern.warning] WARNING: ahci0: ahci_port_reset port 0 the device hardware has been initialized and the power-up diagnostics failed

Shortly thereafter there’s a storm and the power goes out for several hours and drains your entire humongous UPS, and when the power finally comes back on, the system won’t boot — it stops at the grub prompt; issuing a standard boot command fails. You take out the bad disk, and try to boot with the mirror, but the mirror doesn’t boot in normal mode. You scratch your head, and boot successfully in safe mode.

From there, you update the boot archive —

 # bootadm update-archive
 # touch /reconfigure

allowing it to again boot in normal mode.

Then you insert a replacement disk0, boot again, and install GRUB on the new disk (and make sure the existing disk is up-to-date) —

 # cd /boot/grub
 # installgrub -f -m stage[12] [your-boot-device, e.g., /dev/rdsk/c1t0d0s0]
 # installgrub -f -m stage[12] [your-other-boot-device, e.g., /dev/rdsk/c1t1d0s0]

And tell ZFS the drive is again available for use –

 # zpool replace syspool c1t0d0s0

And then you have a happy hypothetical ZFS root mirror again.

Steve Mokris is a developer at Kosada, Inc.