======================================= FreeBSD zfs zpool disk management ======================================= Common commands and how to's for - disk management and partitioning - managing zpool mirrors - managing zpool raidz - managing gmirrors - mirroring and growing efiboot, swap, and zfs partitions - adding, removing, replacing (failed) zpool disks ======================================= disk information --------------------------------------- # scan for (new hotplugged) devices camcontrol rescan all # show devices camcontrol devlist # list devices ls -l /dev /dev/gpt # get some information on a disk diskinfo -v ada0 # get some information on megaraid jbod disks mfiutil show drives mfiutil show volumes # get some smart information on a disk smartctl -a ada0 # megaraid jbod disk smartctl -d sat -x pass0 ======================================= gpart commands --------------------------------------- # show current partition information gpart show gpart show -l gpart show -p gpart show -lp ada1 # manually create the disk partitions with labels gpart destroy ada1 gpart create -s GPT ada1 gpart add -t efi -l efi_4 -b 40 -a 4K -s 260M ada1 <-- efi_4 (4 = reference something that never changes (disk_4.vdi or slot 4) because ada1 might change!) gpart add -t freebsd-boot -l gptboot_4 -a 4K -s 512K ada1 gpart add -t freebsd-swap -l freebsd-swap_4 -a 4K -s 4G ada1 gpart add -t freebsd-zfs -l freebsd-zfs_4 -a 4K -s 20G ada1 gpart show -lp ada1 # copy the disk partitions to another disk including their -l labels gpart backup ada0 | gpart restore -l -F ada1 # change the disk partition labels gpart modify -i 1 -l efi_3 ada1 gpart modify -i 2 -l gptboot_3 ada1 gpart modify -i 3 -l freebsd-swap_3 ada1 gpart modify -i 4 -l freebsd-zfs_3 ada1 gpart show -lp ada1 # delete a partition gpart delete -i 4 ada1 # install bootstrap code into partition 2 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada0 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1 ======================================= gmirror commands --------------------------------------- # status gmirror status # list gmirror list # create mirror for efiboot gmirror label -v efiboot ada0p1 ada1p1 # create mirror for swap gmirror label -v -b prefer -F swap ada0p2 ada1p2 # add to mirror gmirror insert -v efiboot ada2p1 gmirror insert -v swap ada2p3 # forget components that are not currently connected gmirror forget -v swap # remove from mirror gmirror remove -v efiboot ada0p1 gmirror remove -v swap ada0p3 # grow gmirror to maximum size of smallest partition after updating/adding/removing disks gmirror resize -v swap # destroy a mirror gmirror destroy -v swap ======================================= zpool commands --------------------------------------- # create zpool stripe (common for vm guest on zfs host) zpool create -f -m /export -o autoexpand=off -O compression=lz4 -O checksum=sha512 -O atime=on tank ada1 # create zpool mirror (if you must) zpool create -f -m /export -o autoexpand=off -O compression=lz4 -O checksum=sha512 -O atime=on tank mirror ada3 ada4 ada5 # create zpool raidz (common for real hardware) zpool create -f -m /export -o autoexpand=off -O compression=lz4 -O checksum=sha512 -O atime=on tank raidz ada3 ada4 ada5 ada6 ada7 zpool create -f -m /export -o autoexpand=off -O compression=lz4 -O checksum=sha512 -O atime=on tank raidz2 ada3 ada4 ada5 ada6 ada7 # destroy a zpool zpool destroy -f tank # get settings zpool get all tank # change settings zpool set autoreplace=on tank # never have auto expand on as it may cause your disk size dependancy to unexpectedly increase to larger sized disks forever! zpool set autoexpand=off zroot zpool set autoexpand=off tank # when boot fails with "press enter to use single user shell" and then everything is read only you can change zfs to be RW and start fixing things zfs readonly=off zroot/ROOT/default # set a zpool bootable zpool set bootfs=zroot/ROOT/default zroot # any disk used in a zpool will have a label so before the disk is reintroduced to a system you should clear the old label zpool labelclear -f ada0 zpool labelclear -f ada5 # check usage zpool iostat # check history zpool history # check status zpool status -v zpool list -v # zpool replace a disk zpool replace zroot ada0p4 ada2p4 zpool replace tank ada3 ada9 # when the output of zpool status disks don't make sense for replacing zpool status -g zpool replace zroot 12345628799847789273 ada2p4 zpool replace tank 238429089676757789 ada9 # add another disk to a zpool mirror zpool attach zroot ada0p4 ada1p4 zpool attach zroot ada1p4 ada2p4 # remove a disk from a zpool mirror zpool detach zroot ada2p4 zpool detach zroot ada1p4 # start a zpool scrub zpool scrub zroot # upgrading zpool zfs zpool upgrade zpool upgrade -v zpool upgrade -a ======================================= zpool mirror management --------------------------------------- # remove any zpool label before using a disk zpool labelclear -f ada0 zpool labelclear -f ada1 # attach additional disk to mirror <-- partition (size must be >= original) zpool attach zroot ada0p4 ada1p4 # detach disk from mirror zpool detach zroot ada1p4 # replace disk with new disk in same slot zpool replace zroot ada1p4 zpool replace zroot ada1p4 ada1p4 # replace disk with new disk in different slot zpool replace zroot ada1p4 ada3p4 # take disk offline to do disk replacement in same slot zpool offline zroot ada0p4 # bring disk back online (do not grow) zpool online zroot ada0p4 # grow zpool to new size <-- IF YOU DO -e YOU WILL NEVER BE ABLE TO USE A SMALLER ORIGINAL SIZE DISK without the help of zfs send/recv zpool online -e zroot ada0p4 zpool online -e zroot ada1p4 # clear zpool condition zpool clear zroot ======================================= zpool raidz management --------------------------------------- # remove any zpool label before using a disk zpool labelclear -f ada2 # replace disk with new disk in same slot <-- new disk (size must be >= original) zpool replace tank ada2 zpool replace tank ada2 ada2 # replace disk with new disk in different slot <-- new disk (size must be >= original) zpool replace tank ada2 ada3 # take disk offline to do disk replacement in same slot zpool offline tank ada2 # bring disk back online (do not grow) zpool online tank ada2 # grow zpool to new size <-- IF YOU DO -e YOU WILL NEVER BE ABLE TO USE A SMALLER ORIGINAL SIZE DISK without the help of zfs send/recv zpool online -e tank ada2 zpool online -e tank ada3 zpool online -e tank ada4 # clear zpool condition zpool clear tank ======================================= how to fix efi boot partition ada1p1 using gmirror --------------------------------------- # partition disk ada1 first and then... gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada1 mount | grep boot umount /boot/efi gmirror label -v efiboot ada0p1 ada1p1 ls -l /dev/mirror newfs_msdos /dev/mirror/efiboot mkdir /tmp/mirror mount_msdosfs /dev/mirror/efiboot /tmp/mirror mkdir -p /tmp/mirror/efi/boot cp /boot/loader.efi /tmp/mirror/efi/boot/bootx64.efi umount /tmp/mirror ed /etc/fstab /dev/mirror/efiboot /boot/efi msdosfs rw 2 2 ======================================= how to grow zpool zroot mirror or replace failed mirror disk --------------------------------------- # attach new disk(s) camcontrol rescan all camcontrol devlist # create disk geom for all disks gpart create -s GPT ada9 gpart add -t efi -l efiboot9 -b 40 -a 4K -s 260M ada9 gpart add -t freebsd-boot -l gptboot9 -a 4K -s 512K ada9 gpart add -t freebsd-swap -l swap9 -a 4K -s 8G ada9 <-- resize swap size if needed gpart add -t freebsd-zfs -l zfs9 -a 4K ada9 <-- use all remaining free space (size must be >= original) gpart show -lp ada9 # remove old zfs label before using it zpool labelclear -f ada9 # replace all disks one at a time after each finishes resilvering zpool replace zroot ada0p4 ada9p4 # grow zpool to new size <-- Remember ONCE THE ZPOOL HAS GROWN -e IT CAN NOT GO BACK TO SMALLER DISKS without the help of zfs send/recv zpool online -e zroot ada9p4 zpool online -e zroot ada10p4 zpool status zroot # add new disk to efiboot mirror gmirror insert -v efiboot ada9p1 gmirror insert -v efiboot ada10p1 # remove old disk from efiboot mirror gmirror remove -v efiboot ada0p1 gmirror remove -v efiboot ada1p1 # add new disk to swap mirror gmirror insert -v swap ada9p3 gmirror insert -v swap ada10p3 # remove old disk from swap mirror gmirror remove -v swap ada0p3 gmirror remove -v swap ada1p3 # grow swap gmirror to maximum size after changing disks gmirror resize -v swap # use new swap swapinfo -h swapoff -a swapon /dev/mirror/swap ======================================= how to grow zpool raidz or replace failed raidz disk --------------------------------------- # attach new disk(s) camcontrol rescan all camcontrol devlist # replace all disks in the zpool one at a time after each finishes resilvering <-- new disk (size must be >= original) zpool replace tank ada3 ada6 # expand each disk in the zpool to use the new size <-- Remember ONCE THE ZPOOL HAS GROWN -e IT CAN NOT GO BACK TO SMALLER DISKS without the help of zfs send/recv zpool online -e tank ada6 zpool online -e tank ada7 zpool online -e tank ada8 zpool status tank ======================================= enabling zfs --------------------------------------- # enable zfs ed /boot/loader.conf zfs_load="YES" ed /etc/rc.conf zfs_enable="YES" service zfs start ======================================= enabling gmirror --------------------------------------- # enable gmirror ed /boot/loader.conf geom_mirror_load="YES" kldload geom_mirror kldstat | grep geom_mirror ======================================= references --------------------------------------- ed, red(1) - text editor camcontrol(8) - CAM control program diskinfo(8) - get information about disk device smartctl(8) - Control and Monitor Utility for SMART Disks gpart(8) - control utility for the disk partitioning GEOM class gmirror(8) - control utility for mirrored devices pstat, swapinfo(8) - display system data structures swapon, swapctl, swapoff(8) - specify devices for paging and swapping kldload(8) - load a file into the kernel loader.conf(5) - system bootstrap configuration information whatis zpool zpool-features(7) - description of ZFS pool features zpool(8) - configure ZFS storage pools zpool-add(8) - add vdevs to ZFS storage pool zpool-attach(8) - attach new device to existing ZFS vdev zpool-checkpoint(8) - check-point current ZFS storage pool state zpool-clear(8) - clear device errors in ZFS storage pool zpool-create(8) - create ZFS storage pool zpool-destroy(8) - destroy ZFS storage pool zpool-detach(8) - detach device from ZFS mirror zpool-events(8) - list recent events generated by kernel zpool-export(8) - export ZFS storage pools zpool-get, zpool-set(8) - retrieve properties of ZFS storage pools zpool-history(8) - inspect command history of ZFS storage pools zpool-import(8) - import ZFS storage pools or list available pools zpool-initialize(8) - write to unallocated regions of ZFS storage pool zpool-iostat(8) - display logical I/O statistics for ZFS storage pools zpool-labelclear(8) - remove ZFS label information from device zpool-list(8) - list information about ZFS storage pools zpool-offline, zpool-online(8) - take physical devices offline in ZFS storage pool zpool-reguid(8) - generate new unique identifier for ZFS storage pool zpool-remove(8) - remove devices from ZFS storage pool zpool-reopen(8) - reopen vdevs associated with ZFS storage pools zpool-replace(8) - replace one device with another in ZFS storage pool zpool-resilver(8) - resilver devices in ZFS storage pools zpool-scrub(8) - begin or resume scrub of ZFS storage pools zpool-split(8) - split devices off ZFS storage pool, creating new pool zpool-status(8) - show detailed health status for ZFS storage pools zpool-sync(8) - flush data to primary storage of ZFS storage pools zpool-trim(8) - initiate TRIM of free space in ZFS storage pool zpool-upgrade(8) - manage version and feature flags of ZFS storage pools zpool-wait(8) - wait for activity to stop in a ZFS storage pool ======================================= refill beverage =======================================