======================================= freebsd zfs migratable ======================================= This exercise demonstrates a method of migrating freebsd from a vm (or hardware) to a jail and back again. The system to be migrated uses a dataset for the os named zroot and a separate dataset for data named tank. It also has private and public network interfaces. For notes on creating a virtualbox vm see https://www.genunix.com/o1/freebsd_virtualbox_host.txt https://www.genunix.com/o1/freebsd_virtualbox_freebsd_guest.txt ======================================= migrate vm to jail ======================================= An assumption is made that the zfs datasets are created from a standard freebsd installation. In other words, the zfs mountpoints for zroot are "standard". ======================================= backup the vm (bistro) --------------------------------------- Snapshots are created at regular intervals. Here we have snapshot names recycled according to the date. Keep in mind that a snapshot of a dataset cannot be created if it already exists so it must be destroyed first. Also keep in mind that a snapshot of a dataset cannot be destroyed while a zfs send is in progress. "/export/backup" can be remote storage that is accessible form some place on the jail host. Sending the main .capR will most likely be huge and may take some time to complete. zfs destroy -r zroot@01 zfs destroy -r tank@01 zfs snapshot -r zroot@01 zfs snapshot -r tank@01 zfs send -R zroot@01 > /export/backup/studio/vbox_bistro_zroot_01.capR zfs send -R tank@01 > /export/backup/studio/vbox_bistro_tank_01.capR ....................................... Subsequent snapshots can be sent separately as an incremental .capI which will be much faster to complete. This strategy may help you to significantly reduce the down time when doing an actual migration. zfs destroy -r zroot@02 zfs destroy -r tank@02 zfs snapshot -r zroot@02 zfs snapshot -r tank@02 zfs send -R -I zroot@01 zroot@02 > /export/backup/studio/vbox_bistro_zroot_01_02.capI zfs send -R -I tank@01 tank@02 > /export/backup/studio/vbox_bistro_tank_01_02.capI zfs destroy -r zroot@03 zfs destroy -r tank@03 zfs snapshot -r zroot@03 zfs snapshot -r tank@03 zfs send -R -I zroot@02 zroot@03 > /export/backup/studio/vbox_bistro_zroot_02_03.capI zfs send -R -I tank@02 tank@03 > /export/backup/studio/vbox_bistro_tank_02_03.capI ======================================= reference zfs properties of running vm (bistro) --------------------------------------- zfs get -r -t filesystem canmount,mountpoint | grep -v "PROPERTY" | sort -b -t' ' -k 2 -k 1 zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 canmount noauto local zroot/ROOT/13.1-RELEASE_2023-02-01_014610 canmount noauto local zroot/ROOT/default canmount noauto local zroot/usr canmount off local zroot/var canmount off local tank canmount on default tank/backup canmount on default tank/backup/mysql canmount on default tank/http canmount on default tank/mysql canmount on default tank/quarantine canmount on default tank/sftp canmount on default zroot canmount on default zroot/ROOT canmount on default zroot/tmp canmount on default zroot/usr/home canmount on default zroot/usr/ports canmount on default zroot/usr/src canmount on default zroot/var/audit canmount on default zroot/var/crash canmount on default zroot/var/log canmount on default zroot/var/mail canmount on default zroot/var/tmp canmount on default zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 mountpoint / local zroot/ROOT/13.1-RELEASE_2023-02-01_014610 mountpoint / local zroot/ROOT/default mountpoint / local tank mountpoint /export local tank/backup mountpoint /export/backup inherited from tank tank/backup/mysql mountpoint /export/backup/mysql inherited from tank tank/http mountpoint /export/http inherited from tank tank/mysql mountpoint /export/mysql inherited from tank tank/quarantine mountpoint /export/quarantine inherited from tank tank/sftp mountpoint /export/sftp inherited from tank zroot/tmp mountpoint /tmp local zroot/usr mountpoint /usr local zroot/usr/home mountpoint /usr/home inherited from zroot/usr zroot/usr/ports mountpoint /usr/ports inherited from zroot/usr zroot/usr/src mountpoint /usr/src inherited from zroot/usr zroot/var mountpoint /var local zroot/var/audit mountpoint /var/audit inherited from zroot/var zroot/var/crash mountpoint /var/crash inherited from zroot/var zroot/var/log mountpoint /var/log inherited from zroot/var zroot/var/mail mountpoint /var/mail inherited from zroot/var zroot/var/tmp mountpoint /var/tmp inherited from zroot/var zroot mountpoint /zroot local zroot/ROOT mountpoint none local ======================================= restore vm to jail (bistro) --------------------------------------- We will create the zfs dataset structure prior to restoring the .capR and any .capI as required. Receiving the .capR may take some time to complete. zfs create tank/jail/bistro zfs create tank/jail/bistro/disk zfs create tank/jail/bistro/disk/zroot zfs create tank/jail/bistro/disk/tank zfs create tank/jail/bistro/root zfs set canmount=off tank/jail/bistro/root zfs recv -F -u -v tank/jail/bistro/disk/zroot < /export/backup/studio/vbox_bistro_zroot_01.capR zfs recv -F -u -v tank/jail/bistro/disk/tank < /export/backup/studio/vbox_bistro_tank_01.capR ....................................... Receiving any subsequent .capI should be quite quick. zfs recv -F -u -v tank/jail/bistro/disk/zroot < /export/backup/studio/vbox_bistro_zroot_01_02.capI zfs recv -F -u -v tank/jail/bistro/disk/tank < /export/backup/studio/vbox_bistro_tank_01_02.capI zfs recv -F -u -v tank/jail/bistro/disk/zroot < /export/backup/studio/vbox_bistro_zroot_02_03.capI zfs recv -F -u -v tank/jail/bistro/disk/tank < /export/backup/studio/vbox_bistro_tank_02_03.capI ======================================= zfs get --------------------------------------- Get a list of the properties that will have to be tweaked zfs get -r -t filesystem atime,checksum,compress,canmount,mountpoint tank/jail/bistro/disk \ | grep -v \ -e ' PROPERTY' \ -e ' inherited' \ -e ' default' \ -e ' local' \ | sort -b -t' ' -k 2 -k 1 ....................................... tank/jail/bistro/disk/tank atime on received tank/jail/bistro/disk/zroot atime on received tank/jail/bistro/disk/zroot/var/mail atime on received tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 canmount noauto received tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 canmount noauto received tank/jail/bistro/disk/zroot/ROOT/default canmount noauto received tank/jail/bistro/disk/zroot/usr canmount off received tank/jail/bistro/disk/zroot/var canmount off received tank/jail/bistro/disk/tank checksum sha256 received tank/jail/bistro/disk/zroot checksum sha256 received tank/jail/bistro/disk/tank compression lz4 received tank/jail/bistro/disk/zroot compression lz4 received tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 mountpoint / received tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 mountpoint / received tank/jail/bistro/disk/zroot/ROOT/default mountpoint / received tank/jail/bistro/disk/tank mountpoint /export received tank/jail/bistro/disk/zroot/tmp mountpoint /tmp received tank/jail/bistro/disk/zroot/usr mountpoint /usr received tank/jail/bistro/disk/zroot/var mountpoint /var received tank/jail/bistro/disk/zroot mountpoint /zroot received tank/jail/bistro/disk/zroot/ROOT mountpoint none received ======================================= zfs set --------------------------------------- Generate a list of the commands for tweaking the zfs properties and run them. It is optional to check and set atime=on, compression=zstd, and checksum=sha512. zfs get -r -t filesystem atime,checksum,compress,canmount,mountpoint tank/jail/bistro/disk \ | grep -v \ -e ' PROPERTY' \ -e ' inherited' \ -e ' default' \ -e ' local' \ | sed \ -e 's/\([^ ]*\) *\([^ ]*\) *\([^ ]*\) *received.*$/zfs set \2=\3 \1/' \ -e 's/mountpoint=/mountpoint=\/export\/jail\/bistro\/root/' \ -e 's/bistro\/root\/ /bistro\/root /' \ -e 's/\/export\/jail\/bistro\/rootnone /none /' \ -e 's/=noauto \(.*\/ROOT\/default\)/=on \1/' \ -e 's/atime=off/atime=on/' \ -e 's/=sha256/=sha512/' \ -e 's/=lz4/=zstd/' \ | sort -b -t' ' -k 3 -k 4 ....................................... Sample output of generated commands that will need to be run. zfs set atime=on tank/jail/bistro/disk/tank zfs set atime=on tank/jail/bistro/disk/zroot zfs set atime=on tank/jail/bistro/disk/zroot/var/mail zfs set canmount=noauto tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 zfs set canmount=noauto tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 zfs set canmount=off tank/jail/bistro/disk/zroot/usr zfs set canmount=off tank/jail/bistro/disk/zroot/var zfs set canmount=on tank/jail/bistro/disk/zroot/ROOT/default zfs set checksum=sha512 tank/jail/bistro/disk/tank zfs set checksum=sha512 tank/jail/bistro/disk/zroot zfs set compression=zstd tank/jail/bistro/disk/tank zfs set compression=zstd tank/jail/bistro/disk/zroot zfs set mountpoint=/export/jail/bistro/root tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 zfs set mountpoint=/export/jail/bistro/root tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 zfs set mountpoint=/export/jail/bistro/root tank/jail/bistro/disk/zroot/ROOT/default zfs set mountpoint=/export/jail/bistro/root/export tank/jail/bistro/disk/tank zfs set mountpoint=/export/jail/bistro/root/tmp tank/jail/bistro/disk/zroot/tmp zfs set mountpoint=/export/jail/bistro/root/usr tank/jail/bistro/disk/zroot/usr zfs set mountpoint=/export/jail/bistro/root/var tank/jail/bistro/disk/zroot/var zfs set mountpoint=/export/jail/bistro/root/zroot tank/jail/bistro/disk/zroot zfs set mountpoint=none tank/jail/bistro/disk/zroot/ROOT ======================================= reference zfs properties for jail (bistro) --------------------------------------- zfs get -r -t filesystem canmount,mountpoint tank/jail/bistro | grep -v "PROPERTY" | sort -b -t' ' -k 2 -k 1 tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 canmount noauto local tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 canmount noauto local tank/jail/bistro/disk/zroot/usr canmount off local tank/jail/bistro/disk/zroot/var canmount off local tank/jail/bistro/root canmount off local tank/jail/bistro canmount on default tank/jail/bistro/disk canmount on default tank/jail/bistro/disk/tank canmount on default tank/jail/bistro/disk/tank/backup canmount on default tank/jail/bistro/disk/tank/backup/mysql canmount on default tank/jail/bistro/disk/tank/http canmount on default tank/jail/bistro/disk/tank/mysql canmount on default tank/jail/bistro/disk/tank/quarantine canmount on default tank/jail/bistro/disk/tank/sftp canmount on default tank/jail/bistro/disk/zroot canmount on default tank/jail/bistro/disk/zroot/ROOT canmount on default tank/jail/bistro/disk/zroot/tmp canmount on default tank/jail/bistro/disk/zroot/usr/home canmount on default tank/jail/bistro/disk/zroot/usr/ports canmount on default tank/jail/bistro/disk/zroot/usr/src canmount on default tank/jail/bistro/disk/zroot/var/audit canmount on default tank/jail/bistro/disk/zroot/var/crash canmount on default tank/jail/bistro/disk/zroot/var/log canmount on default tank/jail/bistro/disk/zroot/var/mail canmount on default tank/jail/bistro/disk/zroot/var/tmp canmount on default tank/jail/bistro/disk/zroot/ROOT/default canmount on local tank/jail/bistro mountpoint /export/jail/bistro inherited from tank tank/jail/bistro/disk mountpoint /export/jail/bistro/disk inherited from tank tank/jail/bistro/root mountpoint /export/jail/bistro/root inherited from tank tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE-p5_2023-02-20_005552 mountpoint /export/jail/bistro/root local tank/jail/bistro/disk/zroot/ROOT/13.1-RELEASE_2023-02-01_014610 mountpoint /export/jail/bistro/root local tank/jail/bistro/disk/zroot/ROOT/default mountpoint /export/jail/bistro/root local tank/jail/bistro/disk/tank mountpoint /export/jail/bistro/root/export local tank/jail/bistro/disk/tank/backup mountpoint /export/jail/bistro/root/export/backup inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/tank/backup/mysql mountpoint /export/jail/bistro/root/export/backup/mysql inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/tank/http mountpoint /export/jail/bistro/root/export/http inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/tank/mysql mountpoint /export/jail/bistro/root/export/mysql inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/tank/quarantine mountpoint /export/jail/bistro/root/export/quarantine inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/tank/sftp mountpoint /export/jail/bistro/root/export/sftp inherited from tank/jail/bistro/disk/tank tank/jail/bistro/disk/zroot/tmp mountpoint /export/jail/bistro/root/tmp local tank/jail/bistro/disk/zroot/usr mountpoint /export/jail/bistro/root/usr local tank/jail/bistro/disk/zroot/usr/home mountpoint /export/jail/bistro/root/usr/home inherited from tank/jail/bistro/disk/zroot/usr tank/jail/bistro/disk/zroot/usr/ports mountpoint /export/jail/bistro/root/usr/ports inherited from tank/jail/bistro/disk/zroot/usr tank/jail/bistro/disk/zroot/usr/src mountpoint /export/jail/bistro/root/usr/src inherited from tank/jail/bistro/disk/zroot/usr tank/jail/bistro/disk/zroot/var mountpoint /export/jail/bistro/root/var local tank/jail/bistro/disk/zroot/var/audit mountpoint /export/jail/bistro/root/var/audit inherited from tank/jail/bistro/disk/zroot/var tank/jail/bistro/disk/zroot/var/crash mountpoint /export/jail/bistro/root/var/crash inherited from tank/jail/bistro/disk/zroot/var tank/jail/bistro/disk/zroot/var/log mountpoint /export/jail/bistro/root/var/log inherited from tank/jail/bistro/disk/zroot/var tank/jail/bistro/disk/zroot/var/mail mountpoint /export/jail/bistro/root/var/mail inherited from tank/jail/bistro/disk/zroot/var tank/jail/bistro/disk/zroot/var/tmp mountpoint /export/jail/bistro/root/var/tmp inherited from tank/jail/bistro/disk/zroot/var tank/jail/bistro/disk/zroot mountpoint /export/jail/bistro/root/zroot local tank/jail/bistro/disk/zroot/ROOT mountpoint none local ....................................... zfs mount -a ======================================= /export/jail/bistro/root/var/cron/tabs/root --------------------------------------- Edit this file and tripple comment out any cron jobs until you are absolutely sure this machine is ready to go live!!! Remember to uncomment them to restore them when you are ready to go live. ======================================= /export/jail/bistro/root/etc/rc.conf --------------------------------------- Edit the interface names to use the "b" side of the jail epairs ifconfig_epair0b="inet 192.168.101.195 netmask 255.255.255.0" ifconfig_epair10b="inet 111.111.111.123 netmask 255.255.255.248" ======================================= /etc/rc.conf --------------------------------------- Configure the bridges and epairs for the jail. The vm is "bistro" and this example shows how to create epairs for additional jails (mayo). Keep in mind that naming an epair with an additional leading 0 (epair00) will not work. The host server only uses the private interface igb0 with no direct public access. The guest jails will have both private and public access. # jail jail_enable="YES" # only start listed jails on boot jail_list="bistro mayo" # networking for jails ifconfig_igb1="up" # igb0 bridge0 single digit epair = private # igb1 bridge1 double digit epair = public cloned_interfaces="\ bridge0 epair0 epair1 \ bridge1 epair10 epair11" # private ifconfig_bridge0="\ addm igb0 \ addm epair0a \ addm epair1a \ " ifconfig_epair0a="up" ifconfig_epair1a="up" # public ifconfig_bridge1="\ addm igb1 \ addm epair10a \ addm epair11a \ " ifconfig_epair10a="up" ifconfig_epair11a="up" ....................................... service netif restart ======================================= /etc/jail.conf --------------------------------------- exec.clean; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; allow.mount; mount.devfs; path = "/export/jail/${name}/root"; exec.consolelog = "/var/log/jail_${name}.log"; bistro { vnet; vnet.interface = "epair0b"; vnet.interface += "epair10b"; } # mayo { # vnet; # vnet.interface = "epair1b"; # vnet.interface += "epair11b"; # } ....................................... service jail start jls ======================================= upgrade jail (patch releases) --------------------------------------- uanme -apKU freebsd-version -kruj bistro jexec bistro uname -apKU freebsd-update -j bistro fetch freebsd-update -j bistro install service jail stop bistro jls -d service jail start bistro pkg -j bistro update pkg -j bistro upgrade ======================================= upgrade jail (major or minor point releases) --------------------------------------- uanme -apKU freebsd-version -kruj bistro jexec bistro uname -apKU freebsd-update -j bistro fetch freebsd-update -j bistro install service jail stop bistro jls -d service jail start bistro freebsd-update -j bistro upgrade -r 13.2-RELEASE freebsd-update -j bistro install service jail stop bistro jls -d service jail start bistro freebsd-update -j bistro install freebsd-version -kruj bistro jexec bistro uname -apKU pkg -j bistro upgrade -f ======================================= upgrade jail from failed upgrade --------------------------------------- freebsd-update -j bistro --currently-running 13.1-RELEASE -r 13.2-RELEASE upgrade freebsd-update -j bistro install ======================================= migrate jail to vm ======================================= Now that the jail is in perfect running condition it can be migrated back to hardware or a vm. BIOS boot and UEFI boot will both be restored. ======================================= backup the jail (bistro) from the jail host --------------------------------------- For expediency of this excercise we will just demonstrate with the .capR We will stop the jail for totally quiescent datasets. service jail stop bistro zfs destroy -r zroot@04 zfs destroy -r tank@04 zfs snapshot -r zroot@04 zfs snapshot -r tank@04 zfs send -R zroot/export/jail/bistro/disk/zroot@04 > /export/backup/jail_bistro_zroot_04.capR zfs send -R zroot/export/jail/bistro/disk/tank@04 > /export/backup/jail_bistro_tank_04.capR ======================================= create the vm --------------------------------------- Create a disk for zroot and a disk for tank Boot the freebsd installation dvd ....................................... Enter into a ======================================= nfs mount backup server to access .capR --------------------------------------- ifconfig ifconfig vtnet0 192.168.101.195/24 up mkdir /tmp/backup mount 192.168.101.193:/export/backup /tmp/backup ======================================= partition disk for zroot --------------------------------------- camcontrol devlist geom disk list gpart create -s GPT /dev/da0 gpart add -t efi -b 40 -a 4K -s 256M /dev/da0 gpart add -t freebsd-boot -a 4K -s 512K /dev/da0 gpart add -t freebsd-swap -a 4K -s 2G /dev/da0 gpart add -t freebsd-zfs -a 4K /dev/da0 ======================================= bios boot --------------------------------------- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 /dev/da0 ======================================= restore zroot --------------------------------------- mkdir /tmp/zroot mount -t tmpfs tmpfs /tmp/zroot zpool labelclear -f /dev/da0 > /dev/null 2>&1 zpool create \ -O compression=zstd -O checksum=sha512 -O atime=on \ -o autoexpand=off -o autoreplace=on -o failmode=continue -o listsnaps=off \ -m none -R /tmp/zroot zroot /dev/da0p4 zfs recv -F -u -v zroot < /tmp/backup/jail_bistro_zroot_04.capR ======================================= restore tank --------------------------------------- mkdir /tmp/tank mount -t tmpfs tmpfs /tmp/tank zpool labelclear -f /dev/da1 > /dev/null 2>&1 zpool create \ -O compression=zstd -O checksum=sha512 -O atime=on \ -o autoexpand=off -o autoreplace=on -o failmode=continue -o listsnaps=off \ -m none -R /tmp/tank tank /dev/da1 zfs recv -F -u -v tank < /tmp/backup/jail_bistro_tank_04.capR ======================================= zfs adjustments for vm --------------------------------------- zfs set mountpoint=none zroot zfs set mountpoint=none zroot/ROOT zfs set mountpoint=/ zroot/ROOT/default zfs set mountpoint=/tmp zroot/tmp zfs set mountpoint=/usr zroot/usr zfs set mountpoint=/var zroot/var zfs set canmount=noauto zroot/ROOT/default zfs set mountpoint=/export tank ======================================= set bootfs --------------------------------------- zpool set bootfs=zroot/ROOT/default zroot ======================================= uefi boot --------------------------------------- zfs mount zroot/ROOT/default mkdir /tmp/uefi newfs_msdos /dev/da0p1 mount_msdosfs /dev/da0p1 /tmp/uefi mkdir -p /tmp/uefi/EFI/BOOT zfs mount zroot/ROOT/default cp /tmp/zroot/boot/loader.efi /tmp/uefi/EFI/BOOT/BOOTX64.EFI ======================================= /tmp/zroot/etc/rc.conf --------------------------------------- ifconfig_vtnet0="inet 192.168.101.195 netmask 255.255.255.0" ifconfig_vtnet1="inet 111.111.111.123 netmask 255.255.255.248" ....................................... eject the dvd and reboot the vm ======================================= :0) =======================================