This is an AI recipe demonstrating how to move FreeBSD back and forth between hardware and virtual machines and jails by eliminating as many self inflicted restrictions as possible. The word jail will refer to VNET jail throughout. There is a plethora of reasons why having the freedom to move between newer or older hardware or virtual machines or jails is of great value, the least of which enables total disaster recovery solutions that actually work.
Hardware and virtual machines are both quite different than jails. Many things could make the system fail when it is moved back and forth between them, however, this recipe will go over how to do it successfully.
The end result of this study should at least make you aware that if your production environment is not running FreeBSD on ZFS you should seriously make the transition as soon as possible.
A hardware or virtual machine. A FreeBSD installer. It will be assumed that you have computer and network skills and essential *nix skills to understand the concepts and the process. You should also be able to operate a *nix command line text editor of your choice. You are encouraged to read up on FreeBSD and ZFS to understand anything that is not explained.
Download and install the previous version of FreeBSD so you can follow along and perform a system update and bring it to the latest release later in a jail. We shall name it rover. It can be a hardware or virtual machine. This initial FreeBSD installation will be moved everywhere.
It is very good practice keep the os on a dedicated zpool and use a separate zpool for data storage, however, to simplify this recipe we will just use zroot. Keep this in mind for your live production systems as this recipe focuses solely on zroot.
https://download.freebsd.org/releases/ISO-IMAGES/
The best disaster recovery partition scheme to select in the ZFS configuration dialog should be one that provides the most flexibility to maintain bootability when recovering to older and newer hardware.
As a side note, whenever you install onto hardware with multiple disks, make sure you mirror the swap.
Partition Scheme GPT (BIOS+UEFI)
After the installation is complete, check the ZFS mountpoint and canmount properties and save them for future reference.
zfs get -r -t filesystem mountpoint
NAME PROPERTY VALUE SOURCE zroot mountpoint /zroot local zroot/ROOT mountpoint none local zroot/ROOT/default mountpoint / local zroot/tmp mountpoint /tmp local zroot/usr mountpoint /usr local zroot/usr/home mountpoint /usr/home inherited from zroot/usr zroot/usr/ports mountpoint /usr/ports inherited from zroot/usr zroot/usr/src mountpoint /usr/src inherited from zroot/usr zroot/var mountpoint /var local zroot/var/audit mountpoint /var/audit inherited from zroot/var zroot/var/crash mountpoint /var/crash inherited from zroot/var zroot/var/log mountpoint /var/log inherited from zroot/var zroot/var/mail mountpoint /var/mail inherited from zroot/var zroot/var/tmp mountpoint /var/tmp inherited from zroot/var
zfs get -r -t filesystem canmount
NAME PROPERTY VALUE SOURCE zroot canmount on default zroot/ROOT canmount on default zroot/ROOT/default canmount noauto local zroot/tmp canmount on default zroot/usr canmount off local zroot/usr/home canmount on default zroot/usr/ports canmount on default zroot/usr/src canmount on default zroot/var canmount off local zroot/var/audit canmount on default zroot/var/crash canmount on default zroot/var/log canmount on default zroot/var/mail canmount on default zroot/var/tmp canmount on default
Feel free to take as many selfies as you like at any time you like or especially after a change has been performed. Name them appropriately.
zfs snapshot -r zroot@01
Save a full send of the latest snapshot. It can be saved on the host, however it should really be saved to a safe remote (or removable) storage location.
zfs send -R zroot@01 > /export/backup/rover_zroot_01.capR
A jail can be created in a hardware or virtual machine.
This section only applies to jails hosted in a virtual machine. A jail hosted in a virtual machine will require promiscuous mode enabled on the virtual switch for the network to function properly.
Here is an example of how to enable promiscuous mode on the network adapter in VirtualBox.
vboxmanage modifyvm rover --nictype1 virtio --nic1 bridged --bridgeadapter1 igb0 --nicpromisc1 allow-all
Here is an example of how to enable promiscuous mode on the network adapter in VMWare by adding this line to the .vmx file of the virtual machine.
ethernet0.noPromisc = "FALSE"
Prepare the ZFS dataset structure on the host where the jail will live. The jail path will be /jail/rover/root.
zfs create -o mountpoint=/jail zroot/jail
zfs create zroot/jail/rover
zfs create -o canmount=off zroot/jail/rover/root
zfs create zroot/jail/rover/disk
zfs create zroot/jail/rover/disk/zroot
The datasets from ZFS receive should not be mounted until some inspections and changes are made.
zfs recv -F -u zroot/jail/rover/disk/zroot < /export/backup/rover_zroot_01.capR
Check the mountpoint and canmount properties and look for datasets marked received. These will need to be adjusted.
zfs get -r -t filesystem mountpoint zroot/jail/rover/disk/zroot
NAME PROPERTY VALUE SOURCE zroot/jail/rover/disk/zroot mountpoint /zroot received zroot/jail/rover/disk/zroot/ROOT mountpoint none received zroot/jail/rover/disk/zroot/ROOT/default mountpoint / received zroot/jail/rover/disk/zroot/tmp mountpoint /tmp received zroot/jail/rover/disk/zroot/usr mountpoint /usr received zroot/jail/rover/disk/zroot/usr/home mountpoint /usr/home inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/usr/ports mountpoint /usr/ports inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/usr/src mountpoint /usr/src inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/var mountpoint /var received zroot/jail/rover/disk/zroot/var/audit mountpoint /var/audit inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/crash mountpoint /var/crash inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/log mountpoint /var/log inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/mail mountpoint /var/mail inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/tmp mountpoint /var/tmp inherited from zroot/jail/rover/disk/zroot/var
zfs get -r -t filesystem canmount zroot/jail/rover/disk/zroot
NAME PROPERTY VALUE SOURCE zroot/jail/rover/disk/zroot canmount on default zroot/jail/rover/disk/zroot/ROOT canmount on default zroot/jail/rover/disk/zroot/ROOT/default canmount noauto received zroot/jail/rover/disk/zroot/tmp canmount on default zroot/jail/rover/disk/zroot/usr canmount off received zroot/jail/rover/disk/zroot/usr/home canmount on default zroot/jail/rover/disk/zroot/usr/ports canmount on default zroot/jail/rover/disk/zroot/usr/src canmount on default zroot/jail/rover/disk/zroot/var canmount off received zroot/jail/rover/disk/zroot/var/audit canmount on default zroot/jail/rover/disk/zroot/var/crash canmount on default zroot/jail/rover/disk/zroot/var/log canmount on default zroot/jail/rover/disk/zroot/var/mail canmount on default zroot/jail/rover/disk/zroot/var/tmp canmount on default
The mountpoint and canmount properties have to be adjusted for the jail. Basically the datasets marked received need to have their mountpoints changed from root / to the root of the jail path /jail/rover/root.
zfs set mountpoint=/jail/rover/root zroot/jail/rover/disk/zroot/ROOT/default
zfs set mountpoint=/jail/rover/root/zroot zroot/jail/rover/disk/zroot
zfs set mountpoint=/jail/rover/root/tmp zroot/jail/rover/disk/zroot/tmp
zfs set mountpoint=/jail/rover/root/usr zroot/jail/rover/disk/zroot/usr
zfs set mountpoint=/jail/rover/root/var zroot/jail/rover/disk/zroot/var
zfs set canmount=on zroot/jail/rover/disk/zroot/ROOT/default
Some received datasets already have the correct value. We can clear them by updating them with the same value.
zfs set mountpoint=none zroot/jail/rover/disk/zroot/ROOT
zfs set canmount=off zroot/jail/rover/disk/zroot/usr
zfs set canmount=off zroot/jail/rover/disk/zroot/var
Check the results after making changes.
zfs get -r -t filesystem mountpoint zroot/jail/rover/disk/zroot
NAME PROPERTY VALUE SOURCE zroot/jail/rover/disk/zroot mountpoint /jail/rover/root/zroot local zroot/jail/rover/disk/zroot/ROOT mountpoint none local zroot/jail/rover/disk/zroot/ROOT/default mountpoint /jail/rover/root local zroot/jail/rover/disk/zroot/tmp mountpoint /jail/rover/root/tmp local zroot/jail/rover/disk/zroot/usr mountpoint /jail/rover/root/usr local zroot/jail/rover/disk/zroot/usr/home mountpoint /jail/rover/root/usr/home inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/usr/ports mountpoint /jail/rover/root/usr/ports inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/usr/src mountpoint /jail/rover/root/usr/src inherited from zroot/jail/rover/disk/zroot/usr zroot/jail/rover/disk/zroot/var mountpoint /jail/rover/root/var local zroot/jail/rover/disk/zroot/var/audit mountpoint /jail/rover/root/var/audit inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/crash mountpoint /jail/rover/root/var/crash inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/log mountpoint /jail/rover/root/var/log inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/mail mountpoint /jail/rover/root/var/mail inherited from zroot/jail/rover/disk/zroot/var zroot/jail/rover/disk/zroot/var/tmp mountpoint /jail/rover/root/var/tmp inherited from zroot/jail/rover/disk/zroot/var
zfs get -r -t filesystem canmount zroot/jail/rover/disk/zroot
NAME PROPERTY VALUE SOURCE zroot/jail/rover/disk/zroot canmount on default zroot/jail/rover/disk/zroot/ROOT canmount on default zroot/jail/rover/disk/zroot/ROOT/default canmount on local zroot/jail/rover/disk/zroot/tmp canmount on default zroot/jail/rover/disk/zroot/usr canmount off local zroot/jail/rover/disk/zroot/usr/home canmount on default zroot/jail/rover/disk/zroot/usr/ports canmount on default zroot/jail/rover/disk/zroot/usr/src canmount on default zroot/jail/rover/disk/zroot/var canmount off local zroot/jail/rover/disk/zroot/var/audit canmount on default zroot/jail/rover/disk/zroot/var/crash canmount on default zroot/jail/rover/disk/zroot/var/log canmount on default zroot/jail/rover/disk/zroot/var/mail canmount on default zroot/jail/rover/disk/zroot/var/tmp canmount on default
Give everything another glance after they are mounted to make sure everything is correct and not mounted over top of one another.
You will see how nicely the jail properties mimic the host.
zfs mount -a
zfs list -o name,canmount,mounted,mountpoint
NAME CANMOUNT MOUNTED MOUNTPOINT zroot/jail on yes /jail zroot/jail/rover on yes /jail/rover zroot/jail/rover/disk on yes /jail/rover/disk zroot/jail/rover/disk/zroot on yes /jail/rover/root/zroot zroot/jail/rover/disk/zroot/ROOT on no none zroot/jail/rover/disk/zroot/ROOT/default on yes /jail/rover/root zroot/jail/rover/disk/zroot/tmp on yes /jail/rover/root/tmp zroot/jail/rover/disk/zroot/usr off no /jail/rover/root/usr zroot/jail/rover/disk/zroot/usr/home on yes /jail/rover/root/usr/home zroot/jail/rover/disk/zroot/usr/ports on yes /jail/rover/root/usr/ports zroot/jail/rover/disk/zroot/usr/src on yes /jail/rover/root/usr/src zroot/jail/rover/disk/zroot/var off no /jail/rover/root/var zroot/jail/rover/disk/zroot/var/audit on yes /jail/rover/root/var/audit zroot/jail/rover/disk/zroot/var/crash on yes /jail/rover/root/var/crash zroot/jail/rover/disk/zroot/var/log on yes /jail/rover/root/var/log zroot/jail/rover/disk/zroot/var/mail on yes /jail/rover/root/var/mail zroot/jail/rover/disk/zroot/var/tmp on yes /jail/rover/root/var/tmp zroot/jail/rover/root off no /jail/rover/root zroot on yes /zroot zroot/ROOT on no none zroot/ROOT/default noauto yes / zroot/tmp on yes /tmp zroot/usr off no /usr zroot/usr/home on yes /usr/home zroot/usr/ports on yes /usr/ports zroot/usr/src on yes /usr/src zroot/var off no /var zroot/var/audit on yes /var/audit zroot/var/crash on yes /var/crash zroot/var/log on yes /var/log zroot/var/mail on yes /var/mail zroot/var/tmp on yes /var/tmp
The jail will no longer use the machine network interface so update /jail/rover/root/etc/rc.conf to use side b of the jail epair.
ifconfig_epair0b="inet 192.168.x.x netmask 255.255.255.0"
The host machine should be adequately equipped to accommodate a jail.
Add these lines to the host /etc/rc.conf to enable jails, define the bridge, and bring up side a of the epair.
jail_enable="YES" jail_list="rover" cloned_interfaces="bridge0 epair0" ifconfig_bridge0="addm igb0 addm epair0a" ifconfig_epair0a="up"
Configure the vnet jail to use side b of the epair and take notice of the path to the jail root. Add these lines to /etc/jail.conf.
exec.clean;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
allow.mount;
mount.devfs;
path = "/jail/${name}/root";
exec.consolelog = "/var/log/jail_${name}.log";
rover {
vnet;
vnet.interface = "epair0b";
}After a restart of some services the jail can be started.
service devfs restart
service netif restart
service routing restart
service jail start
jls
All ZFS work is performed from the host as usual.
zfs snapshot -r zroot@02
The snapshot of the jail zroot dataset is the one we need to backup.
zfs send -R zroot/jail/rover/disk/zroot@02 > /export/backup/rover_zroot_02.capR
Configure a (hardware or virtual) machine that is adequately equipped.
Boot a FreeBSD installer and select <Install> as if you were installing a brand new system. Use the installer to set up the zpool. I found this to be the fastest and simplest way to set up the disks. When the installation is finished we will essentially be receiving the backup directly into the zroot datasets that were just created with the installer.
Boot the FreeBSD installer again and select <Shell>. Import zroot to a temporary altroot and configure networking to retrieve the backup .capR from nfs storage.
ifconfig igb0 192.168.x.x netmask 255.255.255.0 up
route add default 192.168.x.y
mount -t tmpfs tmpfs /mnt
mkdir /mnt/zroot
mkdir /mnt/backup
mount 192.168.x.z:/backup /mnt/backup
zpool import -N zroot -R /mnt/zroot
zfs recv -F -u zroot < /mnt/backup/rover_zroot_02.capR
Check the mountpoint and canmount properties and look for datasets marked received. These will need to be adjusted.
zfs get -r -t filesystem mountpoint zroot
NAME PROPERTY VALUE SOURCE zroot mountpoint /mnt/zroot/jail/rover/root/zroot received zroot/ROOT mountpoint none local zroot/ROOT/default mountpoint /mnt/zroot local zroot/tmp mountpoint /mnt/zroot/tmp local zroot/usr mountpoint /mnt/zroot/usr local zroot/usr/home mountpoint /mnt/zroot/usr/home inherited from zroot/usr zroot/usr/ports mountpoint /mnt/zroot/usr/ports inherited from zroot/usr zroot/usr/src mountpoint /mnt/zroot/usr/src inherited from zroot/usr zroot/var mountpoint /mnt/zroot/var local zroot/var/audit mountpoint /mnt/zroot/var/audit inherited from zroot/var zroot/var/crash mountpoint /mnt/zroot/var/crash inherited from zroot/var zroot/var/log mountpoint /mnt/zroot/var/log inherited from zroot/var zroot/var/mail mountpoint /mnt/zroot/var/mail inherited from zroot/var zroot/var/tmp mountpoint /mnt/zroot/var/tmp inherited from zroot/var
zfs get -r -t filesystem canmount zroot
NAME PROPERTY VALUE SOURCE
zroot canmount on default
zroot/ROOT canmount on default
zroot/ROOT/default canmount noauto local
zroot/tmp canmount on default
zroot/usr canmount off local
zroot/usr/home canmount on default
zroot/usr/ports canmount on default
zroot/usr/src canmount on default
zroot/var canmount off local
zroot/var/audit canmount on default
zroot/var/crash canmount on default
zroot/var/log canmount on default
zroot/var/mail canmount on default
zroot/var/tmp canmount on defaultThe mountpoints that point to the root of the jail /jail/rover/root will have to be changed to point to the real root /. Of course we are using a temporary mount point which prefixes /mnt to all the displayed values so that part can safely be ignored. Good news, there is only one received dataset to adjust.
zfs set mountpoint=/zroot zroot
The machine will no longer use the jail epair network interface so we will mount zroot/ROOT/default to access and update /etc/rc.conf which will be located at /mnt/zroot/etc/rc.conf.
zfs mount zroot/ROOT/default
ifconfig_igb0="inet 192.168.x.x netmask 255.255.255.0"
Remove the installation media and reboot.
shutdown -r now
All upgrade work is performed from the host machine.
freebsd-version -kruj rover
13.1-RELEASE 13.1-RELEASE 13.1-RELEASE 13.1-RELEASE
jexec rover freebsd-version -kru
13.1-RELEASE 13.1-RELEASE 13.1-RELEASE
The FreeBSD version on the host machine cannot be less than the FreeBSD version of a jail, therefore the host must be upgraded first.
Repeat these next steps until there are no more patch release updates for the current release.
freebsd-update fetch
freebsd-update install
shutdown -r now
freebsd-version -kruj rover
jexec rover freebsd-version -kru
Upgrade to a major or minor point release.
freebsd-update upgrade -r 13.2-RELEASE
freebsd-update install
shutdown -r now
freebsd-update install
shutdown -r now
freebsd-version -kruj rover
jexec rover freebsd-version -kru
It is required to reinstall all packages after a major release upgrade.
pkg-static upgrade -f
shutdown -r now
Again, the FreeBSD version on the host machine cannot be less than the FreeBSD version of a jail.
Repeat the next steps until there are no more patch release updates for the current release.
freebsd-update -j rover fetch
freebsd-update -j rover install
service jail stop rover
jls -d | grep rover
service jail start rover
freebsd-version -kruj rover
jexec rover freebsd-version -kru
Upgrade to a major or minor point release.
freebsd-update -j rover upgrade -r 13.2-RELEASE
freebsd-update -j rover install
service jail stop rover
jls -d | grep rover
service jail start rover
freebsd-update -j rover install
service jail stop rover
jls -d | grep rover
service jail start rover
freebsd-version -kruj rover
jexec rover freebsd-version -kru
It is required to reinstall all packages after a major release upgrade.
pkg -j rover upgrade -f
service jail stop rover
jls -d | grep rover
service jail start rover
This AI recipe is based on taking advantage of ZFS tools like snapshots, send, and receive. If you have ZFS dataset backups you should be confident that they can be restored anywhere. Full backups can be quite large so combining them with incrementals can save lots of time when backing them up and restoring them. The following is a simple example.
You are only as good as your last backup. The backup process could be scripted and run as a cron job.
Recursive snapshots are the way to go. Here they are named after the day they were taken.
zfs snapshot -r zroot@29
zfs snapshot -r zroot@30
zfs snapshot -r zroot@31
zfs snapshot -r zroot@01
zfs snapshot -r zroot@02
zfs snapshot -r zroot@03
zfs snapshot -r zroot@04
zfs snapshot -r zroot@05
A full send will include everything up to the moment the snapshot was taken. Here we do a full send on the first day of the month and store them somewhere on another machine or device.
zfs send -R zroot@01 > /export/backup/rover_zroot_01.capR
An incremental send will include everything between two snapshots. Here we do an incremental send on all the other days of the month and store them somewhere on another machine or device.
zfs send -R -I zroot@01 zroot@02 > /export/backup/rover_zroot_01_02.capI
zfs send -R -I zroot@02 zroot@03 > /export/backup/rover_zroot_02_03.capI
zfs send -R -I zroot@02 zroot@03 > /export/backup/rover_zroot_03_04.capI
zfs send -R -I zroot@03 zroot@04 > /export/backup/rover_zroot_04_05.capI
Receive the latest full backup first and then the incrementals that followed. It is good to receive them unmounted so their mountpoints and other settings can be verified prior to mounting. Remember to check the mountpoint and canmount properties and look for datasets marked received and make adjustments if needed.
zfs recv -F -u zroot < /export/backup/rover_zroot_01.capR
zfs recv -F -u zroot < /export/backup/rover_zroot_01_02.capI
zfs recv -F -u zroot < /export/backup/rover_zroot_02_03.capI
zfs recv -F -u zroot < /export/backup/rover_zroot_03_04.capI
zfs recv -F -u zroot < /export/backup/rover_zroot_04_05.capI
Pour yourself a beverage and start reading the docs.
https://docs.freebsd.org/en/books/handbook/zfs/
https://docs.freebsd.org/en/books/handbook/jails/
https://docs.freebsd.org/en/books/handbook/cutting-edge/
https://man.freebsd.org/cgi/man.cgi?query=zfs
https://man.freebsd.org/cgi/man.cgi?query=freebsd-update
Support this AI research at https://paypal.me/genunix for beer and systems and hot tub.