Create a keyfile encrypted ZFS volume in Ubuntu 22.04

May 27th at 7:57pm

For the first time in nearly a decade I assembled a new desktop workstation to support a growing interest in learning about Artificial Intelligence and Machine Learning from "first principles". This is a departure for me, having been a creature of laptops for many years. To make things more interesting a ZFS pool will be used in place of the usual 'RAID' array for my large storage disks.

This post walks through how to create an encrypted ZFS pool / dataset (still adjusting to the terminology) which is auto-mounted on system boot

References

Volume Setup

  1. First, identify your drives using fdisk -l
    • In my case the 2 drives which I wanted to add to my zfs were /dev/sda and /dev/sdb
  2. Next, create an encryption file
    • Place it in a directory that will be available on boot
    • Ensure the directory and file permissions are setup to prevent other users from viewing or altering your secret
    • For example, it could be stored in /cfg/zfs.enc (or anywhere that makes sense for your system setup
    • I ensured that the permissions were set to 0400
  3. Then, Identify where you want the ZFS dataset to be mounted
    • In my case I elected to mount it under /mnt/bigdisk
  4. Now you can use this command to setup a zfs mirror ( be sure to adjust it based on your needs! You may not want a 'mirror'...)
sudo zpool create -o encryption=on -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=file:///cfg/zfs.enc -m /mnt/bigdisk bigdisk mirror /dev/sda /dev/sdb

Auto-unlock on Boot

(Updated 2024): Edit the systemd configuration for zfs-mount to ensure that the zfs volume will unlock on boot and persist the configuration across system updates/upgrades. My 'original' version below just performs a raw-edit of the systemd service... and I found that I regularly had to go back in and 're-edit' the service configuration, which is non-optimal...

  • sudo systemctl edit zfs-mount
  • Add this line to the configuration:
    • ExecStart=/sbin/zfs mount -a -l

At this point the zfs volume should auto-unlock on boot using your encryption keyfile.

(previous version) Now the only configuration remains is to setup the system so it unlocks the volume on boot:

  • vim /usr/lib/systemd/zfs-mount.service
    • This is the file which controls the zfs mount configuration
  • Add the -l parameter to the zfs mount command. On my system the final configuration looked like this:

  1. [Unit]
    Description=Mount ZFS filesystems
    Documentation=man:zfs(8)
    DefaultDependencies=no
    After=systemd-udev-settle.service
    After=zfs-import.target
    After=systemd-remount-fs.service
    After=zfs-load-module.service
    Before=local-fs.target
    ConditionPathIsDirectory=/sys/module/zfs

    [Service]
    Type=oneshot
    RemainAfterExit=yes
    ExecStart=/sbin/zfs mount -a -l

    [Install]
    WantedBy=zfs.target

At this point in my testing the ZFS volume would unlock on boot/reboot! This compliments the ZFS encryption which was setup at install time on the NVMe drive.

ZFS Troubleshooting

If you find yourself in need of troubleshooting why an encrypted zfs volume won't load, here are some tips:

  1. Check if the key is available:

    • zfs get keystatus aiml
    • If its unavailable / unusable you'll see something like this:
      • NAME PROPERTY VALUE SOURCE
        bgdsk keystatus unavailable -

  2. Check where the keyfile is configured on the volume

    • zfs get encryption,keyformat,keylocation $ZFS_VOLUME_HERE
    • I found that I had relocated the keyfile on one of my volumes, which caused it not to load... this command shows you which key is configured for the particular zfs volume. The output of the command looks like this:
      • NAME PROPERTY VALUE SOURCE
        bgdsk encryption aes-256-gcm -
        bgdsk keyformat raw -
        bgdsk keylocation file:///cfg/vols/old-path.key local

  3. If the keyfile configured in the volume is incorrect, you can correct it like this:

    • sudo zfs set keylocation=file:///cfg/vols/new-path-to.key bgdsk

At this point you can try to manually mount the volume like this:

  • sudo zfs mount $ZFS_VOLUME_HERE

Final Words

ZFS is supposed to be the best thing since sliced bread. With this workstation configuration I'll put it to the test!

As an aside / reference to my future self, this is the system configuration & specs:

  • RTX 4090 FE
  • Ryzen 9 7950X3D
  • Gigabyte X670 AORUS Elite AX (rev. 1.0)
    • Firmware F9a
  • Cosair Vengeance 192GB DDR5 (CMK192GX5M4B5200C38)
  • Samsung 990 Pro 2TB NVMe
  • 2x WD Red Pro WD181KFGX 18TB drives
  • Seasonic PRIME TX-1600 (SSR-1600TR)
  • Fractal Torrent case