NixOS on encrypted ZFS

I wouldn’t call myself a distro hopper. The decision of distribution is solely based on requirements. I have requirements and I want the distribution to fulfill them as much as possible. After 15 years, I know what I want and I go and find it.

In this case, an unexpected project caught my eye. The idea is so radically different that I wasn’t actually searching for it this time. It is one of those times where it found me first.

After looking into Nix and NixOS, I decided it is going to be my distribution of choice on the desktop. I will use that as my test bed before migrating all the serious work there. That’s how I got my first taste of NixOS outside of the deterministic virtualization layer and into the wild.

Requirements

Before installing any new system, I draftdown a list of requirements I would need this system to run. These are things that are very hard to change on the fly in the future without some serious work. Also, things that simply need to be there in this day and age.

Filesystem

I’m a big fan of zfs. I’ve been running it on Linux, since the openzfs project successfully ported it, with no issues. It’s a solid choice for a filesystem and I don’t see a reason not to choose it.

Is it really a requirement ?

Well, yes. By now, openzfs should be accessible to all distributions but my choice of distribution is not usually for the beginner user. I need to know it’s supported and documented. I can figure out the rest from there.

Encryption

This is the first requirement, I always want encryption. The reason why I put it second in the list is that I needed to talk about zfs first.

The zfs filesystem offers encryption. Unfortunately, my research have shown that zfs might not encrypt some metadata. This might not be a big deal but the choice of using Luks is there as well.

With Luks, we can encrypt the whole drive. So let’s do that; Luks with zfs on top.

NixOS

NixOS utilizes Nix to build you an OS from a configuration file. This configuration file is written in the nix language. It is very analogous to written an Ansible playbook but it builds an OS instead.

The idea sounded appealing to me. A good friend of mine, setkeh, gave me a quick and dirty overview, at first. That pushed me into doing more research of my own where I found out that I can spawn off a nix-shell with whatever dependencies I want without having them installed on my system. What a great concept for development or even running applications you don’t really want to run. Java stuff for example.

Anyway, for all of these different reasons I have chosen NixOS to be the OS of choice to go on the desktop.

Installation

After testing NixOS in a VM a few times, I got setkeh on a conference session and we dug into this.

Filesystem partitioning

For the filesystem, we’re going to create two partitions. We need one, vfat, for the boot and another, zfs, for the rest of the filesystem.

We can start by creating the first partition of 1GB.

sgdisk -n3:1M:+1024M -t3:EF00 /dev/disk/by-id/VENDOR-ID

Followed by the rest of the filesystem.

sgdisk -n1:0:0 -t1:BF01 /dev/disk/by-id/VENDOR-ID

Filesystem formatting

Now that we got our partitions creates, let’s go ahead and format them properly.

Starting with the boot partition first.

mkfs.vfat /dev/disk/by-id/VENDOR-ID-part1

Then our zfs partition, but we need to encrypt it first. So, we create the Luks partition.

cryptsetup luksFormat /dev/disk/by-id/VENDOR-ID-part2

At this stage, stage we are done with the filesystem formatting and we need to create the zfs pool. To do so, we need to mount the encrypted root filesystem; Luks.

cryptsetup open --type luks /dev/disk/by-id/VENDOR-ID-part2 crypt

This mounts the filesystem in /dev/mapper/crypt. We’ll use that to create the pool.

zpool create -O mountpoint=none rpool /dev/mapper/crypt
zfs create -o mountpoint=legacy rpool/root
zfs create -o mountpoint=legacy rpool/root/nixos
zfs create -o mountpoint=legacy rpool/home

Filesystem mounting

After creating the filesystem, let’s mount everything.

# Mounting filesystem
mount -t zfs rpool/root/nixos /mnt
mkdir /mnt/home
mkdir /mnt/boot
# Mounting home directory
mount -t zfs rpool/home /mnt/home
# Mounting boot partition
mount /dev/disk/by-id/VENDOR-ID-part1 /mnt/boot

Generating NixOS configuration

At this stage, we need a nix configuration to build our system from. I didn’t have any configuration to start from so I generated one.

nixos-generate-config --root /mnt

NixOS configuration

Due to the weird configuration we’ve had, we need to make a few adjustements to the suggested configuration layed out in the docs.

The required configuration bits to be added to /mnt/etc/nixos/configuration.nix are:

boot.supportedFilesystems = [ "zfs" ];
# Make sure you set the networking.hostId option, which ZFS requires:
networking.hostId = "<random 8-digit hex string>";
# See https://nixos.org/nixos/manual/options.html#opt-networking.hostId for more.

# Use the GRUB 2 boot loader.
boot.loader.grub = {
  enable = true;
  version =2;
  device = "nodev";
  efiSupport = true;
  enableCryptodisk = true;
};

boot.initrd.luks.devices = {
 root = {
   device = "/dev/disk/by-uuid/VENDOR-UUID-part2"; ## Use blkid to find this UUID
   # Required even if we're not using LVM
   preLVM = true;
 };
};

NixOS installation

If we’re done with all of the configuration as described above, we should be able to build a bootable system. Let’s try that out by installing NixOS.

nixos-install

Conclusion

It took a bit of trial and error, and a loooooooot of mounting over and over again. At the end, though, it wasn’t as bad as I thought it would be. I’m still trying to get myself familiarised with NixOS and the new way of doing things. All in all, I would recommend trying NixOS, or the very least Nix.