Instructions to help you setup a ZFS root file system with RAIDZ1 on a nanopi-m4 with a 4 port SATA hat
- NanoPi m4
- 4xSATA hata
- Armbian installation with Bullseye 5.15.x kernel
This installation stores the SPL, u-boot and initramfs files on the SD card and while the root filesystem is stored on the raid drive. U-boot and initramfs will be configured to bypass the root file system on the SD card and to load from the zpool dataset root file system.
- Build Armbian from source - since current versions of Armbian didn't seem to support SD card boot for my NanoPi M4.
- Install necessary ZFS packages
- Create RAIDZ pool and datasets
- Copy of root file system to new raid array
- Setup modules needed for initramfs to load ZFS and rebuild initramfs
- Modify armbianEnv.txt to use bootargs (extraargs) to boot from ZFS ROOT with
- Copy root file system from SD card to zpool dataset using rsync
- Reboot
Before starting it is useful to describe the boot process of this NanoPi M4. After a hardware reset the boot ROM loads a SPL boot loader (secondary program loader - a stripped down version of u-boot) from the SD card, memory is configured and then u-boot is started in full. U-boot then loads the kernel and (in this case) the initial RAM disk initramfs, and then transfers control the kernel.
Download build Armbian using the Armbian Build Documentation.
apt install -t bullseye-backports zfs-dkms zfsutils-linux zfs-initramfs
Identify the four disks to be used in the RAIDZ-1 array, by using the ata-[disk_X] generated by the following command.
ls -lh /dev/disk/by-id/
We will creat a zfs pool called rpool
zpool create -f \
-o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R / \
rpool raidz /dev/disk/by-id/ata-[disk_1] /dev/disk/by-id/ata-[disk_2] /dev/disk/by-id/ata-[disk_3] /dev/disk/by-id/ata-[disk_4]
We will create two datasets with the second being the future root file system, initially it will be mount in /mnt, later this will be changed to '/'.
zfs create -o mountpoint=none rpool/ROOT
zfs create -o mountpoint=/mnt rpool/ROOT/debian-1
It is possible to use the /dev/sdX format however it is not recommended as these names can change when SATA cables are swapped. However swapping formats between /dev/sdX to /dev/disk/by-id/ can be easily be performed by the following commands
zpool export zfspool
zpool import -d /dev/disk/by-id -aN
Make sure the pool is mounted to /mnt with zfs list
The following rsync command copies the files from the existing SD card file system to /mnt.
rsync -aAXv / --exclude={/dev/*,/proc/*,/sys/*,/tmp/*,/run/*,/mnt/*,/media/*,/lost+found} /mnt
The file system table (fstab) does not need the SD card entry and should be commented out. The ZFS module automatically loads the zpool information from zpool.cache and mounts the disk so an entry in /fstab is not needed.
At the moment the ZFS pool is mounted to /mnt for convenience during the setup stage, however in the final setup the rpool/ROOT/debian-1 will need to be mounted as '/'
zfs set mountpoint=/ rpool/ROOT/debian-1
add zfs to the modules to be loaded by initramfs and ensure the /etc/zpool.cache exists, this is generated using the zpool export rpool command.
echo zfs >> /etc/initramfs-tools/modules
Update the initramfs
zpool export rpool
update-initramfs -u
Comment out, rootdev, rootfstype and add extraargs
...
rootdev=UUID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
rootfstype=ext4
...
extraargs=root=rpool/ROOT/debian-1 boot=zfs
During the initramfs boot the kernel will use the 'extraargs' for root and boot supplied by armbianEnv.txt. Adding the parameters with 'extraargs' into armbianEnv.txt means that boot.scr doesn't need to be recreated.
remember any changes to armbianEnv.txt or initramfs need to copied to the SD card.
/boot on /dev/mmcblk1p1 is the device used by u-boot and not on the raid drive.
If the boot process stops here it is most likely the zpool has not been imported correctly. Make sure armbianEnv.txt contains the correct 'extraargs' information so that it can mount the zpool dataset.
(initramfs) zpool import rpool
(initramfs) mount -t zfs rpool/BOOT\debian-1 /root
(initramfs) exit
Manual booting from the SD card while in initramfs
(initramfs) mount /dev/mmcblk1p1 /root
(initramfs) exit
Kernel headers are needed for compiling the ZFS module during apt install zfs-dkms. Use compile options
./compile.sh INSTALL_HEADERS=yes
The package file containing the header files can be retrieved from the ./build/output/debs and then installed manually on to the NanoPi M4 board.
Get u-boot to look for initramfs from raid drive instead of SD card