uboot: add ubootNanoPCT4 for NanoPC-T4

[tmountain]

Motivation for this change

This adds support for the NanoPC-T4 to u-boot. The NanoPC-T4 does not work with upstream u-boot out of the box , so I created a custom repo and applied the necessary changes, which I extracted from the Armbian builder project.

The repo referenced in my PR is the result of applying this patch to the v2020.10 u-boot release.

The NanoPC-T4 will not boot if you build u-boot against a bl31.elf file. It only works if you produce a standalone trust.bin via the rkbin trust_merger utility. I have not been able to determine why this is, but Armbian ships with a custom trust.ini file with the following section which may play a role:

[BL31_OPTION]
SEC=1
PATH=bl31.elf
ADDR=0x10000

The trust_merger utility does not work with Nix out of the box, so I created a separate derivation to wrap it and make it run successfully. This resulted in the need to add a reference to callPackage to the uboot derivation.

If you'd prefer that I reformulate the PR to do the actual patching during the build, I can try to accommodate that request with some basic guidance.

Things done

• Created a ubootNanoPCT4 attribute which builds working images for the NanoPC-T4.

• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested execution of all binary files (usually in ./result/bin/)

[samueldr]

I have marked the previous discussion resolved. Here are the conclusions:

• Booting strictly with mainline u-boot SPL seems to not work
• Booting with u-boot SPL and only rockchip's DDR training (as with Rock64) seems to not work
• Booting with proprietary DDR training and trustzone appears to work

[jakubgs]

Just wanted to say thank you very much @tmountain for this. I was doing some research into running NixOS on my NanoPC T4 but as these kind of devices with uboot are not my area I was mostly just flailing around. Much appreciated, especially with the wiki page instructions. Thanks!

[tmountain]

Just wanted to say thank you very much @tmountain for this. I was doing some research into running NixOS on my NanoPC T4 but as these kind of devices with uboot are not my area I was mostly just flailing around. Much appreciated, especially with the wiki page instructions. Thanks!

I'm happy to hear that you have found some benefit in this work. If this pull request gets merged, the next step is to integrate the NanoPC-T4 with this image builder. That should make the process of installing Nix painless for everyone.

I will continue refining the guide as we make progress. Thank you for the positive feedback.

[jakubgs]

I can confirm this works btw. I got NixOS running on my NanoPC-T4 yesterday using the uboot images provided by @tmountain which I assume were build using this derivation. The NVMe drive is not detected, but that appears to be a kernel driver issue and is separate from uboot.

[tmountain]

I can confirm this works btw. I got NixOS running on my NanoPC-T4 yesterday using the uboot images provided by @tmountain which I assume were build using this derivation. The NVMe drive is not detected, but that appears to be a kernel driver issue and is separate from uboot.

Not sure if it is helpful, but I'm using KingSpec 256GB PCIe NVMe (NE-256 2280), and it works out of the box.

[jakubgs]

I have a Samsung EVO 960 250GB and it only appears with Friendly Core:

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])

Both Armbian and NixOS don't show anything.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/how-to-provide-missing-headers-to-a-kernel-build/11422/1

[tmountain]

Hi @samueldr. Just making sure you aren't waiting on anything my side to merge this. I'm contributing to another project, and waiting on this PR to merge before I submit my changes over there. Not trying to bug you--just making sure that there's nothing left on my side. Thanks.

[Mic92]

@GrahamcOfBorg build ubootNanoPCT4

[Mic92]

Looking forward to have support in https://github.com/Mic92/nixos-aarch64-images

[ghost]

I have a Samsung EVO 960 250GB and it only appears with Friendly Core:

01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 (prog-if 02 [NVM Express])

Both Armbian and NixOS don't show anything.

This was a regression introduced sometime in Linux 5.9.x; the fix wasn't merged until 5.10.17:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=413a2353be6b420b35998de364f79cad24061da7

I confirmed that cherry-picking that one commit and rebuilding the DTB (no need to rebuild the kernel) makes the NVME reappear:

root@carnot:~# lsblk | grep nvme
nvme0n1           259:0    0 465.8G  0 disk
├─nvme0n1p1       259:1    0  93.1G  0 part
└─nvme0n1p2       259:2    0  46.6G  0 part
root@carnot:~# lspci
00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981

Sorry for posting here; I typed in a response to the same question on Armbian's forum but then it started making me jump through a bunch of hoops, captcha's, and click-the-email nonsense and I just gave up.

[jakubgs]

I'm not sure if I was having the same issue, possible. But thanks for bringing this to our attention @a-m-joseph.

My fix was to modify NixOS configuration to build some drivers directly into the kernel:

{
  # Fix for not detecting the NVMe SSD
  boot.kernelPackages = pkgs.linuxPackages_5_10;
  boot.kernelPatches = [{
    name = "pcie-rockchip-config.patch";
    patch = null;
    extraConfig = ''
      NVME_CORE y
      BLK_DEV_NVME y
      NVME_MULTIPATH y
      PCIE_ROCKCHIP y
      PCIE_ROCKCHIP_EP y
      PCIE_ROCKCHIP_HOST y
      PHY_ROCKCHIP_DP y
      PHY_ROCKCHIP_PCIE y
      PHY_ROCKCHIP_USB y
    '';
  }];
}

https://github.com/jakubgs/nixos-nanopc-t4#nixos-config

I will try to use pkgs.linuxPackages_5_11 without the custom config and see how that works.

[Mic92]

I'm not sure if I was having the same issue, possible. But thanks for bringing this to our attention @a-m-joseph.

My fix was to modify NixOS configuration to build some drivers directly into the kernel:

{
  # Fix for not detecting the NVMe SSD
  boot.kernelPackages = pkgs.linuxPackages_5_10;
  boot.kernelPatches = [{
    name = "pcie-rockchip-config.patch";
    patch = null;
    extraConfig = ''
      NVME_CORE y
      BLK_DEV_NVME y
      NVME_MULTIPATH y
      PCIE_ROCKCHIP y
      PCIE_ROCKCHIP_EP y
      PCIE_ROCKCHIP_HOST y
      PHY_ROCKCHIP_DP y
      PHY_ROCKCHIP_PCIE y
      PHY_ROCKCHIP_USB y
    '';
  }];
}

https://github.com/jakubgs/nixos-nanopc-t4#nixos-config

I will try to use pkgs.linuxPackages_5_11 without the custom config and see how that works.

We could have those in nixos-hardware as well.

[jakubgs]

We could have those in nixos-hardware as well.

Sure. Why not, I can make a PR. Although compiling the kernel on the NanoPC-T4 is not the most pleasant experience, especially on a hot day. The poor thing is struggling. And not everyone does use the M.2 slot.

One of these days I need to figure out if there's a way to offload some of the compilation to my x86_64 machine in a VM.

[Mic92]

Maybe we can enable the required kernel modules in our stock kernel?

[jakubgs]

That would be amazing.

[jakubgs]

@a-m-joseph I have tried using Linux kernel 5.11.6 but booting doesn't work using default NixOS kernel:

<<< NixOS Stage 1 >>>

loading module zfs...
loading module spl...
loading module dm_mod...
running udev...
Starting version 246
kbd_mode: KDSKBMODE: Inappropriate ioctl for device
Gstarting device mapper and LVM...
importing root ZFS pool "rpool"...............................................................
cannot import 'rpool': no such pool available
waiting for device /dev/disk/by-uuid/1234-5678 to appear.......................
mounting rpool/root on /...
mount: mounting rpool/root on /mnt-root/ failed: No such file or directory
retrying...
...(omitted)...
mount: mounting rpool/root on /mnt-root/ failed: No such file or directory
retrying...
An error occurred in stage 1 of the boot process, which must mount the
root filesystem on `/mnt-root' and then start stage 2.  Press one
of the following keys:

  r) to reboot immediately
  *) to ignore the error and continue

So it appears booting from the M.2 NVMe does indeed require some modules baked into the kernel.

[ghost]

@a-m-joseph I have tried using Linux kernel 5.11.6 but booting doesn't work using default NixOS kernel:

loading module zfs...
loading module spl...
loading module dm_mod...
running udev...

Correct; you need both:

• the patch (included in 5.11.4+ and 5.10.17+)

and

• either (a) nvme.ko in your initrd or (b) a kernel built with CONFIG_BLK_DEV_NVME=y

So it appears booting from the M.2 NVMe does indeed require some modules baked into the kernel.

Or you can just put the kernel module in your initrd if you don't want to recompile the kernel. This true with all boot devices (including emmc, sd card, usb, etc)... the block device driver for your root filesystem always needs to be either compiled-in or else have its module included in the initrd.

[jakubgs]

@a-m-joseph I have tried your suggestion and added:

boot.initrd.kernelModules = [ "nvme" ];

But it was not enough to make the system detect the NVMe at boot time. I think more might be needed.

[jakubgs]

I've also tested this:

boot.initrd.kernelModules = [ "nvme-core" "nvme" "pcie-rockchip" "pcie-rockchip-host" "phy-rockchip-pcie" ];

And it also didn't work, so I'm not sure what else I can add.

[ghost]

The NanoPC-T4 will not boot if you build u-boot against a bl31.elf file. It only works if you produce a standalone trust.bin via the rkbin trust_merger utility. I have not been able to determine why this is, but Armbian ships with a custom trust.ini file with the following section which may play a role:

@tmountain, thanks again for all the time you put into this.

I've been poking at this issue some more, because I have a pile of NanoPC-T4s and I really like them. I've been able to get the NanoPC to pull a second-stage bootloader off of an SPI chip by wiring it to the 40-pin header on the board (my use case needs to keep the firmware in SPI rather than eMMC for security reasons). It definitely loads, initializes the DRAM, prints useful stuff to the serial console, and starts copying the third-stage bootloader to the DRAM, but then things go all haywire.

Have you considered the possibility that mainline u-boot simply does not have working DDR3 training for RK3399?

As far as I can tell all the rk3399 boards with known-working, blobless mainline u-boot support use DDR4 memory (LPDDR4). For example, RockPro64 and Rock Pi4. The only blobless-boot RK3399 device I know of that uses DDR3 memory is the Samsung gru-kevin -- but that uses coreboot, not u-boot. And I know for a fact that mainline u-boot on gru-kevin does not work (or did not work as of February; there seem to have been a few more commits in March).

I'm starting to suspect that the u-boot folks have never actually tested their RK3399 DDR3 training routines. They seem to have checked in defconfigs for a huge pile of rk3399 boards without testing many of them... basically TODO/FIXME placeholders. At least that's what I suspect.

Let me know if this sounds plausible.

FWIW my interest here is not specifically Nix-related, although I am learning nixpkgs (outside of NixOS).