initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory #84976

sjau · 2020-04-11T05:50:26Z

Describe the bug
Nixos changed from Dropbear to OpenSSH in the initrd. While doing so it requires new keys and new options were add.

However following the instructions still fails to build.

To Reproduce
Steps to reproduce the behavior:

I first generated a key at /root/initrd-openssh-key and added the following to the configuration.nix:

            hostKeys = [ "/root/initrd-openssh-key" ];

Rebuilding failed with this error:

cp: cannot stat '/root/initrd-openssh-key': No such file or directory

However that file exists:

ls -al /root/initrd-openssh-key 
-rwxr-xr-x 1 root root 227 Apr  4 07:18 /root/initrd-openssh-key

Then I followed the instructions from the module file https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/initrd-ssh.nix :

      example = [
        "/etc/secrets/initrd/ssh_host_rsa_key"
        "/etc/secrets/initrd/ssh_host_ed25519_key"
      ];
 [...]
        <prompt># </prompt>ssh-keygen -t rsa -N "" -f /etc/secrets/initrd/ssh_host_rsa_key
        <prompt># </prompt>ssh-keygen -t ed25519 -N "" -f /etc/secrets/initrd/ssh_host_ed_25519_key

So I created the /etc/secrets/initrd folder and run the two commands, which resulted in:

ls -al /etc/secrets/initrd/
total 35
drwxr-xr-x 2 root root    6 Apr 11 07:39 .
drwxr-xr-x 3 root root    3 Apr 11 07:39 ..
-rw------- 1 root root  399 Apr 11 07:39 ssh_host_ed_25519_key
-rw-r--r-- 1 root root   91 Apr 11 07:39 ssh_host_ed_25519_key.pub
-rw------- 1 root root 2590 Apr 11 07:39 ssh_host_rsa_key
-rw-r--r-- 1 root root  563 Apr 11 07:39 ssh_host_rsa_key.pub

I then added the following to my configuration.nix nix:

    boot.initrd.network = {
        enable = true;
        ssh = {
            enable = true;
            port = 2222;
#            hostECDSAKey = /root/initrd-ssh-key;
            hostKeys = [ 
                "/etc/secrets/initrd/ssh_host_rsa_key"
                "/etc/secrets/initrd/ssh_host_ed25519_key"
            ];
            authorizedKeys = [ "${mySecrets.auth_ssh_key1}" "${mySecrets.auth_ssh_key2}" ];
        };
        postCommands = ''
            echo "zfs load-key -a; killall zfs" >> /root.profile
        '';
    };

and rebuild again. This time it failed:

cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory
builder for '/nix/store/pff2dr2i17mjsjavkg6icy5wfixjjlyb-extra-utils.drv' failed with exit code 1
cannot build derivation '/nix/store/i33w1h0asyc33yl1m5f1cl1wpif3xmf0-stage-1-init.sh.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/aqmbw803m7rb4k38lrvfj11zbs1pb81m-initrd-linux-5.4.30.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/zcqgphq575pzzjf7kf7dvcpj5v3zxbbg-nixos-system-subi-20.09pre220429.9b0d2f3fd15.drv': 1 dependencies couldn't be built
error: build of '/nix/store/zcqgphq575pzzjf7kf7dvcpj5v3zxbbg-nixos-system-subi-20.09pre220429.9b0d2f3fd15.drv' failed

Expected behavior
It should properly detect the file provided in the hostKeys.

Metadata
Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

system: "x86_64-linux"
host os: Linux 5.5.13, NixOS, 20.09pre218613.ae6bdcc5358 (Nightingale)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.3.3
channels(hyper): "nixos-19.09pre185402.5eac339829b"
channels(root): "nixos-20.09pre220429.9b0d2f3fd15"
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module: initrd-ssh.nix

The text was updated successfully, but these errors were encountered:

sjau · 2020-04-11T05:52:33Z

@emilazy

sjau · 2020-04-11T07:52:11Z

I see an error in the code:

cp -Lr ${source'} "$out/secrets/${dest}"

https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/stage-1.nix#L148

But even fixing it to:

cp -Lr "${source}" "$out/secrets/${dest}"

won't make the error go away.

Also the mismatch in the examples:

        "/etc/secrets/initrd/ssh_host_ed25519_key"

vs.

        <prompt># </prompt>ssh-keygen -t ed25519 -N "" -f /etc/secrets/initrd/ssh_host_ed_25519_key

didn't fix the problem.

emilazy · 2020-04-11T14:54:57Z

Thanks for the report, and sorry for the broken code; I'll take a look at this. Note that this code path is only triggered when you use a bootloader without native initrd secrets support (which also exposes your host keys to the global Nix store), so using systemd-boot would probably avoid it. This should definitely still work though, so I'll poke at it; the tests are meant to exercise this code path but I guess not well enough.

sjau · 2020-04-11T14:59:17Z

The ECDA key with Dropbear worked just fine.

I fail to see how systemd-boot could be integrated into the initrd to provide early ssh access - which is needed to remote unlock encrypted zfs root.

emilazy · 2020-04-11T15:00:49Z

I meant that the SSH host key copying code that is breaking triggers only when the bootloader you're using doesn't support initrd secrets natively in NixOS, which systemd-boot does but GRUB doesn't.

I'm looking into fixing this, but I suspect setting hostKeys = [ /etc/... ] without the quotes will function as a workaround; it will explicitly copy the host key to the store ahead of time, which matches the behaviour of the test.

sjau · 2020-04-11T15:03:53Z

So I commented out the hostKeys entries and tried to rebuild:

    # Remote ZFS Unlock
    boot.initrd.network = {
        enable = true;
        ssh = {
            enable = true;
            port = 2222;
#            hostECDSAKey = /root/initrd-ssh-key;
#            hostKeys = [ 
#                "/etc/secrets/initrd/ssh_host_rsa_key"
#                "/etc/secrets/initrd/ssh_host_ed_25519_key"
#                "/root/initrd-openssh-key"
#            ];
            authorizedKeys = [ "${mySecrets.auth_ssh_key1}" "${mySecrets.auth_ssh_key2}" ];
        };
        postCommands = ''
            echo "zfs load-key -a; killall zfs" >> /root.profile
        '';
    };

Which ended up in:

error: 
Failed assertions:
- You must now pre-generate the host keys for initrd SSH.
See the boot.inird.network.ssh.hostKeys documentation
for instructions.

(use '--show-trace' to show detailed location information)

emilazy · 2020-04-11T15:06:19Z

Try hostKeys = [ /root/initrd-openssh-key ];.

sjau · 2020-04-11T15:07:16Z

You were right about the quotes. Using:

            hostKeys = [ 
                /etc/secrets/initrd/ssh_host_rsa_key
                /etc/secrets/initrd/ssh_host_ed_25519_key
#                "/root/initrd-openssh-key"
            ];

and the above fix:

cp -Lr "${source}" "$out/secrets/${dest}"

made it build successfully.

sjau · 2020-04-11T15:07:50Z

can't reboot now because I'm in the middle of a resilver and then scrub process.

emilazy · 2020-04-11T15:09:32Z

Okay, so there are two issues here: the typo and #85000. The former is easy to resolve, but I'm not sure about the latter; there's basically nothing that can be done with a bare path only accessible by root inside a sandboxed Nix build. So I think this would probably require a rework of the initrd secrets implementation to fix properly, and in the meantime the best thing that could be done would be to give a more useful error message at NixOS evaluation time if you use a string path when initrd secrets support is disabled; I'll work on the latter.

Edit: there's actually no typo; source' is set a couple lines up.

sjau · 2020-04-11T15:11:07Z

Thanks for your hard work. I'll give feedback about initrd ssh boot once resilver and scrub have completed.

Fixes NixOS#84976.

sjau · 2020-04-14T16:41:45Z

Looks better now. Thanks :)

Fixes #84976. (cherry picked from commit 91c6809)

Fixes NixOS#84976. (cherry picked from commit 91c6809)

baracoder · 2020-06-16T12:05:24Z

@sjau Could you explain, how to work around that problem? I think I am missing something.
I am getting a similar error when trying to set up a new system on nixos-unstable on a remote server with

# I have to use a system without efi support
boot.loader.grub.enable = true;
boot.loader.grub.version = 2;
boot.loader.grub.efiSupport = false;
...
boot.initrd.network = {
  enable = true;
  ssh = {
    enable = true;
    port = 2222;
    hostKeys = [
      ./boot_ssh_host_ed25519_key
      ./boot_ssh_host_rsa_key
    ];
...
};

when running nixos-install I get

updating GRUB 2 menu...
cp: cannot stat '/mnt/etc/nixos/boot_ssh_host_ed25519_key': No such file or directory
failed to create initrd secrets

Edit: I installed 20.03 first, then updated to unstable and was able to build it, placing a copy of the keys everywhere it complained about, eg cp: cannot stat '/mnt/etc/nixos/boot_ssh_host_rsa_key': No such file or directory
I think now I get the issue somewhat, that secrets of every generation are recreated on each build

sjau · 2020-06-16T19:26:19Z

I use:

  # Use the systemd-boot EFI boot loader.
  boot.loader.systemd-boot.enable = true;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.cleanTmpDir = true;

  boot.initrd.network = {
      enable = true;
      ssh = {
          enable = true;
          port = 2222;
          hostKeys = [ /root/ssh_host_ed_25519_key ];
          authorizedKeys = [ "${mySecrets.auth_ssh_key1}" "${mySecrets.auth_ssh_key2}" ];      };
      postCommands = ''
          echo "zfs load-key -a; killall zfs" >> /root/.profile
      '';
  };
  boot.initrd.kernelModules = [ "e1000e" "r8169" ];
  boot.initrd.postDeviceCommands = lib.mkAfter ''
    zfs rollback -r tankTest/encZFS/v/:rootfs@blank
  '';

and on non-EFI

    # Use the GRUB 2 boot loader.
    boot.loader.grub.enable = true;
    boot.loader.grub.version = 2;
    # Define on which hard drive you want to install Grub.
    boot.loader.grub.devices = [
        "/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E4UPT1LH"
        "/dev/disk/by-id/ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E0884719"
    ]; # or "nodev" for efi only

    boot.initrd.network = {
        enable = true;
        ssh = {
            enable = true;
            port = 2222; 
            hostKeys = [ /root/ssh_host_ed_25519_key ];
            authorizedKeys = [ "${mySecrets.auth_ssh_key1}" "${mySecrets.auth_ssh_key2}" ];
        };
        postCommands = ''
            zpool import tankMedia
            echo "zfs load-key -a; killall zfs" >> /root/.profile
        '';
    };
    boot.initrd.kernelModules = [ "e1000e" ];

nixos-discourse · 2021-11-18T21:15:08Z

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/early-boot-remote-decryption/16146/1

this does indeed seem to be important, according to numerous discussions. NixOS/nixpkgs#84976 https://discourse.nixos.org/t/early-boot-remote-decryption/16146

sjau added the 0.kind: bug Something is broken label Apr 11, 2020

emilazy mentioned this issue Apr 11, 2020

boot.initrd.secrets is undocumented and appears broken #41608

Closed

emilazy mentioned this issue Apr 11, 2020

NixOS initrd secrets fallback support shouldn't run as part of a build #85000

Open

emilazy added a commit to emilazy/nixpkgs that referenced this issue Apr 11, 2020

nixos/stage-1: check secret paths before copying

91c6809

Fixes NixOS#84976.

emilazy mentioned this issue Apr 11, 2020

nixos/stage-1: check secret paths before copying #85004

Merged

10 tasks

veprbl added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label Apr 11, 2020

emilazy mentioned this issue Apr 12, 2020

nixos/initrd-ssh: switch from Dropbear to OpenSSH #82603

Merged

10 tasks

lukateras closed this as completed in #85004 Apr 16, 2020

lukateras pushed a commit that referenced this issue Apr 16, 2020

nixos/stage-1: check secret paths before copying

f2d4179

Fixes #84976. (cherry picked from commit 91c6809)

stigok pushed a commit to stigok/nixpkgs that referenced this issue Jun 12, 2020

nixos/stage-1: check secret paths before copying

827a5f6

Fixes NixOS#84976. (cherry picked from commit 91c6809)

wizeman mentioned this issue Oct 23, 2020

20.03 -> 20.09 regression: unable to deploy new (20.09) configuration due to initrd secrets of previous (20.03) configuration #101462

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory #84976

initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory #84976

sjau commented Apr 11, 2020 •

edited

Loading

sjau commented Apr 11, 2020

sjau commented Apr 11, 2020

emilazy commented Apr 11, 2020 •

edited

Loading

sjau commented Apr 11, 2020 •

edited

Loading

emilazy commented Apr 11, 2020

sjau commented Apr 11, 2020 •

edited by SuperSandro2000

Loading

emilazy commented Apr 11, 2020

sjau commented Apr 11, 2020

sjau commented Apr 11, 2020

emilazy commented Apr 11, 2020 •

edited

Loading

sjau commented Apr 11, 2020

sjau commented Apr 14, 2020

baracoder commented Jun 16, 2020 •

edited by SuperSandro2000

Loading

sjau commented Jun 16, 2020 •

edited

Loading

nixos-discourse commented Nov 18, 2021

initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory #84976

initrd ssh: cp: cannot stat '/etc/secrets/initrd/ssh_host_ed25519_key': No such file or directory #84976

Comments

sjau commented Apr 11, 2020 • edited Loading

sjau commented Apr 11, 2020

sjau commented Apr 11, 2020

emilazy commented Apr 11, 2020 • edited Loading

sjau commented Apr 11, 2020 • edited Loading

emilazy commented Apr 11, 2020

sjau commented Apr 11, 2020 • edited by SuperSandro2000 Loading

emilazy commented Apr 11, 2020

sjau commented Apr 11, 2020

sjau commented Apr 11, 2020

emilazy commented Apr 11, 2020 • edited Loading

sjau commented Apr 11, 2020

sjau commented Apr 14, 2020

baracoder commented Jun 16, 2020 • edited by SuperSandro2000 Loading

sjau commented Jun 16, 2020 • edited Loading

nixos-discourse commented Nov 18, 2021

sjau commented Apr 11, 2020 •

edited

Loading

emilazy commented Apr 11, 2020 •

edited

Loading

sjau commented Apr 11, 2020 •

edited

Loading

sjau commented Apr 11, 2020 •

edited by SuperSandro2000

Loading

emilazy commented Apr 11, 2020 •

edited

Loading

baracoder commented Jun 16, 2020 •

edited by SuperSandro2000

Loading

sjau commented Jun 16, 2020 •

edited

Loading