[RFC 0059]: Systemd Service Secrets

[d-goldin]

Hi,

This is a first draft for an RFC about a possible approach to secrets management for systemd services.

Curious to hear your thoughts.

Rendered

[turion]

I find this a great idea in total, and one which would bring NixOS a step closer to a pain-free devops' choice.

Side note: This approach does not prevent one service from reading the secrets of another, e.g. through privilege escalation or remote code execution. But I think that issue is beyond the scope of this RFC, and should better be handled with something like containers.

[turion]

How does the secret store deal with system/main config upgrades? When I uninstall an old service and install a new one that asks for a secret of the same name, does the new service get access to the old secret?
When I uninstall a lot of services, are old secrets garbage collected? Do downgrades always work seamlessly?

Can you outline the possible pitfalls regarding that topic in an extra section?

[d-goldin]

In the current shape, no such management would happen. The secrets would just remain in the secrets store and if a config author decides to re-use them, then so be it. I can include that. I was initially thinking of the drawbacks of "sth that would be worse when adding this", but I guess it depends on the reference point. Definitely good points to add, even if they would remain unhandled.

[turion]

Well, it's maybe a strawman, but if you put a secret in /nix/store, it's going to vanish upon garbage collection when I uninstall the package. If I don't care whether it's readable by root or by all users (e.g. inside a container), this is a drawback now because no such garbage collection would take place.

As in "something that would be worse when adding this", you could also say that people possibly expect that NixOS doesn't do any state management by default and that downgrading will bring their machine back into the exact state, but that's not the case anymore.

[turion]

you could also say that people possibly expect that NixOS doesn't do any state management by default

The difference being that before this RFC, I'd manually handle the secret state and remember to handle it. After all, I'm deleting the line saying service.foo.secretsFile = "/foo/bar";, and when I delete the line, I'll delete the file. This is now not the case anymore. Instead, I need to remember how this semiautomatic secret handling works.

[globin]

if you put a secret in /nix/store

It is exactly the purpose of this RFC not to have secrets in the nix store. This is possible without any problems and if they are referenced in the nixos config they would not be garbage collected as currently is the case. And this can still be achieved with this proposal as before.

If I don't care whether it's readable by root or by all users (e.g. inside a container)

Actually currently the nix store of (nixos) containers is shared with the host and all other containers, so that might be an issue nonetheless.

With regard to secret state handling, a lot of services have the possibility do use a passwordFile, secretFile, etc. but there is no abstraction for handing out permissions to secrets correctly, which is something like the proposal in this RFC is needed. Currently you have to not only drop the files on the machine yourself in the correct destination (same with this RFC, as long as you are using a fetcher that requires that), but also set the correct permissions for the respective service, which might be a chicken-and-egg problem of adding users for service by enabling it and the service requiring the secret.

[turion]

I think it would be better to transition some non-critical services first, to gain some experience without breaking anything. If this works for 10+ regularly used non-critical e.g. webapps, then transition everything to this approach.

[d-goldin]

I will fix the wording on that one. Parts of this section are maybe still a bit too "note to self"-like. What I intended was not for "inclusion into upstream" but more to hypothetically verify this is covering all the most important aspects needed for the most critical services. Your version is definitely better for an actual slow inclusion into upstream step.

[Ericson2314]

The detailed design is quite long. I think it's better to clean up the Nix secrets story before moving downstream, and I suspect trying to do everything at once will leave us stuck.

[globin]

@Ericson2314 I think this proposal is orthogonal to the secrets-in-nix-store issue and that this is quite a nice way to move forward without getting in the way of that, as this RFC abstracts over where the secrets are fetched from. In the case of nix store secrets being implement one could easily add a fetcher to use that, but also keep the abstraction in place for remote secret management systems if for some reason you are required to use that.

But obviously I'd be very happy if #5 or similar would be continued!

[globin]

Side note: This approach does not prevent one service from reading the secrets of another, e.g. through privilege escalation or remote code execution. But I think that issue is beyond the scope of this RFC, and should better be handled with something like containers

@turion I'm not sure how you mean this? This as far as I can see is possible with this RFC as you can also use non-FS fetcher functions. And obviously always depends on how far you want to go with isolation.

[turion]

Side note:

It was really just a comment that all secrets are still root-readable, and that this RFC is orthogonal to more isolation/virtualization.

[aanderse]

From reading this over it seems once a module uses this proposal there is no opt out for the user, correct?

If say the drawback is that this adds more complexity to the system. When I was new to nix I always enjoyed being able to read a modules source and usually understand what is happening... translate things into what a regular distro does. This is an entirely new abstraction that has no equivalent in a regular distro and after reading through it twice I still feel very uncomfortable thinking about how I can explain/justify this mechanism to my co-workers who manage nixos servers with me, but have not dived into nixos enough to understand all the nuances yet. I foresee having to answer many questions to puzzled looks from people wondering why we can't continue to keep doing what we're doing, and wondering why, as non "nix people", they no longer understand module source code.

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

[globin]

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

I think that this rather allows us to consistently use something like vault or some otherwise created secrets folder etc. and not having to depend on every module implementing support for it. It does create a further abstraction but by that allows all modules to have a consistent way of passing secrets and not a different implementation for each of them.

[aanderse]

@globin I meant a single solution like vault could be chosen by the sysadmin for every secret on a machine and then module code could continue as is, unmodified. This provides greater flexibility, though currently at the cost of upfront work for the sysadmin. That being said I concede that the open issues regarding secrets in the nix store have been open for years without resolution, and no one has provided a nice to use module for something like vault that integrates nicely into nixos yet.

I guess some sort of system/module that keeps this logic out of the module system would have been my desired outcome.

I hope my opinion has provided at least minimal value as a differing point of view.

[globin]

The thing is that is not possibly currently.

For all modules that only allow passing secrets by string (secret content) you'd have to modify it to pass in the secret differently as it would otherwise be added to the nix store nonetheless.

This change should happen anyway even without this proposal and a lot of modules now at least allow passing by file, where one can choose to use a non-nix-store-file with more specific permissions or just a pkgs.writeText in one does not care. For the ability to use something like vault one would still have to create new services, similar to the proposal here. That's why I think it would be nice to have an abstraction so that we can standardise all services handling secrets to have a consistent interface.

[aanderse]

@globin I entirely agree that all modules need to replace any password options with passwordFile (or the like) regardless of this RFC, and that is something I (along with many others) have slowly been working toward. I also agree that pkgs.writeText in conjunction with passwordFile is the proper justification for entirely eliminating any password like options, like I proposed when I removed the dbPassword option from the zabbixServer module.

[d-goldin]

@aanderse:

From reading this over it seems once a module uses this proposal there is no opt out for the user, correct?

Do you mean for a module author or a user? If a module is switched to this, then no, without "more involved" fiddling about a user wouldn't be able to change that unless author has created multiple avenues. We could of-course add a "secret store" that is just the nix store as some sort of fallback for scenarios where people don't care and they can at least go and add their secrets to the nix store as before.

If say the drawback is that this adds more complexity to the system. [...]. I foresee having to answer many questions to puzzled looks from people wondering why we can't continue to keep doing what we're doing, and wondering why, as non "nix people", they no longer understand module source code.

This is of course different perceptions, but for me NixOS is already quite an opinionated distribution that does a lot of things quite differently and brings a few abstractions that no other distro has.
Imho pretty much anything new of this sort will have the issue of change/surprise/added complexity.

I wonder, why does this particular code/interface seem more complex than for instance all the module evaluation and merging logic in https://github.com/NixOS/nixpkgs/blob/master/lib/modules.nix? It is of course possible to simplify and document to help people understand it more easily, once we understand what points of confusion exist.

I also wonder if a product similar to vault should be put in the alternatives section. I'm under the impression that while this may not be an entirely solved problem, there are existing solutions out there that can fit.

Well, yes. I thought of this as a given, that people can still go and use vault directly if they want/their software supports it. If all things we are dealing with would support sth like vault, then it would be great. But as @globin said, we'd still want the ability to plug in another service to manage secrets if need be, so some abstraction would likely emerge.

I guess some sort of system/module that keeps this logic out of the module system would have been my desired outcome.

Could you please explain a little bit what exactly you mean? Something like this, but being just some library living separately from NixOS in general, or is this more about how such an approach would be integrated with the module system? For instance, instead of the modules using this interface directly, the user of a module calling some functions on it?

Thanks, I will keep the points in mind when I do the next pass.

[edolstra]

This seems unnecessarily verbose. I would do something like this:

systemd.services.foo = {
  needsSecrets = [ "secret1" ];
  ...
};

and then our systemd module can generate whatever helper units are necessary.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. in

  secretsScope1 = mkSecretsScope {
     loadSecrets = [ "secret1" "secret2" ];
     type = "folder";
   };
  secretsScope2 = mkSecretsScope {
     loadSecrets = [ "secret1" "secret2" ];
     type = "folder";
   };

secretsScope1 and secretsScope2 are actually the same scope.

[d-goldin]

This seems unnecessarily verbose. I would do something like this: [...]

I agree, it's more compact. Why I initially decided against it and for something that modifies the resulting structure separately was to reduce changes to the existing systemd modules. At the same time I thought it would be nice to be able to get the secrets as arguments, which I thought made it nicer to deal with in code. In the needsSecrets case, if I understand it correctly, the user would be requesting a secret by its name, like in the scope creation, but then would have to possibly deal with paths (as strings) to pass the secret as an argument to the service, or load it into an env-var, because there would not be an automatic mapping mechanism anymore. Unless we pack it up into a shell variable or so.

It also avoids imperative-sounding function names like mkSecretsScope which suggest that they allocate a unique new scope, but that's not the case, e.g. ...

I do not necessarily perceive mk* as an imperative terminology, but that depends on the reader. We have a lot of pure mk* functions that construct some structure. But I'm not at all attached to the naming here, so we can change it to whatever seems more suitable if they're still around further down the road.

Edit: In fact, I'm not attached to most of the terms in the RFC, such as "sidecart" and similar. I merely picked them from what I thought would make it easily enough understood. So if there are suggestions to rename things, I'm up for it.

[d-goldin]

@edolstra: Did this sufficiently address your remarks? I'm not super familiar with the inner workings of the process, so for now I just left those as discussion comments here, but I'm willing to incorporate some of the things pointed out into "alternatives" or the core section, if there is some consensus around that.

[d-goldin]

So, getting back to this suggestion - if somebody could point me to the simplest way of implementing it with such an additional field on any systemd service without needing to change too many things in the guts of core modules I'm willing to try and adjust the poc to see how that works. One thing I do kinda like though is the ability to have some at least very basic checking of secrets used, which wouldn't work the same way with just strings.

[edolstra]

It's not clear to me why the helper unit is needed. Can't the keys be fetched using an ExecStartPre=+... command?

More generally, instead of creating our own mechanism for passing keys to services, maybe the kernel keyring mechanism can be used for this? Units would call keyctl request/search/read/... to fetch keys. These keys would either be preloaded into the keyring or produced on demand using the request-key program.

[d-goldin]

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

Regarding kernel keyring mechanism - I agree, it might be a good default backend. The directory based thing was just the dumbest proof-of-concept case I came up with (given that similar approaches are used in nixops and krops/stockholm). Part of the intention is to have a somewhat agnostic interface.

[arianvp]

Kernel keyring mechanism sounds overkill to me. Its usecase is not to communicate files between userland processes, but between userland and kernel drivers. Files are a perfectly sufficient abstraction for passing secrets around in userland.

[flokli]

I have mixed feelings about this.

Using files usually implies having to worry about ACLs and who's allowed to access them. We can cheat by mounting them in a private namespace, but it's still a bit cumbersome.
Sometimes you want to have "use once" properties and provide a new key on every read / issue tokens on access etc. We could cheat again by providing these key files by a fuse filesystem, but then it just gets more complicated.

The kernel keyring might be a good abstraction over all this, it's just not widely adapted currently and lacking real-world usage. Reading keyrings (7) looks promising. In addition to thread/process/session, there's also an upcall feature, bouncing back to userspace to request secrets which could be a request to whatever credentials provider is used.

I'd love to experiment with that a bit, or see some real-world examples. Anybody aware of these?

[poettering]

kernel keyring is very much intended for userspace keys too. See the kerberos stuff that has been ported to use it for that, or systemd's cryptsetup.

I think the kernel keyring has deficiencies (upcalls, yuck! also no namespacing for containers, …), but it probably is the right approach in the long run.

[bennofs]

I briefly tried ExecStartPre, but unfortunately it does not seem to run within the mount namespace, so it doesn't have a access to the PrivateTmp we want. I am not sure if this is intentional or not.

I think that's what the ! prefix is for:

"!" - Similar to the "+" character discussed above this permits invoking command lines with elevated privileges. However, unlike "+" the "!" character exclusively alters the effect of User=, Group= and SupplementaryGroups=, i.e. only the stanzas that affect user and group credentials. Note that this setting may be combined with DynamicUser=, in which case a dynamic user/group pair is allocated before the command is invoked, but credential changing is left to the executed process itself.

[Lassulus]

@dhess did you post a summary of the meetup somewhere else? I can't find one

[dhess]

@Lassulus Sorry, I haven't posted the meeting notes yet. I'll get to it soon.

[dhess]

Sorry for the delay in posting the meeting minute notes, but here they are.

Notes from 2020-05-06 call

Attending:

• @dhess
• @d-goldin
• @flokli
• @Lassulus
• @aanderse
• @andir

@d-goldin reported the following:

• Nobody has stepped up to co-author yet.
• There's now a proof-of-concept repo: https://github.com/d-goldin/nix-svc-secrets/blob/master/lib/secrets.nix
• The Hashicorp Vault implementation in this proof-of-concept implementation isn't very robust at this point.
• @poettering's proposal needs more review, but provides some good primitives. It probably doesn't solve all the NixOS problems, e.g., if we want Vault we'd still need some way of integrating with it. It only provides a few mechanisms. How does a user of a service module describe where secrets would live

@dhess & @flokli agreed to try out the implementation before the next meeting.

@flokli:

• Will open a discussion with @poettering. NixOS should be involved from the start.
• How we expose any systemd hooks from the NixOS module system still needs to be solved.
• Will ask @poettering about other mechanisms, e.g., Vault.

Given @poettering's intention to work on related things in systemd, a vote was taken on this RFC, whether to proceed with the work, pause it for a bit until systemd work is better fleshed-out, or close the proposal. The votes were as follows (@d-goldin did not vote):

• @dhess: continue
• @aanderse: pause for a month or two
• @Lassulus: continue
• @flokli: continue
• @andir: continue

The vote was 4 : 1 : 0 (continue : pause : close), so we'll proceed with the RFC in parallel with the systemd work. Any further work should be done with the intention of integrating with systemd's primitives (if any) when the time comes.

[d-goldin]

@dhess: s/@globin/@andir

[dhess]

@d-goldin Thank you. I've updated the notes. @andir I'm sorry!

[dudebout]

Noting a couple typos.

[Mic92]

@dhess What are next steps for this RFC. Another meeting?

[dhess]

@Mic92 Yes, eventually, but not much has changed since the last one, so I don't know if it's warranted at this time. I haven't had any time to try out the proof-of-concept implementation yet.

@flokli How about you, have you tried it? Also, is there any update on the discussions with upstream?

[aanderse]

Thought I would link this here for future reference. Possibly of interest to people subscribed to this thread: https://cedwards.xyz/protecting-systemd-service-secrets/

[Mic92]

We have now a pass based secret implementation and one based on sops.

[d-goldin]

@dhess, @flokli: Have you had a chance to give it a try, as a follow-up from the last call?

[d-goldin]

As things stand, I don't think this will progress much, so I propose to close this.

[d-goldin]

Thanks for everyone's feedback, ideas and other contributions. I hope they'll be useful at the very least as a reference for other efforts.

[Mic92]

Systemd actually has this feature now: systemd/systemd#16568
It just got merged and we could start using it in NixOS as well.