Managing Secrets
This page contains content which is intended to be merged in the overall documentation at a later time.
Secrets are parts of an application’s configuration (typically Externalized Environment Configuration) that are specific in the following aspects:
-
They need to be protected from unauthorized access
-
They can be dynamic in that sense that values can change during the runtime of your application process, e.g. if a secret is an access token with an expiry time
Protecting secrets is typically done by encryption and/or managing them by a service that performs access control. This creates a typical Chicken and Eggs problem: To get access to a secret, your application will need another secret. This can be a key that allows you to decrypt the secret and/or an access token that your application provides to a secrets management service.
If you pass e.g. a key for decryption via "JAVA_OPTS" to your application but have the key in plain text in a startup script, you may just end up with Security by Obscurity. If an attacker got full access to your environment it is just a question of time until your encrypted secrets are accessible.
Of course there is a way to avoid the Chicken and Eggs problem: Don’t store secrets in your application context but let a runtime "infrastructure" pass them to your application. If you let the infrastructure manage your secrets, it can handle also the renewal of expiring secrets by sending your application a signal that causes your application to accept an updated value for the secret.
The following sections describe variants for this approach. As two examples for the "infrastructrue" we use Kubernetes and the Hashicorp tool stack ("Hashistack").
In this approach the infrastructure directly passes the secret to your application via an environment variable.
This approach works best if you need only access to a very few secrets: Passing a "master secret" to an application that is then used used to access the actual secrets is such a use case.
When Kubernetes launches a Pod, it can pass the value of a secret maintained by the Kubernetes Secrets system as an environment variable. The drawback of this approach with Kubernetes is, that it stores the secrets in the Pod definition - so you have to rely on access control for the Pod definition.
See Secrets Concept description and in particular section Using Secrets as Environment Variables.
When Nomad is used to launch your application, it can pull a token from Vault and pass it via an environment variable named VAULT_TOKEN. The job definition allows a job task to specify the particular token that it requires from Vault. Nomad will automatically retrieve a Vault token for the task and handle token renewal for the task. See Nomad Job Specification and in detail the Vault Stanza. In this case the secret does not appear in the Job definition as it is stored by Vault.
In this approach the "infrastructure" passes the secrets via a file system that is accessible to the application. The storage is temporary: Together with the termination of the application it also removes the file system. During the existence of the temporary file system, protection of your secrets depends on the level of access control that the infrastructure provides. If this is not sufficient for your protection needs, you may consider encryption as an additional means but then you are back at square one.
Kubernetes supports this approach by mounting secrets that are mapped to a path as a volume in a Pod. See Using Secrets as Files from a Pod.
Nomad makes a file system available to tasks which contains a "secrets" directory. This directory is private to each task, not accessible via the "nomad fs" command or filesystem APIs and where possible backed by an in-memory filesystem. It can be used to store secret data that should not be visible outside the task.
Nomad stores a token that it pulls from Vault in the secrets directory.
If your application needs access to multiple secrets and you don’t want to store them as part of your application configuration (e.g. because you want to avoid encryption) you can use a Secrets Management Service that the infrastructure provides. Access to the API of these services is typically controlled using Access Tokens. This means that this approach needs to be combined with one of the basic secret passing mechanisms described above to provide the access token to your application.
TODO: This section duplicates content from ./guide-configuration#password-encryption and also slightly differs. We need to align our approach and avoid such redundancies.
A simple but reasonable approach is to configure the passwords encrypted with a master-password.
The master-password should be a strong secret that is specific for each environment. It must never be committed to version-control.
Instead let the "infrastructure" pass it to your application via an environment variable.
In order to support encrypted passwords in spring-boot application.properties all you need to do is to add jasypt-spring-boot as dependency in your pom.xml(please check for recent version):
<dependency>
<groupId>com.github.ulisesbocchio</groupId>
<artifactId>jasypt-spring-boot-starter</artifactId>
<version>1.17</version>
</dependency>This will smoothly integrate jasypt into your spring-boot application. Read this HOWTO to learn how to encrypt and decrypt passwords using jasypt. Here is a simple example output of an enctrypted password (of course you have to use strong passwords instead of secret and postgres - this is only an example):
----ARGUMENTS-------------------
input: postgres
password: secret
----OUTPUT----------------------
jd5ZREpBqxuN9ok0IhnXabgw7V3EoG2pThe master-password can be configured as "JAVA_OPTS" on your target environment via -Djasypt.encryptor.password=secret (of course you will replace secret with an expansion of the respective environment variable).
Now you are able to put encrypted passwords into your application.properties
spring.datasource.password=ENC(jd5ZREpBqxuN9ok0IhnXabgw7V3EoG2p)To prevent jasypt to throw an exception in dev or test scenarios simply put this in your local config (src/main/config/application.properties and same for test, see above for details):
jasypt.encryptor.password=noneSpring Cloud Vault provides support for externalized Spring configuration in a distributed system using Hashicorp Vault.
See the Quick Start section for details how to use it in your application.
Vault requires an authentication mechanism to authorize client requests. Spring Cloud Vault Config supports multiple authentication mechanisms to authenticate applications with Vault - Token Authentication is the default mechanism.
The Spring Cloud Vault Config documentation provides examples like this to configure the authentication token in your bootstrap.yml file.
spring.cloud.vault:
token: 19aefa97-cccc-bbbb-aaaa-225940e63d76If you use Nomad in combination with Vault, you will use instead the Vault token passing mechanism of Nomad described above.
spring.cloud.vault:
token: ${VAULT_TOKEN}As an alternative you can consider using one of the advanced authentication methods of Vault: If you are using AWS you can use AWS-EC2 authentication that does not require first-deploying, or provisioning security-sensitive credentials.
With every secret, Vault creates a lease: metadata containing information such as a time duration, renewability, and more. Spring Cloud Vault maintains a lease lifecycle beyond the creation of login tokens and secrets. That said, login tokens and secrets associated with a lease are scheduled for renewal just before the lease expires until terminal expiry. See section Lease lifecycle management of Spring Cloud Vault documentation for details.
The Spring Cloud Kubernetes project provides the Secrets PropertySource feature which allows sharing secrets with containers via mounted volumes.
There is a blog of "Red Hat developers" that describes Configuring Spring Boot on Kubernetes With Secrets. It uses the Environment Variables / File System approaches described above.
It is Part-II of a article series where Part-I described how to use ConfigMaps in configuring a spring boot application on Kubernetes. The announced Part-III seems not to be released yet. The author says that it will describe how to use the spring-cloud-kubernetes spring module in more detail.
A similar text can be found in a Red Hat documentation Integrate Spring Boot with Kubernetes.
At the same time multiple projects are working on an integration of Vault with Kubernetes. The most prominent of them is the collaboration of Google and Hashicorp: one of the goals is "Using HashiCorp Vault with Google Cloud and Kubernetes" - see the announcement on the Google Cloud Platform blog.
To access a secret managed by Vault requires an access token. To obtain an access token you need another secret for authentication.
The approach to let Nomad pass a required token to the application, traces back to the question how Nomad gets access to these tokens. For such purposes Vault offers an "auth method" called AppRole. Auth methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user. The AppRole auth method allows machines or apps to authenticate with Vault-defined roles. The role represents a set of policies that define to which secrets Nomad has access.
In a productive system Nomad will operate as a high available clustered service. The credentials required for a successful authentication of Nomad for its AppRole authentication with Vault are passed during the bootstrapping of the cluster. If the access control mechanisms of your platform to protect these bootstrap credentials don’t match your needs you may want to delegate the protection and provisioning to a human user.
This is related to the bootstrap process of Vault: Starting a productive Vault includes a workflow for unsealing the Vault. Unsealing is the process of constructing the key to decrypt the data, allowing access to the Vault. Instead of distributing this master key as a single key to an operator, Vault uses an algorithm known as Shamir’s Secret Sharing to split the key into shards. A certain threshold of shards (e.g. 3 out of 5) is required to reconstruct the master key.
The unseal process can be executed via Vault’s API. This process is stateful: each key can be provided by processes on multiple computers. In theory this means that the bootstrap process could be automated and still have enhanced security by storing each shard of the master key on a distinct machine. In practice Hashicorp at the moment recommends a manual workflow for unsealing. The human users who keep the Vault master key shards will also keep credentials to log on, access an authentication token for Nomad and provide this for the bootstrapping of the Nomad cluster.
If really a fully automated cold boot of a Hashistack cluster is required, a possible workflow that meets also high security needs could look like this:
-
Store the shards of the Vault master key on different machines
-
Protect the shard with the access control mechanisms of the file system and allow access only to system users of system processes that perform the unseal process when the cluster machines boot. (Use encryption to protect the shards? Back to square one!)
-
Split the authentication token into shards using the same "Shamir’s Secret Sharing" approach and protect them the same way as the shards of the Vault master key
-
The distributed system processes that collaborate for the unsealing of the Vault as well collaborate to construct the first authentication token
-
Using this authentication token the bootstrap processes can provide credentials to the bootstrapping of the Nomad servers of the cluster that allow the Nomad servers to authenticate with their Vault AppRole.
This documentation is licensed under the Creative Commons License (Attribution-NoDerivatives 4.0 International).
