[GSoC 2023]: Simple Vault

[solokyo]

Hi,

I am Jichen Xu, glad to be selected as GSoC 2023 contributor. May we have a pleasant and productive collaboration in our work this year.

Currently pgagroal already implemented frontend user mode to allow users access the database without the actual database password, and we can make it a simple vault base on it. This way, we can periodically change the frontend password and improve security.

Several things need to be done:

1. Implement a new program pgagroal-vault as a simple HTTP server so that it can listen and response to user requests.
2. Define and implement messages between pgagroal-vault and pgagroal.
3. Implement dynamic secrets. pgagroal will periodically change front-end secret. Users have to fetch a new secret every n seconds.
4. (Additional Goal) Enable pgagroal-vault to support x.509 authentication.

My proposal is here, you can read it for more details.

Any suggestions would be appreciated.

[jesperpedersen]

👍

[solokyo]

Regarding the vault feature we want to implement, there are some details I'd like to discuss:
Based on our previous discussion, once the vault feature is enabled, the client will first communicate with pgagroal-vault to obtain a one-time password. Then, the client will use this password to communicate with pgagroal and access the database.

To implement this, I plan to create a pgagroal-vault command for the client to initiate requests and a pgagroal-vault-daemon to listen and handle these requests.
Alternatively, we could configure a vault_sock in main.c to listen to these requests, but pgagroal-vault command will still be needed.

Regarding the automatically changing frontend password:
I am considering whether to implement a one-time password or a password that is valid multiple times within a certain period.
This might depend on the way the client interacts with the database.

1. If there are multiple short communications, a timed password might be more reasonable. This way, the client can avoid repeatedly asking the vault for a password.

2. If there's a longer communication, a one-time password might be more appropriate, so the password only changes when necessary, rather than periodically.
   For the existing frontend password, should we upgrade it to an automatically changing password or keep them as two separate features?
   If we choose to upgrade, we might remove the pgagroal_frontend_users.conf and move the mapping of <front_user_name, password_hash> into memory.

[jesperpedersen]

To implement this, I plan to create a pgagroal-vault command for the client to initiate requests and a pgagroal-vault-daemon to listen and handle these requests.
Alternatively, we could configure a vault_sock in main.c to listen to these requests, but pgagroal-vault command will still be needed.

The pgagroal-vault should host the HTTP endpoint that the clients communicate with, as well as the logic to trigger pgagroal to update the front-end passwords every X seconds.

Once the X seconds are up, then pgagroal changes the password which in-turn will invalidate the connection upon return to the pool.

The vault should be a more advanced version of the existing pgagroal_frontend_users.conf functionality.

[solokyo]

Some details i want to discuss:

1. Currently, our HTTP server only has a get_password_by_username service. I plan to support this service through the HTTP GET method with Query parameters, but there is one point to confirm:
   According to the definition in RFC 7231, the GET method should be idempotent. I believe even if the password changes over time, as long as the results of requests at the same time are consistent, it is idempotent.

2. Are there any other services we need to support?
   The current implementation of the HTTP server already has the functionality to parse URLs and query parameters, is able to support a few more services. However, if we have to support many more services, we may need to consider refactoring the code or introducing external dependencies.

3. How many types of messages will there be between vault and pgagroal?
   Assuming that we currently only support get_password_by_username. I think there will at least be a change_password message. Pgagroal can return the latest password in the response message, and if we cache passwords on the vault side, we may only need this one message.

4. Should we implement some sort of "Lazy Cycling"?
   We can only updating the password periodically when a certain username is in use. Suppose we set the password to update every 30s, if no client requests a password for a particular username for more than 1 minute, we can assume that the username is not in use, so we don't send a message to pgagroal to update the password.

5. Should we add any configurable options to vault?
   Options like password_update_cycle would be a good choice. And I assume we also support modifying these configurations via cli at runtime.

6. Do we need to use the mlock system call to prevent map<username, password_hash> from being swapped out due to a page fault?

7. Random password generation function, there is already a static char* generate_password(int pwd_length) in admin.c. I'm considering moving this to security.c for reuse.

I will gradually clarify my TODO list in the discussion and update its progress.
If you want to check my WIP code I can send a PR.

Progress:

• A minimal usable HTTP server, so far without any external dependencies, and it also supports HTTPS.
• Messages between vault and pgagroal.
• Handler functions corresponding to these messages.
• Random password generation on the pgagroal side, and the management of map<username, password_hash>.
• Config options, and the ability to modify the config via CLI at runtime.

[jesperpedersen]

1. Set the Expires HTTP header to the timestamp served
2. We only need to support getting a password for the user aka http://XXX:YYYY/myuser
3. We need to get the password, and a command to change all of them - so 2
4. Nah, just update every X seconds based on the configuration
5. Yeah, the X seconds should be a configuration option - of course the rest; host, port, ...
6. No, just use the struct configuration hierarchy
7. Yes, move it and implement all ASCII password - https://silverhammermba.github.io/password/

[fluca1978]

I'm coming a bit late to this, but I'd like to add a few notes.

First of all, why not extending/refactoring the pgagroal-admin command to enable the management of the valut settings?
I would also like to have a per-user/per-database vault management, so that it is possible to configure a user for all database or only for specific databases only.
I also think that pgagroal_frontend_users.conf should be enhanced with a new field that tells if the user is going to be "vaulted" or not, so that pgagroal knows if the user password should be used statically or dynamically.

Moreover, I'm not sure pgagroal should handle the password/secret expiration. I would let pgagroal to ask the vault if the secret is still valid upon a new connection being returned from the pool.

with regard to HTTP GET being idempotent, I think it is fine to use GET even if that exposes a risk to be flooded with requests for getting a new password with randomly generated users. I would prefer to do that in POST and use a shared secret between the vault and the pgagroal so that the vault can answer only to request coming from a legitimate pgagroal service.

[solokyo]

First of all, why not extending/refactoring the pgagroal-admin command to enable the management of the valut settings?

The current design of vault is very simple: give every user added by pgagroal-admin add-user a dynamic frontend password.
If we are going to implement more features, we might need to add more options to pgagroal-admin.

I also think that pgagroal_frontend_users.conf should be enhanced with a new field that tells if the user is going to be "vaulted" or not, so that pgagroal knows if the user password should be used statically or dynamically.

The vault will be an enhanced version of the current frontend user&password feature. Once the vault is implemented and enabled, there will be no static frontend password.

Moreover, I'm not sure pgagroal should handle the password/secret expiration. I would let pgagroal to ask the vault if the secret is still valid upon a new connection being returned from the pool.

Agree on this, a vault should conceptually keep and manage every secret within itself. The vault that I implementing now is nothing but an HTTP server that passing messages between client and pgagroal.

[jesperpedersen]

The pagroal_froendend_users.conf file splits user and PostgreSQL passwords - that is the main goal. So, this will always be in place for a simple solution.

The vault idea - and this GSoC project - is to provide a simple solution to having the front end password change, and the foundation in pgagroal to have an infrastructure for more advanced vault solutions in the future - how is https://www.vaultproject.io/ going to integrate ?

For now, we should be focused on how to make the user password change, how to serve it, and how to secure it. Yes, maybe the documentation will assume a secure network, but that should be explained. Once the initial solution is in place there are likely going to be many places for improvements - f.ex. I would like to see HTTPS support for starters...

So, having a separate binary with the vault functionality is the best way forward right now - where things are located is a question once pull requests are filed

[jesperpedersen]

Well, https://www.vaultproject.io/ as an example - not as a priority since they are doing the BSL license now

[solokyo]

#378
I opened a new PR. Please have a look.