Authenticated RCE via Patroni HTTP REST API

[german-namestnikov]

Hello,

Seems like there is a way to compromise Patroni nodes with RCE in HTTP REST API. I placed a short write-up in my blog, you will find it here: https://illegalbytes.com/2020-10-14/patroni-remote-code-execution/

Changing the PostgreSQL configuration file with PATCH requests to Patroni HTTP API allows a remote authenticated* attacker to execute arbitrary OS commands. I placed the asterisk near the word 'authenticated' because most of the Patroni clusters I have met in my everyday work had no authentication enabled at least at one node, which potentially makes the whole cluster vulnerable to RCE.

This is only my thoughts, and I haven't checked this yet, but even if you have HTTP REST API authentication enabled on each node, this is still possible to compromise all cluster nodes if you have valid credentials at least for one of them.

I don't have enough experience with Patroni, so, I hope you will help me to determine the possible impact and - which is more important - mitigation.

Thanks!

[CyberDem0n]

Hi @german-namestnikov,

I will disappoint you, but you haven't discovered anything new.

First of all it is possible to protect the REST API with basic auth and/or tls, optionally check client key: https://patroni.readthedocs.io/en/latest/SETTINGS.html#rest-api.
Second, local config file always takes precedence over global configuration, therefore it is enough to have these options defined there: https://patroni.readthedocs.io/en/latest/dynamic_configuration.html
And third, there is a section in the documentation regarding security: https://patroni.readthedocs.io/en/latest/security.html.

This "vulnerability" is more or less the same as using COPY TO PROGRAM when you have postgres credentials.

And maybe a few words about why basic auth is not enabled by default:

1. Without tls basic auth doesn't really improves security (anyone could sniff unencrypted traffic)
2. If there'll be a default password in the config people are too lazy to change it, what is the same as not having a password at all.

[german-namestnikov]

Thanks for the quick and so useful response, @CyberDem0n !

Well, you didn't disappoint me. I found no mentions about this ability before my research - if it is so obvious as COPY TO PROGRAM, why I could not find any publications about this 'feature'? So, I think it is still useful :)

Authentication is Ok (and I agree with your words about default passwords) but seems like people neglect reading documentation and prefer to use ymls you provide by default. This is my own experience and I don't really think everyone has their HTTP API without auth. Just to note.

And thanks for mentioning the Security Considerations page - I am in touch with its author. Of course, I read it before submitting the issue.

Could you please tell me more about the precedence of local configs over global configuration? Do you mean that this is enough to set 'archive_command' in yml to mitigate this technique?

[CyberDem0n]

It is enough to put the following in the config.yam:

postgresql:
  parameters:
    archive_mode: on
    archive_command: ...

The same applies to restore_command and archive_cleanup_command.

In order to change the global configuration one don't even need to use the rest api, writing directly to the config key in DCS will produce exactly the same effect.
The security page was actually created from my comments in one of the previously opened issues. I don't think that someone else but me knows anything more about Patroni :)

[german-namestnikov]

Yep, this one, I know :)
I considered direct writing to DCS during my work as a bit 'out of scope', because I was interested only in Patroni at this step - I have met its REST API during the penetration testing activities.

Ok, to summarize. You don't consider this RCE as vulnerability (1), but if we want to protect ourselves from this 'feature', we could predefine secure *_command parameters in ymls (2). Is that correct?

Finally, should we mention such details as sudden RCE somewhere in documentation? At least, COPY FROM is documented and explained, and some security boundaries exist to prevent malicious use.

Thanks for answers, @CyberDem0n !

[CyberDem0n]

Both correct.
Also, locally defined postgresql.pg_hba (and pg_ident) will not allow overriding it from DCS.

[german-namestnikov]

You prevented me from opening the new issue :D

With your permission, I will list here my last thoughts. If you don't agree, I think you could just close this issue, because no consensus (ha-ha, such a nice word for HA solution security discussion!) is possible.

Let's start from the beginning. We have the Authenticated RCE, which potentially allows a remote attacker to compromise all cluster members.
To successfully exploit the Patroni cluster, the attacker needs some credentials - cert or username/password - for only one node, or at least one node in the cluster without any authentication enabled to conduct an unauthenticated attack.

If we don't consider this as 'vulnerability', let's compare this 'feature' with PostgreSQL's 'COPY FROM PROGRAM' you mentioned here before.

Which functions are used for RCE?

• PostgreSQL - use native functionality with COPY queries.
• Patroni - use native functionality of PostgreSQL, which couldn't be used for RCE in other cases, because RCE is possible only when Patroni exposes several config file values to HTTP API and allows their modifying.

Is this a regular functionality?

• PostgreSQL - yes, the COPY query is documented and possible abuse is well-known.
• Patroni - no. If yes, where I could read about that and possible RCE? Too obvious to write? :)

What is the possible impact of RCE?

• PostgreSQL - establishing malicious control over the single database server.
• Patroni - potential establishing malicious control over the whole cluster.

Who knows about potential malicious use?

• PostgreSQL - anyone, the vendor encourages users to implement security boundaries, also DISA STIG is available.
• Patroni - no one knows, except you and a bunch of guys who are interested in minor implementation details :(

Could we prevent RCE with user policies?

• PostgreSQL - yes, we could protect the superuser account and control the 'pg_execute_server_program' group membership.
• Patroni - no, no user policies, any REST API user (specified by username/password pair or by cert in 'restapi' section of the yml config) is the superuser.

Could we manage authenticated access on the network level?

• PostgreSQL - yes, we could enable network access whitelisting by modifying the pg_hba file.
• Patroni - no, we could just specify the HTTP API IPs.

To summarize, what do affected products offer to mitigate possible security issues?

• PostgreSQL - flexible user access settings, network access policy, good user awareness about possible malicious use.
• Patroni - hardcode some values in yml files, bad user awareness about possible malicious use.

So, where PostgreSQL with their 'controversial' COPY command offers the community many ways to prevent possible malicious use and minimize the possible impact with well-defined and documented security boundaries (and that is why COPY FROM PROGRAM 'feature' is not considered as a vulnerability), Patroni offers nothing, but hardcoding insecure values in config files and keeps their users uninformed about the problem and solution.

I understand that this is a very hard question to consider this behavior as vulnerability or not, that's why I will not interfere if you decide to close this issue.

But I think that it would be the right thing to inform your users about possible security troubles at least, or - I don't hope on that - make your software much secured by default.

Thanks!

[CyberDem0n]

In postgres the superuser is allowed to do anything. It doesn't need to have pg_* roles. These roles allow granting very specific permissions to non-superuser, so they also could call COPY FROM/TO PROGRAM and for example ALTER SYSTEM, which also allows changing GUCs like archive_command. And actually, ALTER SYSTEM exists even longer than COPY TO PROGRAM.

In Patroni there is no need to have flexible roles system. There is simply no demand for allowing certain users to have more permissions than others.

Yes, Patroni allows changing postgres config file on all nodes, but it also allows to have REST API protected. Without having credentials you can't do anything. This is very similar to having superuser credentials in postgres.

The minor difference is pg_hba.conf, which limits access on the network level. It is not hard to implement something similar in Patroni and by default reject access to non-GET endpoints.

In any case I really appreciate the work you have done. Your points about documentation are absolutely fair, it could be improved, but I am not a good writer and also can't dedicate 100% of my time to support and develop Patroni.

[german-namestnikov]

Thanks for the answer.

In postgres the superuser is allowed to do anything. It doesn't need to have pg_* roles. These roles allow granting very specific permissions to non-superuser, so they also could call COPY FROM/TO PROGRAM and for example ALTER SYSTEM, which also allows changing GUCs like archive_command. And actually, ALTER SYSTEM exists even longer than COPY TO PROGRAM.

I don't deny that PostgreSQL superuser has necessary privileges as well I don't deny that there are ways to grant other users enough privileges to execute OS commands via specially crafted queries.

I just note that PostgreSQL offers some security boundaries to make such conditions more secure. I mean user policy, mostly.

Without having credentials you can't do anything. This is very similar to having superuser credentials in postgres.

The word "Authenticated" I used in "Authenticated RCE" means that attacker needs to be authenticated before code execution. How can you see, I don't state that there is any way to execute code without authentication.

The minor difference is pg_hba.conf, which limits access on the network level. It is not hard to implement something similar in Patroni and by default reject access to non-GET endpoints.

Sounds good. I would offer you any help with such a good initiative, but I am afraid that I could break down the whole application with my dirty hands :)

In any case I really appreciate the work you have done. Your points about documentation are absolutely fair, it could be improved, but I am not a good writer and also can't dedicate 100% of my time to support and develop Patroni.

Thanks for the nice words!
I think I could cooperate with @victor-sudakov to improve the Security Considerations page if you accept such activities in your project.

[CyberDem0n]

Any contributions are always welcome!

[german-namestnikov]

Good! So then let's keep this issue opened before I make a proper pull request. Thanks!

[erzhick10]

здравствуйте не подскажите как работает kerberos Postgres в кластере с patroni и будет ли обращение к базе работать через haproxy