Besu always needs cluster level permissions for Kubernetes NAT manager on GKE

[snigdha920]

Description

• Currently the Kubernetes NAT manager checks all namespaces to find the besu service here:

  besu/nat/src/main/java/org/hyperledger/besu/nat/kubernetes/KubernetesNatManager.java

  Line 87 in 9eb3283

  .listServiceForAllNamespaces(

• For this to work, we need to give cluster level permissions to the besu deployment for it to be able to scan all the namespaces
• In some environments, granting cluster level permissions is not authorised (for example, in a lot of banks)
• Even if the besu service is in the same namespace as the besu deployment the NAT manager errors out when we do not grant cluster level permissions since it ALWAYS lists services in all namespaces.
• Falls back to NONE, the enode does not have the correct public IP address anymore, instead it is 0.0.0.0

What we need is the NAT manager to work without the need of cluster level permissions on GKE.

Perhaps a fallback to just checking in the local namespace (i.e the namespace of the besu deployment) before throwing the error?

Acceptance Criteria

• NAT manager works without cluster level permissions on GKE

Steps to Reproduce (Bug)

1. Deploy a besu deployment, service without cluster level permissions
2. The NAT manager errors out, throws the following error:

besu/nat/src/main/java/org/hyperledger/besu/nat/kubernetes/KubernetesNatManager.java

Line 97 in 9eb3283

throw new NatInitializationException(e.getMessage(), e);

and falls back to NONE.
3. The enode url of the node has 0.0.0.0 as the public IP

Expected behavior: [What you expect to happen]

If searching for the service across all namespaces fails, it should look for the service in the local namespace before falling back to NONE.

Actual behavior: [What actually happens]
If searching for the service across all namespaces fails, it errors out, and falls back to NONE.

Frequency: [What percentage of the time does it occur?]

100%

Versions (Add all that apply)

• docker image we use: https://hub.docker.com/layers/hyperledger/besu/22.10.4/images/sha256-036331dda5c27e2d51524614cd628d6f55fc14583b48516e1e019aef0de92a21?context=explore
• Software version: [besu --version]
• Java version: [java -version]
• OS Name & Version: [cat /etc/*release]
• Kernel Version: [uname -a]
• Virtual Machine software & version: [vmware -v]
• Docker Version: [docker version]
• Cloud VM, type, size: [Amazon Web Services I3-large]

Smart contract information (If you're reporting an issue arising from deploying or calling a smart contract, please supply related information)

N/A

Additional Information (Add any of the following or anything else that may be relevant)

N/A

[non-fungible-nelson]

@joshuafernandes - Any opinion here?

[joshuafernandes]

Hi @non-fungible-nelson yes this is an issue and I believe tied to how we do things with K8S RBAC (ie Besu doesn't respect RBAC). The NAT manager works perfectly fine as long as cluster wide RBAC is disabled, but when it is enabled (most cases & this should be the norm to be honest) the permissions it requires are too wide. Have discussed with @matkt a while ago but priorities changed I believe

https://github.com/hyperledger/besu/blob/main/nat/src/main/java/org/hyperledger/besu/nat/kubernetes/KubernetesNatManager.java#L87 should be limited to single namespace imho and we pass the namespace in as another cli arg. In the K8S repo I've turned NAT manager off altogether because of this and prefer to remain secure and manually pass in the addresses via DNS

[non-fungible-nelson]

Thanks for this - leaving open for now. Might make a docs change regarding this for your advice? @alexandratran ping me if you think this makes sense to open a ticket with in besu docs.

[snigdha920]

Hi @non-fungible-nelson yes this is an issue and I believe tied to how we do things with K8S RBAC (ie Besu doesn't respect RBAC). The NAT manager works perfectly fine as long as cluster wide RBAC is disabled, but when it is enabled (most cases & this should be the norm to be honest) the permissions it requires are too wide. Have discussed with @matkt a while ago but priorities changed I believe

https://github.com/hyperledger/besu/blob/main/nat/src/main/java/org/hyperledger/besu/nat/kubernetes/KubernetesNatManager.java#L87 should be limited to single namespace imho and we pass the namespace in as another cli arg. In the K8S repo I've turned NAT manager off altogether because of this and prefer to remain secure and manually pass in the addresses via DNS

How did you use DNS without the NAT manager and cluster level permissions?

I went through your DNS setup and I have the same configuration, but I still get the following issues:

When I try:

• Turning off the NAT manager by settingnat-method=NONE, besu doesn't start with the error:

  The `--Xnat-kube-service-name` parameter is only used in kubernetes mode. Either remove --Xnat-kube-service-name or select the KUBERNETES mode (via --nat--method=KUBERNETES)
  

• Using the K8s NAT manager and starting besu as:

  exec /opt/besu/bin/besu --config-file=/etc/besu/config.toml --Xdns-enabled=true --Xdns-update-enabled=true --Xnat-kube-service-name=${name} 
  

  with the following p2p options in my config:

              'p2p-enabled': true,
              'discovery-enabled': true,
              'p2p-interface': '0.0.0.0',
              'p2p-port': 30303,
              'p2p-host': '0.0.0.0'
              'max-peers': 25,
  

  The NAT manager crashes and doesn't use the DNS config specified because it doesn't have cluster-level permissions.

[joshuafernandes]

Hi @snigdha920 it has been a good while since I used it last and maybe code is changed now, but the two should be treated seperate. In the setup I had in the repo, Besu's NAT manager would fail and then it would lookup things via DNS (more so given it was told what the service name was, it didn't need to do the whole lookup). If this has now changed and the DNS lookup isn't working, then this should be fixed @non-fungible-nelson - effectively Besu in K8S will not work

[snigdha920]

Hi @snigdha920 it has been a good while since I used it last and maybe code is changed now, but the two should be treated seperate. In the setup I had in the repo, Besu's NAT manager would fail and then it would lookup things via DNS (more so given it was told what the service name was, it didn't need to do the whole lookup). If this has now changed and the DNS lookup isn't working, then this should be fixed @non-fungible-nelson - effectively Besu in K8S will not work

Yes, it doesn't work even if I provide the service name. It tries to list services across all namespaces, which causes the permission issue. So effectively besu in k8s doesn't work (without the cluster level permissions)

[cdivitotawela]

@snigdha920 It has been long time since the PR raised. I was checking on this and there was a question whether NAT is required any more?

Apparently NAT is required to communicate cross clusters. We could use Besu for public network (mainnet,testnet) following solution setting nat=NONE.

• Each Besu run on a single Node in Kubernetes cluster
• All Kubernetes nodes(where Besu run) are public and have a public ip
• Pod has a init container to find out the public ip
• Public ip is configured in the Besu configuration

Please refer to the solution in the github repo ethereum-helm-charts. This is the initContainer getting the public ip of the node
statefulset.yml

[jflo]

closed due to inactivity