[Bug] InvalidACL appears in engine after upgrade to Kyuubi 1.6.0

[zhouyifan279]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

Kyuubi Spark SQL engine works using Kyuubi Server Configurations provided below in Kyuubi 1.4.1.

After upgrade to Kyuubi 1.6.0, engine fails to start when set acl for node created on Zookeeper.

Affects Version(s)

1.6.0

Kyuubi Server Log Output

2022-10-21 15:30:53,352 [34154] - INFO  [main:ZooKeeper@442] - Initiating client connection, connectString=dnode6.local:2182,dnode5.local:2182 sessionTimeout=60000 watcher=org.apache.kyuubi.shade.org.apache.curator.ConnectionState@106387cf
2022-10-21 15:30:53,378 [34180] - INFO  [main-SendThread(dnode5.local:2182):ClientCnxn$SendThread@1025] - Opening socket connection to server dnode5.local/172.30.1.45:2182. Will not attempt to authenticate using SASL (unknown error)
2022-10-21 15:30:53,379 [34181] - INFO  [main-SendThread(dnode5.local:2182):ClientCnxn$SendThread@879] - Socket connection established to dnode5.local/172.30.1.45:2182, initiating session
2022-10-21 15:30:53,382 [34184] - INFO  [main:Logging@59] - Service[EngineServiceDiscovery] is initialized.
2022-10-21 15:30:53,382 [34184] - INFO  [main:Logging@59] - Service[SparkTBinaryFrontend] is initialized.
2022-10-21 15:30:53,382 [34184] - INFO  [main:Logging@59] - Service[SparkSQLEngine] is initialized.
2022-10-21 15:30:53,396 [34198] - INFO  [main-SendThread(dnode5.local:2182):ClientCnxn$SendThread@1299] - Session establishment complete on server dnode5.local/172.30.1.45:2182, sessionid = 0x1000139fd3a008d, negotiated timeout = 40000
2022-10-21 15:30:53,400 [34202] - INFO  [main:Logging@59] - Service[SparkSQLOperationManager] is started.
2022-10-21 15:30:53,401 [34203] - INFO  [main:Logging@59] - Service[SparkSQLSessionManager] is started.
2022-10-21 15:30:53,401 [34203] - INFO  [main:Logging@59] - Service[SparkSQLBackendService] is started.
2022-10-21 15:30:53,437 [34239] - INFO  [main-EventThread:ConnectionStateManager@228] - State change: CONNECTED
2022-10-21 15:30:53,448 [34250] - INFO  [Curator-ConnectionStateManager-0:Logging@59] - Zookeeper client connection state changed to: CONNECTED
2022-10-21 15:30:53,500 [34302] - ERROR [main:Logging@77] - Error starting service EngineServiceDiscovery
org.apache.kyuubi.KyuubiException: Failed to create namespace '/kyuubi-cluster_1.6.0-incubating_USER_SPARK_SQL/hive/default'
	at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.createPersistentNode(ZookeeperDiscoveryClient.scala:344)
	at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.registerService(ZookeeperDiscoveryClient.scala:234)
	at org.apache.kyuubi.ha.client.ServiceDiscovery.start(ServiceDiscovery.scala:62)
...
Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /kyuubi-cluster_1.6.0-incubating_USER_SPARK_SQL/bdms_grp.mammut_test/default
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:124)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)

Kyuubi Engine Log Output

No response

Kyuubi Server Configurations

kyuubi.authentication kerberos
kyuubi.frontend.bind.host anode1.local
kyuubi.frontend.bind.port 10009
kyuubi.ha.zookeeper.auth.type KERBEROS
kyuubi.ha.zookeeper.namespace kyuubi-cluster
kyuubi.ha.zookeeper.quorum anode1.local:2182,anode3.local:2182,anode2.local:2182
kyuubi.kinit.keytab /keytab/hive.keytab
kyuubi.kinit.principal hive/anode1.local@HADOOP.COM
kyuubi.session.engine.idle.timeout PT30M

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I can submit a PR independently to fix.
• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[zhouyifan279]

Problem is caused by PR #2993 :

kyuubi.ha.zookeeper.auth.principal & kyuubi.ha.zookeeper.auth.keytab defaults to fallback to kyuubi.kinit.principal & kyuubi.kinit.keytab.
In #2993, we marked kyuubi.kinit.principal & kyuubi.kinit.keytab as server only. Engine can not get principal & keytab and fails when set zookeeper node acl.

[zhouyifan279]

Workaround:

Set kyuubi.ha.zookeeper.auth.principal & kyuubi.ha.zookeeper.auth.keytab in kyuubi-defaults.conf

[zhouyifan279]

cc @cxzl25

[pan3793]

cc @turboFei

[turboFei]

Sorry for the late reply and mistake.

I am ok that we can add kyuubi.kinit.principal & kyuubi.kinit.keytab back to engine side.

[turboFei]

Sorry for the late reply and mistake.

I am ok that we can add kyuubi.kinit.principal & kyuubi.kinit.keytab back to engine side.

After think twice, I think that we should not add kyuubi.kinit.principal & kyuubi.kinit.keytab back to engine side.

It is not security(maybe a CVE), the server keytab has super permission.

If you want to delegate it to engine side, it might be leak to any one that can access the staging dir, for example, the user can get the server keytab from spark staging dir.

So, we should not delegate server keytab to engine side.

Set kyuubi.ha.zookeeper.auth.principal & kyuubi.ha.zookeeper.auth.keytab in kyuubi-defaults.conf

This is the correct way, we need make a dedicated keytab for zk.

How do you think about? @pan3793

[pan3793]

@turboFei's concern sounds reasonable to me, security is more important than compatible, we need a migration guide.

[zhouyifan279]

I'm also agree with @turboFei .
Additionaly, I think we should also remove kyuubi.ha.zookeeper.auth.type in engine side. Then by default, engine won't try to set acl for zookeeper node when it does not have principal & keytab.