[SPARK-49750][DOC] Mention delegation token support in K8s mode #48199
Conversation
|
maybe cc @viirya |
| @@ -947,7 +947,7 @@ mechanism (see `java.util.ServiceLoader`). Implementations of | |||
| `org.apache.spark.security.HadoopDelegationTokenProvider` can be made available to Spark | |||
| by listing their names in the corresponding file in the jar's `META-INF/services` directory. | |||
|
|
|||
| Delegation token support is currently only supported in YARN mode. Consult the | |||
| Delegation token support is currently only supported in YARN and Kubernetes mode. Consult the | |||
| deployment-specific page for more information. | |||
|
|
|||
| The following options provides finer-grained control for this feature: | |||
There was a problem hiding this comment.
There is another place:
Users can exclude Kerberos delegation token renewal at resource scheduler. Currently it is only supported
on YARN.
There was a problem hiding this comment.
Consult the deployment-specific page for more information.
Btw, for YARN, it is true that we have a corresponding section for it as following. Do we have same one for Kubernetes? If so, can you also add the link?
The configuration is covered in the [Running Spark on YARN](running-on-yarn.html#yarn-specific-kerberos-configuration) page.
There was a problem hiding this comment.
@viirya DT renewal exclusion is another feature that is only supported by YARN.
In YARN mode, the RM could take responsibility for the token renewal, in K8s
there isn't a definition of what "external renewal service" is anywhere
ref SPARK-25825
There was a problem hiding this comment.
Btw, for YARN, it is true that we have a corresponding section for it as following. Do we have same one for Kubernetes?
Unlike YARN, the introduction of the K8s Kerberos support was embedded on the security page.
There was a problem hiding this comment.
Unlike YARN, the introduction of the K8s Kerberos support was embedded on the security page.
Do you mean the section "Secure Interaction with Kubernetes"?
|
It appears that StandaloneSchedulerBackend is also supported. |
There was a problem hiding this comment.
+1, LGTM. Thank you, @pan3793 , @viirya , and @yaooqinn .
Let's handle this as a kind of follow up of SPARK-23257. For Standalone cluster, let's proceed to discuss seperately.
Merged to master/3.5/3.4.
Update docs to mention delegation token support in K8s mode. The delegation token support in K8s mode has been implemented since 3.0.0 via SPARK-23257. Yes, docs are updated. Review. No. Closes #48199 from pan3793/SPARK-49750. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit dedf5aa) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Update docs to mention delegation token support in K8s mode. The delegation token support in K8s mode has been implemented since 3.0.0 via SPARK-23257. Yes, docs are updated. Review. No. Closes #48199 from pan3793/SPARK-49750. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit dedf5aa) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit b513297) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? Update docs to mention delegation token support in K8s mode. ### Why are the changes needed? The delegation token support in K8s mode has been implemented since 3.0.0 via SPARK-23257. ### Does this PR introduce _any_ user-facing change? Yes, docs are updated. ### How was this patch tested? Review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48199 from pan3793/SPARK-49750. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
### What changes were proposed in this pull request? Update docs to mention delegation token support in K8s mode. ### Why are the changes needed? The delegation token support in K8s mode has been implemented since 3.0.0 via SPARK-23257. ### Does this PR introduce _any_ user-facing change? Yes, docs are updated. ### How was this patch tested? Review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#48199 from pan3793/SPARK-49750. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
Update docs to mention delegation token support in K8s mode.
Why are the changes needed?
The delegation token support in K8s mode has been implemented since 3.0.0 via SPARK-23257.
Does this PR introduce any user-facing change?
Yes, docs are updated.
How was this patch tested?
Review.
Was this patch authored or co-authored using generative AI tooling?
No.