Batch Jobs Status

[moelhoussein]

Is there any mechanism or plans to have Kyuubi api push the batch status as oppose having to poll it? We are evaluating kyuubi as a Job server, and plan to run hundreds if not thousands of spark jobs. We use airflow. In the current design airflow will have to poll the batch status, which is not very scalable. Any thoughts on this?

[pan3793]

Which resource manager are you using to run spark jobs? YARN or K8s, or something else?

[moelhoussein]

Kubernetes

[pan3793]

Do you hit real performance issues? Kyuubi uses informer to retrieve pod states and keep them in memory, #4625

[moelhoussein]

This purely theoretical now. We are building a custom Airflow Operator to interact with the Kyuubi API. This Operator uses POST batches and GET batch status on the submitted spark application. So we interact directly with Kyuubi API. If you are saying that with the use of Informers in K8s kyuubi should be able to scale, I am okay with that. I just wanted clarification whether continuously polling the API for hundreds of spark applications could cause an issue.

[pan3793]

GET batch status on the submitted spark application

this won't cause any pressure on the K8s API server, Kyuubi server can answer it directly.