KEP-2170: [SDK] Initial implementation of the Kubeflow Training V2 Python SDK #2324

andreyvelich · 2024-11-09T16:31:33Z

Part of: #2216

This is initial implementation for the Kubeflow Training V2 SDK.

We are still designing the APIs, so these functions are not final.
These changes allow us to showcase the Kubeflow Training V2 demo at KubeCon 2024.

We discusses with @tenzen-y that we can add specific label to our runtimes to define whether runtime can be used for pre-training or post-training:

training.kubeflow.org/phase: pre-training
training.kubeflow.org/phase: post-training

/assign @kubeflow/wg-training-leads @varshaprasad96 @akshaychitneni @deepanker13 @helenxie-bit @Electronic-Waste @saileshd1402 @kannon92 @kuizhiqing @shravan-achar @seanlaii

andreyvelich · 2024-11-09T16:34:09Z

/hold

coveralls · 2024-11-09T17:56:40Z

Pull Request Test Coverage Report for Build 12281612841

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 12241759597:	0.0%
Covered Lines:	85
Relevant Lines:	85

💛 - Coveralls

deepanker13 · 2024-11-12T12:23:41Z

sdk_v2/kubeflow/training/api/training_client.py

+
+        # Create the TrainJob.
+        try:
+            self.custom_api.create_namespaced_custom_object(


should we check the status of this api, whether this returns 200 or not

I think, if this API fails it throws an Exception

training-operator/sdk_v2/kubeflow/training/api/training_client.py

Lines 205 to 209 in e9ff7d1

except Exception:

raise RuntimeError(

f"Failed to create {constants.TRAINJOB_KIND}: {self.namespace}/{train_job_name}"

)

deepanker13 · 2024-11-12T13:35:15Z

sdk_v2/kubeflow/training/constants/constants.py

+PHASE_PRE_TRAINING = "pre-training"
+
+# The value indicates that runtime can be used for the model pre-training.
+PHASE_PRE_TRAINING = "post-training"


it should be PHASE_POST_TRAINING I guess

Great catch!

nit: please change the comment as well @andreyvelich
The value indicates that runtime can be used for the model pre-training.
Thanks !

seanlaii · 2024-11-13T03:06:23Z

sdk_v2/kubeflow/training/types/types.py

+@dataclass
+class HuggingFaceDatasetConfig:
+    storage_uri: str
+    access_token: Optional[str] = None
+
+
+@dataclass
+# Configuration for the HuggingFace model provider.
+class HuggingFaceModelInputConfig:
+    storage_uri: str
+    access_token: Optional[str] = None


Do we plan to move HuggingFaceDatasetConfig and HuggingFaceModelConfig to here in this PR?

I have the same question. We have defined dataset and model related classes in pkg/initializerz_v2.

training-operator/pkg/initializer_v2/dataset/config.py

Lines 5 to 9 in 94dee0e

# TODO (andreyvelich): This should be moved under Training V2 SDK.

@dataclass

class HuggingFaceDatasetConfig:

storage_uri: str

access_token: Optional[str] = None

training-operator/pkg/initializer_v2/model/config.py

Lines 5 to 9 in 94dee0e

# TODO (andreyvelich): This should be moved under Training V2 SDK.

@dataclass

class HuggingFaceModelInputConfig:

storage_uri: str

access_token: Optional[str] = None

I guess it would be better if we move these py files in pkg/initializer_v2 to the SDK directory when generating SDK using setup.py. For example: https://github.com/kubeflow/katib/blob/2b41ae62ab3905984e02123218351a703c03bf56/sdk/python/v1beta1/setup.py#L33-L39

Yes, I will move them soon.

@seanlaii @Electronic-Waste I've completed this.

Electronic-Waste

Basically LGTM. I'm willing to help if needed:)

Electronic-Waste · 2024-11-13T16:00:17Z

sdk_v2/kubeflow/training/types/types.py

+@dataclass
+class HuggingFaceDatasetConfig:
+    storage_uri: str
+    access_token: Optional[str] = None
+
+
+@dataclass
+# Configuration for the HuggingFace model provider.
+class HuggingFaceModelInputConfig:
+    storage_uri: str
+    access_token: Optional[str] = None


I have the same question. We have defined dataset and model related classes in pkg/initializerz_v2.

training-operator/pkg/initializer_v2/dataset/config.py

Lines 5 to 9 in 94dee0e

# TODO (andreyvelich): This should be moved under Training V2 SDK.

@dataclass

class HuggingFaceDatasetConfig:

storage_uri: str

access_token: Optional[str] = None

training-operator/pkg/initializer_v2/model/config.py

Lines 5 to 9 in 94dee0e

# TODO (andreyvelich): This should be moved under Training V2 SDK.

@dataclass

class HuggingFaceModelInputConfig:

storage_uri: str

access_token: Optional[str] = None

I guess it would be better if we move these py files in pkg/initializer_v2 to the SDK directory when generating SDK using setup.py. For example: https://github.com/kubeflow/katib/blob/2b41ae62ab3905984e02123218351a703c03bf56/sdk/python/v1beta1/setup.py#L33-L39

saileshd1402 · 2024-11-13T17:39:46Z

sdk_v2/kubeflow/training/api/training_client.py

+
+        return train_job_name
+
+    def list_jobs(self) -> List[types.TrainJob]:


Just a thought, do we want to pass runtimeRef name as optional argument here? might be a good idea if we can get train jobs for a particular runtime?

Sure, good idea.
I added this parameter @saileshd1402

andreyvelich · 2024-11-15T19:28:48Z

I think, this PR should be ready to review to give users provide initial support for Python SDK.
We've already given demo for these APIs at KubeCon 2024 NA, will share the recordings when it is ready.

andreyvelich · 2024-11-18T12:59:07Z

/hold cancel

sdk_v2/kubeflow/training/api/training_client.py

akshaychitneni · 2024-11-18T17:46:46Z

sdk_v2/kubeflow/training/api/training_client.py

+                config.load_incluster_config()
+
+        k8s_client = client.ApiClient(client_configuration)
+        self.custom_api = client.CustomObjectsApi(k8s_client)


Does these client include retries on any failure? May be we could have a wrapper client that would add retries

Yes, I think it is a good idea.
@akshaychitneni Can we have a followup PR to implement it ?
Please can you create an issue to track it ?

sdk_v2/kubeflow/training/api/training_client.py

Electronic-Waste

@andreyvelich Lots of work! I left a few comments for you.

I think we should co-design configs for Pre-Training and Fine-Tuning in the Trainer parameter.

sdk_v2/kubeflow/training/types/types.py

sdk_v2/kubeflow/training/api/training_client.py

sdk_v2/kubeflow/training/types/types.py

hack/python-sdk-v2/gen-sdk.sh

sdk_v2/kubeflow/training/constants/constants.py

tenzen-y · 2024-11-27T14:10:01Z

sdk_v2/kubeflow/training/constants/constants.py

+
+# The label key to identify the device (e.g. GPU, TPU) that is used for training.
+# TODO: Potentially, we should get this data from the Node selectors.
+DEVICE_KEY = "training.kubeflow.org/device"


Here, what exactly does the "device" mean?
Computing Device? Persistent Storage device? Ephemeral Storage device like memory and extended persistent memory? NIC device?

I think, the main goal of this is to show Data Scientists which device will be used by the Training Runtime to perform model training.

This will help them to understand the accelerator (e.g. V100, H100) on which their model training will be run.

In that case, should we name it just an accelerator?
The device sounds weird since you seem to aim support only accelerator including FPGA, and you do not have any plan to support any other special devices.

It's a good point. However, I was just exploring how make it more close to ML frameworks, and they call their API as "device".

PyTorch: https://pytorch.org/docs/stable/generated/torch.cuda.device.html

MLX: https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.set_default_device.html#mlx.core.set_default_device

Thus, I was thinking that device is closer to data scientists expectation when they want to assign their ML code to the specific accelerator. WDYT @tenzen-y ?

For PyTorch, it seems that the device indicates only GPU since the lib name is "cuda.device".
For MLX, I'm not sure which devices can be specified because I can not find any documentation for supported devices at first glance.

As a result, I still would recommend using the accelerator.

For MLX, I'm not sure which devices can be specified because I can not find any documentation for supported devices at first glance

In MLX you can specify CPU or GPU (e.g. Metal). For example: mx.set_default_device(mx.cpu) or mx.set_default_device(mx.gpu): https://github.com/ml-explore/mlx-examples/blob/main/mnist/main.py#L103

On EKS and GKE the label that indicates GPU type on the node is named accelerator k8s.amazonaws.com/accelerator: https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#special-note-on-gpu-instances

However, how we can deal with CPU-based runtimes, e.g. we don't call CPU an "accelerator", isn't ?
Or maybe in the future we can support other devices/accelerators.

@tenzen-y Can we create a tracking issue to discuss this further ?

However, how we can deal with CPU-based runtimes, e.g. we don't call CPU an "accelerator", isn't ?
Or maybe in the future we can support other devices/accelerators.

IIUC, this device field corresponds to the Kubernetes scheduling directives like nodeSelector. So, I'm not sure the reason why we need to provide the knob for CPU workloads.

@tenzen-y The main goal of this device and device count parameter is to show Data Scientists which hardware resources they can utilize while using the existing Runtimes. With that, they can appropriately configure their training code to use those devices. For example, using nccl backend in PyTorch if GPUs are available or using bfloat16 if GPU supports it.

As you can see, the device and device count is part of Runtime and Component classes that Data Scientists can get:

training-operator/sdk_v2/kubeflow/training/types/types.py

Lines 24 to 39 in e9c22b2

class Runtime:

name: str

phase: str

device: str

device_count: str

# Representation for the TrainJob component.

@dataclass

class Component:

name: str

status: str

device: str

device_count: str

pod_name: str

Does it make sense ?

sdk_v2/kubeflow/training/constants/constants.py

sdk_v2/kubeflow/training/api/training_client.py

tenzen-y · 2024-11-27T14:32:55Z

sdk_v2/kubeflow/training/api/training_client.py

+
+        pod_name = None
+        # Get Initializer or Trainer Pod name.
+        for c in self.get_job(name).components:


What if there are multiple Pods for the same role like initializer? How can we identify the Pod name?
We can imagine the situation where the Pods are recreate (but the old one still exist) based on restart, recreate, success, failure policies.

Hmm, it is a good point.
I think, in the current loop it return the following:

Component(name='initializer', status='Failed', device='cpu', device_count='10', pod_name='xe37e16669f7-initializer-0-0-gaf51') Component(name='initializer', status='Succeeded', device='cpu', device_count='10', pod_name='xe37e16669f7-initializer-0-0-hl4wv') Component(name='trainer-node-0', status='Succeeded', device='gpu', device_count='4', pod_name='xe37e16669f7-trainer-node-0-0-ktbt8')

I think, we should discuss if that experience looks good to our users or we should change the output.

WDYT @tenzen-y ?

IIRC, the katib UI had similar problem, and we addressed that, previously.
We might be able to learn something from that.

https://github.com/kubeflow/katib/blob/2b41ae62ab3905984e02123218351a703c03bf56/pkg/ui/v1beta1/backend.go#L720

I see. I guess, that function can return logs from the Failed Pod even if we have another Succeeded Pod since the podList order is alphabetical. Is that understanding correct, @tenzen-y ?
What do you think we should return with get_job() API in that case ?

I see. I guess, that function can return logs from the Failed Pod even if we have another Succeeded Pod since the podList order is alphabetical. Is that understanding correct, @tenzen-y ?

I guess that this current log collection implementation obtains the logs randomly. So, we sometimes fetch the logs from Failed Pods even if the replaced Running or Succeeded Pods, isn't it?

@tenzen-y As you can see right now we fetch logs only from the Trainer component, if follow=True:

training-operator/sdk_v2/kubeflow/training/api/training_client.py

Lines 368 to 377 in e9c22b2

if follow and component == constants.JOB_TRAINER_NODE:

log_streams = []

log_streams.append(

watch.Watch().stream(

self.core_api.read_namespaced_pod_log,

name=pod_name,

namespace=self.namespace,

container=constants.CONTAINER_TRAINER,

)

)

Otherwise, we just return logs from all TrainJob components:

training-operator/sdk_v2/kubeflow/training/api/training_client.py

Lines 410 to 432 in e9c22b2

if component == constants.JOB_INITIALIZER:

logs_dict[constants.CONTAINER_DATASET_INITIALIZER] = (

self.core_api.read_namespaced_pod_log(

name=pod_name,

namespace=self.namespace,

container=constants.CONTAINER_DATASET_INITIALIZER,

)

)

logs_dict[constants.CONTAINER_MODEL_INITIALIZER] = (

self.core_api.read_namespaced_pod_log(

name=pod_name,

namespace=self.namespace,

container=constants.CONTAINER_MODEL_INITIALIZER,

)

)

else:

logs_dict[component + "-" + str(node_index)] = (

self.core_api.read_namespaced_pod_log(

name=pod_name,

namespace=self.namespace,

container=constants.CONTAINER_TRAINER,

)

)

Additionally, by default the Job controller will not create additional Pods if it fails. It will try to re-create the existing Pod until it reaches BackOffLimits.

I am happy to revisit this API in the following PRs if we want to improve it.
WDYT @tenzen-y ?

@Electronic-Waste @akshaychitneni @shravan-achar @droctothorpe @deepanker13 @seanlaii @astefanutti Any thoughts/ideas on the above topic ?

Additionally, by default the Job controller will not create additional Pods if it fails

The behavior of the failure is different based on Pod restart Policy, batch job success, and failure policies. So, we can easily reproduce the unexpected log fetching situations.

I can easily imagine the situations where the v2 SDK users create issues to report unexpected logs seeing.
So, I believe that addressing the issue would be better before the first v2 release.

If you can work on the issue in another PR before we release the first v2 release, I'm ok for now.

Sure, created this @tenzen-y: #2348

sdk_v2/kubeflow/training/types/types.py

hack/update-codegen.sh

sdk_v2/kubeflow/training/constants/constants.py

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Use kube-openapi generator to remove unnecessary models from apimachinery Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y

Thank you for this great contribution!
/lgtm
/approve

google-oss-prow · 2024-12-13T16:53:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-prow bot requested review from jinchihe and kuizhiqing November 9, 2024 16:31

google-oss-prow bot added the size/XL label Nov 9, 2024

google-oss-prow bot added the do-not-merge/hold label Nov 9, 2024

deepanker13 reviewed Nov 12, 2024

View reviewed changes

seanlaii reviewed Nov 13, 2024

View reviewed changes

Electronic-Waste reviewed Nov 13, 2024

View reviewed changes

google-oss-prow bot added size/XXL and removed size/XL labels Nov 13, 2024

saileshd1402 reviewed Nov 13, 2024

View reviewed changes

andreyvelich force-pushed the issue-2216-initial-sdk-api branch from ee57831 to 2abf580 Compare November 18, 2024 12:58

google-oss-prow bot removed the do-not-merge/hold label Nov 18, 2024

akshaychitneni reviewed Nov 18, 2024

View reviewed changes

sdk_v2/kubeflow/training/api/training_client.py Outdated Show resolved Hide resolved

akshaychitneni reviewed Nov 18, 2024

View reviewed changes

andreyvelich commented Nov 25, 2024

View reviewed changes

sdk_v2/kubeflow/training/api/training_client.py Show resolved Hide resolved

Electronic-Waste reviewed Nov 26, 2024

View reviewed changes

sdk_v2/kubeflow/training/types/types.py Show resolved Hide resolved

sdk_v2/kubeflow/training/api/training_client.py Show resolved Hide resolved

Electronic-Waste reviewed Nov 26, 2024

View reviewed changes

sdk_v2/kubeflow/training/types/types.py Show resolved Hide resolved

tenzen-y reviewed Nov 27, 2024

View reviewed changes

andreyvelich force-pushed the issue-2216-initial-sdk-api branch from cc3be21 to e5736c9 Compare November 27, 2024 16:17

Electronic-Waste reviewed Nov 27, 2024

View reviewed changes

sdk_v2/kubeflow/training/types/types.py Outdated Show resolved Hide resolved

andreyvelich mentioned this pull request Nov 27, 2024

KEP-2170: Add Torch Distributed Runtime #2328

Merged

andreyvelich force-pushed the issue-2216-initial-sdk-api branch 2 times, most recently from 6033a72 to dd3a7a6 Compare November 28, 2024 21:58

andreyvelich commented Nov 29, 2024

View reviewed changes

hack/update-codegen.sh Show resolved Hide resolved

astefanutti reviewed Nov 29, 2024

View reviewed changes

sdk_v2/kubeflow/training/constants/constants.py Outdated Show resolved Hide resolved

andreyvelich added 17 commits December 10, 2024 12:39

Fix sed for Linux

b1ef228

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix PHASE_POST_TRAINING const

3525b4d

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add device labels for the runtimes

e874244

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add Components to the TrainJob type

21fa7c8

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Get container devices util

71f1646

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Use SDK assets in initializer

4ad2521

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add the runtime_ref to the list_jobs() API

fd4d5aa

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Check if runtime label presents

59e603e

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Add Trainer arg as part of train API

42fc257

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Rename lora to peft config

2621c51

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Using global prop to remove SDK tests

4e44209

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Fix type for lora_dropout

8fa0d0a

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Import the JobSet models

09473bd

Use kube-openapi generator to remove unnecessary models from apimachinery Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Parse runtime as object

4d5ee70

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Remove old gen_openapi

7989e39

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Rename GPU_DEVICE_LABEL to NVIDIA_GPU_DEVICE_LABEL

b8715bd

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

Run codegen

8e8243c

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

andreyvelich force-pushed the issue-2216-initial-sdk-api branch from e9c22b2 to 8e8243c Compare December 10, 2024 12:44

Install git on initializer images

7c4cb22

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

This was referenced Dec 10, 2024

Upgrade Kubernetes to v1.30.7 kubeflow/katib#2463

Open

[SDK] Get the correct TrainJob components using get_job() API #2348

Open

Rename device to accelerator for Runtime class

52f73da

Signed-off-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

tenzen-y reviewed Dec 13, 2024

View reviewed changes

google-oss-prow bot assigned tenzen-y Dec 13, 2024

google-oss-prow bot added the lgtm label Dec 13, 2024

google-oss-prow bot added the approved label Dec 13, 2024

google-oss-prow bot merged commit ea01481 into kubeflow:master Dec 13, 2024
52 checks passed

andreyvelich deleted the issue-2216-initial-sdk-api branch December 13, 2024 17:58

andreyvelich mentioned this pull request Dec 13, 2024

[SDK] Show available Runtime accelerators to users #2355

Open

	except Exception:
	raise RuntimeError(
	f"Failed to create {constants.TRAINJOB_KIND}: {self.namespace}/{train_job_name}"
	)

	# TODO (andreyvelich): This should be moved under Training V2 SDK.
	@dataclass
	class HuggingFaceDatasetConfig:
	storage_uri: str
	access_token: Optional[str] = None

	# TODO (andreyvelich): This should be moved under Training V2 SDK.
	@dataclass
	class HuggingFaceModelInputConfig:
	storage_uri: str
	access_token: Optional[str] = None


		return train_job_name

		def list_jobs(self) -> List[types.TrainJob]:

	class Runtime:
	name: str
	phase: str
	device: str
	device_count: str


	# Representation for the TrainJob component.
	@dataclass
	class Component:
	name: str
	status: str
	device: str
	device_count: str
	pod_name: str

	if follow and component == constants.JOB_TRAINER_NODE:
	log_streams = []
	log_streams.append(
	watch.Watch().stream(
	self.core_api.read_namespaced_pod_log,
	name=pod_name,
	namespace=self.namespace,
	container=constants.CONTAINER_TRAINER,
	)
	)

	if component == constants.JOB_INITIALIZER:
	logs_dict[constants.CONTAINER_DATASET_INITIALIZER] = (
	self.core_api.read_namespaced_pod_log(
	name=pod_name,
	namespace=self.namespace,
	container=constants.CONTAINER_DATASET_INITIALIZER,
	)
	)
	logs_dict[constants.CONTAINER_MODEL_INITIALIZER] = (
	self.core_api.read_namespaced_pod_log(
	name=pod_name,
	namespace=self.namespace,
	container=constants.CONTAINER_MODEL_INITIALIZER,
	)
	)
	else:
	logs_dict[component + "-" + str(node_index)] = (
	self.core_api.read_namespaced_pod_log(
	name=pod_name,
	namespace=self.namespace,
	container=constants.CONTAINER_TRAINER,
	)
	)

KEP-2170: [SDK] Initial implementation of the Kubeflow Training V2 Python SDK #2324

KEP-2170: [SDK] Initial implementation of the Kubeflow Training V2 Python SDK #2324

Conversation

andreyvelich commented Nov 9, 2024 • edited Loading

andreyvelich commented Nov 9, 2024

coveralls commented Nov 9, 2024 • edited Loading

Pull Request Test Coverage Report for Build 12281612841

Details

💛 - Coveralls

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deepanker13 Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich commented Nov 15, 2024 • edited Loading

andreyvelich commented Nov 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Electronic-Waste left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich Nov 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andreyvelich Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Dec 13, 2024

andreyvelich commented Nov 9, 2024 •

edited

Loading

coveralls commented Nov 9, 2024 •

edited

Loading

deepanker13 Nov 13, 2024 •

edited

Loading

andreyvelich commented Nov 15, 2024 •

edited

Loading

andreyvelich Nov 28, 2024 •

edited

Loading

andreyvelich Nov 27, 2024 •

edited

Loading

andreyvelich Dec 10, 2024 •

edited

Loading