Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap · 2024-08-01T13:47:53Z

Description

This PR supports copying a snapshot to StorPool primary storage in different zones.

Added additional API param storageids in CloudStack API calls:

createSnapshot
copySnapshot
createSnapshotPolicy

The option snapshot.backup.to.secondary does not apply to the copy functionality. The snapshots will be copied only on the required primary storage in a different zone.
The user can create volumes/templates from the copied snapshots. The option to copy the snapshots to both primary and secondary storage is available by providing in the create/copy snapshot the destination zone IDs and storage IDs

For other storage plugins that want to adopt this functionality:

The Primary storage driver should have the capability CAN_COPY_SNAPSHOT_BETWEEN_ZONES
the respective plugin needs to implement the copySnapshot method in their SnapshotStrategy and that the driver can handle the COPY operation

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale

Major
Minor

Screenshots (if appropriate):

How Has This Been Tested?

Manual and smoke tests with StorPool primary storage on multiple zones

codecov · 2024-08-01T13:56:23Z

Codecov Report

Attention: Patch coverage is 4.60358% with 746 lines in your changes missing coverage. Please review.

Project coverage is 15.76%. Comparing base (019f2c6) to head (88d13d2).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...om/cloud/storage/snapshot/SnapshotManagerImpl.java	8.88%	155 Missing and 9 partials ⚠️
...ack/storage/snapshot/StorPoolSnapshotStrategy.java	0.00%	140 Missing ⚠️
...tastore/driver/StorPoolPrimaryDataStoreDriver.java	0.00%	73 Missing ⚠️
...age/collector/StorPoolAbandonObjectsCollector.java	0.00%	63 Missing ⚠️
...loudstack/storage/datastore/util/StorPoolUtil.java	0.00%	36 Missing ⚠️
...n/java/com/cloud/storage/VolumeApiServiceImpl.java	6.25%	29 Missing and 1 partial ⚠️
...org/apache/cloudstack/snapshot/SnapshotHelper.java	21.62%	24 Missing and 5 partials ⚠️
...oudstack/storage/snapshot/SnapshotServiceImpl.java	0.00%	27 Missing ⚠️
...udstack/storage/datastore/util/StorPoolHelper.java	0.00%	22 Missing ⚠️
...e/wrapper/StorPoolModifyStorageCommandWrapper.java	0.00%	21 Missing ⚠️
... and 24 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9478      +/-   ##
============================================
- Coverage     15.78%   15.76%   -0.02%     
- Complexity    12564    12566       +2     
============================================
  Files          5627     5627              
  Lines        492250   492863     +613     
  Branches      61405    62711    +1306     
============================================
+ Hits          77710    77712       +2     
- Misses       406066   406666     +600     
- Partials       8474     8485      +11

Flag	Coverage Δ
uitests	`4.03% <ø> (-0.01%)`	⬇️
unittests	`16.58% <4.60%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-08-21T08:21:53Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-08-22T13:48:15Z

@blueorangutan package

blueorangutan · 2024-08-22T13:50:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-22T15:24:36Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10756

DaanHoogland · 2024-08-23T07:43:27Z

@blueorangutan test

blueorangutan · 2024-08-23T07:44:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-08-23T22:40:23Z

[SF] Trillian test result (tid-11155)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51322 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9478-t11155-kvm-ol8.zip
Smoke tests completed. 137 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_snapshot_usage	`Failure`	38.32	test_usage.py
test_01_vpc_site2site_vpn	`Failure`	308.08	test_vpc_vpn.py

github-actions · 2024-08-29T08:41:48Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java

github-actions · 2024-09-04T08:40:33Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java

sureshanaparti · 2024-09-04T12:06:59Z

...l/src/main/java/org/apache/cloudstack/storage/collector/StorPoolAbandonObjectsCollector.java

+                        continue;
+                    }
+                    for (SnapshotDetailsVO snapshot : snapshotDetails) {
+                        String name = snapshot.getValue().split(";")[0];


make sure the snapshot detail after .split() has length >= 2 & then get name and location from it (chance of Index Out of Bound error).

sureshanaparti · 2024-09-04T12:07:57Z

...l/src/main/java/org/apache/cloudstack/storage/collector/StorPoolAbandonObjectsCollector.java

+        }
+    }
+
+    private static List<String> snapshotsForRcovery(JsonArray arr) {


Suggested change

private static List<String> snapshotsForRcovery(JsonArray arr) {

private static List<String> snapshotsForRecovery(JsonArray arr) {

thanks, it's fixed in the latest commits

sureshanaparti · 2024-09-04T12:14:19Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

+//            logger.info(String.format("Cannot copy snapshot to another storage in different zone. It's not in the right state %s", snapshotOnStore.getStatus()));
+//            return false;
+//        }
+


remove these comments if not needed

thanks, it's removed in the latest commits

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

ui/src/views/storage/FormSchedule.vue

sureshanaparti · 2024-09-04T12:17:20Z

ui/src/views/storage/SnapshotZones.vue

+        const listStoragePools = json.liststoragepoolsresponse.storagepool
+        if (listStoragePools) {
+          this.storagePools = listStoragePools
+          this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)


Suggested change

this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)

this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES_AND_SAME_POOL_TYPE && pool.zoneid !== this.resource.zoneid)

ui/src/views/storage/TakeSnapshot.vue

github-actions · 2024-10-10T20:16:15Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

github-actions · 2024-10-15T07:35:38Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

DaanHoogland · 2024-10-28T15:08:33Z

test/integration/plugins/storpool/sp_util.py

+                    zone_found = True
+                    break
+            if zone_found == False:
+                cls.fail("Unable to find snapshot entry for the zone ID: %s" % zone_id)


@slavkap , lint nitpicking on you (EOL before EOF needed)

DaanHoogland

clgtm, lots of style comments but no blockers

DaanHoogland · 2024-10-28T15:11:57Z

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreDao;
+import org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO;
+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.collections.CollectionUtils;


Suggested change

import org.apache.commons.collections.CollectionUtils;

import org.apache.commons.collections.CollectionUtils;

DaanHoogland · 2024-10-28T15:12:16Z

...chestration/src/main/java/org/apache/cloudstack/engine/orchestration/VolumeOrchestrator.java

@@ -177,6 +107,74 @@
 import com.cloud.vm.dao.UserVmCloneSettingDao;
 import com.cloud.vm.dao.UserVmDao;
 import com.cloud.vm.dao.UserVmDetailsDao;
+import org.apache.cloudstack.api.ApiCommandResourceType;


Suggested change

import org.apache.cloudstack.api.ApiCommandResourceType;

import org.apache.cloudstack.api.ApiCommandResourceType;

DaanHoogland · 2024-10-28T15:13:16Z

...chema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImpl.java

@@ -46,6 +28,22 @@
 import com.cloud.utils.db.SearchCriteria;
 import com.cloud.utils.db.TransactionLegacy;
 import com.cloud.utils.db.UpdateBuilder;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataObjectInStore;


Suggested change

import org.apache.cloudstack.engine.subsystem.api.storage.DataObjectInStore;

import org.apache.cloudstack.engine.subsystem.api.storage.DataObjectInStore;

DaanHoogland · 2024-10-28T15:13:26Z

...chema/src/main/java/org/apache/cloudstack/storage/datastore/db/SnapshotDataStoreDaoImpl.java

+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine.Event;
+import org.apache.cloudstack.engine.subsystem.api.storage.ObjectInDataStoreStateMachine.State;
+import org.apache.commons.collections.CollectionUtils;


Suggested change

import org.apache.commons.collections.CollectionUtils;

import org.apache.commons.collections.CollectionUtils;

DaanHoogland · 2024-10-28T15:14:00Z

...orage/snapshot/src/main/java/org/apache/cloudstack/storage/snapshot/SnapshotServiceImpl.java

+import com.cloud.utils.db.TransactionCallbackNoReturn;
+import com.cloud.utils.db.TransactionStatus;
+import com.cloud.utils.exception.CloudRuntimeException;
+import com.cloud.utils.fsm.NoTransitionException;


Suggested change

import com.cloud.utils.fsm.NoTransitionException;

import com.cloud.utils.fsm.NoTransitionException;

DaanHoogland · 2024-10-28T15:36:53Z

server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

+import org.apache.cloudstack.utils.volume.VirtualMachineDiskInfo;
+import org.apache.commons.collections.CollectionUtils;


Suggested change

import org.apache.cloudstack.utils.volume.VirtualMachineDiskInfo;

import org.apache.commons.collections.CollectionUtils;

import org.apache.cloudstack.utils.volume.VirtualMachineDiskInfo;

import org.apache.commons.collections.CollectionUtils;

DaanHoogland · 2024-10-28T15:37:28Z

server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

+            if (CollectionUtils.isNotEmpty(poolIds)) {
+                throw new InvalidParameterValueException(String.format("%s can not be specified for snapshots linked with snapshot policy", ApiConstants.STORAGE_ID_LIST));
+            }
+            List<SnapshotPolicyDetailVO> poolDetails = snapshotPolicyDetailsDao.findDetails(policyId, ApiConstants.STORAGE_ID);
+            poolIds = poolDetails.stream().map(d -> Long.valueOf(d.getValue())).collect(Collectors.toList());


new method?

DaanHoogland · 2024-10-28T15:38:01Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

@@ -174,6 +108,73 @@
 import com.cloud.vm.snapshot.VMSnapshotVO;
 import com.cloud.vm.snapshot.dao.VMSnapshotDao;
 import com.cloud.vm.snapshot.dao.VMSnapshotDetailsDao;
+import org.apache.cloudstack.acl.SecurityChecker;


Suggested change

import org.apache.cloudstack.acl.SecurityChecker;

import org.apache.cloudstack.acl.SecurityChecker;

DaanHoogland · 2024-10-28T15:38:14Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

+import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
+import org.apache.commons.collections.CollectionUtils;


Suggested change

import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;

import org.apache.commons.collections.CollectionUtils;

import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;

import org.apache.commons.collections.CollectionUtils;

DaanHoogland · 2024-10-28T15:41:01Z

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java

+    private boolean canCopyOnPrimary(List<Long> poolIds, Snapshot snapshot) {
+        List<Long> poolsToBeRemoved = new ArrayList<>();
+        for (Long poolId : poolIds) {
+            PrimaryDataStore dataStore = (PrimaryDataStore) dataStoreMgr.getDataStore(poolId, DataStoreRole.Primary);
+            if (dataStore == null) {
+                poolsToBeRemoved.add(poolId);
+                continue;
+            }
+            SnapshotInfo snapshotInfo = snapshotFactory.getSnapshot(snapshot.getId(), poolId, DataStoreRole.Primary);
+            if (snapshotInfo != null) {
+                logger.debug(String.format("Snapshot [%s] already exist on pool [%s]", snapshot.getUuid(), dataStore.getName()));
+                continue;
+            }
+
+            VolumeVO volume = _volsDao.findById(snapshot.getVolumeId());
+            if (volume == null) {
+                poolsToBeRemoved.add(poolId);
+                continue;
+            }
+            if (!dataStore.getDriver().getCapabilities()
+                    .containsKey(DataStoreCapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES_AND_SAME_POOL_TYPE.toString())
+                    && dataStore.getPoolType() != volume.getPoolType()) {
+                poolsToBeRemoved.add(poolId);
+                logger.debug(String.format("The %s  does not support copy to %s between zones", dataStore.getPoolType(), volume.getPoolType()));
+            }
+        }
+        poolIds.removeAll(poolsToBeRemoved);
+        if (CollectionUtils.isEmpty(poolIds)) {
+            return false;
+        }
+        return true;
+    }


this can be further distilled into smaller methods. Surely some of those can than be reused for the actual copy method.

github-actions · 2024-11-04T14:33:46Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

boring-cyborg bot added component:api component:ceph component:orchestration component:primary-storage component:storage labels Aug 1, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 8cc7b49 to 0df6764 Compare August 14, 2024 09:14

slavkap marked this pull request as ready for review August 14, 2024 09:51

slavkap changed the title ~~[WIP] Support of snapshot copy to StorPool primary storage in different zones~~ Support of snapshot copy to StorPool primary storage in different zones Aug 14, 2024

github-actions bot added the status:has-conflicts label Aug 21, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 0df6764 to e0f4283 Compare August 22, 2024 12:49

github-actions bot removed the status:has-conflicts label Aug 22, 2024

github-actions bot added the status:has-conflicts label Aug 29, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from e0f4283 to 6f209d8 Compare August 30, 2024 09:08

github-actions bot removed the status:has-conflicts label Aug 30, 2024

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Aug 30, 2024

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/snapshot/CopySnapshotCmd.java Outdated Show resolved Hide resolved

rohityadavcloud added this to the 4.20.0.0 milestone Sep 4, 2024

github-actions bot added the status:has-conflicts label Sep 4, 2024

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

.../src/main/java/org/apache/cloudstack/engine/subsystem/api/storage/DataStoreCapabilities.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

server/src/main/java/com/cloud/storage/snapshot/SnapshotManagerImpl.java Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

ui/src/views/storage/FormSchedule.vue Outdated Show resolved Hide resolved

sureshanaparti reviewed Sep 4, 2024

View reviewed changes

ui/src/views/storage/TakeSnapshot.vue Outdated Show resolved Hide resolved

JoaoJandre modified the milestones: 4.20.0.0, 4.21.0.0 Sep 10, 2024

slavkap force-pushed the support-snapshot-copy-on-primary branch from 524f382 to 457b33f Compare September 16, 2024 14:24

github-actions bot removed the status:has-conflicts label Sep 16, 2024

github-actions bot added the status:has-conflicts label Oct 10, 2024

github-actions bot added status:has-conflicts and removed status:has-conflicts labels Oct 14, 2024

slavkap added 5 commits October 21, 2024 16:47

Support of snapshot copy to different StorPool primary storage betwee…

27b9ef4

…n zones Added support to copy a snapshot to another StorPool primary storage in different zones.

added option in the UI for the storage pools

9f643fc

Added drop down to choose the primary storage pools to copy a snapshot Small fixes

Addressed comments

e19923c

Address reviews

156654e

fix snapshot deletion

88d13d2

slavkap force-pushed the support-snapshot-copy-on-primary branch from 457b33f to 88d13d2 Compare October 22, 2024 07:56

github-actions bot removed the status:has-conflicts label Oct 22, 2024

DaanHoogland reviewed Oct 28, 2024

View reviewed changes

DaanHoogland approved these changes Oct 28, 2024

View reviewed changes

github-actions bot added the status:has-conflicts label Nov 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support of snapshot copy to StorPool primary storage in different zones #9478

Support of snapshot copy to StorPool primary storage in different zones #9478

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

sureshanaparti Sep 4, 2024

sureshanaparti Sep 4, 2024

slavkap Sep 16, 2024

sureshanaparti Sep 4, 2024

slavkap Sep 16, 2024

sureshanaparti Sep 4, 2024

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 15, 2024

DaanHoogland Oct 28, 2024

DaanHoogland left a comment

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

DaanHoogland Oct 28, 2024

github-actions bot commented Nov 4, 2024

	private static List<String> snapshotsForRcovery(JsonArray arr) {
	private static List<String> snapshotsForRecovery(JsonArray arr) {

	this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES && pool.zoneid !== this.resource.zoneid)
	this.storagePools = this.storagePools.filter(pool => pool.storagecapabilities.CAN_COPY_SNAPSHOT_BETWEEN_ZONES_AND_SAME_POOL_TYPE && pool.zoneid !== this.resource.zoneid)

	import org.apache.commons.collections.CollectionUtils;

	import org.apache.commons.collections.CollectionUtils;

	import org.apache.cloudstack.api.ApiCommandResourceType;

	import org.apache.cloudstack.api.ApiCommandResourceType;

	import org.apache.cloudstack.engine.subsystem.api.storage.DataObjectInStore;

	import org.apache.cloudstack.engine.subsystem.api.storage.DataObjectInStore;

	import com.cloud.utils.fsm.NoTransitionException;
	import com.cloud.utils.fsm.NoTransitionException;

		import org.apache.cloudstack.utils.volume.VirtualMachineDiskInfo;
		import org.apache.commons.collections.CollectionUtils;

	import org.apache.cloudstack.acl.SecurityChecker;

	import org.apache.cloudstack.acl.SecurityChecker;

		import org.apache.cloudstack.utils.reflectiontostringbuilderutils.ReflectionToStringBuilderUtils;
		import org.apache.commons.collections.CollectionUtils;

Support of snapshot copy to StorPool primary storage in different zones #9478

Are you sure you want to change the base?

Support of snapshot copy to StorPool primary storage in different zones #9478

Conversation

slavkap commented Aug 1, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Aug 1, 2024 • edited Loading

Codecov Report

github-actions bot commented Aug 21, 2024

DaanHoogland commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

blueorangutan commented Aug 22, 2024

DaanHoogland commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

blueorangutan commented Aug 23, 2024

github-actions bot commented Aug 29, 2024

github-actions bot commented Sep 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 10, 2024

github-actions bot commented Oct 15, 2024

Choose a reason for hiding this comment

DaanHoogland left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Nov 4, 2024

slavkap commented Aug 1, 2024 •

edited

Loading

codecov bot commented Aug 1, 2024 •

edited

Loading