feat: add flag to skip deleting obsolete referrers index #957

qweeah · 2023-05-29T02:31:26Z

What this PR does / why we need it:
This PR adds a flag to oras attach, oras cp and oras manifest push. The flag can be used to specify whether to clean up old referrers index when referrers API is not available.

There is not a breaking change and by default, mentioned commands will still do referrers cleanup.

Depends on merging of #967.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #954, fixes #952, fixes #915

Please check the following list:

Does the affected code have corresponding tests, e.g. unit test, E2E test?
Does this change require a documentation update?
Does this introduce breaking changes that would require an announcement or bumping the major version?
Do all new files have an appropriate license header?

codecov-commenter · 2023-05-29T03:58:32Z

Codecov Report

Merging #957 (757f2ae) into main (f9aec91) will increase coverage by 0.30%.
The diff coverage is 85.71%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main     #957      +/-   ##
==========================================
+ Coverage   81.08%   81.39%   +0.30%     
==========================================
  Files          53       54       +1     
  Lines        2792     2816      +24     
==========================================
+ Hits         2264     2292      +28     
+ Misses        358      355       -3     
+ Partials      170      169       -1

Impacted Files	Coverage Δ
cmd/oras/internal/option/referrers.go	`55.55% <55.55%> (ø)`
cmd/oras/internal/errors/errors.go	`100.00% <100.00%> (ø)`
cmd/oras/root/attach.go	`85.48% <100.00%> (+2.98%)`	⬆️
cmd/oras/root/cp.go	`86.86% <100.00%> (+0.49%)`	⬆️
cmd/oras/root/manifest/push.go	`77.27% <100.00%> (+4.40%)`	⬆️

cmd/oras/internal/option/referrers.go

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT

What about the manifest delete path?

cmd/oras/internal/option/referrers.go

qweeah · 2023-06-29T10:28:46Z

What about the manifest delete path?

Referrers index deletion happens after manifest deletion right? If the registry already supports deletion and the token used covers manifest deletion, why would referrers index fail?

Also why would user want to skip referrers index deletion when manifest deletion is successful?

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT · 2023-06-30T06:52:10Z

Referrers index deletion happens after manifest deletion right? If the registry already supports deletion and the token used covers manifest deletion, why would referrers index fail?

LoL. Makes sense.

cmd/oras/internal/option/referrers.go

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT

LGTM

shizhMSFT

LGTM

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

…t#957) Signed-off-by: Billy Zha <jinzha1@microsoft.com> Co-authored-by: Terry Howe <tlhowe@amazon.com>

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

…)" (#1046) Signed-off-by: Billy Zha <jinzha1@microsoft.com>

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah force-pushed the gc-referrers branch from 5ebe86e to 3819f7d Compare May 29, 2023 03:55

qweeah force-pushed the gc-referrers branch from 193980f to 3151255 Compare May 30, 2023 07:25

qweeah marked this pull request as ready for review June 1, 2023 00:33

qweeah requested review from sajayantony, shizhMSFT, SteveLasker and TerryHowe as code owners June 1, 2023 00:33

qweeah force-pushed the gc-referrers branch from d37ec9b to b16519e Compare June 1, 2023 00:33

Wwwsylvia reviewed Jun 2, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Show resolved Hide resolved

qweeah changed the title ~~feat!: add flag for referrers garbage collection~~ feat!: add flag to skip deleting obsolete referrers index Jun 2, 2023

qweeah added 15 commits June 6, 2023 01:32

feat!: add flag for referrers GC

8b22600

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

update go mod

7b7e5a4

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix e2e

e27d723

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

add e2e

57b4e75

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

update deps

adfcca1

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

update dep

cacd6bb

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

update

f0ecebf

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

update version

ffbd8bc

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

mod tidy

d5f1937

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix e2e

abae524

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

rename flag

06d382d

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix: default to true

86f3b1d

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix e2e

19c4516

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

change default value to false

623dc99

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

fix e2e

aa73bc9

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah force-pushed the gc-referrers branch from 1e65493 to aa73bc9 Compare June 6, 2023 01:32

fix e2e

aa91493

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah changed the title ~~feat!: add flag to skip deleting obsolete referrers index~~ feat: add flag to skip deleting obsolete referrers index Jun 6, 2023

refactor: simplify e2e scripts and github action

06af37b

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added 2 commits June 28, 2023 01:37

Merge branch 'main' into gc-referrers

bec6661

Merge branch 'main' into gc-referrers

a69435d

shizhMSFT added this to the v1.1.0 milestone Jun 28, 2023

qweeah mentioned this pull request Jun 28, 2023

Improve the error msg for OCI v1.0 registries or registries that don't support deletion API #915

Closed

1 task

shizhMSFT reviewed Jun 29, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Outdated Show resolved Hide resolved

qweeah added 2 commits June 29, 2023 15:28

Merge remote-tracking branch 'origin-upstream/main' into gc-referrers

fdc59b8

bug fix

6b92395

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT reviewed Jun 30, 2023

View reviewed changes

cmd/oras/internal/option/referrers.go Outdated Show resolved Hide resolved

update comment

757f2ae

Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT approved these changes Jul 3, 2023

View reviewed changes

Merge branch 'main' into gc-referrers

474c798

shizhMSFT approved these changes Jul 3, 2023

View reviewed changes

qweeah merged commit ba60b99 into oras-project:main Jul 3, 2023

This was referenced Jul 5, 2023

Vote for releasing v1.1.0-rc.1 #1002

Closed

bump: tag and release ORAS CLI v1.1.0-rc.1 #1016

Merged

shizhMSFT mentioned this pull request Jul 26, 2023

Request to adjust the release plan #1038

Closed

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

078dd8e

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

fb3dd72

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

feat: add flag to skip deleting obsolete referrers index (oras-projec…

5e6d3bf

…t#957) Signed-off-by: Billy Zha <jinzha1@microsoft.com> Co-authored-by: Terry Howe <tlhowe@amazon.com>

shizhMSFT pushed a commit to shizhMSFT/oras that referenced this pull request Aug 3, 2023

Revert "feat: add flag to skip deleting obsolete referrers index (ora…

6f6c9c1

…s-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

d25f652

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

73b531d

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

06700fb

…as-project#957)" This reverts commit ba60b99. Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added a commit that referenced this pull request Aug 3, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (#957…

43e4832

…)" (#1046) Signed-off-by: Billy Zha <jinzha1@microsoft.com>

ZhongRuoyu mentioned this pull request Aug 7, 2023

oras 1.0.1 Homebrew/homebrew-core#138923

Merged

qweeah added a commit to qweeah/oras that referenced this pull request Aug 9, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

b3ac818

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <jinzha1@microsoft.com>

qweeah added a commit to qweeah/oras that referenced this pull request Aug 9, 2023

revert: "feat: add flag to skip deleting obsolete referrers index (or…

44c61b9

…as-project#957)" (oras-project#1046) Signed-off-by: Billy Zha <jinzha1@microsoft.com>

shizhMSFT mentioned this pull request Jan 8, 2024

manifest delete is leaking indexes when deleting client-managed referrer #1228

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add flag to skip deleting obsolete referrers index #957

feat: add flag to skip deleting obsolete referrers index #957

qweeah commented May 29, 2023 •

edited

Loading

codecov-commenter commented May 29, 2023 •

edited

Loading

shizhMSFT left a comment

qweeah commented Jun 29, 2023 •

edited

Loading

shizhMSFT commented Jun 30, 2023

shizhMSFT left a comment

shizhMSFT left a comment

feat: add flag to skip deleting obsolete referrers index #957

feat: add flag to skip deleting obsolete referrers index #957

Conversation

qweeah commented May 29, 2023 • edited Loading

codecov-commenter commented May 29, 2023 • edited Loading

Codecov Report

shizhMSFT left a comment

Choose a reason for hiding this comment

qweeah commented Jun 29, 2023 • edited Loading

shizhMSFT commented Jun 30, 2023

shizhMSFT left a comment

Choose a reason for hiding this comment

shizhMSFT left a comment

Choose a reason for hiding this comment

qweeah commented May 29, 2023 •

edited

Loading

codecov-commenter commented May 29, 2023 •

edited

Loading

qweeah commented Jun 29, 2023 •

edited

Loading