feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 #3620

pdecat · 2023-06-26T12:44:03Z

This PR upgrades the oras client library to v2 and adds support for OCI registries requiring authentication (tested with Gitlab).

Resolves #2819

pdecat · 2023-06-26T12:47:35Z

go.mod

@@ -32,7 +31,8 @@ require (
 	github.com/mattn/go-isatty v0.0.19
 	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db
 	github.com/olekukonko/tablewriter v0.0.5
-	github.com/opencontainers/image-spec v1.0.2
+	github.com/opencontainers/image-spec v1.1.0-rc.3
+	github.com/oras-project/oras-credentials-go v0.2.0


This dependency is going to be merged into oras-go oras-project/oras-credentials-go#80

Ah!

Is there a timeline for the merge? It wasn't obvious to me from the discussion thread.

Well, they are talking about waiting for v1.0.0-rc.1, currently at v0.2.0.

I asked in their Slack about the timeline.

It should not take too long:

We need to wait for couple of weeks or a month to ensure that all APIs are indeed stable. The stability is currently being validated by oras CLI and notation CLI

binaek · 2023-06-26T12:49:40Z

Thank you for the PR @pdecat .

We have been meaning to get to this, (open issue), but were always held back with the conflicts in go dependency resolution.

pdecat · 2023-06-26T13:04:40Z

We have been meaning to get to this, (open issue),

Did not see that issue, updated PR description to link to it.

but were always held back with the conflicts in go dependency resolution.

I don't see any conflict nowadays, isn't it?

pdecat · 2023-06-26T13:16:58Z

pkg/ociinstaller/ocidownloader.go

+	log.Println("[TRACE] ociDownloader.Pull:", "pulling...")
+
+	copyOpt := oras.DefaultCopyOptions
+	// TODO: use WithTargetPlatform to limit downloads


Not sure if this is desired or not as there are checks later to ensure binaries for several platforms exist.

If this limits the layers downloaded, then we should probably include it

It apparently require more changes than I expected.

As a test, just adding this:

diff --git a/pkg/ociinstaller/ocidownloader.go b/pkg/ociinstaller/ocidownloader.go index bb6f8815..cfb3318a 100644 --- a/pkg/ociinstaller/ocidownloader.go +++ b/pkg/ociinstaller/ocidownloader.go @@ -79,8 +79,10 @@ func (o *ociDownloader) Pull(ctx context.Context, ref string, mediaTypes []strin log.Println("[TRACE] ociDownloader.Pull:", "pulling...") copyOpt := oras.DefaultCopyOptions - // TODO: use WithTargetPlatform to limit downloads - // copyOpt.WithTargetPlatform() + copyOpt.WithTargetPlatform(&ocispec.Platform{ + Architecture: "amd64", + OS: "linux", + }) manifestDescriptor, err := oras.Copy(ctx, repo, tag, fileStore, tag, copyOpt) if err != nil { log.Println("[ERROR] ociDownloader.Pull:", "failed to pull", ref, err)

Results in:

steampipe plugin install csv:latest 2023-06-26 15:09:57.723 UTC [TRACE] steampipe: parse complete after 1 decode passes 2023-06-26 15:09:57.723 UTC [TRACE] steampipe: ensureInstallDir /home/patrick/.steampipe 2023-06-26 15:09:57.727 UTC [TRACE] steampipe: nothing to migrate in /home/patrick/.steampipe/internal/update-check.json 2023-06-26 15:09:57.728 UTC [TRACE] steampipe: No memory limit set 2023-06-26 15:09:57.728 UTC [TRACE] steampipe: ociDownloader.Download: downloading us-docker.pkg.dev/steampipe/plugins/turbot/csv:latest 2023-06-26 15:09:57.728 UTC [TRACE] steampipe: ociDownloader.Pull: preparing to pull ref us-docker.pkg.dev/steampipe/plugins/turbot/csv:latest tag latest destDir /home/patrick/.steampipe/plugins/tmp-0e41-4be8-9d66 2023-06-26 15:09:57.728 UTC [TRACE] steampipe: ociDownloader.Pull: pulling... csv:latest [=========>----------------------------------------------------------] Downloading 2023-06-26 15:09:58.426 UTC [ERROR] steampipe: ociDownloader.Pull: failed to pull us-docker.pkg.dev/steampipe/plugins/turbot/csv:latest fail to recognize platform from unknown config applicati csv:latest [====================================================================] Done Skipped the following plugin: Plugin: csv@latest Reason: fail to recognize platform from unknown config application/vnd.turbot.steampipe.config.v1+json: expect application/vnd.oci.image.config.v1+json

@pdecat can you please remove this comment if it's not working/required?

pdecat · 2023-06-27T08:46:06Z

pkg/ociinstaller/ocidownloader.go

+	}
+
+	// Fetch the config from the file store
+	configData, err := content.FetchAll(ctx, fileStore, manifest.Config)


Simplified code a bit with feedback from the oras-go maintainers: https://cloud-native.slack.com/archives/CJ1KHJM5Z/p1687852628854409?thread_ts=1687832097.450759&cid=CJ1KHJM5Z

pkg/ociinstaller/ocidownloader.go

kaidaguerre

Just one small tweak and I think we're good to merge it

kaidaguerre · 2023-07-12T13:10:39Z

cmd/plugin.go

@@ -409,7 +409,13 @@ func runPluginUpdateCmd(cmd *cobra.Command, args []string) {
 			ref := ociinstaller.NewSteampipeImageRef(p)
 			isExists, _ := plugin.Exists(p)
 			if isExists {
-				runUpdatesFor = append(runUpdatesFor, versionData.Plugins[ref.DisplayImageRef()])
+				if strings.HasPrefix(ref.DisplayImageRef(), "hub.steampipe.io/") {


can we put a const in for hub.steampipe.io/ in the consts package

kaidaguerre · 2023-07-12T13:10:51Z

pkg/ociinstaller/imageref.go

 	pluginNameAndStream := strings.Split(split[len(split)-1], "@")
-
-	return org, pluginNameAndStream[0], pluginNameAndStream[1]
+	if strings.HasPrefix(r.DisplayImageRef(), "hub.steampipe.io/") {


can we put a const in for hub.steampipe.io in the consts package

binaek · 2023-07-12T13:29:53Z

@pdecat We have a couple of tests which verify that OCI installations are working as expected (correct binaries in the expected locations)

Can you rebase your branch with main? That should help verify the work you are doing!

…entication

…ub.steampipe.io

…pipe.io

pdecat · 2023-07-12T13:33:40Z

Added the constant, and rebased on main.

pdecat · 2023-07-12T13:58:38Z

Looks like an acceptance test failed because of Github rate limiting:

Error: fatal: unable to access 'https://github.com/turbot/steampipe/': The requested URL returned error: 429

https://github.com/turbot/steampipe/actions/runs/5532253026/jobs/10094278301?pr=3620#step:2:57

Also, a check is failing on MacOS:

/Users/runner/work/_temp/f5247854-5695-4fc1-8fcd-d6f01fc9338e.sh: line 1: [: -eq: unary operator expected

https://github.com/turbot/steampipe/actions/runs/5532253026/jobs/10094274375?pr=3620#step:15:16

pdecat · 2023-07-12T14:06:05Z

Tests related to this PR seem to pass on Linux:

ok 36 plugin list - output table
ok 37 plugin list - output json
ok 38 plugin list - output table (with a missing plugin)
ok 39 plugin list - output json (with a missing plugin)
ok 40 plugin list - output table (with a failed plugin)
ok 41 plugin list - output json (with a failed plugin)
ok 42 verify that installing plugins creates individual version.json files
ok 43 verify that backfilling of individual plugin version.json works
ok 44 verify that backfilling of individual plugin version.json works where it is only partially backfilled
ok 45 verify that global plugin/versions.json is composed from individual version.json files when it is absent
ok 46 verify that global plugin/versions.json is composed from individual version.json files when it is corrupt
ok 47 verify that composition of global plugin/versions.json works when an individual version.json file is corrupt
ok 48 verify that plugin installed from registry are marked as 'local' when the modtime of the binary is after the install time

https://github.com/turbot/steampipe/actions/runs/5532253026/jobs/10094278681?pr=3620

But not on MacOS:

ok 36 plugin list - output table
ok 37 plugin list - output json
not ok 38 plugin list - output table (with a missing plugin) # timeout after 120s
# (in test file tests/acceptance/test_files/service_and_plugin.bats, line 496)
#   `steampipe plugin uninstall hackernews@0.6.0 --install-dir $tmpdir' failed due to timeout
#
#
# Installed plugin: bitbucket@0.3.1 v0.3.1
# Documentation:    https://hub.steampipe.io/plugins/turbot/bitbucket
#
# Installed plugin: hackernews@0.6.0 v0.6.0
# Documentation:    https://hub.steampipe.io/plugins/turbot/hackernews
#
# /Users/runner/work/steampipe/steampipe/tests/acceptance/test_files/service_and_plugin.bats: line 491: 10387 Terminated: 15          steampipe plugin uninstall hackernews@0.6.0 --install-dir $tmpdir
not ok 39 plugin list - output json (with a missing plugin)
# (in test file tests/acceptance/test_files/service_and_plugin.bats, line 507)
#   `steampipe plugin uninstall hackernews@0.6.0 --install-dir $tmpdir' failed with status 12
#
#
# Installed plugin: bitbucket@0.3.1 v0.3.1
# Documentation:    https://hub.steampipe.io/plugins/turbot/bitbucket
#
# Installed plugin: hackernews@0.6.0 v0.6.0
# Documentation:    https://hub.steampipe.io/plugins/turbot/hackernews
#
# Error: cannot listen on listenAddresses [localhost] and port 9193
not ok 40 plugin list - output table (with a failed plugin)
# (from function `assert_equal' in file tests/acceptance/lib/bats-assert/src/assert_equal.bash, line 40,
#  in test file tests/acceptance/test_files/service_and_plugin.bats, line 525)
#   `assert_equal "$output" "$(cat $TEST_DATA_DIR/expected_plugin_list_table_with_failed_plugins.txt)"' failed
#
#
# Installed plugin: bitbucket@0.3.1 v0.3.1
# Documentation:    https://hub.steampipe.io/plugins/turbot/bitbucket
#
# Installed plugin: hackernews@0.6.0 v0.6.0
# Documentation:    https://hub.steampipe.io/plugins/turbot/hackernews
#
# Error: plugin listing failed - cannot listen on listenAddresses [localhost] and port 9193
#
# -- values do not equal --
#     ],
#     "warnings": null
#   }
# actual (1 lines):
#   Error: plugin listing failed - cannot listen on listenAddresses [localhost] and port 9193
# --
#
ok 42 verify that installing plugins creates individual version.json files
ok 43 verify that backfilling of individual plugin version.json works
ok 44 verify that backfilling of individual plugin version.json works where it is only partially backfilled
ok 45 verify that global plugin/versions.json is composed from individual version.json files when it is absent
ok 46 verify that global plugin/versions.json is composed from individual version.json files when it is corrupt
ok 47 verify that composition of global plugin/versions.json works when an individual version.json file is corrupt
not ok 48 verify that plugin installed from registry are marked as 'local' when the modtime of the binary is after the install time
# (from function `assert_equal' in file tests/acceptance/lib/bats-assert/src/assert_equal.bash, line 40,
#  in test file tests/acceptance/test_files/service_and_plugin.bats, line [68](https://github.com/turbot/steampipe/actions/runs/5532253026/jobs/10094274027?pr=3620#step:13:69)9)
#   `assert_equal "$version" '"local"'' failed
# Error: plugin listing failed - cannot listen on listenAddresses [localhost] and port 9193
#
# -- values do not equal --
# expected : "local"
# actual   :
# --
#

https://github.com/turbot/steampipe/actions/runs/5532253026/jobs/10094274027?pr=3620

binaek

@pdecat requested a couple of (very) small changes. Thank you!

binaek · 2023-07-13T15:13:05Z

pkg/ociinstaller/ocidownloader.go

+	log.Println("[TRACE] ociDownloader.Pull:", "pulling...")
+
+	copyOpt := oras.DefaultCopyOptions
+	// TODO: use WithTargetPlatform to limit downloads


@pdecat can you please remove this comment if it's not working/required?

binaek · 2023-07-13T15:13:53Z

pkg/ociinstaller/ocidownloader.go

 	if err != nil {
-		return &desc, nil, nil, layers, err
+		log.Println("[ERROR] ociDownloader.Pull:", "failed to pull", ref, err)


Make this a TRACE level log. Raw log messages should not end up in the console if we can avoid it.

binaek · 2023-07-13T15:14:17Z

pkg/ociinstaller/ocidownloader.go

+	// Fetch the config from the file store
+	configData, err := content.FetchAll(ctx, fileStore, manifest.Config)
+	if err != nil {
+		log.Println("[ERROR] ociDownloader.Pull:", "failed to fetch config", manifest.Config.MediaType, err)


pdecat commented Jun 26, 2023

View reviewed changes

pdecat force-pushed the oras-go-v2-with-auth branch 2 times, most recently from db001d0 to 4be5a04 Compare June 26, 2023 13:14

pdecat commented Jun 26, 2023

View reviewed changes

pdecat force-pushed the oras-go-v2-with-auth branch from 4be5a04 to 9f0b13f Compare June 26, 2023 15:18

pdecat commented Jun 27, 2023

View reviewed changes

pkg/ociinstaller/ocidownloader.go Outdated Show resolved Hide resolved

pkg/ociinstaller/ocidownloader.go Outdated Show resolved Hide resolved

pkg/ociinstaller/ocidownloader.go Outdated Show resolved Hide resolved

pkg/ociinstaller/ocidownloader.go Show resolved Hide resolved

binaek added this to the 0.22.0 milestone Jun 29, 2023

pdecat force-pushed the oras-go-v2-with-auth branch from 5f43524 to 62caef9 Compare July 7, 2023 06:19

kaidaguerre requested changes Jul 12, 2023

View reviewed changes

binaek changed the title ~~feat: upgrade to oras-go v2 and support OCI registries requiring authentication~~ feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 Jul 12, 2023

pdecat added 6 commits July 12, 2023 15:30

feat: upgrade to oras-go v2 and support OCI registries requiring auth…

6c31ae9

…entication

Do not try to contact update server for plugins not distributed via h…

6a18cd1

…ub.steampipe.io

Simplify manifest config fetching

3c37d91

Return the full path of the plugin if it does not come from hub.steam…

b0e777c

…pipe.io

fix: do not return &manifestDescriptor in case of error

47e6366

fix: extract hub.steampipe.io/ as a constant

2eeffaa

pdecat force-pushed the oras-go-v2-with-auth branch from 62caef9 to 2eeffaa Compare July 12, 2023 13:33

binaek mentioned this pull request Jul 13, 2023

Cannot update a plugin from a third party hub #2152

Closed

binaek suggested changes Jul 13, 2023

View reviewed changes

pdecat added 2 commits July 13, 2023 18:55

Remove TODO about using WithTargetPlatform for OCI downloads

0449afc

Log raw errors at TRACE level

1633356

kaidaguerre merged commit 3c5e98d into turbot:main Jul 14, 2023

pdecat deleted the oras-go-v2-with-auth branch July 14, 2023 09:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 #3620

feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 #3620

pdecat commented Jun 26, 2023 •

edited

Loading

pdecat Jun 26, 2023

binaek Jun 26, 2023

pdecat Jun 26, 2023

pdecat Jun 26, 2023

pdecat Jun 27, 2023

binaek commented Jun 26, 2023

pdecat commented Jun 26, 2023

pdecat Jun 26, 2023

binaek Jun 26, 2023

pdecat Jun 26, 2023 •

edited

Loading

binaek Jul 13, 2023

pdecat Jun 27, 2023

kaidaguerre left a comment

kaidaguerre Jul 12, 2023

pdecat Jul 12, 2023

kaidaguerre Jul 12, 2023

pdecat Jul 12, 2023

binaek commented Jul 12, 2023

pdecat commented Jul 12, 2023

pdecat commented Jul 12, 2023 •

edited

Loading

pdecat commented Jul 12, 2023

binaek left a comment

binaek Jul 13, 2023

binaek Jul 13, 2023

pdecat Jul 13, 2023

binaek Jul 13, 2023

pdecat Jul 13, 2023

feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 #3620

feat: upgrade to oras-go v2 and support OCI registries requiring authentication. Closes #2819 #3620

Conversation

pdecat commented Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

binaek commented Jun 26, 2023

pdecat commented Jun 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdecat Jun 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaidaguerre left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

binaek commented Jul 12, 2023

pdecat commented Jul 12, 2023

pdecat commented Jul 12, 2023 • edited Loading

pdecat commented Jul 12, 2023

binaek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdecat commented Jun 26, 2023 •

edited

Loading

pdecat Jun 26, 2023 •

edited

Loading

pdecat commented Jul 12, 2023 •

edited

Loading