Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zarf package mirror command introduction #1913

Merged
merged 20 commits into from
Sep 6, 2023
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -175,6 +175,7 @@ test-external: ## Run the Zarf CLI E2E tests for an external registry and cluste
@test -s $(ZARF_BIN) || $(MAKE) build-cli
@test -s ./build/zarf-init-$(ARCH)-$(CLI_VERSION).tar.zst || $(MAKE) init-package
@test -s ./build/zarf-package-podinfo-flux-$(ARCH).tar.zst || $(ZARF_BIN) package create examples/podinfo-flux -o build -a $(ARCH) --confirm
@test -s ./build/zarf-package-argocd-$(ARCH).tar.zst || $(ZARF_BIN) package create examples/argocd -o build -a $(ARCH) --confirm
cd src/test/external && go test -failfast -v -timeout 30m

## NOTE: Requires an existing cluster and
Expand Down
1 change: 1 addition & 0 deletions docs/2-the-zarf-cli/100-cli-commands/zarf_package.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ Zarf package commands for creating, deploying, and inspecting packages
* [zarf package deploy](zarf_package_deploy.md) - Deploys a Zarf package from a local file or URL (runs offline)
* [zarf package inspect](zarf_package_inspect.md) - Displays the definition of a Zarf package (runs offline)
* [zarf package list](zarf_package_list.md) - Lists out all of the packages that have been deployed to the cluster (runs offline)
* [zarf package mirror-resources](zarf_package_mirror-resources.md) - Mirrors a Zarf package's resources from a local file or URL to specified registries and repos
* [zarf package publish](zarf_package_publish.md) - Publishes a Zarf package to a remote registry
* [zarf package pull](zarf_package_pull.md) - Pulls a Zarf package from a remote registry and save to the local file system
* [zarf package remove](zarf_package_remove.md) - Removes a Zarf package that has been deployed already (runs offline)
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# zarf package mirror-resources
<!-- Auto-generated by hack/gen-cli-docs.sh -->

Mirrors a Zarf package's resources from a local file or URL to specified registries and repos

## Synopsis

Unpacks resources and dependencies from a Zarf package archive and mirrors them into the specified
registries and repos within the target environment

```
zarf package mirror-resources [ PACKAGE ] [flags]
```

## Options

```
--components string Comma-separated list of components to mirror. This list will be respected regardless of a component's 'required' status.
--confirm Confirms package deployment without prompting. ONLY use with packages you trust. Skips prompts to review SBOM, configure variables, select optional components and review potential breaking changes.
--git-push-password string Password for the push-user to access the git server
--git-push-username string Username to access to the git server Zarf is configured to use. User must be able to create repositories via 'git push' (default "zarf-git-user")
--git-url string External git server url to use for this Zarf cluster
-h, --help help for mirror-resources
--no-img-checksum Turns off the addition of a checksum to image tags (as would be used by the Zarf Agent) while mirroring images.
--registry-push-password string Password for the push-user to connect to the registry
--registry-push-username string Username to access to the registry Zarf is configured to use (default "zarf-push")
--registry-url string External registry url address to use for this Zarf cluster
```

## Options inherited from parent commands

```
-a, --architecture string Architecture for OCI images and Zarf packages
--insecure Allow access to insecure registries and disable other recommended security enforcements such as package checksum and signature validation. This flag should only be used if you have a specific reason and accept the reduced security posture.
-l, --log-level string Log level when running Zarf. Valid options are: warn, info, debug, trace (default "info")
--no-color Disable colors in output
--no-log-file Disable log file creation
--no-progress Disable fancy UI progress bars, spinners, logos, etc
--oci-concurrency int Number of concurrent layer operations to perform when interacting with a remote package. (default 3)
--tmpdir string Specify the temporary directory to use for intermediate files
--zarf-cache string Specify the location of the Zarf cache directory (default "~/.zarf-cache")
```

## SEE ALSO

* [zarf package](zarf_package.md) - Zarf package commands for creating, deploying, and inspecting packages
46 changes: 46 additions & 0 deletions src/cmd/package.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ import (
var includeInspectSBOM bool
var outputInspectSBOM string
var inspectPublicKey string
var noImgChecksum bool

var packageCmd = &cobra.Command{
Use: "package",
Expand Down Expand Up @@ -102,6 +103,28 @@ var packageDeployCmd = &cobra.Command{
},
}

var packageMirrorCmd = &cobra.Command{
Use: "mirror-resources [ PACKAGE ]",
Aliases: []string{"mr"},
Short: lang.CmdPackageMirrorShort,
Long: lang.CmdPackageMirrorLong,
Args: cobra.MaximumNArgs(1),
Run: func(cmd *cobra.Command, args []string) {
pkgConfig.PkgOpts.PackagePath = choosePackage(args)

pkgConfig.PkgSource = pkgConfig.PkgOpts.PackagePath

// Configure the packager
pkgClient := packager.NewOrDie(&pkgConfig)
defer pkgClient.ClearTempPaths()

// Deploy the package
if err := pkgClient.Mirror(noImgChecksum); err != nil {
message.Fatalf(err, lang.CmdPackageDeployErr, err.Error())
}
Racer159 marked this conversation as resolved.
Show resolved Hide resolved
},
}

var packageInspectCmd = &cobra.Command{
Use: "inspect [ PACKAGE ]",
Aliases: []string{"i"},
Expand Down Expand Up @@ -268,6 +291,7 @@ func init() {
rootCmd.AddCommand(packageCmd)
packageCmd.AddCommand(packageCreateCmd)
packageCmd.AddCommand(packageDeployCmd)
packageCmd.AddCommand(packageMirrorCmd)
packageCmd.AddCommand(packageInspectCmd)
packageCmd.AddCommand(packageRemoveCmd)
packageCmd.AddCommand(packageListCmd)
Expand All @@ -277,6 +301,7 @@ func init() {
bindPackageFlags(v)
bindCreateFlags(v)
bindDeployFlags(v)
bindMirrorFlags(v)
bindInspectFlags(v)
bindRemoveFlags(v)
bindPublishFlags(v)
Expand Down Expand Up @@ -334,6 +359,27 @@ func bindDeployFlags(v *viper.Viper) {
deployFlags.MarkHidden("sget")
}

func bindMirrorFlags(v *viper.Viper) {
mirrorFlags := packageMirrorCmd.Flags()

// Always require confirm flag (no viper)
mirrorFlags.BoolVar(&config.CommonOptions.Confirm, "confirm", false, lang.CmdPackageDeployFlagConfirm)

mirrorFlags.BoolVar(&noImgChecksum, "no-img-checksum", false, lang.CmdPackageMirrorFlagNoChecksum)

mirrorFlags.StringVar(&pkgConfig.PkgOpts.OptionalComponents, "components", v.GetString(common.VInitComponents), lang.CmdPackageMirrorFlagComponents)
Racer159 marked this conversation as resolved.
Show resolved Hide resolved

// Flags for using an external Git server
mirrorFlags.StringVar(&pkgConfig.InitOpts.GitServer.Address, "git-url", v.GetString(common.VInitGitURL), lang.CmdInitFlagGitURL)
mirrorFlags.StringVar(&pkgConfig.InitOpts.GitServer.PushUsername, "git-push-username", v.GetString(common.VInitGitPushUser), lang.CmdInitFlagGitPushUser)
mirrorFlags.StringVar(&pkgConfig.InitOpts.GitServer.PushPassword, "git-push-password", v.GetString(common.VInitGitPushPass), lang.CmdInitFlagGitPushPass)

// Flags for using an external registry
mirrorFlags.StringVar(&pkgConfig.InitOpts.RegistryInfo.Address, "registry-url", v.GetString(common.VInitRegistryURL), lang.CmdInitFlagRegURL)
mirrorFlags.StringVar(&pkgConfig.InitOpts.RegistryInfo.PushUsername, "registry-push-username", v.GetString(common.VInitRegistryPushUser), lang.CmdInitFlagRegPushUser)
mirrorFlags.StringVar(&pkgConfig.InitOpts.RegistryInfo.PushPassword, "registry-push-password", v.GetString(common.VInitRegistryPushPass), lang.CmdInitFlagRegPushPass)
}

func bindInspectFlags(v *viper.Viper) {
inspectFlags := packageInspectCmd.Flags()
inspectFlags.BoolVarP(&includeInspectSBOM, "sbom", "s", false, lang.CmdPackageInspectFlagSbom)
Expand Down
2 changes: 0 additions & 2 deletions src/cmd/prepare.go
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,6 @@ func init() {
prepareCmd.AddCommand(prepareFindImages)
prepareCmd.AddCommand(prepareGenerateConfigFile)

v.SetDefault(common.VPkgCreateSet, map[string]string{})

prepareFindImages.Flags().StringVarP(&repoHelmChartPath, "repo-chart-path", "p", "", lang.CmdPrepareFlagRepoChartPath)
// use the package create config for this and reset it here to avoid overwriting the config.CreateOptions.SetVariables
prepareFindImages.Flags().StringToStringVar(&pkgConfig.CreateOpts.SetVariables, "set", v.GetStringMapString(common.VPkgCreateSet), lang.CmdPrepareFlagSet)
Expand Down
1 change: 1 addition & 0 deletions src/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ const (

ZarfDeployStage = "Deploy"
ZarfCreateStage = "Create"
ZarfMirrorStage = "Mirror"
)

// Zarf Constants for In-Cluster Services.
Expand Down
7 changes: 7 additions & 0 deletions src/config/lang/english.go
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,10 @@ const (
CmdPackageDeployLong = "Unpacks resources and dependencies from a Zarf package archive and deploys them onto the target system.\n" +
"Kubernetes clusters are accessed via credentials in your current kubecontext defined in '~/.kube/config'"

CmdPackageMirrorShort = "Mirrors a Zarf package's resources from a local file or URL to specified registries and repos"
CmdPackageMirrorLong = "Unpacks resources and dependencies from a Zarf package archive and mirrors them into the specified \n" +
"registries and repos within the target environment"

CmdPackageInspectShort = "Displays the definition of a Zarf package (runs offline)"
CmdPackageInspectLong = "Displays the 'zarf.yaml' definition for the specified package and optionally allows SBOMs to be viewed"

Expand Down Expand Up @@ -261,6 +265,9 @@ const (
CmdPackageDeployInvalidCLIVersionWarn = "CLIVersion is set to '%s' which can cause issues with package creation and deployment. To avoid such issues, please set the value to the valid semantic version for this version of Zarf."
CmdPackageDeployErr = "Failed to deploy package: %s"

CmdPackageMirrorFlagComponents = "Comma-separated list of components to mirror. This list will be respected regardless of a component's 'required' status."
CmdPackageMirrorFlagNoChecksum = "Turns off the addition of a checksum to image tags (as would be used by the Zarf Agent) while mirroring images."

CmdPackageInspectFlagSbom = "View SBOM contents while inspecting the package"
CmdPackageInspectFlagSbomOut = "Specify an output directory for the SBOMs from the inspected Zarf package"
CmdPackageInspectFlagValidate = "Validate any checksums and signatures while inspecting the package"
Expand Down
92 changes: 92 additions & 0 deletions src/pkg/packager/mirror.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2021-Present The Zarf Authors

// Package packager contains functions for interacting with, managing and deploying Zarf packages.
package packager

import (
"fmt"
"strings"

"github.com/defenseunicorns/zarf/src/config"
"github.com/defenseunicorns/zarf/src/pkg/message"
"github.com/defenseunicorns/zarf/src/pkg/utils/helpers"
"github.com/defenseunicorns/zarf/src/types"
)

// Mirror pulls resources from a package (images, git repositories, etc) and pushes them to remotes in the air gap without deploying them
func (p *Packager) Mirror(noImgChecksum bool) (err error) {
spinner := message.NewProgressSpinner("Mirroring zarf package %s", p.cfg.PkgOpts.PackagePath)
Racer159 marked this conversation as resolved.
Show resolved Hide resolved
defer spinner.Stop()

if helpers.IsOCIURL(p.cfg.PkgOpts.PackagePath) {
err := p.SetOCIRemote(p.cfg.PkgOpts.PackagePath)
if err != nil {
return err
}
}

if err := p.loadZarfPkg(); err != nil {
return fmt.Errorf("unable to load the Zarf Package: %w", err)
}

if err := ValidatePackageSignature(p.tmp.Base, p.cfg.PkgOpts.PublicKeyPath); err != nil {
return err
}

// Confirm the overall package mirror
if !p.confirmAction(config.ZarfMirrorStage, p.cfg.SBOMViewFiles) {
return fmt.Errorf("mirror cancelled")
}

state := &types.ZarfState{
RegistryInfo: p.cfg.InitOpts.RegistryInfo,
GitServer: p.cfg.InitOpts.GitServer,
ArtifactServer: p.cfg.InitOpts.ArtifactServer,
}
p.cfg.State = state

// Filter out components that are not compatible with this system if we have loaded from a tarball
p.filterComponents(true)
requestedComponentNames := getRequestedComponentList(p.cfg.PkgOpts.OptionalComponents)

for _, component := range p.cfg.Pkg.Components {
if len(requestedComponentNames) == 0 || helpers.SliceContains(requestedComponentNames, component.Name) {
if err := p.mirrorComponent(component, noImgChecksum); err != nil {
return err
}
}
}

return nil
}

// mirrorComponent mirrors a Zarf Component.
func (p *Packager) mirrorComponent(component types.ZarfComponent, noImgChecksum bool) error {
message.Debugf("packager.mirrorComponent(%#v, %#v", p.tmp, component)
Racer159 marked this conversation as resolved.
Show resolved Hide resolved

componentPath, err := p.createOrGetComponentPaths(component)
if err != nil {
return fmt.Errorf("unable to create the component paths: %w", err)
}

// All components now require a name
message.HeaderInfof("📦 %s COMPONENT", strings.ToUpper(component.Name))

hasImages := len(component.Images) > 0
hasRepos := len(component.Repos) > 0

if hasImages {
if err := p.pushImagesToRegistry(component.Images, noImgChecksum); err != nil {
return fmt.Errorf("unable to push images to the registry: %w", err)
}
}

if hasRepos {
if err = p.pushReposToRepository(componentPath.Repos, component.Repos); err != nil {
return fmt.Errorf("unable to push the repos to the repository: %w", err)
}
}
Racer159 marked this conversation as resolved.
Show resolved Hide resolved

return nil
}
1 change: 1 addition & 0 deletions src/test/e2e/22_git_and_gitops_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,7 @@ func waitFluxPodInfoDeployment(t *testing.T) {

// Tests the URL mutation for GitRepository CRD for Flux.
stdOut, stdErr, err = e2e.Kubectl("get", "gitrepositories", "podinfo", "-n", "flux-system", "-o", "jsonpath={.spec.url}")
require.NoError(t, err, stdOut, stdErr)
expectedMutatedRepoURL := fmt.Sprintf("%s/%s/podinfo-1646971829.git", config.ZarfInClusterGitServiceURL, config.ZarfGitPushUser)
require.Equal(t, expectedMutatedRepoURL, stdOut)

Expand Down
6 changes: 5 additions & 1 deletion src/test/external/common.go
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: 2021-Present The Zarf Authors

// Package external provides a test for the external init flow.
// Package external provides a test for interacting with external resources
package external

import (
"context"
"path"
"strings"
"testing"
"time"

"github.com/defenseunicorns/zarf/src/pkg/utils/exec"
"github.com/defenseunicorns/zarf/src/test"
)

var zarfBinPath = path.Join("../../../build", test.GetCLIName())

func verifyKubectlWaitSuccess(t *testing.T, timeoutMinutes time.Duration, args []string, onTimeout string) bool {
return verifyWaitSuccess(t, timeoutMinutes, "kubectl", args, "condition met", onTimeout)
}
Expand Down
73 changes: 0 additions & 73 deletions src/test/external/ext_in_cluster_init_test.go

This file was deleted.

Loading