Big Bang OCI compose and Big Bang --differential fix

docs/3-create-a-zarf-package/2-zarf-components.md

@@ -164,11 +164,11 @@ The [`podinfo-flux`](/examples/podinfo-flux/) example showcases a simple GitOps
 <Properties item="ZarfComponent" include={["import"]} />
 
 <Tabs queryString="import-examples">
-<TabItem value="Path">
+<TabItem value="Local Path">
 <ExampleYAML example="composable-packages" component="games" />
 </TabItem>
-<TabItem value="OCI">
-<ExampleYAML example="composable-packages" component="chart-via-oci" />
+<TabItem value="OCI URL">
+<ExampleYAML example="composable-packages" component="wordpress" />
 </TabItem>
 </Tabs>
 

docs/5-zarf-tutorials/2-deploying-zarf-packages.md

@@ -28,7 +28,7 @@ If you do not provide the path to the package as an argument to the `zarf packag
 
 <iframe src="/docs/tutorials/package_deploy_suggest.html" width="100%" height="120px"></iframe>
 
-By hitting 'tab', you can use the arrow keys to select which package you want to deploy. Since we are deploying the Helm OCI chart package in this tutorial, we will select that package and hit 'enter'.
+By hitting 'tab', you can use the arrow keys to select which package you want to deploy. Since we are deploying the WordPress package in this tutorial, we will select that package and hit 'enter'.
 
 <iframe src="/docs/tutorials/package_deploy_wordpress_suggestions.html" width="100%" height="150px"></iframe>
 

docs/5-zarf-tutorials/7-publish-and-deploy.md

@@ -1,4 +1,4 @@
-# Store & Deploy Packages with OCI
+# Publish & Deploy Packages w/OCI
 
 ## Introduction
 
@@ -71,6 +71,14 @@ To modify, edit `zarf.yaml` and re-run `zarf package create .`
 
 :::
 
+:::info
+
+The above publish command publishes a fully-built package that is ready to deploy to a cluster, but this isn't the only kind of package you can publish to an OCI registry.
+
+If you specify a path to a directory containing a `zarf.yaml` (as you would on `zarf package create`) you can create a `skeleton` package that is importable into other packages using [component composability](../../examples/composable-packages/README.md).  These can be inspected like normal Zarf packages but cannot be deployed as they haven't been fully created yet.
+
+:::
+
 ### Inspect Package
 
 [CLI Reference](../2-the-zarf-cli/100-cli-commands/zarf_package_inspect.md)

examples/composable-packages/README.md

@@ -2,20 +2,21 @@ import ExampleYAML from "@site/src/components/ExampleYAML";
 
 # Composable Packages
 
-This example demonstrates using Zarf to compose existing zarf packages into another package.  It uses the existing [zarf game](../dos-games/) example by simply adding an `import` and `path` in the new [zarf.yaml](zarf.yaml).
+This example demonstrates using Zarf to import components from existing Zarf package definitions while merging overrides to add or change functionality.  It uses the existing [DOS games](../dos-games/README.md) and [WordPress](../wordpress/README.md) examples by simply adding `import` keys in the new [zarf.yaml](zarf.yaml) file.
 
-## Example Prerequisites
+The `import` key in Zarf supports two modes to pull in a component:
 
-Creating this example requires a locally hosted container registry that has the `helm-charts` skeleton package published and available. You can do this by running the following commands:
+1. The `path` key allows you to specify a path to a directory that contains the `zarf.yaml` that you wish to import on your local filesystem.  This allows you to have a common component that you can reuse across multiple packages *within* a project.
 
-```bash
-docker run -d -p 5000:5000 --restart=always --name registry registry:2
-zarf package publish examples/helm-charts oci://127.0.0.1:5000 --insecure
-```
+2. The `url` key allows you to specify an `oci://` URL to a skeleton package that was published to an OCI registry.  Skeleton packages are special package bundles that contain the `zarf.yaml` package definition and any local files referenced by that definition at publish time.  This allows you to version a set of components and import them into multiple packages *across* projects.
 
-:::note
+:::tip
 
-Import paths must be statically defined at create time.  You cannot use [variables](../variables/) in them.
+You can create a skeleton package from a `zarf.yaml` by pointing `zarf package publish` at the directory that contains it:
+
+```bash
+zarf package publish path/containing/package/definition oci://your-registry.com
+```
 
 :::
 
@@ -27,4 +28,35 @@ To view the example in its entirety, select the `Edit this page` link below the
 
 :::
 
+:::note
+
+Creating this example requires a locally hosted container registry that has the `wordpress` skeleton package published and available. You can do this by running the following commands:
+
+```bash
+docker run -d -p 5000:5000 --restart=always --name registry registry:2
+zarf package publish examples/wordpress oci://127.0.0.1:5000 --insecure
+```
+
+You will also need to pass the `--insecure` flag to `zarf package create` to pull from the `http` registry:
+
+```bash
+zarf package create examples/composable-packages/ --insecure
+```
+
+:::
+
 <ExampleYAML example="composable-packages" showLink={false} />
+
+:::info
+
+As you can see in the example, the `import` key can be combined with other keys to merge components together.  This can be done as many components deep as you wish and in the end will generate one main `zarf.yaml` file with all of the defined resources included.
+
+This is useful if you want to slightly tweak a given component while maintaining a common core.
+
+:::
+
+:::note
+
+The import `path` or `url` must be statically defined at create time.  You cannot use [package templates](../variables/README.md#create-time-package-configuration-templates) within them.
+
+:::

examples/composable-packages/quake-service.yaml

@@ -0,0 +1,18 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: quake
+  annotations:
+    zarf.dev/connect-description: "Play quake!!!"
+    zarf.dev/connect-url: "?quake"
+  labels:
+    # Enables "zarf connect quake"
+    zarf.dev/connect-name: quake
+spec:
+  selector:
+    app: game
+  ports:
+    - name: http
+      port: 8000
+      protocol: TCP
+      targetPort: 8000

examples/composable-packages/zarf.yaml

@@ -4,16 +4,38 @@ metadata:
   description: Demo Zarf package composability
 
 components:
-  - name: games
+  - name: local-games-path
+    # The component logic keys ('required', 'group', and 'default') always override those of the imported package
     required: true
-    description: Example of a composed package with a unique description for this component
+    # group: ""         # the initial value overrides the child component
+    # default: false    # the initial value overrides the child component
+    description: "Example of a local composed package with a unique description for this component"
     import:
+      # The local relative path to the folder containing this component's package definition
       path: ../dos-games
-      # Example optional custom name to point to in the imported package
+      # Example optional custom name to point to in the imported package (default is to use this component's name)
       name: baseline
+    # Array keys are appended to the end of anything the imported component defined (in this case an extra service)
+    manifests:
+      - name: additional-games
+        namespace: dos-games
+        files:
+          - quake-service.yaml
 
-  - name: chart-via-oci
-    required: false
+  - name: oci-wordpress-url
+    # The component logic keys ('required', 'group', and 'default') always override those of the imported package
+    # required: false   # the initial value overrides the child component
+    # group: ""         # the initial value overrides the child component
+    # default: false    # the initial value overrides the child component
     import:
-      url: oci://localhost:5000/helm-charts:0.0.1-skeleton
-      name: demo-helm-local-chart
+      # The URL to the skeleton package containing this component's package definition
+      url: oci://localhost:5000/wordpress:16.0.4-skeleton
+      # Example optional custom name to point to in the imported package (default is to use this component's name)
+      name: wordpress
+    # Array keys are appended to the end of anything the imported component defined (in this case an override of the blog name)
+    actions:
+      onDeploy:
+        before:
+          - cmd: ./zarf tools kubectl get -n dos-games deployment -o jsonpath={.items[0].metadata.creationTimestamp}
+            setVariables:
+              - name: WORDPRESS_BLOG_NAME

src/cmd/initialize.go

@@ -47,6 +47,8 @@ var initCmd = &cobra.Command{
 			message.Fatal(err, err.Error())
 		}
 
+		pkgConfig.PkgSourcePath = pkgConfig.DeployOpts.PackagePath
+
 		// Ensure uppercase keys from viper
 		viperConfig := utils.TransformMapKeys(v.GetStringMapString(V_PKG_DEPLOY_SET), strings.ToUpper)
 		pkgConfig.DeployOpts.SetVariables = utils.MergeMap(viperConfig, pkgConfig.DeployOpts.SetVariables)

src/extensions/bigbang/bigbang.go

@@ -37,7 +37,7 @@ var tenMins = metav1.Duration{
 	Duration: 10 * time.Minute,
 }
 
-// Run Mutates a component that should deploy Big Bang to a set of manifests
+// Run mutates a component that should deploy Big Bang to a set of manifests
 // that contain the flux deployment of Big Bang
 func Run(YOLO bool, tmpPaths types.ComponentPaths, c types.ZarfComponent) (types.ZarfComponent, error) {
 	var err error
@@ -244,6 +244,41 @@ func Run(YOLO bool, tmpPaths types.ComponentPaths, c types.ZarfComponent) (types
 	return c, nil
 }
 
+// Skeletonize mutates a component so that the valuesFiles can be contained inside a skeleton package
+func Skeletonize(tmpPaths types.ComponentPaths, c types.ZarfComponent) (types.ZarfComponent, error) {
+	for valuesIdx, valuesFile := range c.Extensions.BigBang.ValuesFiles {
+		// Define the name as the file name without the extension.
+		baseName := strings.TrimSuffix(valuesFile, filepath.Ext(valuesFile))
+
+		// Replace non-alphanumeric characters with a dash.
+		baseName = nonAlphnumeric.ReplaceAllString(baseName, "-")
+
+		// Add the skeleton name prefix.
+		skelName := fmt.Sprintf("bb-ext-skeleton-values-%s.yaml", baseName)
+
+		rel := filepath.Join(types.TempFolder, skelName)
+		dst := filepath.Join(tmpPaths.Base, rel)
+
+		if err := utils.CreatePathAndCopy(valuesFile, dst); err != nil {
+			return c, err
+		}
+
+		c.Extensions.BigBang.ValuesFiles[valuesIdx] = rel
+	}
+
+	return c, nil
+}
+
+// Compose mutates a component so that the valuesFiles are relative to the parent importing component
+func Compose(pathAncestry string, c types.ZarfComponent) types.ZarfComponent {
+	for valuesIdx, valuesFile := range c.Extensions.BigBang.ValuesFiles {
+		parentRel := filepath.Join(pathAncestry, valuesFile)
+		c.Extensions.BigBang.ValuesFiles[valuesIdx] = parentRel
+	}
+
+	return c
+}
+
 // isValidVersion check if the version is 1.54.0 or greater.
 func isValidVersion(version string) (bool, error) {
 	specifiedVersion, err := semver.NewVersion(version)

src/extensions/bigbang/manifests.go

@@ -116,7 +116,7 @@ func manifestValuesFile(path string) (secret corev1.Secret, err error) {
 	}
 
 	// Define the name as the file name without the extension.
-	baseName := strings.TrimSuffix(filepath.Base(path), filepath.Ext(path))
+	baseName := strings.TrimSuffix(path, filepath.Ext(path))
 
 	// Replace non-alphanumeric characters with a dash.
 	baseName = nonAlphnumeric.ReplaceAllString(baseName, "-")

src/extensions/bigbang/test/bigbang_test.go

@@ -56,9 +56,6 @@ func TestReleases(t *testing.T) {
 	pkgPath := fmt.Sprintf("zarf-package-big-bang-test-amd64-%s.tar.zst", previous)
 	zarfExec(t, "package", "deploy", pkgPath, "--confirm")
 
-	// Remove the previous version package
-	_ = os.RemoveAll(pkgPath)
-
 	// HACK: scale down the flux deployments due to very-low CPU in the test runner
 	fluxControllers := []string{"helm-controller", "source-controller", "kustomize-controller", "notification-controller"}
 	for _, deployment := range fluxControllers {
@@ -71,13 +68,16 @@ func TestReleases(t *testing.T) {
 	// Build the latest version
 	bbVersion = fmt.Sprintf("--set=BB_VERSION=%s", latest)
 	bbMajor = fmt.Sprintf("--set=BB_MAJOR=%s", latest[0:1])
-	zarfExec(t, "package", "create", "../src/extensions/bigbang/test/package", bbVersion, bbMajor, "--confirm")
+	zarfExec(t, "package", "create", "../src/extensions/bigbang/test/package", bbVersion, bbMajor, "--differential", pkgPath, "--confirm")
+
+	// Remove the previous version package
+	_ = os.RemoveAll(pkgPath)
 
 	// Clean up zarf cache now that all packages are built to reduce disk pressure
 	zarfExec(t, "tools", "clear-cache")
 
 	// Deploy the latest version
-	pkgPath = fmt.Sprintf("zarf-package-big-bang-test-amd64-%s.tar.zst", latest)
+	pkgPath = fmt.Sprintf("zarf-package-big-bang-test-amd64-%s-differential-%s.tar.zst", previous, latest)
 	zarfExec(t, "package", "deploy", pkgPath, "--confirm")
 
 	// Cluster info

src/pkg/packager/compose.go

@@ -88,6 +88,10 @@ func (p *Packager) getChildComponent(parent types.ZarfComponent, pathAncestry st
 
 	var cachePath string
 	if parent.Import.URL != "" {
+		if !strings.HasSuffix(parent.Import.URL, skeletonSuffix) {
+			return child, fmt.Errorf("import URL must be a 'skeleton' package: %s", parent.Import.URL)
+		}
+
 		// Save all the OCI imported components into our build data
 		p.cfg.Pkg.Build.OCIImportedComponents[parent.Import.URL] = childComponentName
 
@@ -265,6 +269,8 @@ func (p *Packager) fixComposedFilepaths(pathAncestry string, child types.ZarfCom
 		child.CosignKeyPath = composed
 	}
 
+	child = p.composeExtensions(pathAncestry, child)
+
 	return child, nil
 }
 

src/pkg/packager/create.go

@@ -65,6 +65,7 @@ func (p *Packager) Create(baseDir string) error {
 		return err
 	}
 
+	// Compose components into a single zarf.yaml file
 	if err := p.composeComponents(); err != nil {
 		return err
 	}
@@ -74,7 +75,12 @@ func (p *Packager) Create(baseDir string) error {
 		return fmt.Errorf("unable to fill values in template: %s", err.Error())
 	}
 
-	// Remove unnecessary repos and images if we are building a differential package
+	// After templates are filled process any create extensions
+	if err := p.processExtensions(); err != nil {
+		return err
+	}
+
+	// After we have a full zarf.yaml remove unnecessary repos and images if we are building a differential package
 	if p.cfg.CreateOpts.DifferentialData.DifferentialPackagePath != "" {
 		// Verify the package version of the package we're using as a 'reference' for the differential build is different than the package we're building
 		// If the package versions are the same return an error
@@ -91,20 +97,6 @@ func (p *Packager) Create(baseDir string) error {
 		}
 	}
 
-	// Create component paths and process extensions for each component.
-	for i, c := range p.cfg.Pkg.Components {
-		componentPath, err := p.createOrGetComponentPaths(c)
-		if err != nil {
-			return err
-		}
-
-		// Process any extensions.
-		p.cfg.Pkg.Components[i], err = p.processExtensions(p.cfg.Pkg.Metadata.YOLO, componentPath, c)
-		if err != nil {
-			return fmt.Errorf("unable to process extensions: %w", err)
-		}
-	}
-
 	// Perform early package validation.
 	if err := validate.Run(p.cfg.Pkg); err != nil {
 		return fmt.Errorf("unable to validate package: %w", err)