#208. Basic Composability (#351)

* #208. config: Add SetComponents method to update the components in the private config. package/create.Create: Add logic to expand locally imported components within the parent build config. types: Add Import field to Zarf.Component as expirement (expected to change).

* #208. ADD packager/compose.go: Created compose logic for imported components and nested import components. UPDATE types.go: Added ZarfImport type with a path variable as the associated type with ZarfComponent.Import field (this sets up sha verification additions in the future, still need team imput). UPDATE Packager/Common.go: pulled confirming optional component logic out of getValidComponents to play with composed package validation (need to validate expected behavior). UPDATE Packager/Create.go: Replaced the initial compose logic with call to GetComposedAssets(). ADD examples/compose-example: a simple composition example for explicit example.

* #208. UPDATE cli/config: Delete SetSeedImages method as it is no longer needed. UPDATE packager/compose: removed the logic working with SeedImages as it is not necessary for composition. UPDATE packager/create.go: seperate the GetComposedAssets call from the seed images. Set seedImages variable using the config.GetSeedImages() method.

* #208. ADD e2e_composability_test: set up basic tests to ensure that the composed packages flux and games are properly deployed together. UPDATE examples/Makefile with the package-example-composability command and added that command to the package-examples command. RENAME examples/composable-example -> examples/composable-packages

* #208, #351. UPDATE packager/validate: add the ValidateImportPackage method for use with compose.go. UPDATE packager/compose: add the hasValidSubPackage method and replaced the hasSubPackage calls. Updated the hasSubPackage method to validate only that the Component.Import field exists. Clean up index names in prepComponentToCompose.

* #208, #351. UPDATE docer/components: Added new section on composing packages. ADD examples/composable-packages/README: with instructions on running doom using composition from the games example zarf.yaml. UPDATE examples/composable-packages/zarf.yaml: removed the flux package as the game is a more clear example and still shows the same composability without the need of flux since the games package does not rely on flux

* #208, #351. UPDATE packager/compose: added the shouldAddImportedPackage method when pulling in imported components. Updated shouldAddImportedPackage method to no longer prompt if --confirm flag is passed

* #208, #351. FIX packager/compose: to no longer add the optional composed package when the user chooses to not import that composed packages.

* #208. UPDATE e2e_composability_test: removed the tests verifying flux due to remove of flux composed component in the composability example.

* #208, #351. UPDATE Makefile: add package-example-compose for e2e tests. Update .github/workflows/test*: add package-compose-example to make packages stage. UPDATE packager/compose: rename GetComposedAssets to GetComposedComponents UPDATE reference in packager/create. refactored and rename hasValidSubPackage to validateOrBail. Rename shouldAddImportedPackage to shouldComposePackage. Extracted component required logic to componentConfirmedForInclusion. UPDATE e2e/e2e_composability_test: to work with new testing strategy.

.github/workflows/test-k3d.yml

@@ -15,6 +15,6 @@ jobs:
       - name: Build CLI
         run: make build-cli-linux
       - name: Make Packages
-        run: make init-package package-example-game package-example-data-injection package-example-gitops-data
+        run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
       - name: Run Tests
         run: TESTDISTRO=k3d make test-e2e

.github/workflows/test-k3s.yml

@@ -15,7 +15,7 @@ jobs:
       - name: Build CLI
         run: make build-cli-linux
       - name: Make Packages
-        run: make init-package package-example-game package-example-data-injection package-example-gitops-data
+        run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
       - name: Run Tests
         # NOTE: "PATH=$PATH" preserves the default user $PATH. This is needed to maintain the version of go installed in a previous step
         run: sudo env "PATH=$PATH" TESTDISTRO=k3s make test-e2e

.github/workflows/test-kind.yml

@@ -15,6 +15,6 @@ jobs:
       - name: Build CLI
         run: make build-cli-linux
       - name: Make Packages
-        run: make init-package package-example-game package-example-data-injection package-example-gitops-data
+        run: make init-package package-example-game package-example-data-injection package-example-gitops-data package-example-compose
       - name: Run Tests
         run: TESTDISTRO=kind make test-e2e

Makefile

@@ -78,6 +78,10 @@ package-example-gitops-data:
 package-example-tiny-kafka:
 	cd examples/tiny-kafka && ../../$(ZARF_BIN) package create --confirm && mv zarf-package-* ../../build/
 
+.PHONY: package-example-compose
+package-example-compose:
+	cd examples/composable-packages && ../../$(ZARF_BIN) package create --confirm && mv zarf-package-* ../../build/
+
 # TODO: This can be cleaned up a little more when `zarf init` is able to provide the path to the `zarf-init.tar.zst`
 .PHONY: test-new-e2e
 test-e2e: ## Run e2e tests on a KiND cluster. All dependencies are assumed to be built and in the ./build directory
@@ -97,5 +101,7 @@ test-e2e: ## Run e2e tests on a KiND cluster. All dependencies are assumed to be
 	@if [ ! -f ./build/zarf-package-gitops-service-data.tar.zst ]; then\
 		$(MAKE) package-example-gitops-data;\
 	fi
-
+	@if [ ! -f ./build/zarf-package-compose-example.tar.zst ]; then\
+		$(MAKE) package-example-compose;\
+	fi
 	cd test/e2e && cp ../../build/zarf-init.tar.zst . && go test ./... -v -timeout 2400s && rm zarf-init.tar.zst

cli/config/config.go

@@ -116,6 +116,10 @@ func GetComponents() []types.ZarfComponent {
 	return config.Components
 }
 
+func SetComponents(components []types.ZarfComponent) {
+	config.Components = components
+}
+
 func GetBuildData() types.ZarfBuildData {
 	return config.Build
 }

cli/internal/packager/common.go

@@ -110,19 +110,7 @@ func getValidComponents(allComponents []types.ZarfComponent, requestedComponentN
 					}
 				}
 			} else {
-				// Present the users with the component details one more time
-				displayComponent := component
-				displayComponent.Description = ""
-				content, _ := yaml.Marshal(displayComponent)
-				utils.ColorPrintYAML(string(content))
-				message.Question(fmt.Sprintf("%s: %s", component.Name, component.Description))
-
-				// Since no requested components were provided, prompt the user
-				prompt := &survey.Confirm{
-					Message: "Deploy this component?",
-					Default: component.Default,
-				}
-				_ = survey.AskOne(prompt, &confirmComponent)
+				confirmComponent = ConfirmOptionalComponent(component)
 			}
 		}
 
@@ -150,6 +138,23 @@ func getValidComponents(allComponents []types.ZarfComponent, requestedComponentN
 	return validComponentsList
 }
 
+// Confirm optional component
+func ConfirmOptionalComponent(component types.ZarfComponent) (confirmComponent bool) {
+	displayComponent := component
+	displayComponent.Description = ""
+	content, _ := yaml.Marshal(displayComponent)
+	utils.ColorPrintYAML(string(content))
+	message.Question(fmt.Sprintf("%s: %s", component.Name, component.Description))
+
+	// Since no requested components were provided, prompt the user
+	prompt := &survey.Confirm{
+		Message: "Deploy this component?",
+		Default: component.Default,
+	}
+	_ = survey.AskOne(prompt, &confirmComponent)
+	return confirmComponent
+}
+
 // HandleIfURL If provided package is a URL download it to a temp directory
 func HandleIfURL(packagePath string, shasum string, insecureDeploy bool) (string, func()) {
 	// Check if the user gave us a remote package

cli/internal/packager/compose.go

@@ -0,0 +1,113 @@
+package packager
+
+import (
+	"github.com/defenseunicorns/zarf/cli/config"
+	"github.com/defenseunicorns/zarf/cli/internal/message"
+	"github.com/defenseunicorns/zarf/cli/internal/packager/validate"
+	"github.com/defenseunicorns/zarf/cli/internal/utils"
+	"github.com/defenseunicorns/zarf/cli/types"
+)
+
+func GetComposedComponents() (components []types.ZarfComponent) {
+	for _, component := range config.GetComponents() {
+		// Check for standard component.
+		if !hasComposedPackage(&component) {
+			// Append standard component to list.
+			components = append(components, component)
+		} else if shouldComposePackage(&component) { // Validate and confirm inclusion of imported package.
+			// Expand and add components from imported package.
+			importedComponents := getSubPackageAssets(component)
+			components = append(components, importedComponents...)
+		}
+	}
+	// Update the parent package config with the expanded sub components.
+	// This is important when the deploy package is created.
+	config.SetComponents(components)
+	return components
+}
+
+// Returns true if import field is populated.
+func hasComposedPackage(component *types.ZarfComponent) bool {
+	return component.Import != types.ZarfImport{}
+}
+
+// Validates and confirms inclusion of imported package.
+func shouldComposePackage(component *types.ZarfComponent) bool {
+	validateOrBail(component)
+	return componentConfirmedForInclusion(component)
+}
+
+// Returns true if confirm flag is true, the component is required, or the user confirms inclusion.
+func componentConfirmedForInclusion(component *types.ZarfComponent) bool {
+	return config.DeployOptions.Confirm || component.Required || ConfirmOptionalComponent(*component)
+}
+
+// Validates the sub component, exits program if validation fails.
+func validateOrBail(component *types.ZarfComponent) {
+	if err := validate.ValidateImportPackage(component); err != nil {
+		message.Fatalf(err, "Invalid import definition in the %s component: %s", component.Name, err)
+	}
+}
+
+// Get expanded components from imported component.
+func getSubPackageAssets(importComponent types.ZarfComponent) (components []types.ZarfComponent) {
+	// Read the imported package.
+	importedPackage := getSubPackage(&importComponent)
+	// Iterate imported components.
+	for _, componentToCompose := range importedPackage.Components {
+		// Check for standard component.
+		if !hasComposedPackage(&componentToCompose) {
+			// Doctor standard component name and included files.
+			prepComponentToCompose(&componentToCompose, importedPackage.Metadata.Name, importComponent.Import.Path)
+			components = append(components, componentToCompose)
+		} else if shouldComposePackage(&componentToCompose) {
+			// Recurse on imported components.
+			components = append(components, getSubPackageAssets(componentToCompose)...)
+		}
+	}
+	return components
+}
+
+// Reads the locally imported zarf.yaml
+func getSubPackage(component *types.ZarfComponent) (importedPackage types.ZarfPackage) {
+	utils.ReadYaml(component.Import.Path+"zarf.yaml", &importedPackage)
+	return importedPackage
+}
+
+// Updates the name and sets all local asset paths relative to the importing package.
+func prepComponentToCompose(component *types.ZarfComponent, parentPackageName string, importPath string) {
+	// Prefix component name with parent package name to distinguish similarly named components.
+	component.Name = parentPackageName + "-" + component.Name
+
+	// Prefix composed component file paths.
+	for fileIdx, file := range component.Files {
+		component.Files[fileIdx].Source = getComposedFilePath(file.Source, importPath)
+	}
+
+	// Prefix non-url composed component chart values files.
+	for chartIdx, chart := range component.Charts {
+		for valuesIdx, valuesFile := range chart.ValuesFiles {
+			component.Charts[chartIdx].ValuesFiles[valuesIdx] = getComposedFilePath(valuesFile, importPath)
+		}
+	}
+
+	// Prefix non-url composed manifest files and kustomizations.
+	for manifestIdx, manifest := range component.Manifests {
+		for fileIdx, file := range manifest.Files {
+			component.Manifests[manifestIdx].Files[fileIdx] = getComposedFilePath(file, importPath)
+		}
+		for kustomIdx, kustomization := range manifest.Kustomizations {
+			component.Manifests[manifestIdx].Kustomizations[kustomIdx] = getComposedFilePath(kustomization, importPath)
+		}
+	}
+}
+
+// Prefix file path with importPath if original file path is not a url.
+func getComposedFilePath(originalPath string, pathPrefix string) string {
+	// Return original if it is a remote file.
+	if utils.IsUrl(originalPath) {
+		return originalPath
+	}
+	// Add prefix for local files.
+	return pathPrefix + originalPath
+}

cli/internal/packager/create.go

@@ -30,10 +30,10 @@ func Create() {
 	tempPath := createPaths()
 	defer tempPath.clean()
 
+	components := GetComposedComponents()
+	seedImages := config.GetSeedImages()
 	packageName := config.GetPackageName()
 	dataInjections := config.GetDataInjections()
-	seedImages := config.GetSeedImages()
-	components := config.GetComponents()
 	configFile := tempPath.base + "/zarf.yaml"
 
 	config.SetAcrch()

cli/internal/packager/validate/validate.go

@@ -2,8 +2,11 @@ package validate
 
 import (
 	"fmt"
+	"strings"
+
 	"github.com/defenseunicorns/zarf/cli/config"
 	"github.com/defenseunicorns/zarf/cli/internal/message"
+	"github.com/defenseunicorns/zarf/cli/internal/utils"
 	"github.com/defenseunicorns/zarf/cli/types"
 )
 
@@ -81,3 +84,33 @@ func validateManifest(manifest types.ZarfManifest) error {
 
 	return nil
 }
+
+func ValidateImportPackage(composedComponent *types.ZarfComponent) error {
+	intro := fmt.Sprintf("imported package %s", composedComponent.Name)
+	path := composedComponent.Import.Path
+	packageSuffix := "zarf.yaml"
+
+	// ensure path exists
+	if !(len(path) > 0) {
+		return fmt.Errorf("%s must include a path", intro)
+	}
+
+	// remove zarf.yaml from path if path has zarf.yaml suffix
+	if strings.HasSuffix(path, packageSuffix) {
+		path = strings.Split(path, packageSuffix)[0]
+	}
+
+	// add a forward slash to end of path if it does not have one
+	if !strings.HasSuffix(path, "/") {
+		path = path + "/"
+	}
+
+	// ensure there is a zarf.yaml in provided path
+	if utils.InvalidPath(path + packageSuffix) {
+		return fmt.Errorf("invalid file path \"%s\" provided directory must contain a valid zarf.yaml file", composedComponent.Import.Path)
+	}
+
+	// replace component path with doctored path
+	composedComponent.Import.Path = path
+	return nil
+}

cli/types/types.go

@@ -53,6 +53,9 @@ type ZarfComponent struct {
 
 	// Scripts are custom commands that run before or after package deployment
 	Scripts ZarfComponentScripts `yaml:"scripts,omitempty"`
+
+	// Import refers to another zarf.yaml package.
+	Import ZarfImport `yaml:"import,omitempty"`
 }
 
 // ZarfManifest defines raw manifests Zarf will deploy as a helm chart
@@ -141,3 +144,8 @@ type ZarfDeployOptions struct {
 	Components    string
 	ApplianceMode bool
 }
+
+// ZarfImport structure for including imported zarf packages
+type ZarfImport struct {
+	Path string `yaml:"path"`
+}

docs/components.md

@@ -37,6 +37,17 @@ These optional components are listed below along with the "magic strings" you pa
 
  
 
+## Composing Package Components
+Existing components and packages within a zarf.yaml can be composed in new packages. This can be achieved by using the import field and providing a path the zarf.yaml you wish to compose. Checkout the  [composable-packages](../examples/composable-packages/zarf.yaml) example.
+```yaml
+components:
+  - name: flux
+    import: 
+     path: 'path/to/flux/package/directory/'
+```
+
+ 
+
 ## Further reading
 
 For more detail—like which components are on/off by default—there's no better place to check than the source: [zarf.yaml](../zarf.yaml).

examples/Makefile

@@ -56,7 +56,7 @@ vm-destroy: ## Cleanup plz
 	@vagrant destroy -f
 
 .PHONY: package-examples
-package-examples: package-example-big-bang package-example-software-factory package-example-data-injection package-example-game package-example-gitops-data package-example-single-big-bang-package package-example-tiny-kafka package-example-postgres-operator ## Create zarf packages from all examples
+package-examples: package-example-big-bang package-example-software-factory package-example-data-injection package-example-game package-example-gitops-data package-example-single-big-bang-package package-example-tiny-kafka package-example-postgres-operator package-example-composability ## Create zarf packages from all examples
 
 .PHONY: package-example-big-bang
 package-example-big-bang: ## Create the Big Bang Core example
@@ -101,3 +101,7 @@ package-example-tiny-kafka: ## Create the Tiny Kafka example
 .PHONY: package-example-postgres-operator
 package-example-postgres-operator: ## Create the Postgres Operator example
 	cd postgres-operator && $(ZARF_BIN) package create --confirm && mv zarf-package-* ../sync/
+
+.PHONY: package-example-composability
+package-example-composability: ## Create the Composable example
+	cd composable-packages && $(ZARF_BIN) package create --confirm && mv zarf-package-* ../sync/