fix(oscal): deterministic OSCAL model write (#553)

* fix(component): initial component deterministic sort

* fix(component): testing compose

* fix(component): testing compose with common write function

* fix(results): deterministic sorting of assessment-results

* fix(docs): add component definition oscal doc

* fix(compose): remove compose write function in favor of common function

* fix(cleanup): remove istio composed test file

* fix(assessment): add unit test for assessment results determinism

* fix(oscal): add unit test for MakeComponentDeterministic

* fix(docs): Update docs/oscal/component-definition.md

Co-authored-by: Megan Wolf <97549300+meganwolf0@users.noreply.github.com>

* fix(docs): update component definition docs for sort specifics

---------

Co-authored-by: Megan Wolf <97549300+meganwolf0@users.noreply.github.com>

cspell.json

@@ -0,0 +1,11 @@
+{
+    "version": "0.2",
+    "ignorePaths": [],
+    "dictionaryDefinitions": [],
+    "dictionaries": [],
+    "words": [
+        "OSCAL"
+    ],
+    "ignoreWords": [],
+    "import": []
+}

docs/oscal/assessment-results.md

@@ -1,12 +1,12 @@
 # Assessment Results
 
-An [Assessment Result](https://pages.nist.gov/OSCAL/resources/concepts/layer/assessment/assessment-results/) is an OSCAL-specific model to report on the specific assessment outcomes of a system. In Lula, the `validate` function creates an `assessment-result` object to enumerate the asseement of the input controls provided by the `component-definition`. These are reported as finding that are `satisfied` or `not-satisfied` as a result of the observations performed by the Lula validations.
+An [Assessment Result](https://pages.nist.gov/OSCAL/resources/concepts/layer/assessment/assessment-results/) is an OSCAL model to report on the specific assessment outcomes of a system. In Lula, the `validate` command creates an `assessment-result` object to enumerate the assessment of the input controls provided by the `component-definition`. These are reported as findings that are `satisfied` or `not-satisfied` as a result of the observations performed by the Lula validations.
 ```mermaid
 flowchart TD
     A[Assessment Results]-->|compose|C[Finding 1]
     A[Assessment Results]-->|compose|G[Finding 2]
-    B(Control)-->|satsified by|C
-    B(Control)-->|satsified by|G
+    B(Control)-->|satisfied by|C
+    B(Control)-->|satisfied by|G
     C -->|compose|D[Observation 1]
     C -->|compose|E[Observation 2]
     C -->|compose|F[Observation 3]
@@ -19,13 +19,20 @@ flowchart TD
 ```
 
 ## Observation Results
-Based on the structure outlined, the results of the observations impact the findings, which in turn result in the decision for the control as `satisfied` or `not-satisfied`. The observations are aggregated to the findings as `and` operations, such that if a single observation is `not-satisifed` then the associated finding is marked as `not-satisfied`.
+Based on the structure outlined, the results of the observations impact the findings, which in turn result in the decision for the control as `satisfied` or `not-satisfied`. The observations are aggregated to the findings as `and` operations, such that if a single observation is `not-satisfied` then the associated finding is marked as `not-satisfied`.
 
-The way Lula performs evaluations default to a conservative reporting of a `not-satisified` observation. The only `satisfied` observations occur when a domain provides resources and those resources are evaluated by the policy such that the policy will pass. If a Lula Validation [cannot be evaluated](#not-satisfied-conditions) then it will by default return a `not-satisfied` result.
+The way Lula performs evaluations default to a conservative reporting of a `not-satisfied` observation. The only `satisfied` observations occur when a domain provides resources and those resources are evaluated by the policy such that the policy will pass. If a Lula Validation [cannot be evaluated](#not-satisfied-conditions) then it will by default return a `not-satisfied` result.
 
 ### Not-satisfied conditions
 The following conditions enumerate when the Lula Validation will result in a `not-satisfied` evaluation. These cases exclude the case where the Lula validation policy has been evaluated and returned a failure.
 - Malformed Lula validation -> bad validation structure
 - Missing resources -> No resources are found as input to the policy
 - Missing reference -> If a remote or local reference is invalid
-- Executable validations disallowed -> If a validation is executable but has not been allowed to run
+- Executable validations disallowed -> If a validation is executable but has not been allowed to run
+
+## Structure
+The primary structure for Lula production and operation of `assessment-results` for determinism is as follows:
+- Results are sorted by `start` time in descending order
+- Findings are sorted by `target.target-id` in ascending order
+- Observations are sorted by `collected` time in ascending order
+- Back Matter Resources are sorted by `title` in ascending order.

docs/oscal/component-definition.md

@@ -0,0 +1,16 @@
+# Component Definition
+
+A [Component Definition](https://pages.nist.gov/OSCAL/resources/concepts/layer/implementation/component-definition/) is an OSCAL model for capturing control information that pertains to a specific component/capability of a potential system. It can largely be considered the modular and re-usable model for use across many systems. In Lula, the `validate` command will process a `component-definition`, iterate through all `implemented-requirements` to discover Lula validations, and execute those validations to produce `observations`. 
+
+## Components/Capabilities and Control-Implementations
+
+The modularity of `component-definitions` allows for the specification of one to many components or capabilities that include one to many `control-implementations`.
+
+By allowing for many `control-implementations`, a given component or capability can have information as to its compliance with many different regulatory standards. 
+
+## Structure
+The primary structure for Lula production and operations of `component-definitions` for determinism is as follows:
+- Components/Capabilities are sorted by `title` in ascending order (Case Sensitive Sorting).
+- Control Implementations are sorted by `source` in ascending order.
+- Implemented Requirements are sorted by `control-id` in ascending order.
+- Back Matter Resources are sorted by `title` in ascending order (Case Sensitive Sorting).

src/cmd/tools/compose.go

@@ -1,19 +1,16 @@
 package tools
 
 import (
-	"bytes"
 	"errors"
 	"fmt"
 	"os"
 	"path/filepath"
 	"strings"
 
-	"github.com/defenseunicorns/go-oscal/src/pkg/files"
-	oscalTypes_1_1_2 "github.com/defenseunicorns/go-oscal/src/types/oscal-1-1-2"
 	"github.com/defenseunicorns/lula/src/pkg/common/composition"
+	"github.com/defenseunicorns/lula/src/pkg/common/oscal"
 	"github.com/defenseunicorns/lula/src/pkg/message"
 	"github.com/spf13/cobra"
-	"gopkg.in/yaml.v3"
 )
 
 type composeFlags struct {
@@ -81,7 +78,7 @@ func Compose(inputFile, outputFile string) error {
 	}
 
 	// Write the composed OSCAL model to a file
-	err = WriteComposedOscalModel(model, outputFile, inputFile)
+	err = oscal.WriteOscalModel(outputFile, model)
 	if err != nil {
 		return err
 	}
@@ -93,25 +90,3 @@ func Compose(inputFile, outputFile string) error {
 func GetDefaultOutputFile(inputFile string) string {
 	return strings.TrimSuffix(inputFile, filepath.Ext(inputFile)) + "-composed" + filepath.Ext(inputFile)
 }
-
-// WriteComposedOscalModel writes the composed OSCAL model to a file
-func WriteComposedOscalModel(model *oscalTypes_1_1_2.OscalCompleteSchema, outputFile string, inputFile string) (err error) {
-	var b bytes.Buffer
-
-	yamlEncoder := yaml.NewEncoder(&b)
-	yamlEncoder.SetIndent(2)
-	yamlEncoder.Encode(model)
-
-	outputFileName := outputFile
-	if outputFileName == "" {
-		outputFileName = GetDefaultOutputFile(inputFile)
-	}
-
-	message.Infof("Writing Composed OSCAL Component Definition to: %s", outputFileName)
-
-	err = files.WriteOutput(b.Bytes(), outputFileName)
-	if err != nil {
-		return err
-	}
-	return nil
-}

src/pkg/common/oscal/assessment-results.go

@@ -3,6 +3,7 @@ package oscal
 import (
 	"fmt"
 	"slices"
+	"sort"
 	"time"
 
 	"github.com/defenseunicorns/go-oscal/src/pkg/uuid"
@@ -249,6 +250,55 @@ func EvaluateResults(thresholdResult *oscalTypes_1_1_2.Result, newResult *oscalT
 	return status, categorizedResultComparisons, nil
 }
 
+func MakeAssessmentResultsDeterministic(assessment *oscalTypes_1_1_2.AssessmentResults) {
+
+	// Sort Results
+	slices.SortFunc(assessment.Results, func(a, b oscalTypes_1_1_2.Result) int { return b.Start.Compare(a.Start) })
+
+	for _, result := range assessment.Results {
+		// sort findings by target id
+		if result.Findings != nil {
+			findings := *result.Findings
+			sort.Slice(findings, func(i, j int) bool {
+				return findings[i].Target.TargetId < findings[j].Target.TargetId
+			})
+			result.Findings = &findings
+		}
+		// sort observations by collected time
+		if result.Observations != nil {
+			observations := *result.Observations
+			slices.SortFunc(observations, func(a, b oscalTypes_1_1_2.Observation) int { return a.Collected.Compare(b.Collected) })
+			result.Observations = &observations
+		}
+
+		// Sort the include-controls in the control selections
+		controlSelections := result.ReviewedControls.ControlSelections
+		for _, selection := range controlSelections {
+			if selection.IncludeControls != nil {
+				controls := *selection.IncludeControls
+				sort.Slice(controls, func(i, j int) bool {
+					return controls[i].ControlId < controls[j].ControlId
+				})
+				selection.IncludeControls = &controls
+			}
+		}
+	}
+
+	// sort backmatter
+	if assessment.BackMatter != nil {
+		backmatter := *assessment.BackMatter
+		if backmatter.Resources != nil {
+			resources := *backmatter.Resources
+			sort.Slice(resources, func(i, j int) bool {
+				return resources[i].Title < resources[j].Title
+			})
+			backmatter.Resources = &resources
+		}
+		assessment.BackMatter = &backmatter
+	}
+
+}
+
 // findAndSortResults takes a map of results and returns a list of thresholds and a sorted list of results in order of time
 func findAndSortResults(resultMap map[string]*oscalTypes_1_1_2.AssessmentResults) ([]*oscalTypes_1_1_2.Result, []*oscalTypes_1_1_2.Result) {
 

src/pkg/common/oscal/assessment-results_test.go

@@ -1,6 +1,7 @@
 package oscal_test
 
 import (
+	"slices"
 	"testing"
 	"time"
 
@@ -34,18 +35,38 @@ var findingMapFail = map[string]oscalTypes_1_1_2.Finding{
 	},
 }
 
+var findings = []oscalTypes_1_1_2.Finding{
+	{
+		Target: oscalTypes_1_1_2.FindingTarget{
+			TargetId: "ID-1",
+			Status: oscalTypes_1_1_2.ObjectiveStatus{
+				State: "satisfied",
+			},
+		},
+	},
+	{
+		Target: oscalTypes_1_1_2.FindingTarget{
+			TargetId: "ID-2",
+			Status: oscalTypes_1_1_2.ObjectiveStatus{
+				State: "not-satisfied",
+			},
+		},
+	},
+}
+
+// Delineate between these two observations based on the description
 var observations = []oscalTypes_1_1_2.Observation{
 	{
 		Collected:   time.Now(),
 		Methods:     []string{"TEST"},
-		UUID:        uuid.NewUUID(),
-		Description: "test description",
+		UUID:        "4344e734-63d7-4bda-81f1-b805f60fdbf5",
+		Description: "test description first",
 	},
 	{
 		Collected:   time.Now(),
 		Methods:     []string{"TEST"},
-		UUID:        uuid.NewUUID(),
-		Description: "test description",
+		UUID:        "1ac95fcc-1adb-4a25-89a7-08a708def2f3",
+		Description: "test description second",
 	},
 }
 
@@ -448,3 +469,83 @@ func TestEvaluateResultsNewFindings(t *testing.T) {
 	}
 
 }
+
+func TestMakeAssessmentResultsDeterministic(t *testing.T) {
+	// reverse the order
+	slices.Reverse(findings)
+	slices.Reverse(observations)
+
+	// Will already be in reverse order
+	var results = []oscalTypes_1_1_2.Result{
+		{
+			Start:        time.Now(),
+			UUID:         "d66c9509-cb92-4597-86f8-6e6623ea9154",
+			Findings:     &findings,
+			Observations: &observations,
+		},
+		{
+			Start:        time.Now(),
+			UUID:         "28174d67-06a7-4c7c-be04-1edf437d4ece",
+			Findings:     &findings,
+			Observations: &observations,
+		},
+	}
+
+	var assessment = oscalTypes_1_1_2.AssessmentResults{
+		Results: results,
+	}
+
+	oscal.MakeAssessmentResultsDeterministic(&assessment)
+
+	if len(assessment.Results) < 2 {
+		t.Fatalf("Expected 2 results, got %d", len(assessment.Results))
+	}
+
+	// Assessment-Results.Results are sorted newest to oldest
+	var resultExpected = []string{"28174d67-06a7-4c7c-be04-1edf437d4ece", "d66c9509-cb92-4597-86f8-6e6623ea9154"}
+	//Verify order
+
+	for key, id := range resultExpected {
+
+		if assessment.Results[key].UUID != id {
+			t.Fatalf("Expected UUID %q, got %q", id, assessment.Results[key].UUID)
+		}
+
+		assessmentResult := assessment.Results[key]
+		if assessmentResult.Findings == nil {
+			t.Fatal("Expected findings, got nil")
+		}
+
+		assesmentFindings := *assessmentResult.Findings
+
+		if len(assesmentFindings) != 2 {
+			t.Fatalf("Expected 2 findings, got %d", len(findings))
+		}
+
+		var findingExpected = []string{"ID-1", "ID-2"}
+
+		for key, id := range findingExpected {
+			if assesmentFindings[key].Target.TargetId != id {
+				t.Fatalf("Expected finding %q, got %q", id, assesmentFindings[key].Target.TargetId)
+			}
+		}
+
+		if assessmentResult.Observations == nil {
+			t.Fatal("Expected observations, got nil")
+		}
+
+		assessmentObservations := *assessmentResult.Observations
+
+		if len(assessmentObservations) != 2 {
+			t.Fatalf("Expected 2 observations, got %d", len(assessmentObservations))
+		}
+
+		var observationExpected = []string{"4344e734-63d7-4bda-81f1-b805f60fdbf5", "1ac95fcc-1adb-4a25-89a7-08a708def2f3"}
+
+		for key, id := range observationExpected {
+			if assessmentObservations[key].UUID != id {
+				t.Fatalf("Expected observation %q, got %q", id, assessmentObservations[key].UUID)
+			}
+		}
+	}
+}

src/pkg/common/oscal/complete-schema.go

@@ -73,6 +73,14 @@ func WriteOscalModel(filePath string, model *oscalTypes_1_1_2.OscalModels) error
 			return err
 		}
 	}
+	// If the deterministic update is applied here - Lula will fix OSCAL that was previously written
+	// or generated outside of Lula workflows
+	switch modelType {
+	case "component":
+		MakeComponentDeterminstic(model.ComponentDefinition)
+	case "assessment-results":
+		MakeAssessmentResultsDeterministic(model.AssessmentResults)
+	}
 
 	var b bytes.Buffer