Agnostic Vars

Table of Contents

Description
- Features
- Example of agnosticv repositories
agnosticv CLI
- Install
- Usage
Build
File naming convention
Merging strategies
- Default
- Custom merge strategies
See also

Description

Agnosticv is a generic way of organizing and merging YAML files to manage a Catalog.

Currently the project is composed of:

CLI agnosticv, a generic implementation.
agnosticv-operator an OpenShift operator that uses the CLI and generates OpenShift Custom Resources for Babylon based on the content of the agnosticv repositories. The operator is more opinionated than the CLI as it expects a specific file structure in the agnosticV repository. We use it to manage the catalog of our demo portal RHPDS.

Features

Merge YAML files automatically following a particular convention, see below.
Support for includes.
Inject information about last change (commit, author, date) when merging the vars of a catalog item
When listing catalog items, filter catalog items using JMESPath expressions (--has flag)
Configure custom policies or behavior, per repository:
- Configure merge strategies
- Validate variables using OpenAPI v3 schemas. Just place schemas in a .schemas directory at the top-level of the agnosticv repository.
- Configuration file /.agnosticv.yaml

Example of agnosticv repositories

public: see in cli/fixtures for an example of structure of a catalog
private GPTE agnosticv repository
private Red Hat Summit 2021 repository
private 2020Summit-AgnosticV repository
private RHTR_2020_agnosticV repository

`agnosticv` CLI

Install

Download the binary for your architecture (linux, mac, windows, ..) from release page.

Usage

The CLI agnosticv has the following capabilities:

list all the catalog items present in a directory
merge and print the vars of an item of the catalog

Usage

Usage of agnosticv:
-debug
Debug mode
-git
Perform git operations to gather and inject information into the merged vars like 'last_update'. Git operations are slow so this option is automatically disabled for listing. (default true)
-has value
Use with --list only. Filter catalog items using a JMESPath expression.
Can be used several times (act like AND).

Examples:
--has __meta__.catalog
--has "env_type == 'ocp-clientvm'"
--has "to_string(worker_instance_count) == '2'"

-list
List all the catalog items present in current directory.
-merge string
Merge and print variables of a catalog item.
-or-related value
Use with --list only. Same as --related except it appends the related files to the list instead of reducing it.

Example:
--list --related dir/common.yaml --or-related includes/foo.yaml
List all catalog items under dir/ and also all catalog items that include includes/foo.yaml

Can be used several times (act like OR).
-related value
Use with --list only. Filter output and display only related catalog items.
A catalog item is related to FILE if:
- it includes FILE as a common file
- it includes FILE via #include
- FILE is description.adoc or description.html

Example:
--list --related dir/common.yaml --related includes/foo.yaml
List all catalog items under dir/ that also include includes/foo.yaml

Can be used several times (act like AND).
-root string
The top directory of the agnosticv files. Files outside of this directory will not be merged.
By default, it's empty, and the scope of the git repository is used, so you should not
need this parameter unless your files are not in a git repository, or if you want to use a subdir. Use -root flag with -merge.
-validate
Validate variables against schemas present in .schemas directory. (default true)
-version
Print build version.

List catalog items in local directory

cli $ ./agnosticv --list
fixtures/gpte/OCP_CLIENTVM/dev.yaml
fixtures/gpte/OCP_CLIENTVM/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG/test.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_AWS/test.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/dev.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/prod.yaml
fixtures/test/BABYLON_EMPTY_CONFIG_OSP/test.yaml

Merge and print the vars of a catalog item

cli $ ./agnosticv --merge fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
---
# MERGED: (1)
# fixtures/common.yaml
# fixtures/test/account.yaml
# fixtures/test/account.meta.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/common.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG_AWS/prod.yaml
__meta__:
  deployer:
    scm_ref: test-empty-config-test-0.5
    scm_type: git
    scm_url: https://github.com/redhat-cop/agnosticd.git
    type: agnosticd
  secrets:
  - from-top-common.yml
  - name: gpte
  [...] output omitted

Merge list: gives information about how files were merged to produce the final set of variables

Note	`common.yaml` files are always included when merging. `agnosticv` searches for those files as long as it is in the same git repository. If the files are not versioned with git, it is possible to "chroot" the search using the `--root` parameter.

Build

cd cli
go get
go build -o agnosticv

File naming convention

Nomenclature

Catalog item, or Leaf: a file considered a catalog item. It appears when listing an agnosticv repository with agnosticv --list. To print the content of a catalog item, you run agnosticv --merge PATH. That merges common files, meta files, included files, and finally, the leaf to produce the catalog item.
Common file: a file that is automatically included in the merge list. Ex: common.yaml
Related file: a file that is related to a catalog item. Ex: description.adoc
Included file: any file that is included in the merge list using the #include PATH feature
Meta file: Any file ending with .meta.yml or .meta.yaml. It contains the value of the meta dictionary.

Common files

Some files are automatically included in the merge list to produce the final catalog item. The following names are valid common YAML files:

common.yaml
common.yml
account.yaml
account.yml

They can be placed at any level in the agnosticv repository.

Includes

Files included in the merge list using the #include PATH feature. See the dedicated section below.

Usually, we place them in an includes/ directory at the top of the agnosticv repository.

`#include` merge feature

syntax: #include FILENAME
where: In any file

identation is ignored

#include /file.yaml

And:

    #include /file.yaml

are the same.

FILENAME is added to the merge list right before current file regardless of the position of #include in the file. In other words, current file vars take precedence over included files vars.
- That’s also why you should put all your includes at the top of the file.
if FILENAME starts with / then path is absolute to the AgnosticV repo.
- if not, the path is relative to the current file

Example

gpte/OCP4_WORKSHOP/prod.yaml

#include /includes/file1.yaml
#include /includes/file2.yaml

cloud_provider: ec2
key_name: opentlc_admin_backdoor
repo_method: file

subdomain_base_suffix: .example.opentlc.com
HostedZoneId: Z3IHLWJZOU9SRT

agnosticv_meta:
  deploy_with: babylon

includes/file1.yaml with vars at the "agnosticd" level

var1: value1
var2: value2

includes/file2.yaml with meta vars

agnosticv_meta:
  secrets:
    - somesecret

__meta__:
  secrets:
    - name: somesecret
      namespace: gpte

The merge list will be:

/common.yaml
/gpte/account.yaml
/gpte/OCP4_WORKSHOP/common.yaml
/includes/file1.yaml
/includes/file2.yaml
/gpte/OCP4_WORKSHOP/prod.yaml

Meta files

For any common file, leaf file, or included file, you can create an associated meta file to be automatically included. The meta file can contain the value of the __meta__ dictionary. It is convenient to separate that special dictionary from the other variables.

For example, the following files are valid meta files:

common.meta.yml meta file for common.yml
account.meta.yml meta file for account.yml
dev.meta.yml meta file for dev.yml

Warning

you can only put the content of the __meta__ variable in a meta file.

example1: content of meta file

__meta__:
  secrets:
    - name: mysecret

example2: with the content of __meta__ directly, same as example1

secrets:
  - name: mysecret

example3: wrong meta file - This will fail

__meta__:
  secrets:
    - name: mysecret

another_var: value (1)

other variables than __meta__ are not allowed

Related Files

Related files are helpful in estimating when a catalog item was last changed. All catalog items related to that file are considered touched if a related file is touched.
To print all catalog items related to a file, run agnosticv --list --related-to RELATED_FILE
All included files of a catalog item are automatically considered related files.
All common files of a catalog item are automatically considered related files.
It is possible to add custom related files using the config file; see section below.
It is possible to load related files into the merged vars using the config file; see section below.

Configuration

You can add custom related files using the related_files configuration option in a .agnosticv.yaml configuration file. It contains a list of filenames that if present in the same directory of a catalog item, will be considered as related to that catalog item.

You can decide to load the content of the relative file into a path by specifying the path as a JSON Pointer format. The destination path will be a dictionary and the content will be loaded into the 'content_key'.

Here is an example of configuration:

/.agnosticv.yaml

# For any catalog item, consider those files in same directory as related files:
related_files_v2:
  - file: service-ready-message-template.html.j2
    load_into: /__meta__/catalog/message_templates/service_ready
    content_key: template
    set:
      format: jinja2
      output_format: html
  - file: description.txt

When merging, that will produce:

__meta__:
  catalog:
    message_templates:
      service_ready:
        format: jinja2
        output_format: html
        template: |
          <<content of the file>>

Leaf files

The "leaf" files, or catalog items, are just the rest of the YAML files, having one of these extensions:

yml
yaml

You can list all catalog items in a directory by using --list parameter: agnosticv --list

Files ignored

Any dotfile is ignored. Ex: .git
Any directory named includes is reserved to includes. The files in those directories are never considered as catalog items.
Any file containing:
```
#agnosticv catalog_item false
```
is ignored. It is not considered a catalog item.

Merging strategies

Default

When it comes to merging variables, there are different possible strategies.

The default is the following:

What Dictionaries Lists Strings / Numbers

__meta__ and agnosticv_meta dictionaries

merge

append

replace

Rest of the vars

Same behavior as if you were using ansible extra vars

replace

Custom merge strategies

It is possible to extend agnosticV and define the merge strategy to use on what variable or part of a dictionary variable.

To do that, you can define the custom merge strategies in any schema in the .schemas directory. Just use the x-merge keyword at the beginning of the schema. x-merge is a list of strategies. Each strategy defines a path and a strategy name to apply to that path. path is a JSON Pointer. For the list of strategies, see section below.

.schema/schema.yaml example of x-merge custom strategy

type: object
x-merge:
  - path: /__meta__/access_control # (1)
    strategy: overwrite # (2)
properties:

The path of the variable or key of dictionnary, as a JSON Pointer, to apply the custom strategy against.
When merging, agnosticv will overwrite the content of __meta__.access_control instead of merging it.

For example, with the schema above and following merge list:

# MERGED:
# fixtures/common.yaml
# fixtures/test/account.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG/common.yaml
# fixtures/test/BABYLON_EMPTY_CONFIG/prod.yaml

The value of __meta__.access_control from prod.yaml will take precedence and overwrite.

Here are the available custom merge strategies:

Strategy	Can be applied to	Dictionaries	Lists	Strings / Numbers
`overwrite`	List or Dict	replace	replace	replace
`merge`	List or Dict	Merge	Append	replace
`merge-no-append`	Dict	Merge	replace	replace
`strategic-merge`	List or Dict	Strategic Merge ^[1]	Strategic Merge ^[1]	replace

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

readme.adoc

readme.adoc

Agnostic Vars

Description

Features

Example of agnosticv repositories

`agnosticv` CLI

Install

Usage

Build

File naming convention

Nomenclature

Common files

Includes

`#include` merge feature

Example

Meta files

Configuration

Leaf files

Files ignored

Merging strategies

Default

Custom merge strategies

See also

Files

readme.adoc

Latest commit

History

readme.adoc

File metadata and controls

Agnostic Vars

Description

Features

Example of agnosticv repositories

agnosticv CLI

Install

Usage

Build

File naming convention

Nomenclature

Common files

Includes

#include merge feature

Example

Meta files

Related Files

Configuration

Leaf files

Files ignored

Merging strategies

Default

Custom merge strategies

See also

`agnosticv` CLI

`#include` merge feature