Add support for otk-defined distributions (COMPOSER-2299) #797
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achilleas-k
requested review from
ondrejbudai,
mvo5,
thozza,
schuellerf and
supakeen
achilleas-k
force-pushed
the
otk
branch
2 times, most recently
from
6b3fae0
to
9b1a67c
Compare
|
Oh nice. Tests are failing now because we depend on I'll probably mock the otk calls for testing the |
achilleas-k
changed the title
|
This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days. |
|
This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days. |
Make the distro field of the manifest.Manifest struct private and make sure it's set using the manifest.New() constructor. The intention is to make manifest.Manifest an interface and getting rid of private fields is the first step towards this.
manifest.Manifest is now an interface with one implementation called IntManifest (Internal Manifest), which is the previous Manifest type. All functions that interact with pointers to the old concrete Manifest type now interact with the interface instead.
The OTKManifest is a second implementation of the manifest.Manifest interface that will be used to generate manifests from otk yaml files. The type only implements the Serialize() method. None of the other methods are important (at least for now). The content spec methods in particular are not needed since otk will handle content resolution (depsolving, container and ostree commit resolves) itself.
The directory structure follows the assumptions of the test_otk_distro.
New package that loads a distribution from otk yaml files.
Distribution-level properties, like name, codename, versions, etc, are
loaded from a file called 'properties.yaml' in the root of the
distribution's directory. An error is returned if this file is missing
or if any of the required properties are undefined/empty.
The distro creates a distro->arch->imagetype hierarchy based on the
contents of the filesystem tree. It assumes a directory structure
`<distroid>/<archname>/<imagetypename>.yaml`. For example:
fedora-40/
properties.yaml
x86_64/
qcow2.yaml
ami.yaml
aarch64/
qcow2.yaml
The top-level directory is used as the unique distro ID, whereas the
name is defined in the properties.yaml. The distinction is useful for
separating the on-disk, unique ID of a distro from the display name.
Test that the distro is loaded with the expected properties, arches, and image types.
Add the DistroFactory() function to otkdistro, which will enable loading distros defined using otk files alongside the internally defined ones using the same mechanism. The root of the otk path is hard-coded to "./otk", but we will have to make this configurable.
With this addition, for every file found in the repository configuration path (for this project, "test/data/repositories/" by default), the distrofactory will try to load an otk distro at "otk/<distroid>/".
When initialising an OTKManifest, add the Blueprint to it as well, so we can create variables (defines, modifications) based on it before compiling.
New otk package for managing otk-related things. The otkCustomizationsFile type handles creating an otk file with defines for customizations and a single otk.include. The purpose is to use this to create a 'customizations.yaml' file that 'otk.define's all the variables represented by a blueprint then 'otk.include' the file we want to compile, effectively setting variables defined by the user in the compilation of the manifest. Currently it only writes a few simple customizations as a proof of concept, but it will grow to cover all potential customizations.
When serializing an OTKManifest, create a customizations file that includes 'otk.define's to represent the options of a blueprint, then 'otk.include' the original entrypoint to generate a manifest with the user customizations included.
Define the expected file contents as a map[string]interface{} and
deserialize the file contents before comparing for convenience and nicer
error output (compared to strings).
Add tests for otk serialization with some simple blueprint customizations. The entrypoint contents and expected serialized manifests are defined as map[string]interface for convenience and nicer error output (compared to strings).
Remove test/otk/fakedistro and instead create it when running the unit test. This will make it easier to evolve the test to cover more features.
|
Rebased and resolved conflicts. |
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Sep 30, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
language: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
language: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just regrep what "modifications" vars are being used.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Sep 30, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
language: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
language: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list what "modifications" vars are being used for a given
image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Sep 30, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
language: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
language: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list what "modifications" vars are being used for a given
image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Oct 1, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
language: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
language: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list what "modifications" vars are being used for a given
image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Oct 1, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
locale: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
locale: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list (via otk) what "modifications" vars are being used for a
given image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Oct 7, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
locale: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
locale: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list (via otk) what "modifications" vars are being used for a
given image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
mvo5
added a commit
to mvo5/osbuild-otk
that referenced
this pull request
Oct 7, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
locale: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
locale: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list (via otk) what "modifications" vars are being used for a
given image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
supakeen
pushed a commit
to mvo5/osbuild-otk
that referenced
this pull request
Oct 10, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
locale: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
locale: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list (via otk) what "modifications" vars are being used for a
given image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
|
This PR is stale because it has been open 30 days with no activity. Remove "Stale" label or comment or this will be closed in 7 days. |
github-merge-queue bot
pushed a commit
to osbuild/otk
that referenced
this pull request
Oct 31, 2024
This commit adds support for user modifications. The
idea behind modifications is that those are variables
that users want to change to customize the imagess.
This appears uses the existing otk.defines mechanism
and adds a convention around the naming for modifications.
We will use the customizations from blueprints and map
them to modifcations. A simple example would be:
```yaml
otk.define:
user:
modifications:
locale: nl_NL.UTF-8
```
In our otk files we will have defaults for all supported
modifications and then join the defaults with the user
define modifcations.
```yaml
otk.define:
default:
modifications:
locale: en_US.UTF-8
modifications:
otk.op.join:
values:
- ${default.modifications}
- ${user.modifications}
```
This allows us to follow the idea from the images PR#797
(osbuild/images#797):
```yaml
otk.define:
user:
modifications:
hostname: "hal-9000"
kernel_append: "debug"
keyboard: "uk"
otk.include: `/otkroot/fedora-40/x86_64/qcow2.yaml`
```
and because the otk.op.join will merge two tree this should
work.
We could also (later) provide a convinience helper like:
```
$ otk compile --preload/--include "my-customizatins.yaml" centos-9-x86_64-ami.yaml
```
which would just preload/include (terms stawman) the customizations
and then proceed processing the centos-9-x86_64-ami.yaml file.
To see what modifications are supported by a given image we would
just list (via otk) what "modifications" vars are being used for a
given image type.
Note that this commit cheats by pre-defining an empty
"user.modifications" var. This avoids having to deal with the issue
of undefined variables and conditionals.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds the initial (but not complete) support for defining distributions using otk yaml files. I've been experimenting with multiple ideas for how to do this and I think this is the nicest to work with while keeping disruption of current internal distro definitions to a minimum.
There's still a lot more to do, but I wanted to get this in front of people to discuss design choices before moving on with the full solution. Meanwhile, I intend to start converting image types to otk files to evaluate the integration.
Notable design choices worth discussing
<distro ID>/<architecture>/<image type name>.yaml.properties.yamlin its root (<distro ID>/properties.yaml) which defines values required by osbuild-composer. Currently this includesname(which can differ from the distro ID),os_version,release_version, andrunner.module_platform_idis definitely not needed since otk is taking care of depsolving for us. But we will almost certainly need a way to define the canonical distro name in order to do host distro selection.Keep in mind that while we transition to a fully otk-defined state, we will need to keep the instantiation and serialisation flow of the internal manifests and a lot of the interfaces associated with that. Once we finish transitioning, a lot of this should become simpler.
Blueprint customizations
When compiling an otk manifest, an
otk-customizations.yamlis created in a temporary directory which includes all the Blueprint customizations and anotk.includeto the target image type entrypoint. The customizations file is used as the entrypoint to the compile command.For example, given the following Blueprint:
when compiling a Fedora 40 x86-64 qcow2, an
otk-customizations.yamlis created with:(where
/otkrootis the root where we define our otk distros)and the manifest is compiled with:
The
qcow2yaml (or the files it includes) should then be properly handling these values. For example:One issue with this approach however is that this puts customizations at the lowest priority since they're defined first, so any defines that follow will override the value. One way around this issue is prefixing these or namespacing them under an object that is treated with higher priority (e.g.
modifications.keyboard), which is something we already discussed in the otk specifications.