feat: Look for recipes in ./recipes/, build files in ./files/, and Containerfiles in ./containerfiles/

[gmpinder]

Recipe files can now be put into their own directory ./recipes/. This directory is NEVER copied into the build so changes to a recipe will no longer cause cache misses for builds. Here is an example of my build changing the second to last module and only requiring the last 2 RUN layers to be run again.

 => CACHED [stage-config 1/1] COPY ./config /config                                                                                                       0.0s
 => CACHED [stage-modules 1/2] COPY --from=ghcr.io/blue-build/modules:latest /modules /modules                                                            0.0s
 => CACHED [stage-modules 2/2] COPY ./modules /modules                                                                                                    0.0s
 => CACHED [stage-keys 1/1] COPY cosign.pub /keys/jp-desktop-gaming.pub                                                                                   0.0s
 => CACHED [stage-4  2/16] RUN --mount=type=bind,from=stage-keys,src=/keys,dst=/tmp/keys   mkdir -p /usr/etc/pki/containers/   && cp /tmp/keys/* /usr/et  0.0s
 => CACHED [stage-bins 1/3] COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /bins/cosign                                                         0.0s
 => CACHED [stage-bins 2/3] COPY --from=docker.io/mikefarah/yq /usr/bin/yq /bins/yq                                                                       0.0s
 => CACHED [stage-bins 3/3] COPY --from=ghcr.io/blue-build/cli:main-installer /out/bluebuild /bins/bluebuild                                              0.0s
 => CACHED [stage-4  3/16] RUN --mount=type=bind,from=stage-bins,src=/bins,dst=/tmp/bins   mkdir -p /usr/bin/   && cp /tmp/bins/* /usr/bin/   && ostree   0.0s
 => CACHED [stage-4  4/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  5/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  6/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  7/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  8/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4  9/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 10/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 11/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 12/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 13/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => CACHED [stage-4 14/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind  0.0s
 => [stage-4 15/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind,from=  33.4s
 => [stage-4 16/16] RUN   --mount=type=tmpfs,target=/var   --mount=type=bind,from=stage-config,src=/config,dst=/tmp/config,rw   --mount=type=bind,from=s  0.7s

Making it so that an upgrade only required pulling those last 2 layers:

$> rpm-ostree upgrade
Pulling manifest: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop
Importing: ostree-image-signed:docker://registry.gitlab.com/wunker-bunker/wunker-os/jp-laptop (digest: sha256:01073d98adf4041f3b91840af1135b15aee97db90de6bc3a3846d67c07345e6a)
ostree chunk layers already present: 65
custom layers already present: 16
custom layers needed: 2 (255.0 MB)

Support was also added to put all build files into ./files/ instead of ./config/. This is an all or nothing operation, meaning if there exists a directory of files then the config directory will be completely ignored. Work will have to be done in https://github.com/blue-build/modules to allow users to put their files directly in ./files/ and not ./files/files for the files module or ./files/scripts for the scripts module.

Support was also added to move the ./config/containerfiles/ directory to the root of the project. Now the directories you can find in the root of projects are:

files/
containerfiles/
recipes/

[gmpinder]

Just realized I didn't do the same for the containerfile module. Let me fix that

[fiftydinar]

I agree with this one. It improves the builds as you mentioned & directory layout looks cleaner to manage.

The only thing is that putting usr for example directly in files folder can get lost if user has a lot of folder inclusions from other modules, like systemd, gschema-overrides & similar.
This is very apparent in cases when there are multiple custom image variants.

Maybe files/os_files/ is better solution here?

[xynydev]

@fiftydinar In my view, after modules are refactored.

The files/ directory could look like this:

• usr/
• gschema-overrides/
• systemd/

Or it could look like this:

• system_files/
  • usr/
  • etc/
• modules/
  • systemd/
  • gschema/

etc. etc.

And so, it would be up to the maintainer to decide how to arrange their repository.

[fiftydinar]

The files/ directory could look like this:

...

Or it could look like this:

- `system_files/`
  
  - `usr/`
  - `etc/`

- `modules/`
  
  - `systemd/`
  - `gschema/`

I like this one

[gmpinder]

Yeah the general idea still is to keep all the files in one directory still so the CLI doesn't have to keep track of logic that determines where to pull files from

[xynydev]

Cool, yeah! I think I've come around on the ./recipes/ folder naming, and it's my understanding that this PR doesn't have any breaking changes, so I'll approve. We should discuss the details of this transition elsewhere.