Copier support

[augi]

Tell us more.

Hi, we are using copier to scaffold new projects. It would be great if Renovate could update the copier-generated projects, so it would execute the copier with the new version of template, using the stored answers.

[github-actions[bot]]

Hi there,

You're asking us to support a new package manager. We need to know some basic information about this package manager first. Please copy/paste the new package manager questionnaire, and fill it out in full.

Once the questionnaire is filled out we'll decide if we want to support this new manager.

Good luck,

The Renovate team

[secustor]

You should be able to do it with a combination of using a customManager and postUpgradeTasks.

For native support fill out the questionare.

[augi]

New package manager questionnaire

Did you read our documentation on adding a package manager?

• I've read the adding a package manager documentation.

Basics

What's the name of the package manager?

Copier

What language(s) does this package manager support?

It is a tool that generates/scaffolds any text files, according to the template. The templates are Git repos, and they use Git tags for versioning.

How popular is this package manager?

Currently, 1.3k stars on GitHub, and 139 forks.

Does this language have other (competing?) package managers?

• Yes (give names).

• Cookiecutter
• Yeoman

What are the big selling points for this package manager?

It is not for updating dependencies, but it is intended to keep the scaffolded code up-to-date.

For example, README file is part of scaffolding, and if a badge needs to be updated, the automatic re-run of copier (executed by Renovatebot) would prepare a PR to update the badge.

Detecting package files

What kind of package files, and names, does this package manager use?

Every project scaffolded using copier has the following file: .copier-answers.yml

Which fileMatch pattern(s) should Renovate use?

Probably something like .copier-answers.yml

Do many users need to extend the fileMatch pattern for custom file names?

Probably something like (.*)\.copier-answers.yml$

• Yes, provide details.
• No.

Is the fileMatch pattern going to get many "false hits" for files that have nothing to do with package management?

IMHO not.

Parsing and Extraction

Can package files have "local" links to each other that need to be resolved?

No.

Package file parsing method

The package files should be:

• Parsed together (in serial).
• Parsed independently.

Which format/syntax does the package file use?

• JSON
• TOML
• YAML
• Custom (explain below)

How should we parse the package files?

• Off the shelf parser.
• Using regex.
• Custom-parsed line by line.
• Other.

Does the package file have different "types" of dependencies?

• Yes, production and development dependencies.
• No, all dependencies are treated the same.

List all the sources/syntaxes of dependencies that can be extracted

The YAML file contains the following fields:

• _commit, typically Git tag, e.g. v2.7.0. PEP 440
• _src_path: The template path, e.g. git@github.com:pawamoy/copier-poetry.git`

Describe which types of dependencies above are supported and which will be implemented in future

The template doesn't have to be a Git repo, it could be also a local path. But this is not interesting for Renovate, IMHO.

When executing copier, it uses the latest version according to PEP-440, but e.g. latest commit from a branch could be specified manually: https://copier.readthedocs.io/en/latest/generating/#copying-dirty-changes

Versioning

What versioning scheme does the package file(s) use?

PEP 440

Does this versioning scheme support range constraints, like ^1.0.0 or 1.x?

• Supports range constraints (for example: ^1.0.0 or 1.x), provide details.
• No.

Lookup

Is a new datasource required?

• Yes, provide details.
• No.

Will users want (or need to) set a custom host or custom registry for Renovate's lookup?

• Yes, provide details.
• IMHO same as when updating e.g. Git submodule.
• No.

Where can Renovate find the custom host/registry?

• No custom host or registry is needed.
• In the package file(s), provide details.
• IMHO same as e.g. Git submodule.
• In some other file inside the repository, provide details.
• User needs to configure Renovate where to find the information, provide details.

Are there any constraints in the package files that Renovate should use in the lookup procedure?

• Yes, there are constraints on the parent language (for example: supports only Python v3.x), provide details.
• copier requires Python 3.8 or newer
• Yes, there are constraints on the parent platform (for example: only supports Linux, Windows, etc.), provide details.
• Yes, some other kind of constraint, provide details.
• Requires Git 2.27 or newer.
• No constraints.

Will users need the ability to configure language or other constraints using Renovate config?

• Yes, provide details.
• No.

Artifacts

Does the package manager use a lock file or checksum file?

• Yes, uses lock file.
• Yes, uses checksum file.
• Yes, uses lock file and checksum file.
• No lock file or checksum.

Is the locksum or checksum mandatory?

• Yes, locksum is mandatory.
• Yes, checksum is mandatory.
• Yes, lock file and checksum are mandatory.
• No mandatory locksum or checksum.
• Package manager does not use locksums or checksums.

If lockfiles or checksums are used: what tool and exact commands should Renovate use to update one (or more) package versions in a dependency file?

Package manager cache

Does the package manager use a cache?

• Yes, provide details.
• No.

If the package manager uses a cache, how can Renovate control the cache?

• Package manager does not use a cache.
• Controlled via command line interface, provide details.
• Controlled via environment variables, provide details.

Should Renovate keep a cache?

• Yes, ignore/disable the cache.
• Same as for e.g. Git submodules.
• No.

Generating a lockfile from scratch

Renovate can perform "lock file maintenance" by getting the package manager to generate a lockfile from scratch.
Can the package manager generate a lockfile from scratch?

• Yes, explain which command Renovate should use to generate the lockfile.
• No, the package manager does not generate a lockfile from scratch.
• No, the package manager does not use lockfiles.

Other

What else should we know about this package manager?

[augi]

Thank you for your swift responses! 👍 Indeed, I think that this could be doable using the mentioned parameters, but I believe it would be nice to have support for copier.

Actually, it would be sufficient for me to have an example on how to configure Renovatebot for copier (so I wouldn't have to fight with regex etc.) 🙏

[ypid-work]

Seems the GitLab Infra team is already doing that. See https://gitlab.com/gitlab-com/gl-infra/common-ci-tasks/-/merge_requests/146

[lkubb]

Per #29215 (comment), we understandably need to discuss the necessity of each option I'm adding in my Copier manager PR to avoid configuration bloat.

I'll provide an overview of each option and my rationale for adding it. This is detailed, but I hope it provides necessary context.

Edit: I removed all Copier-specific options to avoid stalling the base functionality any further. Some of the now missing functionality can be implemented in Copier instead, the rest should be triaged separately.

---

Overview of Copier (roughly)

Copier templates are repositories with templated paths and file contents, along with a special configuration file (copier.yml). Versions are identified by git tags following PEP440. After generating a project, .copier-answers.yml records the template source, version, and provided template inputs. When updating a project, Copier extracts customizations against the current version, regenerates the project from the new version, and applies the customizations diff.

---

In my PR, I'm adding the following options:

copierOptions

A container for the following options:

copierOptions.data

Description: Pass arbitrary template inputs which override the current ones from .copier-answers.yml. They are ignored if not part of the template.

Rationale:

• Allows centrally managing and updating some inputs. Example: The https://github.com/salt-extensions/ organization uses https://github.com/salt-extensions/salt-extension-copier. When the minimum required Salt version of managed extensions should be updated after a release goes EOL, it could define this in a central Renovate configuration instead of manually updating each generated repository. Note that it should not be done in the template itself since there are other consumers of the Copier template with different requirements.
• Helps when template updates define new required inputs.
• Useful if template requirements change and migrations are disabled or forgotten in the template. Example: The mentioned template increased the minimum required Python version to 3.8, but forgot to add a migration. Updates would crash with the default inputs, but passing python_requires=3.8 allows updates as usual.

Without: Works, but significant manual intervention can be required at specific times.

copierOptions.recopy

Description: Disables the smart update algorithm, overwriting any changes made to templated files in the generated project.

Rationale: The smart update algorithm can result in merge conflicts. The copier manager adds them to the PR anyways with a warning (they are not always reliable, and even when they are, it helps receiving the PR). When a user wants to avoid merge conflicts at all cost, this can help.

Without: When automatically merging Renovate PRs, you need a CI check for merge conflict markers. Some manual intervention might be required in specific cases.

Note: [1]

copierOptions.skip

Description: When specific paths already exist in the generated project, don't follow the template for them anymore.

Rationale: Sometimes, users want to override templated files completely. Templates often contain files with example contents intended to be overridden. When either change in the template, it would cause unwanted conflicts.

Without: This could only be defined by the template, which still solves the example file problem. Overriding other files causes conflicts, thus some projects would often require manual intervention. [2]

Note: [1]

copierOptions.exclude

Description: Never copy specific paths from the template.

Rationale: When templates contain example files that are removed in the generated project, Copier sometimes re-adds them on every update.

Without: For some projects, each PR would require manual intervention to avoid adding unwanted files. This obviously cannot be defined in the template. [2]

Note: The described behavior might be caused by incorrect usage, but it's a very frequent headache from my experience. [3]

---

[1]: Personally, I would not need this, but thought it would be a frequent requirement.
[2]: Not respected from the answers file, but I think this could be implemented in Copier.
[3]: I think this can be caused by not initially committing and then removing the files from the repository. It's a bug in Copier (copier-org/copier#1718).

---

Replaced

copierTrust --> allowScripts

Description: Allow running migrations/tasks (shell commands) and custom Jinja extensions (Python code from the template repository).

Rationale: Many templates rely on some of the mentioned functionality.

Without: Templates with custom Jinja extensions would not work. When tasks and migrations need to be run, the update fails.

Note: This option allows running arbitrary code from the template repository, making it a self-hosted option.

Removed

copierOptions.dataFile

Description: Read template inputs from a YAML file.

Rationale: Initially thought to be useful, but reconsidered its necessity since.

Without: Define the inputs in copierOptions.data.

copierOptions.skipTasks

Description: When copierTrust is enabled, skip running tasks.

Rationale: Thought it would be handy in specific circumstances.

Without: Still functional for all intended purposes.

[viceice]

copierTrust

use existing allowScripts option

https://docs.renovatebot.com/self-hosted-configuration/#allowscripts

[lkubb]

Thanks for the hint, changed! I removed dataFile and skipTasks as well since they are not essential.

[m-birke]

Hi guys,

could we continue here? Updating a copier template is one of my main use cases for using renovate.
Thanks and KR

[m-birke]

Also fine for me to have the tasks option in it. If we would go with 0 options, my proposal is to skip tasks because trusting and executing tasks per default is not preferable.

[lkubb]

Ah, I see. Tasks are only run by Copier when --trust is passed, which it is not by default (can be enabled in the administrator options via allowScripts). --skip-tasks does not have any effect by itself.

I'll leave it as copier update --defaults --skip-answered then. Thanks for answering!

[m-birke]

nice, thanks @lkubb for doing the work

[m-birke]

one more thing:

if tasks are specified but --trust is not set, copier throws an error, so allowScripts = False probably should set --skip-tasks while allowScripts = True should set --trust

[lkubb]

Good observation.

The desired behavior will currently not work because of copier-org/copier#1598, but should once that issue is fixed.

Adding --skip-tasks by default would however make Renovate require the latest Copier release. I wonder if it makes more sense to add this behavior once it has an effect.

[lkubb]

If anyone followed this discussion, but not the issue/PR:

Initial Copier support was released today in version 38.19! Hope it's useful to some people. :)