Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make it possible to validate Dependabot config before it lands on main #4605

Open
saramah opened this issue Jan 8, 2022 · 17 comments
Open

Make it possible to validate Dependabot config before it lands on main #4605

saramah opened this issue Jan 8, 2022 · 17 comments
Labels
EE Engineering Efficiency F: configuration-file Keep Exempt this from being marked by stalebot T: bug 🐞 Something isn't working T: feature-improvement Requests to improve existing features

Comments

@saramah
Copy link

saramah commented Jan 8, 2022 •
edited
Loading

There used to be a dependabot config validator at https://dependabot.com/docs/config-file/validator/, but it has gone missing. Is there a replacement?

references:
https://stackoverflow.com/questions/63943201/how-do-i-test-dependabot-before-merging-config
https://gh.neting.ccmunity/t/how-to-validate-dependabot-config-file/218636
@saramah saramah added the T: bug 🐞 Something isn't working label Jan 8, 2022
@mark-stopka
Copy link

Same issue here...

@mark-stopka mark-stopka mentioned this issue Jan 25, 2022
Closed
@jurre
Copy link
Member

jurre commented Jan 25, 2022

The Dependabot app registers a "CheckRun" in GH whenever the config file changes: https://github.com/dependabot/dependabot-core/runs/4327679442

Does that help?

@hfhbd
Copy link
Contributor

hfhbd commented Jan 27, 2022

@jurre Not really, as the CheckRun only runs at the main branch, not in a PR. This is even visible in your linked example:
https://github.com/dependabot/dependabot-core/pull/4454/checks, in the checks there is no validator.

See also: #3878

@jurre
Copy link
Member

jurre commented Jan 27, 2022

@hfhbd ah yes, that's still not working :( I'll see about getting that prioritized.

@jurre
Copy link
Member

jurre commented Jan 27, 2022

In an effort to be transparent, I just investigated this a little and it seems the issue that prevents the check from running on PRs is fairly deep inside GitHub itself, and will require us to rejig some things internally so unfortunately not an easy fix. Will still try to get it prioritized either way, but just might take a little longer for us to roll out a fix because it involves changing a bunch of moving parts :(

@mark-stopka mark-stopka mentioned this issue Jan 27, 2022
Open
@jurre jurre added T: feature-improvement Requests to improve existing features needs-triage ❓ labels Jan 28, 2022
activescott added a commit to activescott/agentmarkdown that referenced this issue Feb 21, 2022
@activescott
chore(build): remove dependabot validate as it no longer works
13ee487 
due to marocchino/validate-dependabot#233 and dependabot/dependabot-core#4605
@activescott activescott mentioned this issue Feb 21, 2022
Merged
activescott added a commit to activescott/agentmarkdown that referenced this issue Feb 22, 2022
@activescott
chore(build): remove dependabot validate as it no longer works
c96cb11 
due to marocchino/validate-dependabot#233 and dependabot/dependabot-core#4605
@brrygrdn
Copy link
Contributor

👋🏻 Just removing the needs-triage as we've been looking at this internally. There is some work we want to do around Dependabot configuration and enablement, so I've made a note of this problem as part of that. I don't have anything to share right now, but this is something we'd like to do better on.

@brrygrdn brrygrdn self-assigned this Feb 25, 2022
@jlosito jlosito mentioned this issue Jul 19, 2022
Merged
@jlosito
Copy link

jlosito commented Jul 19, 2022

@jurre @brrygrdn I understand that some additional rejigging would need to be done, but would there be the possibility to release some sort of cli tool which people could run in order to validate the configuration file? Something such as dp-lint /path/to/.github/dependabot.yml.

@ChrisBAshton ChrisBAshton mentioned this issue Aug 3, 2022
Closed
@jeffwidman jeffwidman mentioned this issue Aug 9, 2022
Closed
@jeffwidman jeffwidman mentioned this issue Aug 23, 2022
Closed
@jeffwidman jeffwidman changed the title dependabot validator has gone missing Validate Dependabot config before it lands on main Aug 26, 2022
@jeffwidman jeffwidman changed the title Validate Dependabot config before it lands on main Make it possible to validate Dependabot config before it lands on main Aug 26, 2022
@jeffwidman
Copy link
Member

I realize the original post title was asking for a stand-alone validator, but I suspect the real intent here is to make it possible to validate the config before it lands on master/main.

That could be implemented via a standalone validator, but as discussed later in this issue's comments, it could also be via a check that runs on the PR before merge.

So I reworded the title to focus on the problem, rather than describing only one possible fix implementation.

And yes, we do very much want to resolve the hurdles mentioned in #4605 (comment) and get this resolved.

@jeffwidman
Copy link
Member

I just noticed that the GH action that validates dependabot.yaml just ran on a PR:
https://github.com/dependabot/dependabot-core/pull/5513/checks?check_run_id=8648478306

I'm a bit surprised because AFAIK we didn't make the the internal changes described in #4605 (comment).

The only thing that is different from a normal PR is that I updated the PR with a rebase off main... so perhaps that triggered it? 😕

@jeffwidman jeffwidman mentioned this issue Oct 6, 2022
Closed
1 task
@jeffwidman
Copy link
Member

jeffwidman commented Oct 13, 2022 •
edited
Loading

I just noticed the check also ran on this PR branch: https://github.com/dependabot/cli/pull/17/checks?check_run_id=8862312262

And that PR doesn't touch dependabot.yml file... BUT, interestingly, this PR that I merged 15 mins before does change the dependabot file: dependabot/cli#18

Given that we haven't made internal changes, I suspect this is caused by an unrelated race condition of some sort, but I don't have time to dig into that right now. So just leaving a breadcrumb if we keep seeing this occasionally and want to investigate.

@pkanal pkanal mentioned this issue Dec 14, 2022
Merged
pkanal added a commit to honeycombio/honeycomb-opentelemetry-node that referenced this issue Dec 14, 2022
@pkanal
fix: add semantic commit prefix to dependabot PRs (#131)
71112b4 
## Which problem is this PR solving?
Our dependabot PRs are failing CI checks because of the PR title
validation check. Instead of getting the PR title validation check to
ignore dependabot PRs, I thought it would be better to add the correct
semantic prefix to dependabot PRs instead.

- Updates honeycombio/telemetry-team#389

## Short description of the changes
- Adds semantic prefix `maint` to all dependabot commits / PRs

## How to verify that this has the expected result
Unfortunately, it doesn't seem like we can validate dependabot config
changes without merging them in according to [this
issue](dependabot/dependabot-core#4605).

I think this should work as expected based on:
- [The dependabot docs for
`commit-message`](https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#commit-message)
- [The code adds a `:` and a space automatically

](https://github.com/dependabot/dependabot-core/blob/main/common/spec/dependabot/pull_request_creator/pr_name_prefixer_spec.rb#L248)
@mhuster23 mhuster23 mentioned this issue Jan 16, 2024
Merged
@thehowl thehowl mentioned this issue Feb 9, 2024
Merged
@rocodes rocodes mentioned this issue Feb 12, 2024
Merged
1 task
@timstibbs-hippo timstibbs-hippo mentioned this issue Feb 15, 2024
Open
thehowl added a commit to gnolang/gno that referenced this issue Feb 22, 2024
@thehowl
ci: use action to validate dependabot.yml (#1642)
91aa2af 
following [this
comment](#1628 (comment))
from @kristovatlas

This action seems to only validate the JSON schema. For full validation,
it's on GitHub to provide this -- tracked in this issue:
dependabot/dependabot-core#4605
leohhhn pushed a commit to leohhhn/gno that referenced this issue Feb 29, 2024
@thehowl @leohhhn
ci: use action to validate dependabot.yml (gnolang#1642)
ce0209f 
following [this
comment](gnolang#1628 (comment))
from @kristovatlas

This action seems to only validate the JSON schema. For full validation,
it's on GitHub to provide this -- tracked in this issue:
dependabot/dependabot-core#4605
@bastelfreak bastelfreak mentioned this issue Mar 17, 2024
Closed
didrocks added a commit to ubuntu/authd that referenced this issue Mar 18, 2024
@didrocks
Always validate dependabot config in CI
254e294 
Current upstream validator is still under work and tracked as
dependabot/dependabot-core#4605.
didrocks added a commit to ubuntu/authd that referenced this issue Mar 18, 2024
@didrocks
Always validate dependabot config in CI
3f9df8f 
Current upstream validator is still under work and tracked as
dependabot/dependabot-core#4605.
didrocks added a commit to ubuntu/authd that referenced this issue Mar 18, 2024
@didrocks
Always validate dependabot config in CI
6ff288c 
Current upstream validator is still under work and tracked as
dependabot/dependabot-core#4605.
kachick referenced this issue in kachick/wait-other-jobs Apr 7, 2024
@kachick
Fix wrong update for dependabot.yml
f6f76c9
@jonjanego jonjanego added the Keep Exempt this from being marked by stalebot label May 2, 2024
@aduth aduth mentioned this issue May 10, 2024
Merged
@avatus avatus mentioned this issue Jun 13, 2024
Merged
avatus added a commit to gravitational/teleport that referenced this issue Jun 13, 2024
@avatus
Add yarn ecosystem to dependabot.yaml
0820864 
This PR should enable dependabot to check our web UI ecosystem. We use
yarn workspaces so (I believe) we have to target each individual
package.json in order for it to be updated. As far as I'm aware, I
haven't found a way to [validate a dependabot.yaml file before letting
it run](dependabot/dependabot-core#4605) so
this may need to be iterated on.

I've tried to split up the responsibility for each package around the
web team (connect team takes /teleterm, ryan helps with /build).

Change ecosystem name to npm

According to the docs, although yarn is a valid ecosystem, the job is
now asking for npm. I'm not sure if this will use yarn to update the
yarn.lock or not
github-merge-queue bot pushed a commit to gravitational/teleport that referenced this issue Jun 14, 2024
@avatus
Add yarn ecosystem to dependabot.yaml (#42942)
e464034 
This PR should enable dependabot to check our web UI ecosystem. We use
yarn workspaces so (I believe) we have to target each individual
package.json in order for it to be updated. As far as I'm aware, I
haven't found a way to [validate a dependabot.yaml file before letting
it run](dependabot/dependabot-core#4605) so
this may need to be iterated on.

I've tried to split up the responsibility for each package around the
web team (connect team takes /teleterm, ryan helps with /build).

Change ecosystem name to npm

According to the docs, although yarn is a valid ecosystem, the job is
now asking for npm. I'm not sure if this will use yarn to update the
yarn.lock or not
@nickzelei
Copy link

I ran into this myself after merging a bad config into main (which was previously valid somehow).
I put up another PR to try and fix it and suddenly the same job that didn't run on my previous PR is now running to validate the dependabot config.

Some weird behavior here. Why didn't it run previously on the PR that ended up breaking, but now it runs on the subsequent PR to fix it?

@mironiasty mironiasty mentioned this issue Jul 8, 2024
Merged
@open-dynaMIX open-dynaMIX mentioned this issue Aug 2, 2024
Open
@jamescrowley jamescrowley mentioned this issue Aug 5, 2024
Merged
@bugron
Copy link

bugron commented Sep 1, 2024 •
edited
Loading

I'm sharing this here in case anyone finds it useful.

If your Dependabot config file is located at .github/dependabot.yml (notice the extension) simply run the following command in the root of your repository. See the docs regarding passing another path.

npx -y @bugron/validate-dependabot-yaml@latest

I also needed to validate a Dependabot config file without committing to the default branch first so I've published a CLI tool @bugron/validate-dependabot-yaml that does exactly that and more. It is capable of logging validation errors both in Markdown and JSON so can naturally used in GH Actions for sticky comments and post-processing errors. Note, the tool isn't perfect, but should do the job.

fquffio added a commit to chialab/aws-ecr-get-login-password that referenced this issue Sep 2, 2024
@fquffio
fix: what better way to test @dependabot config than pushing to main …
40e86e0 
…until it works?

Ref dependabot/dependabot-core#4605
fquffio added a commit to chialab/aws-ecr-get-login-password that referenced this issue Sep 2, 2024
@fquffio
fix: what better way to test @dependabot config than pushing to main …
20b0ca6 
…until it works? 2/n

Ref dependabot/dependabot-core#4605
eskultety added a commit to eskultety/cachi2 that referenced this issue Sep 9, 2024
@eskultety
.github: dependabot: Fix the 'exclude_patterns' config typo
5978de9 
s/exclude_patterns/exclude-patterns [1].

Since an official native validation solution doesn't exist yet [2] that
would work reliably, @bugron/validate-dependabot-yaml was used to
validate this typo fix.

[1] https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups
[2] dependabot/dependabot-core#4605
[3] https://www.npmjs.com/package/@bugron/validate-dependabot-yaml

Signed-off-by: Erik Skultety <eskultet@redhat.com>
eskultety added a commit to eskultety/cachi2 that referenced this issue Sep 9, 2024
@eskultety
.github: dependabot: Fix the 'exclude_patterns' config typo
7ad6340 
s/exclude_patterns/exclude-patterns [1].

Since an official native validation solution doesn't exist yet [2] that
would work reliably, @bugron/validate-dependabot-yaml was used to
validate this typo fix.

fixes: 24f35ee

[1] https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups
[2] dependabot/dependabot-core#4605
[3] https://www.npmjs.com/package/@bugron/validate-dependabot-yaml

Signed-off-by: Erik Skultety <eskultet@redhat.com>
@eskultety eskultety mentioned this issue Sep 9, 2024
Merged
4 tasks
github-merge-queue bot pushed a commit to containerbuildsystem/cachi2 that referenced this issue Sep 9, 2024
@eskultety
.github: dependabot: Fix the 'exclude_patterns' config typo
b3862f2 
s/exclude_patterns/exclude-patterns [1].

Since an official native validation solution doesn't exist yet [2] that
would work reliably, @bugron/validate-dependabot-yaml was used to
validate this typo fix.

fixes: 24f35ee

[1] https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups
[2] dependabot/dependabot-core#4605
[3] https://www.npmjs.com/package/@bugron/validate-dependabot-yaml

Signed-off-by: Erik Skultety <eskultet@redhat.com>
midnightercz pushed a commit to midnightercz/cachi2 that referenced this issue Sep 12, 2024
@eskultety @midnightercz
.github: dependabot: Fix the 'exclude_patterns' config typo
e2acb7f 
s/exclude_patterns/exclude-patterns [1].

Since an official native validation solution doesn't exist yet [2] that
would work reliably, @bugron/validate-dependabot-yaml was used to
validate this typo fix.

fixes: 24f35ee

[1] https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file#groups
[2] dependabot/dependabot-core#4605
[3] https://www.npmjs.com/package/@bugron/validate-dependabot-yaml

Signed-off-by: Erik Skultety <eskultet@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
EE Engineering Efficiency F: configuration-file Keep Exempt this from being marked by stalebot T: bug 🐞 Something isn't working T: feature-improvement Requests to improve existing features
Projects
Dependabot
Status: No status
Milestone
No milestone
Development

No branches or pull requests
16 participants
@saramah @jeffwidman @abdulapopoola @jurre @brrygrdn @jonjanego @larouxn @wesley-harding @nickzelei @bugron @mark-stopka @hfhbd @jlosito @MuhammedMahrous @maxmoehl and others