Terraform provider: flapping branch state when combined with other updates

[msw-kialo]

How are you running Renovate?

Self-hosted, but reproduced with RenovateApp on GitHub

If you're self-hosting Renovate, tell us what version of Renovate you run.

31.15.0

Please select which platform you are using if self-hosting.

github.com

If you're self-hosting Renovate, tell us what version of the platform you run.

No response

Describe the bug

Terraform provider updates with rangeStrategy update-lockfile are unreliable when grouped with other updates (like a regex manager to update the Terraform version). Renovate flaps the branch between the complete updates and only the Terraform artifacts updates.

I stumbled across this with self-hosted renovate but was able to reproduce it with the renovate app on Github, too. See msw-kialo/renovate-flapping-grouped-terraform-updates#1 for a reproduction and a sample PR.

Furthermore, it makes it impossible to request the upgrade out-of-schedule if prCreation is set to not-pending: it would require two runs with the same outcome. This way I encountered it the first time. But that is not really relevant to me (although it made debugging/testing it more challenging).

Relevant debug logs

Logs

https://app.renovatebot.com/dashboard#github/msw-kialo/renovate-flapping-grouped-terraform-updates/545881011

Effect msw-kialo/renovate-flapping-grouped-terraform-updates#1

Note the difference between 2 file(s) to commit and the wrong commit that only includes the one artifact file.

DEBUG: Setting current branch to main(branch="renovate/terraform")
DEBUG: latest commit(branch="renovate/terraform")
{
  "branchName": "main",
  "latestCommitDate": "2022-01-03T17:23:25+01:00"
}
DEBUG: branchExists=true(branch="renovate/terraform")
DEBUG: dependencyDashboardCheck=undefined(branch="renovate/terraform")
DEBUG: PR rebase requested=false(branch="renovate/terraform")
DEBUG: Checking if PR has been edited(branch="renovate/terraform")
DEBUG: Branch has not been modified(branch="renovate/terraform")
{
  "branchName": "renovate/terraform"
}
DEBUG: Found existing branch PR(branch="renovate/terraform")
DEBUG: Checking schedule(at any time, null)(branch="renovate/terraform")
DEBUG: No schedule defined(branch="renovate/terraform")
DEBUG: Branch already exists(branch="renovate/terraform")
DEBUG: Endpoint needs paid GitHub plan(branch="renovate/terraform")
{
  "path": "repos/msw-kialo/renovate-flapping-grouped-terraform-updates/branches/main/protection"
}
DEBUG: Branch protection: Do not have permissions to detect branch protection(branch="renovate/terraform")
DEBUG: Skipping stale branch check due to rebaseWhen=auto(branch="renovate/terraform")
DEBUG: Branch does not need rebasing(branch="renovate/terraform")
DEBUG: Using reuseExistingBranch: true(branch="renovate/terraform")
DEBUG: manager.getUpdatedPackageFiles() reuseExistinbranch=true(branch="renovate/terraform")
DEBUG: isLockFileUpdate without updateLockedDependency(branch="renovate/terraform")
{
  "manager": "terraform"
}
DEBUG: Branch dep is already updated(packageFile="versions.sh", branch="renovate/terraform")
{
  "depName": "hashicorp/terraform"
}
DEBUG: No content changed(packageFile="versions.sh", branch="renovate/terraform")
{
  "depName": "hashicorp/terraform"
}
DEBUG: terraform.updateArtifacts(terraform/main.tf)(branch="renovate/terraform")
DEBUG: Updated 1 package files(branch="renovate/terraform")
DEBUG: Updated 1 lock files(branch="renovate/terraform")
{
  "updatedArtifacts": [
    "terraform/.terraform.lock.hcl"
  ]
}
DEBUG: getClosedPrs(): no graphql data(branch="renovate/terraform")
DEBUG: Getting comments for #1(branch="renovate/terraform")
DEBUG: Found 0 comments(branch="renovate/terraform")
DEBUG: 2 file(s) to commit(branch="renovate/terraform")
DEBUG: Committing files to branch renovate/terraform(branch="renovate/terraform")
DEBUG: Setting git author name(branch="renovate/terraform")
{
  "gitAuthorName": "Renovate Bot"
}
DEBUG: Setting git author email(branch="renovate/terraform")
{
  "gitAuthorEmail": "bot@renovateapp.com"
}
DEBUG: git commit(branch="renovate/terraform")
{
  "deletedFiles": [],
  "ignoredFiles": [],
  "result": {
    "author": null,
    "branch": "renovate/terraform",
    "commit": "e60b8f1",
    "root": false,
    "summary": {
      "changes": 1,
      "insertions": 12,
      "deletions": 0
    }
  }
}
DEBUG: git push(branch="renovate/terraform")
{
  "result": {
    "pushed": [],
    "branch": {
      "local": "renovate/terraform",
      "remote": "renovate/terraform",
      "remoteName": "origin"
    },
    "ref": {
      "local": "refs/remotes/origin/renovate/terraform"
    },
    "remoteMessages": {
      "all": []
    }
  }
}
INFO: Branch updated(branch="renovate/terraform")
{
  "commitSha": "e60b8f1"
}
DEBUG: Checking if we can automerge branch(branch="renovate/terraform")
DEBUG: mergeStatus=no automerge(branch="renovate/terraform")
DEBUG: Ensuring PR(branch="renovate/terraform")
DEBUG: There are 0 errors and 0 warnings(branch="renovate/terraform")
DEBUG: Found existing PR(branch="renovate/terraform")
DEBUG: Processing existing PR(branch="renovate/terraform")
DEBUG: Pull Request #1 does not need updating(branch="renovate/terraform")
DEBUG: PR is not configured for automerge(branch="renovate/terraform")

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction repository in the bug description

[viceice]

What's the log when it fails?

[msw-kialo]

That is the failing / incorrect log. Previously, renovate opened the PR with two changed files (version.sh from the regex manager, and one terraform/.terraform.lock.hcl). So on this following run, nothing should be changed: current and available versions did not change.

The regex manager correctly determines the branch is still up-to-date:

DEBUG: Branch dep is already updated(packageFile="versions.sh", branch="renovate/terraform")
{
  "depName": "hashicorp/terraform"
}
DEBUG: No content changed(packageFile="versions.sh", branch="renovate/terraform")
{
  "depName": "hashicorp/terraform"
}

But the Terraform manager does not, and changes files:

DEBUG: terraform.updateArtifacts(terraform/main.tf)(branch="renovate/terraform")
DEBUG: Updated 1 package files(branch="renovate/terraform")
DEBUG: Updated 1 lock files(branch="renovate/terraform")
{
  "updatedArtifacts": [
    "terraform/.terraform.lock.hcl"
  ]
}

Afterwards, only this one file is committed and incorrectly force-pushed on the branch. Compare the original commit with the force-pushed commit. One contains correctly both updates, the other one only the terraform provide update but not the regex update.

On the next run, renovate will correct the branch and incomplete it again on the following and so on. If you like, I can trigger more example runs (and also leave it in the "correct" state if you prefer).

[yafanasiev]

Just hit this issue - self-hosted renovate 32.232.0, terraform manager. We have a monorepo and a few package rules grouping terraform manager updates, like so:

    {
      "groupName": "terraform dependencies (internal)",
      "matchManagers": [
        "terraform"
      ],
      "matchPaths": [
        "internal"
      ],
      "addLabels": [
        "internal"
      ]
    },
    {
      "groupName": "terraform dependencies (live)",
      "matchManagers": [
        "terraform"
      ],
      "matchPaths": [
        "live"
      ],
      "addLabels": [
        "live"
      ]
    }

with rangeStrategy set to update-lockfile each run of renovate creates a new commit, first with all the changes and then with lockfile changes only, same as in the original issue. We switched to pin for now because it suits our workflow, but having update-lockfile working properly would be great.

[github-actions]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[msw-kialo]

@rarkins The linked repository still outlines the issue. We are still observing it with that configuration on our self-hosted instance.
However, it is apparently since #17848. On github.com, renovate always stops at DEBUG: branch.isUpToDate(): using cached result "true"(branch="renovate/terraform"). Our self-hosted instance doesn't have a long living cache.
I dig a bit through the code to understand how long living this cache is but failed. It is still a cache and will expire eventually, isn't it? I tried waiting a few days and updating the base branch.
Is there a way to invalidate the cache for the github.com renovate instance?

[rarkins]

branch.isUpToDate() will return true unless the list of packages/versions changes in the branch. There's no expiry on that. You can force override that by ticking the rebase/retry checkbox in any PR.

[msw-kialo]

Thanks. So it should be fixed. I will look out for the next PR for our self-hosted installation and close / update the ticket accordingly.

[rarkins]

Just so I understand, do you mean that you turned on repositoryCache functionality locally and you are hoping it solves the problem for you?

[msw-kialo]

It was not. Your response didn't include that it is driven by repositoryCache. I only wanted to make sure to ensure we run on the newest renovate version and take a closer look at the next terraform update PR (scheduled one for once a month).

But now, I suspect I have to? The reproducible repository is apparently no longer sufficient — so a repositoryCache is required for correct functionality here.

[rarkins]

If the hosted app doesn't reproduce the problem (because it uses repositoryCache) then try to reproduce using the CLI you run yourself. I would like to fix any problem even if it only happens when repository cache is disabled

[msw-kialo]

I have just check running renovate (v34.50.0) without a (persistent) branch cache on the linked reproducible repository. And it still triggers this issues (the branch is updated to only include the terraform artifact update without the regex terraform bump).

Log of the second renovate run (branch is actually up-to-date but still partially rebuild)

DEBUG: Found PR #4 (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: GET https://api.github.com/repos/msw-kialo/renovate-flapping-grouped-terraform-updates/branches/main/protection = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=304) (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: No branch protection found (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Skipping behind base branch check due to rebaseWhen=auto (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: isBranchConflicted(main, self-hosted-renovate/terraform) (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: branch.isConflicted(): using git to calculate (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Setting git author name: renovatebot-msw-kialo[bot] (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Setting git author email: renovatebot-msw-kialo[bot]@users.noreply.github.com (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: branch.isConflicted(): false (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Branch does not need rebasing (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Using reuseExistingBranch: true (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Setting current branch to main (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: latest commit (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "branchName": "main",
       "latestCommitDate": "2022-11-28T13:47:47+01:00"
DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=true (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: isLockFileUpdate without updateLockedDependency (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "manager": "terraform"
DEBUG: Branch dep hashicorp/terraform in versions.sh is already updated (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: No content changed (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, packageFile=versions.sh, branch=self-hosted-renovate/terraform)
       "depName": "hashicorp/terraform"
DEBUG: terraform.updateArtifacts(terraform/main.tf) (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Updated 1 package files (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Updated 1 lock files (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "updatedArtifacts": ["terraform/.terraform.lock.hcl"]
DEBUG: Getting comments for #4 (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Found 0 comments (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: 2 file(s) to commit (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: Preparing files for committing to branch self-hosted-renovate/terraform (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: git commit (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "deletedFiles": [],
       "ignoredFiles": [],
       "result": {
         "author": null,
         "branch": "self-hosted-renovate/terraform",
         "commit": "5f8f42a6ad6ceb4f973a965725fe814f8bc1b88c",
         "root": false,
         "summary": {"changes": 1, "insertions": 12, "deletions": 2}
       }
DEBUG: Pushing branch self-hosted-renovate/terraform (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
DEBUG: git push (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "result": {
         "pushed": [],
         "ref": {"local": "refs/remotes/origin/self-hosted-renovate/terraform"},
         "remoteMessages": {"all": []}
       }
 INFO: Branch updated (repository=msw-kialo/renovate-flapping-grouped-terraform-updates, branch=self-hosted-renovate/terraform)
       "commitSha": "5f8f42a6ad6ceb4f973a965725fe814f8bc1b88c"

See msw-kialo/renovate-flapping-grouped-terraform-updates#4

[github-actions]

When a bug has been marked as needing a reproduction, it means nobody can work on it until one is provided. In cases where no reproduction is possible, or the issue creator does not have the time to reproduce, we unfortunately need to close such issues as they are non-actionable and serve no benefit by remaining open. This issue will be closed after 7 days of inactivity.

[rarkins]

I have classified this as low priority if it only occurs when grouping AND with regex manager