Atlantis unlock fails to delete plan of environment with project name #3845

the-nando · 2023-10-11T16:55:56Z

#3750 changed the behaviour of atlantis unlock which used to trigger a deletion of the whole directory where the PR is checked out in favour of only deleting the plan file.
The delete lock command however doesn't handle configuration with project names configured, which causes issues: https://github.com/runatlantis/atlantis/blob/v0.26.0/server/events/delete_lock_command.go#L60

Reproduction Steps

Atlantis version: v0.26.0

Given an atlantis.yaml:

---
version: 3
projects:
  - name: dev 
    dir: clusters/dev/config
    workflow: dev
    terraform_version: v1.5.7
    autoplan:
      enabled: true
      when_modified:
      - "*.tf*"

When an atlantis unlock is issued:

{"level":"info","ts":"2023-10-11T15:53:45.296Z","caller":"events/unlock_command_runner.go:35","msg":"Unlocking all locks","json":{"repo":"<repo>","pull":"1004"}}
{"level":"info","ts":"2023-10-11T15:53:45.300Z","caller":"events/working_dir.go:418","msg":"Deleting plan: /home/atlantis/.atlantis/repos/<repo>/1004/default/clusters/dev/config/default.tfplan","json":{}}
{"level":"warn","ts":"2023-10-11T15:53:45.300Z","caller":"events/delete_lock_command.go:68","msg":"Failed to delete plan: remove /home/atlantis/.atlantis/repos/<repo>/1004/default/clusters/dev/config/default.tfplan: no such file or directory","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultDeleteLockCommand).DeleteLocksByPull\n\t/tmp/atlantis-v0.26.0/server/events/delete_lock_command.go:68\ngh.neting.cc/runatlantis/atlantis/server/events.(*UnlockCommandRunner).Run\n\t/tmp/atlantis-v0.26.0/server/events/unlock_command_runner.go:37\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/tmp/atlantis-v0.26.0/server/events/command_runner.go:328"}
{"level":"error","ts":"2023-10-11T15:53:45.300Z","caller":"events/unlock_command_runner.go:40","msg":"failed to delete locks by pull remove /home/atlantis/.atlantis/repos/<repo>/1004/default/clusters/dev/config/default.tfplan: no such file or directory","json":{"repo":"<repo>","pull":"1004"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*UnlockCommandRunner).Run\n\t/tmp/atlantis-v0.26.0/server/events/unlock_command_runner.go:40\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\t/tmp/atlantis-v0.26.0/server/events/command_runner.go:328"}

The plan file is actually named dev-default.tfplan after fmt.Sprintf("%s-%s.tfplan", projName, workspace).
A workaround is to remove project names from the Atlantis config and rely only on dir names.

The text was updated successfully, but these errors were encountered:

jamengual · 2023-10-11T17:31:52Z

@X-Guardian do you think you could check if this is the case?

X-Guardian · 2023-10-11T18:41:32Z

Yes, the Atlantis locks controller has no knowledge of projects.

X-Guardian · 2023-10-20T10:26:52Z

Several tickets have been raised bout this same issue, so here is a clarification of the current situation:

The file name of the terraform plan file produced by Atlantis for each project differs depending upon the following conditions:

If there is no repo level atlantis.yaml file or within the atlantis.yaml file the project that is being planned has no name property, then the plan file will be called <workspace>.tfplan.
If within the atlantis.yaml file the project has a name property, then the plan file will be called <projectName>.<workspace>.tfplan

you can see the code for this here:

atlantis/server/core/runtime/runtime.go

Lines 89 to 97 in f34e5db

    
           // GetPlanFilename returns the filename (not the path) of the generated tf plan 
        
           // given a workspace and project name. 
        
           func GetPlanFilename(workspace string, projName string) string { 
        
           	if projName == "" { 
        
           		return fmt.Sprintf("%s.tfplan", workspace) 
        
           	} 
        
           	projName = strings.Replace(projName, "/", planfileSlashReplace, -1) 
        
           	return fmt.Sprintf("%s-%s.tfplan", projName, workspace) 
        
           }

The problem is that the locks controller functions have no knowledge of the project name, so if this is set, the deletion of the plan file will fail in the unlock functions. To fix this would need the locks controller functions plus other higher level functions updating with details of the project including the project name.

I expect there are more bugs within the locks controller when multiple projects are specified for the same directory and different project names.

Easy workaround for now is if you are not using multiple projects for the same directory within your atlantis.yaml, don't specify the name property for the project, as it is not adding any value and will cause this issue.

jamengual · 2023-10-20T19:35:40Z

@GenPage

velvetzhero · 2023-12-08T15:09:20Z

We have multiple projects for the same directories, and we want to use atlantis to lock prs, it's very convenient..
So I've tested around, and while it's true that it prints out a simple Failed to delete PR locks output, and the logs points to the wrong tfplan file ( while it should be for us too: prj-name-workspace.tfplan) :

{"level":"error","ts":"2023-12-08T13:45:05.176Z","caller":"events/unlock_command_runner.go:40","msg":"failed to delete locks by pull remove /atlantis/repos/XXX/ops/1969/staging/terraform/XXX/staging.tfplan │
│ no such file or directory","json":

The lock is removed!
Can you confirm the problem is not effectively about the lock, but the plan being removed ? What are the implication of running a pre_hook to remove the plan it if it exists ? Just not to get the error message I mean.
My second question may sounds a bit naive, but I am looking for a quick workaround as you may have noticed..Would using an extra_args such as -out=<workspace>.tfplan could do it ?

Not a big deal really, just trying my luck frankly..

aston-r · 2023-12-14T09:57:14Z

@X-Guardian

Easy workaround for now is if you are not using multiple projects for the same directory within your atlantis.yaml, don't specify the name property for the project, as it is not adding any value and will cause this issue.

Unfortunately this workaround does not suit me. Without project name it is really inconvenient since the path is really long and in gh issue summary(CI block check) contains base path that is common for all all projects + ...As a result I do not know what the project this(Not informative). Also previously i used atlantis plan -p and without project name the command is different and need to pass directory and workspace.

Moreover I liked that Atlantis removed the whole workspace during unlock(when I need to free disk space) and for me it would be better to control via option(like -cleanup-worksapce-on-unclock): #3751

Are you planing to fix this?

jakubjakubik · 2023-12-14T10:17:52Z

Easy workaround for now is if you are not using multiple projects for the same directory within your atlantis.yaml, don't specify the name property for the project, as it is not adding any value and will cause this issue.

The workaround doesn't cover my use case as well. Is there any way to get to the previous behaviour?

jamengual · 2023-12-14T16:56:01Z

For those who have issues with the way that unlock is working now, I will ask that if you have the time and golang knowledge to fix it, please create a PR for it.
#3751 fixed other issues with unlock that people took for granted, but in fact, this behaviour was not correct from the beginning.

We are trying to avoid adding more flags so we could add this to the repo.config if is needed.

grimm26 · 2024-01-09T15:56:15Z

For those who have issues with the way that unlock is working now, I will ask that if you have the time and golang knowledge to fix it, please create a PR for it. #3751 fixed other issues with unlock that people took for granted, but in fact, this behaviour was not correct from the beginning.

We are trying to avoid adding more flags so we could add this to the repo.config if is needed.

Sure. Shall I create a revert of #3751?

X-Guardian · 2024-01-09T16:18:26Z

No

grimm26 · 2024-01-09T21:29:56Z

No

OK, meanwhile I will have to rollback to 0.25.0 because I have a pileup of PRs that won't unlock that I keep having to unlock manually.

albertorm95 · 2024-01-10T15:54:55Z

No

OK, meanwhile I will have to rollback to 0.25.0 because I have a pileup of PRs that won't unlock that I keep having to unlock manually.

+1 with this, we use multiple projects in the same directory and its not rmving the plan file, we will need to rollback to 0.25.0 😭

velvetzhero · 2024-01-11T08:55:25Z

I expect there are more bugs within the locks controller when multiple projects are specified for the same directory and different project names.

If it was a quick fix, it would have been done already. Better to rollback to 0.25 and wait for someone to pick this up someday...

grimm26 · 2024-01-22T23:16:28Z

FYI, I do not use the name parameter and I still hit this issue. Snippet from my root level atlantis.yaml

version: 3
projects:
- dir: align/production/align-home
  terraform_version: v1.3.8
  workflow: terragrunt
  autoplan:
    enabled: true
    when_modified:
    - '*.hcl'
    - '*.tf'
    - atlantis.yaml

GenPage · 2024-01-26T18:35:52Z

Hello everyone, we will revert #3751. I approved the change and actually was worried about this very issue. We have been working to better understand the differences between locks and filesystem operations. I apologize for the disruptions.

The code is spread out across the lock controllers, command controllers, and a special library for file system operation for "WorkingDir". I appreciate everyones patience as we continue to streamline and optimize these operations. It makes more sense to properly support project locks across the codebase before making these changes.

GenPage · 2024-01-26T22:28:51Z

To clarify, this revert may not result in a "full revert" but there will be at a minimum, a revert in functionality. We are still debating options internally on the core team. Most likely solution is a new PR that will keep both behaviors, gated by a configuration flag.

dimisjim · 2024-03-15T14:21:37Z

this issue seems to still be there in latest v0.27.2
When trying "atlantis unlock", I get a reply: "Failed to delete PR locks"

but lock is properly unlocked:

My atlantis repo config looks like this:

version: 3
projects:
- name: main
  workflow: main
  dir: .
  autoplan:
    when_modified: ["*.tf", "env-vars/main.tfvars"]

workflows:
  main:
    plan:
      steps:
      - init:
          extra_args: ["-backend-config=backend-configs/main.hcl", "-reconfigure", "-upgrade"]
      - plan:
          extra_args: ["-var-file=env-vars/main.tfvars"]
    apply:
      steps:
      - apply

jamengual · 2024-03-15T17:24:51Z

these tests you have done have been always on the same PR? you could try to create another PR and see how it works or close and reopen your current PR. if this is a brand-new setup, you should not have this issue.

…

On Fri, Mar 15, 2024 at 7:22 AM Dimitris Moraitidis < ***@***.***> wrote: this issue seems to still be there. When trying "atlantis unlock", I see: "Failed to delete PR locks" but lock is properly unlocked: image.png (view on web) <https://github.com/runatlantis/atlantis/assets/6890249/df9f6614-1f06-41c1-bbe9-0582f2c38ec0> My atlantis repo config looks like this: version: 3 projects: - name: main workflow: main dir: . autoplan: when_modified: ["*.tf", "env-vars/main.tfvars"] workflows: main: plan: steps: - init: extra_args: ["-backend-config=backend-configs/main.hcl", "-reconfigure", "-upgrade"] - plan: extra_args: ["-var-file=env-vars/main.tfvars"] apply: steps: - apply — Reply to this email directly, view it on GitHub <#3845 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3ERC6JFD7M2IP5FEBKLTYYL7YRAVCNFSM6AAAAAA54J56LSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZG43TIMJQG4> . You are receiving this because you commented.Message ID: ***@***.***>

dimisjim · 2024-03-17T10:46:30Z

@jamengual Indeed, this did not happen in a new PR, ensuring that it was opened after atlantis was upgraded to 0.27.2

Thanks!

jamengual · 2024-03-17T12:31:54Z

I'm glad to hear is working

ritaCanavarro · 2024-04-18T09:48:42Z

Hi @jamengual :)

We just upgraded our Atlantis version to get the fix for this issue and unfortunately it came back after some time, i.e, we tried to run the atlantis unlock on a PR and then we got the Failed to delete PR locks message. Checking our Atlantis logs, we found the following:

failed to delete locks by pull remove /atlantis-data/repos/*****/****/1006/default/staging/*****.tfplan: no such file or directory

Have you seen any error similar to this one since this fix is live?

dimisjim · 2024-04-18T09:52:20Z

I've also noticed this happening, but only on a closed PR

ritaCanavarro · 2024-04-18T13:35:07Z

I've also noticed this happening, but only on a closed PR

Thanks for sharing. In our case, the PR had only been opened the day before :/

jamengual · 2024-04-18T17:01:05Z

the unlock was run on a project that had a different project/workflow configuration?

amontalban · 2024-04-18T19:39:44Z

Just upgraded to 0.27.2 from 0.25.0 today and saw the same Failed to delete PR locks error while they disappeared from Atlantis UI. Unfortunately, I do not have access to the FS because we use EFS and ECS.

ritaCanavarro · 2024-04-19T07:11:29Z

the unlock was run on a project that had a different project/workflow configuration?

No, in this case it was on the same project.

jippi · 2024-05-03T22:24:54Z

There is a fair chance #4502 will fix this (and a couple of other ensure stuff is deleted issues we've seen)

the-nando added the bug Something isn't working label Oct 11, 2023

X-Guardian mentioned this issue Oct 17, 2023

Comment atlantis unlock returns Failed to delete PR locks every time #3856

Closed

nitrocode mentioned this issue Oct 20, 2023

atlantis unlock results in misleading error: Failed to delete PR locks #3877

Closed

nitrocode added the regression Bug introduced in a new version label Oct 20, 2023

jamengual mentioned this issue Oct 20, 2023

Can`t discard plan via WEB and apply job is never ended (v0.25.0) #3734

Closed

X-Guardian mentioned this issue Nov 22, 2023

fix: Running 'atlantis unlock' on a PR Causes The Whole Working Directory to be Deleted #3751

Merged

nitrocode mentioned this issue Dec 26, 2023

Atlantis fails to release locks #4058

Closed

GenPage added this to Atlantis Roadmap Jan 23, 2024

github-project-automation bot moved this to Backlog in Atlantis Roadmap Jan 23, 2024

lukemassa mentioned this issue Feb 6, 2024

feat: Add logic for project name into lock #4192

Merged

lukemassa closed this as completed in #4192 Feb 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Atlantis unlock fails to delete plan of environment with project name #3845

Atlantis unlock fails to delete plan of environment with project name #3845

the-nando commented Oct 11, 2023 •

edited

Loading

jamengual commented Oct 11, 2023

X-Guardian commented Oct 11, 2023

X-Guardian commented Oct 20, 2023

jamengual commented Oct 20, 2023

velvetzhero commented Dec 8, 2023 •

edited

Loading

aston-r commented Dec 14, 2023

jakubjakubik commented Dec 14, 2023

jamengual commented Dec 14, 2023

grimm26 commented Jan 9, 2024

X-Guardian commented Jan 9, 2024

grimm26 commented Jan 9, 2024

albertorm95 commented Jan 10, 2024

velvetzhero commented Jan 11, 2024 •

edited

Loading

grimm26 commented Jan 22, 2024 •

edited by nitrocode

Loading

GenPage commented Jan 26, 2024

GenPage commented Jan 26, 2024

dimisjim commented Mar 15, 2024 •

edited

Loading

jamengual commented Mar 15, 2024 via email

dimisjim commented Mar 17, 2024

jamengual commented Mar 17, 2024

ritaCanavarro commented Apr 18, 2024

dimisjim commented Apr 18, 2024

ritaCanavarro commented Apr 18, 2024

jamengual commented Apr 18, 2024

amontalban commented Apr 18, 2024

ritaCanavarro commented Apr 19, 2024

jippi commented May 3, 2024

Atlantis unlock fails to delete plan of environment with project name #3845

Atlantis unlock fails to delete plan of environment with project name #3845

Comments

the-nando commented Oct 11, 2023 • edited Loading

Reproduction Steps

jamengual commented Oct 11, 2023

X-Guardian commented Oct 11, 2023

X-Guardian commented Oct 20, 2023

jamengual commented Oct 20, 2023

velvetzhero commented Dec 8, 2023 • edited Loading

aston-r commented Dec 14, 2023

jakubjakubik commented Dec 14, 2023

jamengual commented Dec 14, 2023

grimm26 commented Jan 9, 2024

X-Guardian commented Jan 9, 2024

grimm26 commented Jan 9, 2024

albertorm95 commented Jan 10, 2024

velvetzhero commented Jan 11, 2024 • edited Loading

grimm26 commented Jan 22, 2024 • edited by nitrocode Loading

GenPage commented Jan 26, 2024

GenPage commented Jan 26, 2024

dimisjim commented Mar 15, 2024 • edited Loading

jamengual commented Mar 15, 2024 via email

dimisjim commented Mar 17, 2024

jamengual commented Mar 17, 2024

ritaCanavarro commented Apr 18, 2024

dimisjim commented Apr 18, 2024

ritaCanavarro commented Apr 18, 2024

jamengual commented Apr 18, 2024

amontalban commented Apr 18, 2024

ritaCanavarro commented Apr 19, 2024

jippi commented May 3, 2024

the-nando commented Oct 11, 2023 •

edited

Loading

velvetzhero commented Dec 8, 2023 •

edited

Loading

velvetzhero commented Jan 11, 2024 •

edited

Loading

grimm26 commented Jan 22, 2024 •

edited by nitrocode

Loading

dimisjim commented Mar 15, 2024 •

edited

Loading