API plan/apply should allow for the pre/posthook workflows

[Dirrk]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

When I run a plan via the api it does not run the prehook workflows (which I rely on to generate my atlantis.yaml). This basically removes the ability to use anything but the default terraform version for terragrunt using the terraform$${ATLANTIS_TERRAFORM_VERSION} value. Further, I cannot set the terraform_version in the api plan call itself which would provide a work around.

Reproduction Steps

1. Enable the api on atlantis
2. Curl the api to plan

curl --request POST 'http://localhost:4141/api/plan' \
  --header 'X-Atlantis-Token: REMOVED' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "Repository": "example/terraform",
    "Ref": "main",
    "Type": "Github",
    "Paths": [{
      "Directory": "aws/example",
      "Workspace": "default"
    }]
}'

Returns:
{"Error":null,"Failure":"","ProjectResults":[{"Command":1,"SubCommand":"","RepoRelDir":"aws/example","Workspace":"default","Error":{},"Failure":"","PlanSuccess":null,"PolicyCheckResults":null,"ApplySuccess":"","VersionSuccess":"","ImportSuccess":null,"StateRmSuccess":null,"ProjectName":""}],"PlansDeleted":false}

Viewing the job logs:

time=2024-01-31T18:51:33Z level=info msg=All variables passed in by terragrunt are in use. prefix=[/atlantis-data/repos/example/terraform/0/default/aws/example] 
time=2024-01-31T18:51:33Z level=info msg=All required inputs are passed in by terragrunt prefix=[/atlantis-data/repos/example/terraform/0/default/aws/example] 

Error: Unsupported Terraform Core version

  on provider.tf line 16, in terraform:
  16:   required_version = "= 1.4.6"

This configuration does not support Terraform version 0.13.5. To proceed,
either choose another supported Terraform version or update this version
constraint. Version constraints are normally set for good reason, so updating
the constraint may lead to other errors or unexpected behavior.

time=2024-01-31T18:51:33Z level=error msg=1 error occurred:
        * exit status 1

Logs

{"level":"error","ts":"2024-01-31T19:48:18.759Z","caller":"vcs/instrumented_client.go:239","msg":"Unable to update status at url: https://atlantis.example.network/jobs/333919bf-b7e1-4016-bff9-c01616ca33d6, error: POST https://api.github.com/repos/example/terraform/statuses/main: 422 Validation Failed [{Resource:Status Field:sha Code:custom Message:sha must be a 40 character SHA1}]","json":{"repository":"example/terraform","pull-num":"0","src":"atlantis/plan: aws/example/default","description":"Plan failed.","state":"failed","url":"https://atlantis.example.network/jobs/333919bf-b7e1-4016-bff9-c01616ca33d6"},"stacktrace":"github.com/runatlantis/atlantis/server/events/vcs.(*InstrumentedClient).UpdateStatus\n\t/home/runner/work/atlantis/atlantis/server/events/vcs/instrumented_client.go:239\ngh.neting.cc/runatlantis/atlantis/server/events/vcs.(*ClientProxy).UpdateStatus\n\t/home/runner/work/atlantis/atlantis/server/events/vcs/proxy.go:84\ngh.neting.cc/runatlantis/atlantis/server/events.(*DefaultCommitStatusUpdater).UpdateProject\n\t/home/runner/work/atlantis/atlantis/server/events/commit_status_updater.go:103\ngh.neting.cc/runatlantis/atlantis/server/jobs.(*JobURLSetter).SetJobURLWithStatus\n\t/home/runner/work/atlantis/atlantis/server/jobs/job_url_setter.go:41\ngh.neting.cc/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).updateProjectPRStatus\n\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:187\ngh.neting.cc/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:164\ngh.neting.cc/runatlantis/atlantis/server/events.RunAndEmitStats\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:74\ngh.neting.cc/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).apiPlan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:152\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).Plan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:90\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212\ngh.neting.cc/urfave/negroni/v3.(*Negroni).UseHandler.Wrap.func1\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngh.neting.cc/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\t/home/runner/work/atlantis/atlantis/server/middleware.go:70\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Recovery).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Negroni).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009"}
{"level":"error","ts":"2024-01-31T19:48:18.759Z","caller":"events/project_command_runner.go:188","msg":"updating project PR status%!(EXTRA *github.ErrorResponse=POST https://api.github.com/repos/example/terraform/statuses/main: 422 Validation Failed [{Resource:Status Field:sha Code:custom Message:sha must be a 40 character SHA1}])","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).updateProjectPRStatus\n\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:188\ngh.neting.cc/runatlantis/atlantis/server/events.(*ProjectOutputWrapper).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/project_command_runner.go:164\ngh.neting.cc/runatlantis/atlantis/server/events.RunAndEmitStats\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:74\ngh.neting.cc/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).apiPlan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:152\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).Plan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:90\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212\ngh.neting.cc/urfave/negroni/v3.(*Negroni).UseHandler.Wrap.func1\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngh.neting.cc/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\t/home/runner/work/atlantis/atlantis/server/middleware.go:70\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Recovery).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Negroni).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009"}
{"level":"error","ts":"2024-01-31T19:48:18.759Z","caller":"events/instrumented_project_command_runner.go:78","msg":"Error running plan operation: running \"terragrunt init -input=false -upgrade --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION}\" in \"/atlantis-data/repos/example/terraform/0/default/aws/example\": exit status 1: running \"terragrunt init -input=false -upgrade --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION}\" in \"/atlantis-data/repos/example/terraform/0/default/aws/example\": \n\nError: Unsupported Terraform Core version\n\n  on provider.tf line 16, in terraform:\n  16:   required_version = \"= 1.4.6\"\n\nThis configuration does not support Terraform version 0.13.5. To proceed,\neither choose another supported Terraform version or update this version\nconstraint. Version constraints are normally set for good reason, so updating\nthe constraint may lead to other errors or unexpected behavior.\n\ntime=2024-01-31T19:48:18Z level=error msg=1 error occurred:\n\t* exit status 1\n\n\n\nIn API call\nTF Version = 0.13.5\ngenerated atlantis.yaml\n\ntime=2024-01-31T19:48:18Z level=info msg=All variables passed in by terragrunt are in use. prefix=[/atlantis-data/repos/example/terraform/0/default/aws/example] \ntime=2024-01-31T19:48:18Z level=info msg=All required inputs are passed in by terragrunt prefix=[/atlantis-data/repos/example/terraform/0/default/aws/example] \n","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:78\ngh.neting.cc/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\t/home/runner/work/atlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).apiPlan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:152\ngh.neting.cc/runatlantis/atlantis/server/controllers.(*APIController).Plan\n\t/home/runner/work/atlantis/atlantis/server/controllers/api_controller.go:90\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2136\ngh.neting.cc/gorilla/mux.(*Router).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/gorilla/mux@v1.8.1/mux.go:212\ngh.neting.cc/urfave/negroni/v3.(*Negroni).UseHandler.Wrap.func1\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:59\ngh.neting.cc/urfave/negroni/v3.HandlerFunc.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:33\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP\n\t/home/runner/work/atlantis/atlantis/server/middleware.go:70\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Recovery).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/recovery.go:210\ngh.neting.cc/urfave/negroni/v3.middleware.ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:51\ngh.neting.cc/urfave/negroni/v3.(*Negroni).ServeHTTP\n\t/home/runner/go/pkg/mod/github.com/urfave/negroni/v3@v3.0.0/negroni.go:111\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.21.5/x64/src/net/http/server.go:2009"}

Environment details

Version 0.27.1
Helm Chart Version 4.19.0 (via Argocd onto EKS)
atlantis.yaml = empty we rely on it to be generated using terragrunt-atlantis-config during the prehook stage

server side config is mostly here:

pre_workflow_hooks:
  - run: terragrunt-atlantis-config generate --output atlantis.yaml --autoplan=true --parallel=true --ignore-parent-terragrunt --create-project-name --cascade-dependencies=false --ignore-dependency-blocks=true --automerge=true
workflows:
  default:
    plan:
      steps:
        - run: |
            if [ "$$PULL_NUM" == "0" ]; then
              terragrunt-atlantis-config generate --output atlantis.yaml --autoplan=true --parallel=true --ignore-parent-terragrunt --create-project-name --cascade-dependencies=false --ignore-dependency-blocks=true --automerge=true
            fi
        - run: terragrunt validate-inputs --terragrunt-strict-validate --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION}
        - run: terragrunt hclfmt --terragrunt-check --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION}
        - run: terragrunt init -input=false -upgrade --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION}
        - run: terragrunt plan -input=false --terragrunt-tfpath terraform${ATLANTIS_TERRAFORM_VERSION} --out $PLANFILE

Additional Context

As you can see from the above I also manually generate the atlantis.yaml as the first step of the plan but at that point the repoconfig would have already been parsed so it doesn't do it again between run commands. Ideally when I run api/plan it would work exactly as if I made a noop PR against the repo and ran atlantis plan -d path. However at minimum the api should support the terraform_version and probably any other context related variables that would be defined for a project in the atlantis.yaml

[albertorm95]

I think this should be close now? I haven't tested this
@jamengual

[jamengual]

closed via #4482