Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"File exists" errors when un-taring golangci-lint #244

Closed
andig opened this issue May 24, 2021 · 39 comments
Closed

"File exists" errors when un-taring golangci-lint #244

andig opened this issue May 24, 2021 · 39 comments
Labels
area: cache duplicate This issue or pull request already exists

Comments

@andig
Copy link

andig commented May 24, 2021

Follow-up to #135, #156 and #221.

Using v2, I'm still seeing these errors:

- name: Lint
  uses: golangci/golangci-lint-action@v2
  with:
    version: v1.37

Example is in https://github.com/andig/evcc/pull/1075/checks?check_run_id=2655591730

Apart from fixing using tar- why not just use go install instead?

@andig
Copy link
Author

andig commented May 24, 2021

Looking at https://pipelines.actions.githubusercontent.com/D1tYAQGUpNaqlpNKaplcT4IVn3rJSlK5kNFys0tSrNSiqbBdeP/_apis/pipelines/1/runs/3119/signedlogcontent/4?urlExpires=2021-05-24T11%3A43%3A31.7174061Z&urlSigningMethod=HMACV1&urlSignature=fl95dXdfFb1YndQYp1OdBV8sg4IEP%2Fdu9Jw4VQz1%2B8c%3D I'm a bit confused what is really going on here:

2021-05-24T11:30:59.9592849Z Requested golangci-lint 'v1.37', using 'v1.37.1', calculation took 101ms
2021-05-24T11:30:59.9593619Z Installing golangci-lint v1.37.1...
2021-05-24T11:30:59.9594890Z Downloading https://github.com/golangci/golangci-lint/releases/download/v1.37.1/golangci-lint-1.37.1-linux-amd64.tar.gz ...
2021-05-24T11:30:59.9604130Z [command]/usr/bin/tar xz --overwrite --warning=no-unknown-keyword -C /home/runner -f /home/runner/work/_temp/699e2cb2-9f46-4ed3-ba70-55cf1b855115
2021-05-24T11:30:59.9607159Z Installed golangci-lint into /home/runner/golangci-lint-1.37.1-linux-amd64/golangci-lint in 879ms
2021-05-24T11:30:59.9607999Z Received 50331648 of 370718094 (13.6%), 47.8 MBs/sec
2021-05-24T11:31:00.0583863Z Received 180355072 of 370718094 (48.7%), 85.8 MBs/sec
2021-05-24T11:31:01.0577699Z Received 301989888 of 370718094 (81.5%), 95.9 MBs/sec
2021-05-24T11:31:01.8965434Z Received 370718094 of 370718094 (100.0%), 92.0 MBs/sec
2021-05-24T11:31:01.8967722Z Cache Size: ~354 MB (370718094 B)
2021-05-24T11:31:01.9067971Z [command]/usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/ce838b4c-e0e7-4371-b6a5-e0bbc5c6235e/cache.tzst -P -C /home/runner/work/evcc/evcc
2021-05-24T11:31:02.8551876Z /usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/.gitignore: Cannot open: File exists
2021-05-24T11:31:02.8562145Z ##[error]/usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/conn.go: Cannot open: File exists
2021-05-24T11:31:02.8575744Z ##[error]/usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/examples/echo/server.go: Cannot open: File exists
2021-05-24T11:31:02.8578177Z ##[error]/usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/examples/echo/main.go: Cannot open: File exists
2021-05-24T11:31:02.8580108Z /usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/examples/echo/README.md: Cannot open: File exists
2021-05-24T11:31:02.8581576Z ##[error]/usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/examples/echo/server_test.go: Cannot open: File exists
2021-05-24T11:31:02.8583504Z /usr/bin/tar: ../../../go/pkg/mod/nhooyr.io/websocket@v1.8.7/examples/README.md: Cannot open: File exists

It seems that the install actually went fine but there's a follow-on step that breaks?

@seh
Copy link

seh commented Jun 10, 2021

I see similar errors why I try to use this action. Here's a sample of the first two complaints:

  /usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/154ee373-4600-4395-b547-3643dff1d28a/cache.tzst -P -C /home/runner/work/my-repo/my-repo
  Error: /usr/bin/tar: ../../../go/pkg/mod/google.golang.org/grpc@v1.37.0/server.go: Cannot open: File exists
  Error: /usr/bin/tar: ../../../go/pkg/mod/google.golang.org/grpc@v1.37.0/attributes/attributes_test.go: Cannot open: File exists

@seh
Copy link

seh commented Jun 10, 2021

Related issues: #23, #135

@andig
Copy link
Author

andig commented Jun 10, 2021

maintainers: any chance to get this fixed? Has a number of past and current related issues.

updated sorry for the bad @

@seh
Copy link

seh commented Jun 10, 2021

I see previous discussion about not running certain Go toolchain commands ahead of golangci-lint in the same GitHub Actions job. In my case, the only thing I run before this action is go list -find -json ./... in order to ensure that there are actually some Go files available on the current branch. If I try to run golangci-lint when there are no Go files available, it fails, instead of succeeding trivially. When go list either produces no output or its "GoFiles" object field has no entries, I skip running golangci-lint.

@seh
Copy link

seh commented Jun 10, 2021

Oh, I just noticed that this action runs actions/setup-go itself. I was taking care of that myself before running this action, because I need to run go list first as explained earlier.

Is it possible to influence which version of Go this action installs? I don't see mention of such a parameter in the action.yaml file.

@ifree92
Copy link

ifree92 commented Jul 22, 2021

As a "very hot" solution I have just disabled package caching:

skip-pkg-cache: true

taoufik07 added a commit to taoufik07/terraform-provider-gandi that referenced this issue Oct 21, 2021
taoufik07 added a commit to taoufik07/terraform-provider-gandi that referenced this issue Oct 21, 2021
bors bot added a commit to onflow/flow-go that referenced this issue Oct 22, 2021
1509: Update golangci, trying to fix lint r=Kay-Zee a=Kay-Zee

This updates the golangci action to just use the latest v2 action, which is recommended for the action.

Also, skipping the PKG cache because of all the File Exists error. Will have to follow: golangci/golangci-lint-action#244 to see when this is resolved. (though lint is one of the faster jobs, so it's fine even if we skip the cache)

Lastly, I'm suggesting we skip the `unused` lint for TEST FILES ONLY, as there are many cases where we want to use `t.Skip()` to quarantine, but then it causes "unused" errors. This will resolve those cases and better allow us to use `t.Skip()`

Co-authored-by: Kay-Zee <kan@axiomzen.co>
bors bot added a commit to onflow/flow-go that referenced this issue Oct 22, 2021
1509: Update golangci, trying to fix lint r=Kay-Zee a=Kay-Zee

This updates the golangci action to just use the latest v2 action, which is recommended for the action.

Also, skipping the PKG cache because of all the File Exists error. Will have to follow: golangci/golangci-lint-action#244 to see when this is resolved. (though lint is one of the faster jobs, so it's fine even if we skip the cache)

Lastly, I'm suggesting we skip the `unused` lint for TEST FILES ONLY, as there are many cases where we want to use `t.Skip()` to quarantine, but then it causes "unused" errors. This will resolve those cases and better allow us to use `t.Skip()`

Co-authored-by: Kay-Zee <kan@axiomzen.co>
bors bot added a commit to onflow/flow-go that referenced this issue Oct 22, 2021
1509: Update golangci, trying to fix lint r=Kay-Zee a=Kay-Zee

This updates the golangci action to just use the latest v2 action, which is recommended for the action.

Also, skipping the PKG cache because of all the File Exists error. Will have to follow: golangci/golangci-lint-action#244 to see when this is resolved. (though lint is one of the faster jobs, so it's fine even if we skip the cache)

Lastly, I'm suggesting we skip the `unused` lint for TEST FILES ONLY, as there are many cases where we want to use `t.Skip()` to quarantine, but then it causes "unused" errors. This will resolve those cases and better allow us to use `t.Skip()`

Co-authored-by: Kay-Zee <kan@axiomzen.co>
@directionless
Copy link

This appears to still be present in v1.42

My build depends on generated files. As several linters seem to examine the full call tree, they will always fail without the generated content. Creating something of a chicken and egg problem.

@andig
Copy link
Author

andig commented Oct 29, 2021

Looks like this issue has not really caught attention with the maintainers ;)

@SVilgelm
Copy link
Member

SVilgelm commented Nov 1, 2021

there is a special skip-go-installation parameter to use already installed go

      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v2
        with:
          version: latest
          skip-go-installation: true

@SVilgelm
Copy link
Member

SVilgelm commented Nov 1, 2021

This appears to still be present in v1.42

My build depends on generated files. As several linters seem to examine the full call tree, they will always fail without the generated content. Creating something of a chicken and egg problem.

you are right, there are several linters that require whole project, like unused. In this case you need to generate all files first, then run golangci-lint. And I don't see the chicken and egg problem here. Your goal is in checking that code is compilable and is valid. So if you run go generate first and if fails, then it is an expected and valid result of a workflow.
if the files were successfully generated, then linter can do its work

andig added a commit to evcc-io/evcc that referenced this issue Nov 2, 2021
@andig
Copy link
Author

andig commented Nov 2, 2021

there is a special skip-go-installation parameter to use already installed go

      - uses: actions/checkout@v2
      - uses: actions/setup-go@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v2
        with:
          version: latest
          skip-go-installation: true

In evcc-io/evcc@1557bb0 I've added skip-go-installation: true, but build log is still full or errors: https://github.com/evcc-io/evcc/runs/4081084055?check_suite_focus=true, see Lint step.

In fact, the errors are not caused by the Go installation:

  /usr/bin/tar --use-compress-program zstd -d -xf /home/runner/work/_temp/8b8190cf-ca27-4b77-9d43-823bd24af1dd/cache.tzst -P -C /home/runner/work/evcc/evcc
  Error: /usr/bin/tar: ../../../go/pkg/mod/gosrc.io/xmpp@v0.5.1/xmpp_transport.go: Cannot open: File exists
  Error: /usr/bin/tar: ../../../go/pkg/mod/gosrc.io/xmpp@v0.5.1/router.go: Cannot open: File exists

@andig
Copy link
Author

andig commented Nov 2, 2021

@sg0hsmt @abennett any chance to look at this issue?

@SVilgelm
Copy link
Member

SVilgelm commented Nov 2, 2021

I checked the logs, so the linter works fine, no issues at all. The errors are related to usage of cache and I found that you are caching the go folder in another step, so I would recommend you to remove that step, it is not needed: https://github.com/evcc-io/evcc/blob/master/.github/workflows/build.yml#L18-L26

so, your step restores the cache, then you call lint action that tries to restore the cache as well and of course it fails due to the files already exist.

@andig
Copy link
Author

andig commented Nov 2, 2021

Thank you for the comment. I still feel this is bad behaviour.

Use case: My code contained generated files and I'd also like to lint those (might have gotten the templates wrong). I can only generate them when the cache is already populated.
I might also simply chose to run the linter as a later step in the build when the cache would already be populated.

Why does the linter think that it needs to restore the cache? Imho that's not its task and if it's optional behaviour then it would help have control over it?

@SVilgelm
Copy link
Member

SVilgelm commented Nov 2, 2021

have you tried to disable cache?:

          skip-pkg-cache: true
          skip-build-cache: true

the action is designed to be run alone first, since you have more advanced usage you need to tweak the action's and linter's configs accordingly

@andig
Copy link
Author

andig commented Nov 2, 2021

That leads to:

Run golangci/golangci-lint-action@v2
prepare environment
run golangci-lint
  Running [/home/runner/golangci-lint-1.42.1-linux-amd64/golangci-lint run --out-format=github-actions] in [] ...
  level=error msg="Timeout exceeded: try increasing it by passing --timeout option"
  Error: golangci-lint exit with code 4
  Ran golangci-lint in 76614ms

with the manual caching step before looking like this:

- name: Cache Go modules
  uses: actions/cache@v2
  with:
    path: |
      ~/go/pkg/mod
      ~/.cache/go-build
    key: ${{ runner.os }}-${{ matrix.go-version }}-go-${{ hashFiles('**/go.sum') }}
    restore-keys: |
      ${{ runner.os }}-${{ matrix.go-version }}-go-

This cache will of course be cold when the go.sum changes.

I've finally moved the linter to a later position after the first build step and then it succeeds.

@StevenACoffman
Copy link
Contributor

Why can we not add --overwrite to tar?

@SVilgelm
Copy link
Member

Because it is not our code, we use GitHub cache package

@StevenACoffman
Copy link
Contributor

StevenACoffman commented Nov 14, 2021

Looks like someone tried to fix this upstream for this very package here: actions/toolkit#717 but that PR was superseded by actions/toolkit#807 which was merged on May 21, 2021.

However, the last release of @actions/cache was 1.0.7 which was Tuesday, April 13, 2021, prior to the merge of the fix. 😞

As a result, this will not be fixed until a new upstream release is cut and this updates to use it, but no other code needs to be changed.

@andig
Copy link
Author

andig commented Jan 21, 2023

@maintainers could we please reopen? IMHO we have the bad choices of slow CI due to excessive logging or slow CI due do omitting the caches, both are not good options.

sharnoff added a commit to neondatabase/autoscaling that referenced this issue Apr 19, 2023
Currently we have ~50k lines of tar extraction errors with "File exists"
when the golangi-lint-action runs. There doesn't seem to be a clean way
around this, other than disabling its cache.

See also:
golangci/golangci-lint-action#244 (comment)

Example of the offending run:
https://github.com/neondatabase/autoscaling/actions/runs/4740610494/jobs/8416629913
sharnoff added a commit to neondatabase/autoscaling that referenced this issue Apr 19, 2023
Currently we have ~50k lines of tar extraction errors with "File exists"
when the golangi-lint-action runs. There doesn't seem to be a clean way
around this, other than disabling its cache.

See also:
golangci/golangci-lint-action#244 (comment)

Example of the offending run:
https://github.com/neondatabase/autoscaling/actions/runs/4740610494/jobs/8416629913
gzurowski added a commit to gzurowski/gonic that referenced this issue May 18, 2023
@ldez
Copy link
Member

ldez commented Jun 19, 2023

duplicate of #135

@ldez ldez added the duplicate This issue or pull request already exists label Jun 19, 2023
sir-gon pushed a commit to sir-gon/algorithm-exercises-go that referenced this issue Aug 25, 2023
sir-gon pushed a commit to sir-gon/algorithm-exercises-go that referenced this issue Aug 25, 2023
andrew-farries added a commit to xataio/pgroll that referenced this issue Sep 14, 2023
Disabling the cache during the `setup-go` step was probably done to
avoid errors from the `golangci-lint` step
golangci/golangci-lint-action#807,
golangci/golangci-lint-action#244), but the
`setup-go` step isn't actually needed before running `golangci-lint`.
prdoyle added a commit to prdoyle/terraform-provider-bosk that referenced this issue Dec 4, 2023
budimanjojo added a commit to budimanjojo/talhelper that referenced this issue Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: cache duplicate This issue or pull request already exists
Projects
None yet
Development

No branches or pull requests