Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git.apache.org down, use github for mods #9990

Closed

Conversation

mikecook
Copy link

@mikecook mikecook commented Sep 4, 2019

Community Note

  • Please vote on this pull request by adding a 👍 reaction to the original pull request comment to help the community and maintainers prioritize this request
  • Please do not leave "+1" comments, they generate extra noise for pull request followers and do not help prioritize the request

Relates hashicorp/terraform#22664

@mikecook mikecook requested a review from a team September 4, 2019 19:08
@ghost ghost added the size/XS Managed by automation to categorize the size of a PR. label Sep 4, 2019
@bflad
Copy link
Contributor

bflad commented Sep 4, 2019

Hi @mikecook 👋 Can you confirm that setting the GOPROXY=https://proxy.golang.org environment variable also works around this issue?

@bflad bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Sep 4, 2019
@gcase555
Copy link

gcase555 commented Sep 4, 2019

Its worth mentioning that the official documentation points people to pull the dependency from github, not the apache mirror https://thrift.apache.org/lib/go#using-thrift-with-go

@bflad I think this approach is more aligned with what is officially recommended

@ghost ghost removed the waiting-response Maintainers are waiting on response from community or contributor. label Sep 4, 2019
@mikecook mikecook force-pushed the apache_down_use_github branch from 36c48fe to eb1d8b7 Compare September 5, 2019 17:59
@mikecook
Copy link
Author

mikecook commented Sep 5, 2019

@bflad If you're using go modules then there's no good solution without this patch.

go1.12.9 GO111MODULE=on GOPROXY="https://proxy.golang.org" make build doesn't work. (410)
go1.12.9 GO111MODULE=on GOPROXY="https://proxy.golang.org,direct" make build doesn't work. (cannot have comma)
go1.13.0 GO111MODULE=on GOPROXY="https://proxy.golang.org,direct" make build doesn't work. (golangci-lint deps do not match version-control timestamp)

Deepdive below

Note: Output is dockerized for repeatability, scroll right to see the full command lines to duplicate. All use GO111MODULE=on

go1.12.9 (the latest 1.12.x) does not (and will not) support a goproxy comma delimited list.
go1.13.0 (the latest 1.13.x) does support a goproxy comma delimited list.
golang/go#26334 (where they add goproxy list support for 1.13.x and confirm they will not add goproxy list support for 1.12.x)

Without a comma delimited list that contains direct, as soon as https://proxy.golang.org/ responds 404/410 for any dep that it doesn't have yet, a build will fail.

go1.12.9, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.12.9-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on ; export GOPROXY="https://proxy.golang.org" ; go env | grep "GOPROXY" ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ export 'GOPROXY=https://proxy.golang.org'
+ grep GOPROXY
+ go env
GOPROXY="https://proxy.golang.org"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: finding github.com/kubernetes-sigs/aws-iam-authenticator v0.3.1-0.20181019024009-82544ec86140
go: finding github.com/golangci/gosec v0.0.0-20180901114220-8afd9cbb6cfb
<snip>
go: finding golang.org/x/net v0.0.0-20190213061140-3a22650c66bd
go: github.com/terraform-providers/terraform-provider-template@v2.1.2+incompatible: unexpected status (https://proxy.golang.org/github.com/terraform-providers/terraform-provider-template/@v/v2.1.2+incompatible.info): 410 Gone
go: error loading module requirements
make: *** [GNUmakefile:10: build] Error 1

With a comma delimited list (just to see what happens), on go1.12.x, the build will fail: cannot have comma.

go1.12.9, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org,direct"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.12.9-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on ; export GOPROXY="https://proxy.golang.org,direct" ; go env | grep "GOPROXY" ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ export 'GOPROXY=https://proxy.golang.org,direct'
+ go env
+ grep GOPROXY
GOPROXY="https://proxy.golang.org,direct"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: github.com/hashicorp/go-version@v1.2.0: invalid $GOPROXY setting: cannot have comma
<snip>
go: github.com/terraform-providers/terraform-provider-template@v2.1.2+incompatible: invalid $GOPROXY setting: cannot have comma
go: error loading module requirements
make: *** [GNUmakefile:10: build] Error 1

In go1.13.0 "The default setting for GOPROXY is "https://proxy.golang.org,direct", which means to try the Go module mirror run by Google and fall back to a direct connection if the proxy reports that it does not have the module (HTTP error 404 or 410)." (quoted from: https://tip.golang.org/cmd/go/#hdr-Module_downloading_and_verification )

But go1.13.x goproxy can't seem to manage to work for golangci-lint golangci/golangci-lint#595 (Issue has pivoted over the months from 410 to does not match version-control timestamp)

go1.13.0, aws provider v2.26.0 without patch, GOPROXY="https://proxy.golang.org,direct"

$ docker run --rm -it -v `pwd`:/go/src/github.com/terraform-providers/terraform-provider-aws -w=/go/src/github.com/terraform-providers/terraform-provider-aws golang:1.13.0-alpine3.10 sh -c "apk add --no-cache make git bash; set -x ; git clean -fx ; git checkout -f v2.26.0 ; export GO111MODULE=on  ; go env | grep GOPROXY ; make build"
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
<snip>
OK: 30 MiB in 26 packages
+ git clean -fx
+ git checkout -f v2.26.0
HEAD is now at f3ad9853b v2.26.0
+ export 'GO111MODULE=on'
+ go env
+ grep GOPROXY
GOPROXY="https://proxy.golang.org,direct"
+ make build
==> Checking that code complies with gofmt requirements...
go install
go: github.com/golangci/golangci-lint@v1.17.1 requires
	github.com/go-critic/go-critic@v0.0.0-20181204210945-1df300866540: invalid pseudo-version: does not match version-control timestamp (2019-05-26T07:48:19Z)
make: *** [GNUmakefile:10: build] Error 1

Which leaves no good option besides this patch to replace git.apache.org with apache's github.

@mikecook mikecook force-pushed the apache_down_use_github branch from eb1d8b7 to 4595e09 Compare September 5, 2019 18:22
@ghost ghost added size/M Managed by automation to categorize the size of a PR. and removed size/XS Managed by automation to categorize the size of a PR. labels Sep 5, 2019
@bflad
Copy link
Contributor

bflad commented Sep 5, 2019

Regarding Go 1.13, we do not currently support that version yet for development and testing with this project, especially due to the the golangci-lint situation. See #9992 for updates there. 👍 We are hoping they fix their Go module and cut a bug fix release within a reasonable timeframe before attempting any workarounds in our Go modules setup.

One important note here is that our Makefile does not currently force usage of -mod=vendor even though we have everything vendored in the repository. Given that, setting GOFLAGS=-mod=vendor may also help in your build environment so it uses the local vendored files instead of reaching out at all.

@bflad
Copy link
Contributor

bflad commented Sep 6, 2019

This situation should be resolved through a chain of dependency updates starting with hashicorp/go-getter#204 then hashicorp/terraform then here. 😄

bflad added a commit that referenced this pull request Sep 9, 2019
…2c24e25f

Reference: #9990
Reference: #9991

Updated via:

```console
$ go get github.com/hashicorp/terraform@5fb1e0867836ff6f6890fd1f02c7a1342c24e25f
$ go mod tidy
$ go mod vendor
```
@bflad
Copy link
Contributor

bflad commented Oct 8, 2019

Hi folks 👋 Apologies for the delayed response here -- the transitive dependency on thrift (git.apache.org/thrift and github.com/apache/thrift) was removed in #10367. Thank you for contributing this when it was really problematic before.

@bflad bflad closed this Oct 8, 2019
@ghost
Copy link

ghost commented Nov 8, 2019

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Nov 8, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
size/M Managed by automation to categorize the size of a PR.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants