make start fails with /usr/local/go/pkg/linux_amd64/os/user.a: permission denied

[yunhailuo]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:
My issue is similar to #1154. I started a new issue because that one is closed and my versions are all new. Please let me know if I shouldn't. This is part of the output from make start

...
writing new private key to 'argo-server.key'
-----
# If we're making a dev build, we build this locally (this will be faster due to existing Go build caches).
if [ true = true ]; then make dist/argoexec-linux-amd64 && mv dist/argoexec-linux-amd64 argoexec; fi
make[1]: Entering directory '/home/yunhai/go/src/github.com/yunhailuo/argo'
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -i -ldflags '-X github.com/argoproj/argo.version=latest -X github.com/argoproj/argo.buildDate=2020-05-26T03:10:40Z -X github.com/argoproj/argo.gitCommit=fa69c1bb7157e19755eea669bf44434e2bedd157 -X github.com/argoproj/argo.gitTreeState=clean -extldflags "-static" -X github.com/argoproj/argo.gitTag=fa69c1bb' -o dist/argoexec-linux-amd64 ./cmd/argoexec
github.com/argoproj/argo/workflow/executor/os-specific
go build net: copying /home/yunhai/.cache/go-build/90/909cb6374a29ecb0678f93b25ad908bc12664d75553c51adbf20ab2508b08209-d: open /usr/local/go/pkg/linux_amd64/net.a: permission denied
go build os/user: copying /home/yunhai/.cache/go-build/14/142a7f8d90f425f0dddc363c6a10966b2ca543622775f3eb4c4e5b154910495b-d: open /usr/local/go/pkg/linux_amd64/os/user.a: permission denied
make[1]: *** [Makefile:232: dist/argoexec-linux-amd64] Error 1
make[1]: Leaving directory '/home/yunhai/go/src/github.com/yunhailuo/argo'
make: *** [Makefile:239: dist/executor-image.latest] Error 2

What you expected to happen:
Build successfully

How to reproduce it (as minimally and precisely as possible):

1. minikube start
2. Fresh clone of the repo
3. make start

Anything else we need to know?:
N/A

Environment:

• Argo version:

$ argo version
argo: v2.8.0
  BuildDate: 2020-05-11T22:55:16Z
  GitCommit: 8f696174746ed01b9bf1941ad03da62d312df641
  GitTreeState: clean
  GitTag: v2.8.0
  GoVersion: go1.13.4
  Compiler: gc
  Platform: linux/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2020-05-20T12:52:00Z"
  compiler: gc
  gitCommit: 2e7996e3e2712684bc73f0dec0200d64eec7fe40
  gitTreeState: clean
  gitVersion: v1.18.3
  goVersion: go1.13.9
  major: "1"
  minor: "18"
  platform: linux/amd64
serverVersion:
  buildDate: "2020-04-16T11:48:36Z"
  compiler: gc
  gitCommit: 52c56ce7a8272c798dbc29846288d7cd9fbae032
  gitTreeState: clean
  gitVersion: v1.18.2
  goVersion: go1.13.9
  major: "1"
  minor: "18"
  platform: linux/amd64

• Linux version

Ubuntu 20.04 LTS

• Go version

$ go version
go version go1.14.3 linux/amd64

• Kubernetes cluster

$ virtualbox --help
Oracle VM VirtualBox VM Selector v6.1.6_Ubuntu
(C) 2005-2020 Oracle Corporation
All rights reserved.

No special options.

If you are looking for --startvm and related options, you need to use VirtualBoxVM.

$ minikube version
minikube version: v1.10.1
commit: 63ab801ac27e5742ae442ce36dff7877dcccb278

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

we develop on OS-X, so it looks like the Makefile does not work correctly for Linux - would you like to submit a PR to fix this?

[yunhailuo]

we develop on OS-X, so it looks like the Makefile does not work correctly for Linux - would you like to submit a PR to fix this?

Yes, that will be my pleasure. But I don't really know how to fix this in the first place. Could you give some guidance?

I Googled a little and couldn't figure out. I found this most relevant: golang/go#24674 (comment) but I couldn't understand why it needs to write (the directory and files do have read permission open).

[alexec]

@MrXinWang - you run Linux I think? Would you be able to help please?

[MrXinWang]

@alexec Yes of course. I will try to reproduce this issue today.

[yunhailuo]

Hi @MrXinWang , thank you for offering to help.

What I found is golang/go#37962 (comment)
Based on Jay's comment, I removed -i from all go build commands in Makefile. My build seems to go through though I'm not 100% sure:

1. I have Argo on https://localhost:2746. But for some reasons I can't get token

   FATA[0000] could not find a token
   

2. I have MinIO on http://localhost:9000 and can get in use admin/password

Is removing -i a right patch here?

[MrXinWang]

Hi @alexec ! I tried to reproduce this on my machine and it seems there isn't any problems in the Makefile, as I did not met the error mentioned by @yunhailuo. It seems that his error occured in building dist/argoexec-linux-amd64 binary and it seems I can build on my machine.

@yunhailuo It seems that your golang might be the problem, as you also mentioned in your comment. I would suggest you completely remove golang and install it again? I will attach what I did below:

My test cluster is two x86_64 completely newly started ubuntu 18.04 virtual machines with newly installed docker 19.03.9, k8s 1.14.9 (CNI is calico), golang 1.14.3.

Then I just run make cli && make build && make start on the master node.

And speaking of how to install golang:

# Remove old golang, for me it is: rm -rf /usr/local/go 
wget https://dl.google.com/go/go1.14.3.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.14.3.linux-amd64.tar.gz

# Add these lines to your $HOME/.bashrc
export PATH=$PATH:/usr/local/go/bin/:/go/bin
export GOPATH=/go
export GOROOT=/usr/local/go

source $HOME/.bashrc

Another suggestion is to use the root user? or run sudo make build && sudo make start? As I used the root user in my VM...

Is removing -i a right patch here?

No I don't think so (as I didn't do anything with the Makefile and all binaries can still be built), @alexec can you please confirm?

[yunhailuo]

Thank you @MrXinWang . I think the root privilege is the only solution so far. With my previous settings, sudo make start seems to go through fine. With your setting, I have to build as root user; otherwise it will fail pretty early since you have /go/bin under root. It makes sense since I hit permission denied errors.

I was trying to follow the contributor seminar on YouTube and didn't notice Alex make as root user. Now that I watch it again, it might be he installed GO under his home directory instead of /usr/local/go thus he did have write privilege to GOROOT? Reading through some issues and comments, I was wondering if root user privilege or writing to GOROOT are right thing to do. Maybe it is indeed the right way to go here.

[MrXinWang]

@yunhailuo That absolutely makes sense. Congratulations.

I would say it is definitely possible to change the value of GOROOT and GOPATH...and of course you can put it under your home directory where you have privilege to access anything. However I did not tried before so I am also not sure. As long as all environment variables and (symbolic links) are configured correctly, there shouldn't be any problems.

Also the reason why I put everything like above is that I just simply followed the tutorial in https://golang.org/doc/install#tarball, no any other special reasons...

[yunhailuo]

Thank you @MrXinWang . I think it is clear and all worked out. I still couldn't get token. I'll close this one and open another issue if needed.

[yunhailuo]

In case this matters, the reason I got

FATA[0000] could not find a token

is I used minikube and by default the minikube user has no authentication set up. To fix this I did the following two and redo a clean start. Both of them works:

1. kubectl config set-credentials minikube --username=minikube --password=password
2. kubectl config set-credentials minikube --token="Basic bWluaWt1YmU6cGFzc3dvcmQ="

[Bobgy]

For anyone else bumping into this, I managed to avoid the permission issue by installing go in $HOME/bin/go, rather than /usr/local/go.