Significant performance issues with goimports and modules

[rliebz]

Running goimports outside of the GOPATH has a significant performance hit, off by an order of magnitude compared to other linters. No other linter seems to have any significant difference off the GOPATH.

I've run a quick benchmark against this repo after removing .golangci.yml for a clean configuration, with times taken from the -v output. I've gotten similar results after repeating multiple times, and similar results using another repo.

Some numbers, which seem to indicate that only goimports + modules has the issue:

command	on GOPATH	time (seconds)
golangci-lint run -v --enable-all --disable goimports ./...	yes	5.671473167
golangci-lint run -v --disable-all --enable goimports ./...	yes	3.489298274
golangci-lint run -v --enable-all --disable goimports ./...	no	5.844223245
golangci-lint run -v --disable-all --enable goimports ./...	no	46.860324028

1. Version of golangci-lint: golangci-lint --version

$ golangci-lint --version
golangci-lint has version 1.11 built from 17508ab on 2018-10-28T15:21:50Z

2. Go environment: go version && go env

$ go version && go env
go version go1.11.1 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/rliebz/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/rliebz/Projects/gocode"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/rliebz/Projects/tusk/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/wb/_ss422w92c78hrs6vf8456_r0000gn/T/go-build844491563=/tmp/go-build -gno-record-gcc-switches -fno-common"

3. Verbose output of running: golangci-lint run -v --disable-all --enable goimports ./... with no .golangci.yml

INFO [config_reader] Config search paths: [./ /Users/rliebz/Playground/golangci-lint /Users/rliebz/Playground /Users/rliebz /Users /] 
INFO [lintersdb] Active 1 linters: [goimports]    
INFO [loader] Go packages loading at mode load files took 1.090516742s 
INFO [loader/astcache] Parsed AST of all pkg.GoFiles: [/Users/rliebz/Playground/golangci-lint/cmd/golangci-lint/main.go] for 151.122µs 
... (a lot of really fast AST parsing)
INFO [runner/skip dirs] sorted abs args: [/Users/rliebz/Playground/golangci-lint] 
INFO [runner] worker.6 took 12.8µs                
INFO [runner] worker.3 took 3.341µs               
INFO [runner] worker.7 took 4.922µs               
INFO [runner] worker.8 took 23.615µs              
INFO [runner] worker.5 took 26.35µs               
INFO [runner] worker.2 took 47.4µs                
INFO [runner] worker.1 took 56.66µs               
INFO [runner] worker.4 took 45.557783839s with stages: goimports: 45.557748359s 
INFO [runner] Workers idle times: #1: 45.557597123s, #2: 45.557601806s, #3: 45.557703824s, #5: 45.557607879s, #6: 45.557729237s, #7: 45.557666091s, #8: 45.557627599s 
INFO [runner] processing took 9.564µs with stages: nolint: 3.071µs, path_prettifier: 2.688µs, max_same_issues: 1.04µs, skip_dirs: 477ns, autogenerated_exclude: 359ns, skip_files: 264ns, cgo: 262ns, diff: 251ns, exclude: 247ns, max_from_linter: 239ns, max_per_file_from_linter: 186ns, source_code: 181ns, uniq_by_line: 165ns, path_shortener: 134ns 
INFO Memory: 469 samples, avg is 69.7MB, max is 70.1MB 
INFO Execution took 46.860324028s

[jirfag]

hi, thank you for the verbose description!

1. does it reproduce on the repeated run?
2. how much lines of code does project have?
3. how long simple find . -name "*.go" | xargs -t goimports -w work?
4. is it much faster in GOPATH?

[jirfag]

I have reproduced it myself

1. yes
2. 9 kLoC
3. time sh -c 'find . -name "*.go"| fgrep -v vendor | xargs goimports -w' took 43s
4. it's much faster in GOPATH

[jirfag]

It's the issue with goimports.
They already have a WIP patch, I'll look at it and try to merge.

[otto-md]

As part of this commit golang.org/x/tools was updated to 6c7e314b6563. This includes golang/tools@a2dc476 which reverted the very nice improvement to import grouping that was introduced in 0e2be20

Is there any way of getting this feature back? Having working import grouping is more important to me than keeping import comments, which was the reason for the revert upstream.

[jirfag]

yep, it contains bugs (described in golang/go#20818). We could merge it when it would be fixed

[jirfag]

Reverting fix because it's buggy, waiting for a final fix in x/tools

[jirfag]

Merged a new fix for goimports, it should work fast again