[0.9] blob download progress broken #2248

tonistiigi · 2021-07-13T05:46:35Z

Looks like after #2203 the image pulls do not show progress output anymore when layer blobs are pulled.

Before:

 => [git 1/2] FROM docker.io/library/alpine:3.14@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0                                              0.4s
 => => resolve docker.io/library/alpine:3.14@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0                                                  0.0s
 => => sha256:58ab47519297212468320b23b8100fc1b2b96e8d342040806ae509a778a0a07a 2.71MB / 2.71MB                                                                        0.3s
 => => extracting sha256:58ab47519297212468320b23b8100fc1b2b96e8d342040806ae509a778a0a07a                                                                             0.1s
 => [buildkit-export 1/3] FROM docker.io/tonistiigi/alpine:3.14@sha256:9447926dd3a4312576b65d3ec9eb4a3b0219614afad538c2bb30c5d2b3c2ac2c                               0.0s
 => => resolve docker.io/tonistiigi/alpine:3.14@sha256:9447926dd3a4312576b65d3ec9eb4a3b0219614afad538c2bb30c5d2b3c2ac2c                                               0.0s
 => [golatest 1/1] FROM docker.io/library/golang:1.16-alpine@sha256:84b4f06d7246ad4e87293cf2c1d4b34802621c99c49456a926463b805429f0ee                                  3.0s
 => => resolve docker.io/library/golang:1.16-alpine@sha256:84b4f06d7246ad4e87293cf2c1d4b34802621c99c49456a926463b805429f0ee                                           0.0s
 => => sha256:5f19e5d43a5efe6c746a1cfa2fcf1862bcc4925567d86a5c59e84ff2e1e1e9ff 155B / 155B                                                                            0.2s
 => => sha256:8ec328162280f19fa491a9223100438095ce2a77e7b9281df21e7be3bd6a7bc4 101.14MB / 101.14MB                                                                    3.0s
 => => sha256:705bf5f5ed4dc23e2dc33456ba3ccbef3ecabf22614e760a8f4c1e3a93a6c59b 154B / 154B                                                                            0.3s
 => => sha256:f7ad96702509b3443881ad70a623b6184f9936c6b4cd986694020c95f3a4441c 281.67kB / 281.67kB                                                                    0.3s
 => => extracting sha256:f7ad96702509b3443881ad70a623b6184f9936c6b4cd986694020c95f3a4441c                                                                             0.0s
 => => extracting sha256:705bf5f5ed4dc23e2dc33456ba3ccbef3ecabf22614e760a8f4c1e3a93a6c59b                                                                             0.0s
 => => extracting sha256:8ec328162280f19fa491a9223100438095ce2a77e7b9281df21e7be3bd6a7bc4                                                                             0.0s
 => FROM docker.io/tonistiigi/binfmt:buildkit@sha256:81a03e6630e9c39df109bf24ae8c807881c4fd1703084827d855f8093cc7ab7a                                                 1.1s
 => => resolve docker.io/tonistiigi/binfmt:buildkit@sha256:81a03e6630e9c39df109bf24ae8c807881c4fd1703084827d855f8093cc7ab7a                                           0.8s
 => => sha256:3938f66b01d0bdf237060e40ebc3345a553f0be16cbc45a5eb35f391a6280bf5 10.59MB / 10.59MB                                                                      0.8s
 => => extracting sha256:3938f66b01d0bdf237060e40ebc3345a553f0be16cbc45a5eb35f391a6280bf5                                                                             0.3s
 => [xx 1/1] FROM docker.io/tonistiigi/xx@sha256:1e96844fadaa2f9aea021b2b05299bc02fe4c39a92d8e735b93e8e2b15610128                                                     0.2s
 => => resolve docker.io/tonistiigi/xx@sha256:1e96844fadaa2f9aea021b2b05299bc02fe4c39a92d8e735b93e8e2b15610128                                                        0.0s
 => => sha256:2d0d8fa641cc2f38ede622ec565c4f557f6c6bea57e50f223d0fd557a0f68e03 13.40kB / 13.40kB                                                                      0.2s

After:

 => [buildkit-export 1/3] FROM docker.io/tonistiigi/alpine:3.14@sha256:9447926dd3a4312576b65d3ec9eb4a3b0219614afad538c2bb30c5d2b3c2ac2c                               0.0s
 => => resolve docker.io/tonistiigi/alpine:3.14@sha256:9447926dd3a4312576b65d3ec9eb4a3b0219614afad538c2bb30c5d2b3c2ac2c                                               0.0s
 => [git 1/2] FROM docker.io/library/alpine:3.14@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0                                              0.0s
 => => resolve docker.io/library/alpine:3.14@sha256:234cb88d3020898631af0ccbbcca9a66ae7306ecd30c9720690858c1b007d2a0                                                  0.0s
 => FROM docker.io/tonistiigi/binfmt:buildkit@sha256:81a03e6630e9c39df109bf24ae8c807881c4fd1703084827d855f8093cc7ab7a                                                 0.0s
 => => resolve docker.io/tonistiigi/binfmt:buildkit@sha256:81a03e6630e9c39df109bf24ae8c807881c4fd1703084827d855f8093cc7ab7a                                           0.9s
 => [xx 1/1] FROM docker.io/tonistiigi/xx@sha256:1e96844fadaa2f9aea021b2b05299bc02fe4c39a92d8e735b93e8e2b15610128                                                     0.0s
 => => resolve docker.io/tonistiigi/xx@sha256:1e96844fadaa2f9aea021b2b05299bc02fe4c39a92d8e735b93e8e2b15610128                                                        0.0s
 => [golatest 1/1] FROM docker.io/library/golang:1.16-alpine@sha256:84b4f06d7246ad4e87293cf2c1d4b34802621c99c49456a926463b805429f0ee                                  0.0s
 => => resolve docker.io/library/golang:1.16-alpine@sha256:84b4f06d7246ad4e87293cf2c1d4b34802621c99c49456a926463b805429f0ee                                           0.0s
 => [auth] tonistiigi/binfmt:pull token for registry-1.docker.io                                                                                                      0.0s

PTAL @coryb @sipsma

The text was updated successfully, but these errors were encountered:

fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com>

* Remove duplicate "not found" from some error messages I noticed this when building a Dockerfile that failed because a file didn't exist, so went through error messages that looked like they had a duplicate "not found" in the output; [+] Building 0.9s (6/9) => [internal] load build definition from Dockerfile 0.2s => => transferring dockerfile: 306B 0.0s => [internal] load .dockerignore 0.1s => => transferring context: 2B 0.0s => [internal] load metadata for docker.io/library/alpine:latest 0.0s => CACHED [1/5] FROM docker.io/library/alpine 0.0s => [internal] load build context 0.6s => => transferring context: 701B 0.5s => ERROR [2/5] ADD no-such-file.txt / 0.0s ------ > [2/5] ADD no-such-file.txt /: ------ failed to compute cache key: "/no-such-file.txt" not found: not found Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * git: support subdir component Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update SSH agent socket handling to support Windows OpenSSH using named pipes. Signed-off-by: Siebe Schaap <siebe@digibites.nl> * Update Windows named pipe handling for SSH forwarding to use x/sys/windows rather than syscall. Signed-off-by: Siebe Schaap <siebe@digibites.nl> * Rename SSH agent forwarding helper parsePlatformSocketPath() to getWindowsPipeDialer(). Signed-off-by: Siebe Schaap <siebe@digibites.nl> * Fix regular expression to test for Windows named pipe in SSH agent path; allowed backslashes in host. Signed-off-by: Siebe Schaap <siebe@digibites.nl> * update to go 1.16 This updates all occurrences of Go 1.13 to Go 1.16; also updated the code that's used to redact credentials in URLs to use the Go implementation. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * go.mod: update docker and docker cli to v20.10.7 - pkg/signal: ignore SIGURG on all platforms - pkg/archive: use v2 capabilities in layer archives - update ishidawataru/sctp to fix possible socket leak when bind fails docker/cli: - config: print deprecation warning when falling back to ~/.dockercfg Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * hack: update proto generators for arm64 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * test: fix mirror cache handling Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * cache: fix possible nil dereferences Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * move RegistryConfig to resolver package This allows using the resolver package without having to import the buildkit daemon configuration. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * vendor: fix broken gosum Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add hack/shell helper for dev shell environment Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * session: make sure all token request keep correct context Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * filesync: ensure sendclose is always called Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable stargz snapshotter to use session for snapshot auth Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Implement low-level parser primitives for heredocs This provides the basic functionality for the parser to recognize and parse provided heredocs in supported commands. Signed-off-by: Justin Chadwell <me@jedevc.com> * Add support for parsing heredocs in ADD/COPY and RUN Signed-off-by: Justin Chadwell <me@jedevc.com> * Integrate heredoc support into ADD/COPY and RUN This modifies the command structures to support inline files, as well as provides the logic to compile them down into appropriate LLB definitions. Signed-off-by: Justin Chadwell <me@jedevc.com> * Add integration tests for dockerfile heredocs Signed-off-by: Justin Chadwell <me@jedevc.com> * Update docker/docker to master / v21.xx-dev (post libnetwork integration) full diff: https://github.com/docker/docker/compare/v20.10.7..0ad2293d0e5b This applies the same / similar local changes to buildkit as were made in the branch that moby is currently vendoring: moby/buildkit@244e8cd...cpuguy83:update_libnetwork_import Unfortunately, this (again) requires a "replace" rule (probably until we tagged a release with this change). Go mod refuses to pick a version from master, and (without the replace rule), does weird, reverting docker to a very old version, predating the sirupsen rename: > [vendored 4/4] RUN --mount=target=/src,rw --mount=target=/go/pkg/mod,type=cache go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out: #10 0.500 go: finding github.com/docker/docker 8dbd90ec00daa26dc45d7da2431c965dec99e8b4 #10 0.599 warning: ignoring symlink /src/examples/kube-consistent-hash #10 1.027 go: github.com/moby/buildkit/util/testutil/integration imports #10 1.027 github.com/docker/docker/testutil/daemon imports #10 1.027 github.com/docker/docker/opts imports #10 1.027 github.com/docker/libnetwork/ipamutils imports #10 1.027 github.com/docker/libnetwork/osl imports #10 1.027 github.com/Sirupsen/logrus: github.com/Sirupsen/logrus@v1.8.1: parsing go.mod: #10 1.027 module declares its path as: github.com/sirupsen/logrus #10 1.027 but was required as: github.com/Sirupsen/logrus ------ error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out]: exit code: 1 make: *** [vendor] Error 1 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Fix doc about Stargz Snapshotter and registry authentication Signed-off-by: ktock <ktokunaga.mail@gmail.com> * dockerfile: add documentation for here-docs Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add differ support for local source Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: don't allow content diff for now Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Fix file modes with remote ADD commands Signed-off-by: Justin Chadwell <me@jedevc.com> * cache: avoid concurrent maps write on prune remove() needs to be called while holding the manager lock Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * metadata: hold lock on storageitem update The locks usage is mixed up because two locks separate locks are actually needed. With a specific lock, calls to SetValue can be protected. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Dockerfile: install fuse-overlayfs from apk fuse-overlayfs is available as an apk since Alpine 3.14. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * vendor: update vt100 to fork and remove replace rule Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add support for opentelemetry Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * replace opentracing with opentelemetry tracers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * hack: avoid updating generated proto version This can be done as a separate change when needed. Also should analyze if this would affect the gogo incompatibility issues with newer proto. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * session: avoid tracing health checkpoint Upstream fixes needed for cleaner solution Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * disable otel error logging Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * correctly validate span from context Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: ensure config resolve errors keep source location Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove go fork for building windows/arm64 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Dockerfile: remove nsswitch as default in go1.16 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * retryhandler: use net.ErrClosed for error check Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * replace uses of deprecated containerd/sys.RunningInUserNS() This utility was moved to a separate package, which has no dependencies. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * integration: add common context base to all integration tests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: update opentelemetry to 1.0.0-rc Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add current tracing context detection and exec propagation Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit bc9a83144c83e9fd78007b7bfe92e8082c59d40e) * add transform package to convert from otlp Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: add delegated exporter Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * otlgrpc: provide a way to get otlp client from grpc conn Hopefully this can be removed with a future upstream change that could make this configurable. The package also needs internal dependency that is copied in. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable collecting traces via control api Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * support collecting traces from llb.Exec Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * client: pass delegated exporter as parameter Avoid client package having dependency on global detect package. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update runc binary to v1.0.0 GA Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * handle unconfigured spans without errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to vertex and validate Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to async llb Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: ensure meta resolver uses platform form constraints Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * flightcontrol: reduce contention between goroutines Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Avoid nil pointer dereference when copying from image with no layers Fix this panic when copying from an image with no layers: ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xdd8c17] goroutine 326 [running]: github.com/moby/buildkit/cache/contenthash.(*cacheManager).Checksum(0xc0005ec030, 0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:95 +0x37 github.com/moby/buildkit/cache/contenthash.Checksum(0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:59 +0xd5 github.com/moby/buildkit/solver/llbsolver.NewContentHashFunc.func1.1(0x0, 0x4425d6) /src/solver/llbsolver/result.go:59 +0x20a golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00056a360, 0xc000594510) /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59 created by golang.org/x/sync/errgroup.(*Group).Go /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66 ``` When the path is "/", we allow it because it's a noop. Based on moby#2185 Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add test for copying from scratch Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Check that scratch is mounted as empty dir Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Make error message consistent when layer is empty Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Test with tonistiigi/test:nolayers as well Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * ensure containerd io is complete and closed before returning Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.Controller should own the progress.Writer to prevent leaks Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.FromContext returns a writer factory this allows progress.Controller to manage the writer lifecycle Signed-off-by: Cory Bennett <cbennett@netflix.com> * contenthash: use SeekLowerBound to seek radix tree Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: fix git version detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add support for heredocs with ONBUILD Signed-off-by: Justin Chadwell <me@jedevc.com> * dockerfile: use none differ for dockerfile/dockerignore This avoids wrong metadata matches on small files Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * progressui: print logs for failed step as summary in plain mode Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * grpcerrors: avoid rpc error wrapping in error messages Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * exec: improve error message on exec errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Improve heredoc parsing to allow more generic words Previously, heredoc names were restricted to simple alphanumeric strings. However, heredocs should support much more complex use-cases, including quoting anywhere, as well as allowing special symbols like `.` for easily expressing file extensions. This patch adds support for these more complex cases, by using the shell lexer to parse each heredoc name. Additionally, we include improvements to the lexer to optionally preserve escape tokens to avoid problems when lexing words that have already been lexed before. Signed-off-by: Justin Chadwell <me@jedevc.com> * Improve progress and history messages for heredoc-related commands Signed-off-by: Justin Chadwell <me@jedevc.com> * Remove unneeded Finalize method from ImmutableRef. Finalize was only used outside the cache package in one place, which called it with the commit arg set to false. The code path followed when commit==false turned out to essentially be a no-op because it set "retain cache" to true if it was already set to true. It was thus safe to remove the only external call to it and remove it from the interface. This should be helpful for future efforts to simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist due to the "lazy commit" feature that Finalize is tied into. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Fix ref leak if fileop ref fails to mount. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * add error suggest pkg Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: suggest mistyped flag names Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: provide suggestions for mount options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: add tests for error suggestions Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: remove unnecessary error wrappings Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable riscv64 build Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update QEMU emulators Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * dockerfile: move run network to stable channel Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Automatically detect default git branch Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked. Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Moved getDefaultBranch function to gitsource It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Fix tests Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * git: fix default branch detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable to forcefully specify compression type Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Add full timestamp to logs Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Remove meaningless encode Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Ignore missing providers for blobs w/ same chainid. GetByBlob checks to see if there are any other blobs with the same (uncompressed) ChainID and, if so, reuses their unpacked snapshot if it exists. The problem is if this code finds a match, it was trying to get the matching record, but couldn't do so when the match is lazy because the caller doesn't necessarily have descriptor handlers setup for it. This commit changes the behavior to just ignore any match with the same ChainID that's also lazy as they just aren't usable for the snapshot-reuse optimization. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * authprovider: handle eaccess on storing token seeds Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * log with traceID and spanID Signed-off-by: Morlay <morlay.null@gmail.com> * github: update CI buildkit to v0.9.0-rc1 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * initial version of github cache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: add goactionscache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * caps: add cap for gha cache backend Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove tracetransform package Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: increase default idle conns reuse The current default were even lower than stdlib defaults. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * refactor to use util/bklog instead of using logurs directly Signed-off-by: Morlay <morlay.null@gmail.com> * GitHub Actions cache docs Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Skips getting UID/GUID if passwd/group file is not found When running a WORKDIR instruction, buildkit will create that folder and chown it to the currently set user. For this, it will try to read the /etc/passwd file to get the proper UID, and if that user is not found in the file, the root user will be considered as the owner. However, Windows image do not have that file, which will result in an error while building the image. We can consider not finding the /etc/passwd file as the same as not finding the user in the file, which would solve this issue. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com> * add per domain semaphore to limit concurrent connections This is a safer alternative until we figure out why http.Transport based limiting fails. Some connections like cache export/import do not have a domain key atm and these connections use global pool. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update to github.com/containerd/containerd v1.5.3 Signed-off-by: coryb <cbennett@netflix.com> * vendor: update go-actions-cache with custom client support Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: update to otelhttp roundtripper Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enhance test matrix Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * fix dropped pull progress output due to canceled context fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com> * Add span for layer export This can be a significant amount of time that isn't currently accounted for in traces. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * new implementation for limiting tcp connections The previous implementation had many issues. Eg. on fetch, even if the data already existed and no remote connections were needed the request would still be waiting in the queue. Or if two fetches of same blob happened together they would take up two places in queue although there was only one remote request. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * limited: allow extra high-priority connection for json requests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * ensure wrappers support seeking to continue partial downloads Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * contentutil: change offset to int64 to simplify Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Exporter config digest typo Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * daemonless: wait for daemon to finish before exit Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * github: update CI buildkit to v0.9.0 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add docs for new config options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add ktock and crazy-max to maintainers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update Dockerfile references to use 1.3 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * docs: update images-readme to v0.9 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Bump to codecov/codecov-action v2 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * wip Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com> Co-authored-by: Siebe Schaap <siebe@digibites.nl> Co-authored-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> Co-authored-by: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com> Co-authored-by: ktock <ktokunaga.mail@gmail.com> Co-authored-by: Justin Chadwell <me@jedevc.com> Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Aaron Lehmann <alehmann@netflix.com> Co-authored-by: Cory Bennett <cbennett@netflix.com> Co-authored-by: Erik Sipsma <erik@sipsma.dev> Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com> Co-authored-by: Levi Harrison <levisamuelharrison@gmail.com> Co-authored-by: masibw <masi19bw@gmail.com> Co-authored-by: Morlay <morlay.null@gmail.com> Co-authored-by: CrazyMax <1951866+crazy-max@users.noreply.github.com> Co-authored-by: Claudiu Belu <cbelu@cloudbasesolutions.com>

* hack: update proto generators for arm64 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * test: fix mirror cache handling Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * cache: fix possible nil dereferences Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * move RegistryConfig to resolver package This allows using the resolver package without having to import the buildkit daemon configuration. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * vendor: fix broken gosum Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add hack/shell helper for dev shell environment Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * session: make sure all token request keep correct context Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * filesync: ensure sendclose is always called Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable stargz snapshotter to use session for snapshot auth Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Implement low-level parser primitives for heredocs This provides the basic functionality for the parser to recognize and parse provided heredocs in supported commands. Signed-off-by: Justin Chadwell <me@jedevc.com> * Add support for parsing heredocs in ADD/COPY and RUN Signed-off-by: Justin Chadwell <me@jedevc.com> * Integrate heredoc support into ADD/COPY and RUN This modifies the command structures to support inline files, as well as provides the logic to compile them down into appropriate LLB definitions. Signed-off-by: Justin Chadwell <me@jedevc.com> * Add integration tests for dockerfile heredocs Signed-off-by: Justin Chadwell <me@jedevc.com> * Update docker/docker to master / v21.xx-dev (post libnetwork integration) full diff: https://github.com/docker/docker/compare/v20.10.7..0ad2293d0e5b This applies the same / similar local changes to buildkit as were made in the branch that moby is currently vendoring: moby/buildkit@244e8cd...cpuguy83:update_libnetwork_import Unfortunately, this (again) requires a "replace" rule (probably until we tagged a release with this change). Go mod refuses to pick a version from master, and (without the replace rule), does weird, reverting docker to a very old version, predating the sirupsen rename: > [vendored 4/4] RUN --mount=target=/src,rw --mount=target=/go/pkg/mod,type=cache go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out: #10 0.500 go: finding github.com/docker/docker 8dbd90ec00daa26dc45d7da2431c965dec99e8b4 #10 0.599 warning: ignoring symlink /src/examples/kube-consistent-hash #10 1.027 go: github.com/moby/buildkit/util/testutil/integration imports #10 1.027 github.com/docker/docker/testutil/daemon imports #10 1.027 github.com/docker/docker/opts imports #10 1.027 github.com/docker/libnetwork/ipamutils imports #10 1.027 github.com/docker/libnetwork/osl imports #10 1.027 github.com/Sirupsen/logrus: github.com/Sirupsen/logrus@v1.8.1: parsing go.mod: #10 1.027 module declares its path as: github.com/sirupsen/logrus #10 1.027 but was required as: github.com/Sirupsen/logrus ------ error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c go mod tidy && go mod vendor && mkdir /out && cp -r go.mod go.sum vendor /out]: exit code: 1 make: *** [vendor] Error 1 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Fix doc about Stargz Snapshotter and registry authentication Signed-off-by: ktock <ktokunaga.mail@gmail.com> * dockerfile: add documentation for here-docs Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add differ support for local source Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: don't allow content diff for now Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Fix file modes with remote ADD commands Signed-off-by: Justin Chadwell <me@jedevc.com> * cache: avoid concurrent maps write on prune remove() needs to be called while holding the manager lock Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * metadata: hold lock on storageitem update The locks usage is mixed up because two locks separate locks are actually needed. With a specific lock, calls to SetValue can be protected. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Dockerfile: install fuse-overlayfs from apk fuse-overlayfs is available as an apk since Alpine 3.14. Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * vendor: update vt100 to fork and remove replace rule Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add support for opentelemetry Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * replace opentracing with opentelemetry tracers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * hack: avoid updating generated proto version This can be done as a separate change when needed. Also should analyze if this would affect the gogo incompatibility issues with newer proto. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * session: avoid tracing health checkpoint Upstream fixes needed for cleaner solution Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * disable otel error logging Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * correctly validate span from context Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: ensure config resolve errors keep source location Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove go fork for building windows/arm64 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Dockerfile: remove nsswitch as default in go1.16 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * retryhandler: use net.ErrClosed for error check Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * replace uses of deprecated containerd/sys.RunningInUserNS() This utility was moved to a separate package, which has no dependencies. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * integration: add common context base to all integration tests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: update opentelemetry to 1.0.0-rc Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add current tracing context detection and exec propagation Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit bc9a83144c83e9fd78007b7bfe92e8082c59d40e) * add transform package to convert from otlp Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: add delegated exporter Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * otlgrpc: provide a way to get otlp client from grpc conn Hopefully this can be removed with a future upstream change that could make this configurable. The package also needs internal dependency that is copied in. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable collecting traces via control api Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * support collecting traces from llb.Exec Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * client: pass delegated exporter as parameter Avoid client package having dependency on global detect package. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update runc binary to v1.0.0 GA Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * handle unconfigured spans without errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to vertex and validate Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to async llb Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: ensure meta resolver uses platform form constraints Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * flightcontrol: reduce contention between goroutines Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Avoid nil pointer dereference when copying from image with no layers Fix this panic when copying from an image with no layers: ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xdd8c17] goroutine 326 [running]: github.com/moby/buildkit/cache/contenthash.(*cacheManager).Checksum(0xc0005ec030, 0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:95 +0x37 github.com/moby/buildkit/cache/contenthash.Checksum(0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:59 +0xd5 github.com/moby/buildkit/solver/llbsolver.NewContentHashFunc.func1.1(0x0, 0x4425d6) /src/solver/llbsolver/result.go:59 +0x20a golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00056a360, 0xc000594510) /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59 created by golang.org/x/sync/errgroup.(*Group).Go /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66 ``` When the path is "/", we allow it because it's a noop. Based on moby#2185 Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add test for copying from scratch Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Check that scratch is mounted as empty dir Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Make error message consistent when layer is empty Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Test with tonistiigi/test:nolayers as well Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * ensure containerd io is complete and closed before returning Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.Controller should own the progress.Writer to prevent leaks Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.FromContext returns a writer factory this allows progress.Controller to manage the writer lifecycle Signed-off-by: Cory Bennett <cbennett@netflix.com> * contenthash: use SeekLowerBound to seek radix tree Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: fix git version detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add support for heredocs with ONBUILD Signed-off-by: Justin Chadwell <me@jedevc.com> * dockerfile: use none differ for dockerfile/dockerignore This avoids wrong metadata matches on small files Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * progressui: print logs for failed step as summary in plain mode Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * grpcerrors: avoid rpc error wrapping in error messages Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * exec: improve error message on exec errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Improve heredoc parsing to allow more generic words Previously, heredoc names were restricted to simple alphanumeric strings. However, heredocs should support much more complex use-cases, including quoting anywhere, as well as allowing special symbols like `.` for easily expressing file extensions. This patch adds support for these more complex cases, by using the shell lexer to parse each heredoc name. Additionally, we include improvements to the lexer to optionally preserve escape tokens to avoid problems when lexing words that have already been lexed before. Signed-off-by: Justin Chadwell <me@jedevc.com> * Improve progress and history messages for heredoc-related commands Signed-off-by: Justin Chadwell <me@jedevc.com> * Remove unneeded Finalize method from ImmutableRef. Finalize was only used outside the cache package in one place, which called it with the commit arg set to false. The code path followed when commit==false turned out to essentially be a no-op because it set "retain cache" to true if it was already set to true. It was thus safe to remove the only external call to it and remove it from the interface. This should be helpful for future efforts to simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist due to the "lazy commit" feature that Finalize is tied into. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Fix ref leak if fileop ref fails to mount. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * add error suggest pkg Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: suggest mistyped flag names Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: provide suggestions for mount options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: add tests for error suggestions Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: remove unnecessary error wrappings Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable riscv64 build Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update QEMU emulators Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * dockerfile: move run network to stable channel Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Automatically detect default git branch Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked. Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Moved getDefaultBranch function to gitsource It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Fix tests Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * git: fix default branch detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable to forcefully specify compression type Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Add full timestamp to logs Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Remove meaningless encode Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Ignore missing providers for blobs w/ same chainid. GetByBlob checks to see if there are any other blobs with the same (uncompressed) ChainID and, if so, reuses their unpacked snapshot if it exists. The problem is if this code finds a match, it was trying to get the matching record, but couldn't do so when the match is lazy because the caller doesn't necessarily have descriptor handlers setup for it. This commit changes the behavior to just ignore any match with the same ChainID that's also lazy as they just aren't usable for the snapshot-reuse optimization. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * authprovider: handle eaccess on storing token seeds Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * log with traceID and spanID Signed-off-by: Morlay <morlay.null@gmail.com> * github: update CI buildkit to v0.9.0-rc1 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * initial version of github cache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: add goactionscache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * caps: add cap for gha cache backend Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove tracetransform package Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: increase default idle conns reuse The current default were even lower than stdlib defaults. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * refactor to use util/bklog instead of using logurs directly Signed-off-by: Morlay <morlay.null@gmail.com> * GitHub Actions cache docs Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Skips getting UID/GUID if passwd/group file is not found When running a WORKDIR instruction, buildkit will create that folder and chown it to the currently set user. For this, it will try to read the /etc/passwd file to get the proper UID, and if that user is not found in the file, the root user will be considered as the owner. However, Windows image do not have that file, which will result in an error while building the image. We can consider not finding the /etc/passwd file as the same as not finding the user in the file, which would solve this issue. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com> * add per domain semaphore to limit concurrent connections This is a safer alternative until we figure out why http.Transport based limiting fails. Some connections like cache export/import do not have a domain key atm and these connections use global pool. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update to github.com/containerd/containerd v1.5.3 Signed-off-by: coryb <cbennett@netflix.com> * vendor: update go-actions-cache with custom client support Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: update to otelhttp roundtripper Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enhance test matrix Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * fix dropped pull progress output due to canceled context fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com> * Add span for layer export This can be a significant amount of time that isn't currently accounted for in traces. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * new implementation for limiting tcp connections The previous implementation had many issues. Eg. on fetch, even if the data already existed and no remote connections were needed the request would still be waiting in the queue. Or if two fetches of same blob happened together they would take up two places in queue although there was only one remote request. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * limited: allow extra high-priority connection for json requests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * ensure wrappers support seeking to continue partial downloads Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * contentutil: change offset to int64 to simplify Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Exporter config digest typo Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * daemonless: wait for daemon to finish before exit Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * github: update CI buildkit to v0.9.0 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add docs for new config options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add ktock and crazy-max to maintainers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update Dockerfile references to use 1.3 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * docs: update images-readme to v0.9 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Bump to codecov/codecov-action v2 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * build(deps): bump github.com/containerd/containerd from 1.5.3 to 1.5.4 Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.5.3 to 1.5.4. - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.5.3...v1.5.4) --- updated-dependencies: - dependency-name: github.com/containerd/containerd dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * util/tracing: remove incorrect import enforcing comment This import comment caused compilation of buildx to fail if `GO111MODULE` was set to `off`: Without `GO111MODULE` set (but with `-mod=vendor`: echo $GO111MODULE export PKG=github.com/docker/buildx export LDFLAGS="-X ${PKG}/version.Version=$(git describe --match 'v[0-9]*' --always --tags) -X ${PKG}/version.Revision=$(git rev-parse HEAD) -X ${PKG}/version.Package=${PKG}" GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx bin/docker-buildx version github.com/docker/buildx v0.6.0 d9ee3b134cbc2d09513fa7fee4176a3919e05887 When setting `GO111MODULE=off`, it fails on the incorrect import path in the vendored file (looks like GO111MODULE=on ignores import-path comments?): export GO111MODULE=off root@5a55ec1c1eed:/go/src/github.com/docker/buildx# GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx vendor/github.com/moby/buildkit/client/client.go:20:2: code in directory /go/src/github.com/docker/buildx/vendor/github.com/moby/buildkit/util/tracing/otlptracegrpc expects import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/connection/connection.go:33:2: found import comments "go.opentelemetry.io/otel/exporters/otlp/internal/otlpconfig" (options.go) and "go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig" (optiontypes.go) in /go/src/github.com/docker/buildx/vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Fix protoc link Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Allow ExitError type to be transmitted over GRPC This will allow clients to retrieve exit error codes returned during a solve without parsing the error messages. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Update to github.com/opencontainers/runc v1.0.1 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Split cache options doc for each exporter Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Set default socket permissions to 660 The systemd default is 666, it seems. Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com> * fix SecurityMode being dropped on gateway container Start Signed-off-by: Cory Bennett <cbennett@netflix.com> * bump containerd from 1.5.4 to 1.5.5 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * go.mod: golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c In preparation of replacing the deprecated github.com/docker/docker/pkg/signal, which uses this version (updating it separately for easier review). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * replace use of deprecated github.com/docker/docker/pkg/signal This package was moved to a separate module in github.com/moby/sys/signal Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Additional tests and cleanup for cache/contenthash This adds a little extra testing around ** patterns, and adds a (currently skipped) test for copying directories under symlinks (moby#2300). It removes an extra call to `filepath.FromSlash` in `shouldIncludePath` and an unused argument to that function. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * all: remove duplicate imports Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the specs-go package import alias to ocispecs ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1 opencontainers /image-spec/specs-go/v1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * hack/dockerfiles: upgrade golangci-lint version to v1.41.1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: enable importas and add settings for specs-go package Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the go-digest package import alias to digest Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: add go-digest importas setting Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * Fix IncludePattern/ExcludePattern matching The transformation to rootedPatterns seems very wrong and inconsistent with what the copy logic did. Change it to match the copy logic, and add more testing. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * dockerfile: fix parsing required key without value Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * generated files: use "go install" to install binaries Now that this repository moved to go1.16, we can use 'go install' to install these binaries. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * util/stack: update protoc options to work with newer versions Generating the util/stack protos failed when updating protoc-gen-go to v1.5.2; it looks like this is the only proto that's not generated using protoc-gen-gogo): util/stack/generate.go protoc-gen-go: unable to determine Go import path for "stack.proto" Please specify either: • a "go_package" option in the .proto source file, or • a "M" argument on the command line. See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information. --go_out: protoc-gen-go: Plugin failed with status code 1. util/stack/generate.go:3: running "protoc": exit status 1 Newer protobuf versions expect a go package to be set. Other .proto files in this repository use the bare package name, but with protoc-gen-go v1.5.2, this produces an error (package names must at least have a "/"). In addition to including the option to the .proto file also changes the generated result (`options go_package "<package name>"`). Using the `-go_opt=M<package name>` option on the other hand, didn't change the result (while still on protoc-gen-go v1.3.5), so I used that option instead. protoc-gen-go v1.5.2 also changed the behavior where the generated file is stored, seemingly relative to the `../../vendor` path specified. This coud be fixed either by setting `--go_out=../../`, which was a bit counter-intuitive, or setting the `--go_opt=paths=source_relative` option. The latter also prevented v1.5.2 from storing the file in `utils/stack/github.com/moby/buildkit/utils/stack/` (sigh). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * add missing ExtraHosts to gateway exec Also adding tests for ExtraHosts and NetMode via gateway exec Signed-off-by: Cory Bennett <cbennett@netflix.com> * add gateway.exec.extrahosts capability Signed-off-by: Cory Bennett <cbennett@netflix.com> * cache: Fix flightcontrol use in computeBlobChain. Previously, the flightcontrol group was being given a key just set to the ref's ID, which meant that concurrent calls using different values of compressionType, createIfNeeded and forceCompression would incorrectly be de-duplicated. The change here splits up the flightcontrol group into a few separate calls and ensures that all the correct input variables are put into the flightcontrol keys. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * solver: include cachemap index in flightcontrol. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * pull: use resolvemode in flightcontrol key. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * util: remove outdated flightcontrol test assertion. The test was making an assertion that is no longer expected to always be true after moby#2195, which purposely made flightcontrol less deterministic. This lead to occasional failures. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * update go to 1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * gomod: update to go1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * cmd/buildkitd: replace BurntSushi/toml with pelletier/go-toml The BurntSushi/toml project has been deprecated, and the ecosystem is converging on using pelletier/go-toml as the "canonical" replacement. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * control: fix 64bit alignment for buildcount Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Use fixed fileutils matching functions This is important for two reasons: 1) Keeps caching logic consistent with recent fsutil changes to use these functions (also vendored here). 2) Allows us to move forward with removal of the original buggy Matches implementation in moby/moby. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add `estargz` compression type Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> Co-authored-by: Tõnis Tiigi <tonistiigi@gmail.com> Co-authored-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Co-authored-by: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com> Co-authored-by: ktock <ktokunaga.mail@gmail.com> Co-authored-by: Justin Chadwell <me@jedevc.com> Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Aaron Lehmann <alehmann@netflix.com> Co-authored-by: Cory Bennett <cbennett@netflix.com> Co-authored-by: Erik Sipsma <erik@sipsma.dev> Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com> Co-authored-by: Levi Harrison <levisamuelharrison@gmail.com> Co-authored-by: masibw <masi19bw@gmail.com> Co-authored-by: Morlay <morlay.null@gmail.com> Co-authored-by: CrazyMax <1951866+crazy-max@users.noreply.github.com> Co-authored-by: Claudiu Belu <cbelu@cloudbasesolutions.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com> Co-authored-by: CrazyMax <github@crazymax.dev> Co-authored-by: Koichi Shiraishi <zchee.io@gmail.com>

* dockerfile: fix git version detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add support for heredocs with ONBUILD Signed-off-by: Justin Chadwell <me@jedevc.com> * dockerfile: use none differ for dockerfile/dockerignore This avoids wrong metadata matches on small files Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * progressui: print logs for failed step as summary in plain mode Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * grpcerrors: avoid rpc error wrapping in error messages Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * exec: improve error message on exec errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Improve heredoc parsing to allow more generic words Previously, heredoc names were restricted to simple alphanumeric strings. However, heredocs should support much more complex use-cases, including quoting anywhere, as well as allowing special symbols like `.` for easily expressing file extensions. This patch adds support for these more complex cases, by using the shell lexer to parse each heredoc name. Additionally, we include improvements to the lexer to optionally preserve escape tokens to avoid problems when lexing words that have already been lexed before. Signed-off-by: Justin Chadwell <me@jedevc.com> * Improve progress and history messages for heredoc-related commands Signed-off-by: Justin Chadwell <me@jedevc.com> * Remove unneeded Finalize method from ImmutableRef. Finalize was only used outside the cache package in one place, which called it with the commit arg set to false. The code path followed when commit==false turned out to essentially be a no-op because it set "retain cache" to true if it was already set to true. It was thus safe to remove the only external call to it and remove it from the interface. This should be helpful for future efforts to simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist due to the "lazy commit" feature that Finalize is tied into. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Fix ref leak if fileop ref fails to mount. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * add error suggest pkg Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: suggest mistyped flag names Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: provide suggestions for mount options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: add tests for error suggestions Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: remove unnecessary error wrappings Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable riscv64 build Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update QEMU emulators Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * dockerfile: move run network to stable channel Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Automatically detect default git branch Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked. Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Moved getDefaultBranch function to gitsource It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Fix tests Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * git: fix default branch detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable to forcefully specify compression type Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Add full timestamp to logs Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Remove meaningless encode Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Ignore missing providers for blobs w/ same chainid. GetByBlob checks to see if there are any other blobs with the same (uncompressed) ChainID and, if so, reuses their unpacked snapshot if it exists. The problem is if this code finds a match, it was trying to get the matching record, but couldn't do so when the match is lazy because the caller doesn't necessarily have descriptor handlers setup for it. This commit changes the behavior to just ignore any match with the same ChainID that's also lazy as they just aren't usable for the snapshot-reuse optimization. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * authprovider: handle eaccess on storing token seeds Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * log with traceID and spanID Signed-off-by: Morlay <morlay.null@gmail.com> * github: update CI buildkit to v0.9.0-rc1 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * initial version of github cache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: add goactionscache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * caps: add cap for gha cache backend Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove tracetransform package Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: increase default idle conns reuse The current default were even lower than stdlib defaults. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * refactor to use util/bklog instead of using logurs directly Signed-off-by: Morlay <morlay.null@gmail.com> * GitHub Actions cache docs Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Skips getting UID/GUID if passwd/group file is not found When running a WORKDIR instruction, buildkit will create that folder and chown it to the currently set user. For this, it will try to read the /etc/passwd file to get the proper UID, and if that user is not found in the file, the root user will be considered as the owner. However, Windows image do not have that file, which will result in an error while building the image. We can consider not finding the /etc/passwd file as the same as not finding the user in the file, which would solve this issue. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com> * add per domain semaphore to limit concurrent connections This is a safer alternative until we figure out why http.Transport based limiting fails. Some connections like cache export/import do not have a domain key atm and these connections use global pool. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update to github.com/containerd/containerd v1.5.3 Signed-off-by: coryb <cbennett@netflix.com> * vendor: update go-actions-cache with custom client support Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: update to otelhttp roundtripper Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enhance test matrix Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * fix dropped pull progress output due to canceled context fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com> * Add span for layer export This can be a significant amount of time that isn't currently accounted for in traces. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * new implementation for limiting tcp connections The previous implementation had many issues. Eg. on fetch, even if the data already existed and no remote connections were needed the request would still be waiting in the queue. Or if two fetches of same blob happened together they would take up two places in queue although there was only one remote request. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * limited: allow extra high-priority connection for json requests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * ensure wrappers support seeking to continue partial downloads Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * contentutil: change offset to int64 to simplify Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Exporter config digest typo Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * daemonless: wait for daemon to finish before exit Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * github: update CI buildkit to v0.9.0 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add docs for new config options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add ktock and crazy-max to maintainers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update Dockerfile references to use 1.3 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * docs: update images-readme to v0.9 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Bump to codecov/codecov-action v2 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * build(deps): bump github.com/containerd/containerd from 1.5.3 to 1.5.4 Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.5.3 to 1.5.4. - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.5.3...v1.5.4) --- updated-dependencies: - dependency-name: github.com/containerd/containerd dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * util/tracing: remove incorrect import enforcing comment This import comment caused compilation of buildx to fail if `GO111MODULE` was set to `off`: Without `GO111MODULE` set (but with `-mod=vendor`: echo $GO111MODULE export PKG=github.com/docker/buildx export LDFLAGS="-X ${PKG}/version.Version=$(git describe --match 'v[0-9]*' --always --tags) -X ${PKG}/version.Revision=$(git rev-parse HEAD) -X ${PKG}/version.Package=${PKG}" GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx bin/docker-buildx version github.com/docker/buildx v0.6.0 d9ee3b134cbc2d09513fa7fee4176a3919e05887 When setting `GO111MODULE=off`, it fails on the incorrect import path in the vendored file (looks like GO111MODULE=on ignores import-path comments?): export GO111MODULE=off root@5a55ec1c1eed:/go/src/github.com/docker/buildx# GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx vendor/github.com/moby/buildkit/client/client.go:20:2: code in directory /go/src/github.com/docker/buildx/vendor/github.com/moby/buildkit/util/tracing/otlptracegrpc expects import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/connection/connection.go:33:2: found import comments "go.opentelemetry.io/otel/exporters/otlp/internal/otlpconfig" (options.go) and "go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig" (optiontypes.go) in /go/src/github.com/docker/buildx/vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Fix protoc link Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Allow ExitError type to be transmitted over GRPC This will allow clients to retrieve exit error codes returned during a solve without parsing the error messages. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Update to github.com/opencontainers/runc v1.0.1 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Split cache options doc for each exporter Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Set default socket permissions to 660 The systemd default is 666, it seems. Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com> * fix SecurityMode being dropped on gateway container Start Signed-off-by: Cory Bennett <cbennett@netflix.com> * bump containerd from 1.5.4 to 1.5.5 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * go.mod: golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c In preparation of replacing the deprecated github.com/docker/docker/pkg/signal, which uses this version (updating it separately for easier review). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * replace use of deprecated github.com/docker/docker/pkg/signal This package was moved to a separate module in github.com/moby/sys/signal Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Additional tests and cleanup for cache/contenthash This adds a little extra testing around ** patterns, and adds a (currently skipped) test for copying directories under symlinks (moby#2300). It removes an extra call to `filepath.FromSlash` in `shouldIncludePath` and an unused argument to that function. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * all: remove duplicate imports Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the specs-go package import alias to ocispecs ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1 opencontainers /image-spec/specs-go/v1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * hack/dockerfiles: upgrade golangci-lint version to v1.41.1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: enable importas and add settings for specs-go package Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the go-digest package import alias to digest Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: add go-digest importas setting Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * Fix IncludePattern/ExcludePattern matching The transformation to rootedPatterns seems very wrong and inconsistent with what the copy logic did. Change it to match the copy logic, and add more testing. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * dockerfile: fix parsing required key without value Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * generated files: use "go install" to install binaries Now that this repository moved to go1.16, we can use 'go install' to install these binaries. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * util/stack: update protoc options to work with newer versions Generating the util/stack protos failed when updating protoc-gen-go to v1.5.2; it looks like this is the only proto that's not generated using protoc-gen-gogo): util/stack/generate.go protoc-gen-go: unable to determine Go import path for "stack.proto" Please specify either: • a "go_package" option in the .proto source file, or • a "M" argument on the command line. See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information. --go_out: protoc-gen-go: Plugin failed with status code 1. util/stack/generate.go:3: running "protoc": exit status 1 Newer protobuf versions expect a go package to be set. Other .proto files in this repository use the bare package name, but with protoc-gen-go v1.5.2, this produces an error (package names must at least have a "/"). In addition to including the option to the .proto file also changes the generated result (`options go_package "<package name>"`). Using the `-go_opt=M<package name>` option on the other hand, didn't change the result (while still on protoc-gen-go v1.3.5), so I used that option instead. protoc-gen-go v1.5.2 also changed the behavior where the generated file is stored, seemingly relative to the `../../vendor` path specified. This coud be fixed either by setting `--go_out=../../`, which was a bit counter-intuitive, or setting the `--go_opt=paths=source_relative` option. The latter also prevented v1.5.2 from storing the file in `utils/stack/github.com/moby/buildkit/utils/stack/` (sigh). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * add missing ExtraHosts to gateway exec Also adding tests for ExtraHosts and NetMode via gateway exec Signed-off-by: Cory Bennett <cbennett@netflix.com> * add gateway.exec.extrahosts capability Signed-off-by: Cory Bennett <cbennett@netflix.com> * cache: Fix flightcontrol use in computeBlobChain. Previously, the flightcontrol group was being given a key just set to the ref's ID, which meant that concurrent calls using different values of compressionType, createIfNeeded and forceCompression would incorrectly be de-duplicated. The change here splits up the flightcontrol group into a few separate calls and ensures that all the correct input variables are put into the flightcontrol keys. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * solver: include cachemap index in flightcontrol. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * pull: use resolvemode in flightcontrol key. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * util: remove outdated flightcontrol test assertion. The test was making an assertion that is no longer expected to always be true after moby#2195, which purposely made flightcontrol less deterministic. This lead to occasional failures. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * update go to 1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * gomod: update to go1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Follow links in includedPaths to resolve incorrect caching when source path is behind symlink As discussed in moby#2300, includedPaths does not resolve symlinks when looking up the source path in the prefix tree. If the user requests a path that involves symlinks (for example, /a/foo when a symlink /a -> /b exists), includedPaths will not find it, and will expect nothing to be copied. This does not match the actual copy behavior implemented in fsutil, which will follow symlinks in prefix components of a given path, so it can end up caching an empty result even though the copy will produce a non-empty result, which is quite bad. To fix this, use getFollowLinks to resolve the path before walking it. In the wildcard case, this is done to the non-wildcard prefix of the path (if any), which matches the behavior in fsutil. Fixes the repro case here: https://gist.github.com/aaronlehmann/64054c9a2cff0d27e200cc107bba3d69 Fixes moby#2300 Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * cmd/buildkitd: replace BurntSushi/toml with pelletier/go-toml The BurntSushi/toml project has been deprecated, and the ecosystem is converging on using pelletier/go-toml as the "canonical" replacement. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * control: fix 64bit alignment for buildcount Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Use fixed fileutils matching functions This is important for two reasons: 1) Keeps caching logic consistent with recent fsutil changes to use these functions (also vendored here). 2) Allows us to move forward with removal of the original buggy Matches implementation in moby/moby. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add `estargz` compression type Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Refactor cache metadata interface. There are a few goals with this refactor: 1. Remove external access to fields that no longer make sense and/or won't make sense soon due to other potential changes. For example, there can now be multiple blobs associated with a ref (for different compression types), so the fact that you could access the "Blob" field from the Info method on Ref incorrectly implied there was just a single blob for the ref. This is on top of the fact that there is no need for external access to blob digests. 2. Centralize use of cache metadata inside the cache package. Previously, many parts of the code outside the cache package could obtain the bolt storage item for any ref and read/write it directly. This made it hard to understand what fields are used and when. Now, the Metadata method has been removed from the Ref interface and replaced with getters+setters for metadata fields we want to expose outside the package, which makes it much easier to track and understand. Similar changes have been made to the metadata search interface. 3. Use a consistent getter+setter interface for metadata, replacing the mix of interfaces like Metadata(), Size(), Info() and other inconsistencies. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Use containerd/pkg/seccomp.IsEnabled() This replaces the local SeccompSupported() utility for the implementation in containerd, which performs the same check. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Compute diff from the upper dir of overlayfs-based snapshotter Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * go.mod: github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 full diff: moby/term@bea5bbe...3f7ff69 updates Azure/go-ansiterm to fix integer overflow on arm Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * go.mod: split the indirect packages After go1.17, all indirect packages are listed in the go.mod file. In addition, has been introduced the ability to list indirect packages separately. Split the indirect packages to make the dependency packages clearer. Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * exporter: support creating blobs with zstd compression Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update getremote test for zstd Estargz support has been removed from this test as implementation does not guarantee digest stability and only reason it passed were the exceptions in the test via variant map that ignored cases where timing resulted the digest to go wrong. This needs to be addressed in the follow up if we want to keep estargz support. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add test case for symlink which is not final path component before wildcard Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * hack: allow mounting in workdir in shell Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Handle the case of multiple path component symlinks (including last component) in wildcard prefix Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Use getFollowLinksWalked Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * bklog: only log tracing ids when span exporter not nil Signed-off-by: Morlay <morlay.null@gmail.com> * Refactor url redacting util Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Clean up old TODOs Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Move config parsing to a dedicated pkg Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Generate and embed build sources Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * resolver: use different mutext for handlers and hosts hosts mutex is called on initialization, meaning `GetResolver` might block if it is in the middle of auth exchange. This is currently bad in the case where Job initialization needs to register a name before timeout is reached. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: make sure authorizer is not overwritten on other resolvers Authorizer stores the current session.Group so if it is overwritten for another resolver it means that session might have been dropped and authentication will fail. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * solver: increase timeout for job registration Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * go.mod: sort and move self-managed indirect dependencies to first block Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * Fix issues moby#1980 and moby#2198 Signed-off-by: Jonathan Giannuzzi <jonathan@giannuzzi.me> * Add BUILDKIT_SANDBOX_HOSTNAME build-arg Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Fix estargz compression loses the original tar metadata Currently, eStargz compression doesn't preserve the original tar metadata (header bytes and their order). This causes failure of `TestGetRemote` because an uncompressed blob converted from a gzip blob provides different digset against the one converted from eStargz blob even if their original tar (computed by differ) are the same. This commit solves this issue by fixing eStargz to preserve original tar's metadata that is modified by eStargz. Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Enhance ANSI color for progress ui Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Move resolver config to a dedicated package Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Standard user umask for git process Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * make sure ci runs on version branches Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * return an error instead of panicking when failing to get edge Signed-off-by: Maxime Lagresle <maxime@angel.co> * Add support for shm size Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * gha: handle already exist error on save Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * don't cast Value when pipe is errored Signed-off-by: Maxime Lagresle <maxime@angel.co> * gha: handle missing blob gracefully FromRemote now calls CheckDescriptor to validate if the blob still exists. Otherwise cache loading fallback does not get triggered because cache is actually lazily pulled in only on exporting phase. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * solver: make sure previous error gets reset This happens for example when cache loading fails but then fallback step execution succeeds. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: update go-actions-cache to 4d48f2ff Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Differ: write diff to the content store over bufio writer Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Do not enable overlayfs differ for fuse-overlayfs-snapshotter Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Converter: make sure uncompressed digest annotation is set Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Use gha cache on CI Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Creating tcp socket without using go-connections. Signed-off-by: Jacob MacElroy <jacob@okteto.com> * limited: fix possible deadlock when pushhandler calls fetcher Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * README.md: improve "Building multi-platform images" section Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * Add support for ulimit Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * solver: fix exporters unsafely sharing records Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * fix: provide only available capabilities to insecure environment The problem this change is trying to fix are the environments where some capabilities are already dropped, so they can't be granted to the job with `--security=insecure`. I know that probably fixed set of capabilities was implemented to provide a stable build environment, but at the same time this breaks environments with reduced capabilities. Signed-off-by: Andrey Smirnov <andrey.smirnov@talos-systems.com> * client: allow setting custom dialer for session endpoint Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add size to tmpfs mounts Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * deduplicate mounts Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * use bytes as given size for tmpfs mount Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * use `opts.MemBytes` for tmpfs size run mount instruction Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Re-add Finalize method to ImmutableRef. It turns out that while Buildkit code did not need this method to be public, moby code does still use it, so we have to re-add it after its removal in moby#2216 (commit b85ef15). This commit is not a revert because some of the changes are still desireable, namely the removal of the "commit" parameter which didn't serve any purpose. Signed-off-by: Erik Sipsma <erik@sipsma.dev> Co-authored-by: Tonis Tiigi <tonistiigi@gmail.com> Co-authored-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> Co-authored-by: Justin Chadwell <me@jedevc.com> Co-authored-by: Erik Sipsma <erik@sipsma.dev> Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com> Co-authored-by: Levi Harrison <levisamuelharrison@gmail.com> Co-authored-by: ktock <ktokunaga.mail@gmail.com> Co-authored-by: masibw <masi19bw@gmail.com> Co-authored-by: Morlay <morlay.null@gmail.com> Co-authored-by: CrazyMax <1951866+crazy-max@users.noreply.github.com> Co-authored-by: Claudiu Belu <cbelu@cloudbasesolutions.com> Co-authored-by: coryb <cbennett@netflix.com> Co-authored-by: Aaron Lehmann <alehmann@netflix.com> Co-authored-by: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com> Co-authored-by: CrazyMax <github@crazymax.dev> Co-authored-by: Koichi Shiraishi <zchee.io@gmail.com> Co-authored-by: Jonathan Giannuzzi <jonathan@giannuzzi.me> Co-authored-by: Maxime Lagresle <maxime@angel.co> Co-authored-by: Jacob MacElroy <jacob@okteto.com> Co-authored-by: Andrey Smirnov <andrey.smirnov@talos-systems.com>

* integration: add common context base to all integration tests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: update opentelemetry to 1.0.0-rc Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add current tracing context detection and exec propagation Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> (cherry picked from commit bc9a83144c83e9fd78007b7bfe92e8082c59d40e) * add transform package to convert from otlp Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: add delegated exporter Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * otlgrpc: provide a way to get otlp client from grpc conn Hopefully this can be removed with a future upstream change that could make this configurable. The package also needs internal dependency that is copied in. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable collecting traces via control api Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * support collecting traces from llb.Exec Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * client: pass delegated exporter as parameter Avoid client package having dependency on global detect package. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update runc binary to v1.0.0 GA Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> * handle unconfigured spans without errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to vertex and validate Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: add constraints to async llb Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * llb: ensure meta resolver uses platform form constraints Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * flightcontrol: reduce contention between goroutines Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Avoid nil pointer dereference when copying from image with no layers Fix this panic when copying from an image with no layers: ``` panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x50 pc=0xdd8c17] goroutine 326 [running]: github.com/moby/buildkit/cache/contenthash.(*cacheManager).Checksum(0xc0005ec030, 0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:95 +0x37 github.com/moby/buildkit/cache/contenthash.Checksum(0x1682c00, 0xc000842140, 0x0, 0x0, 0xc0005d4023, 0x1, 0x0, 0x0, 0x0, 0x0, ...) /src/cache/contenthash/checksum.go:59 +0xd5 github.com/moby/buildkit/solver/llbsolver.NewContentHashFunc.func1.1(0x0, 0x4425d6) /src/solver/llbsolver/result.go:59 +0x20a golang.org/x/sync/errgroup.(*Group).Go.func1(0xc00056a360, 0xc000594510) /src/vendor/golang.org/x/sync/errgroup/errgroup.go:57 +0x59 created by golang.org/x/sync/errgroup.(*Group).Go /src/vendor/golang.org/x/sync/errgroup/errgroup.go:54 +0x66 ``` When the path is "/", we allow it because it's a noop. Based on moby#2185 Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add test for copying from scratch Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Check that scratch is mounted as empty dir Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Make error message consistent when layer is empty Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Test with tonistiigi/test:nolayers as well Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * ensure containerd io is complete and closed before returning Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.Controller should own the progress.Writer to prevent leaks Signed-off-by: Cory Bennett <cbennett@netflix.com> * [moby#2112] progress.FromContext returns a writer factory this allows progress.Controller to manage the writer lifecycle Signed-off-by: Cory Bennett <cbennett@netflix.com> * contenthash: use SeekLowerBound to seek radix tree Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: fix git version detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add support for heredocs with ONBUILD Signed-off-by: Justin Chadwell <me@jedevc.com> * dockerfile: use none differ for dockerfile/dockerignore This avoids wrong metadata matches on small files Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * progressui: print logs for failed step as summary in plain mode Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * grpcerrors: avoid rpc error wrapping in error messages Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * exec: improve error message on exec errors Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Improve heredoc parsing to allow more generic words Previously, heredoc names were restricted to simple alphanumeric strings. However, heredocs should support much more complex use-cases, including quoting anywhere, as well as allowing special symbols like `.` for easily expressing file extensions. This patch adds support for these more complex cases, by using the shell lexer to parse each heredoc name. Additionally, we include improvements to the lexer to optionally preserve escape tokens to avoid problems when lexing words that have already been lexed before. Signed-off-by: Justin Chadwell <me@jedevc.com> * Improve progress and history messages for heredoc-related commands Signed-off-by: Justin Chadwell <me@jedevc.com> * Remove unneeded Finalize method from ImmutableRef. Finalize was only used outside the cache package in one place, which called it with the commit arg set to false. The code path followed when commit==false turned out to essentially be a no-op because it set "retain cache" to true if it was already set to true. It was thus safe to remove the only external call to it and remove it from the interface. This should be helpful for future efforts to simplify the equal{Mutable,Immutable} fields in cacheRecord, which exist due to the "lazy commit" feature that Finalize is tied into. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Fix ref leak if fileop ref fails to mount. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * add error suggest pkg Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: suggest mistyped flag names Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: provide suggestions for mount options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: add tests for error suggestions Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * dockerfile: remove unnecessary error wrappings Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * enable riscv64 build Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update QEMU emulators Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * dockerfile: move run network to stable channel Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Automatically detect default git branch Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked. Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Moved getDefaultBranch function to gitsource It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * Fix tests Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com> * git: fix default branch detection Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enable to forcefully specify compression type Signed-off-by: ktock <ktokunaga.mail@gmail.com> * Add full timestamp to logs Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Remove meaningless encode Signed-off-by: Yamazaki Masashi <masi19bw@gmail.com> * Ignore missing providers for blobs w/ same chainid. GetByBlob checks to see if there are any other blobs with the same (uncompressed) ChainID and, if so, reuses their unpacked snapshot if it exists. The problem is if this code finds a match, it was trying to get the matching record, but couldn't do so when the match is lazy because the caller doesn't necessarily have descriptor handlers setup for it. This commit changes the behavior to just ignore any match with the same ChainID that's also lazy as they just aren't usable for the snapshot-reuse optimization. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * authprovider: handle eaccess on storing token seeds Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * log with traceID and spanID Signed-off-by: Morlay <morlay.null@gmail.com> * github: update CI buildkit to v0.9.0-rc1 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * initial version of github cache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * vendor: add goactionscache Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * caps: add cap for gha cache backend Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * remove tracetransform package Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: increase default idle conns reuse The current default were even lower than stdlib defaults. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * refactor to use util/bklog instead of using logurs directly Signed-off-by: Morlay <morlay.null@gmail.com> * GitHub Actions cache docs Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Skips getting UID/GUID if passwd/group file is not found When running a WORKDIR instruction, buildkit will create that folder and chown it to the currently set user. For this, it will try to read the /etc/passwd file to get the proper UID, and if that user is not found in the file, the root user will be considered as the owner. However, Windows image do not have that file, which will result in an error while building the image. We can consider not finding the /etc/passwd file as the same as not finding the user in the file, which would solve this issue. Signed-off-by: Claudiu Belu <cbelu@cloudbasesolutions.com> * add per domain semaphore to limit concurrent connections This is a safer alternative until we figure out why http.Transport based limiting fails. Some connections like cache export/import do not have a domain key atm and these connections use global pool. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update to github.com/containerd/containerd v1.5.3 Signed-off-by: coryb <cbennett@netflix.com> * vendor: update go-actions-cache with custom client support Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * tracing: update to otelhttp roundtripper Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Enhance test matrix Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * fix dropped pull progress output due to canceled context fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com> * Add span for layer export This can be a significant amount of time that isn't currently accounted for in traces. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * new implementation for limiting tcp connections The previous implementation had many issues. Eg. on fetch, even if the data already existed and no remote connections were needed the request would still be waiting in the queue. Or if two fetches of same blob happened together they would take up two places in queue although there was only one remote request. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * limited: allow extra high-priority connection for json requests Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * ensure wrappers support seeking to continue partial downloads Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * contentutil: change offset to int64 to simplify Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Exporter config digest typo Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * daemonless: wait for daemon to finish before exit Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * github: update CI buildkit to v0.9.0 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add docs for new config options Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * add ktock and crazy-max to maintainers Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Update Dockerfile references to use 1.3 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * docs: update images-readme to v0.9 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Bump to codecov/codecov-action v2 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * build(deps): bump github.com/containerd/containerd from 1.5.3 to 1.5.4 Bumps [github.com/containerd/containerd](https://github.com/containerd/containerd) from 1.5.3 to 1.5.4. - [Release notes](https://github.com/containerd/containerd/releases) - [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md) - [Commits](containerd/containerd@v1.5.3...v1.5.4) --- updated-dependencies: - dependency-name: github.com/containerd/containerd dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * util/tracing: remove incorrect import enforcing comment This import comment caused compilation of buildx to fail if `GO111MODULE` was set to `off`: Without `GO111MODULE` set (but with `-mod=vendor`: echo $GO111MODULE export PKG=github.com/docker/buildx export LDFLAGS="-X ${PKG}/version.Version=$(git describe --match 'v[0-9]*' --always --tags) -X ${PKG}/version.Revision=$(git rev-parse HEAD) -X ${PKG}/version.Package=${PKG}" GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx bin/docker-buildx version github.com/docker/buildx v0.6.0 d9ee3b134cbc2d09513fa7fee4176a3919e05887 When setting `GO111MODULE=off`, it fails on the incorrect import path in the vendored file (looks like GO111MODULE=on ignores import-path comments?): export GO111MODULE=off root@5a55ec1c1eed:/go/src/github.com/docker/buildx# GOFLAGS=-mod=vendor go build -o bin/docker-buildx -ldflags "${LDFLAGS}" ./cmd/buildx vendor/github.com/moby/buildkit/client/client.go:20:2: code in directory /go/src/github.com/docker/buildx/vendor/github.com/moby/buildkit/util/tracing/otlptracegrpc expects import "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/connection/connection.go:33:2: found import comments "go.opentelemetry.io/otel/exporters/otlp/internal/otlpconfig" (options.go) and "go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig" (optiontypes.go) in /go/src/github.com/docker/buildx/vendor/go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlpconfig Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Fix protoc link Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Allow ExitError type to be transmitted over GRPC This will allow clients to retrieve exit error codes returned during a solve without parsing the error messages. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Update to github.com/opencontainers/runc v1.0.1 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Split cache options doc for each exporter Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Set default socket permissions to 660 The systemd default is 666, it seems. Signed-off-by: Anders F Björklund <anders.f.bjorklund@gmail.com> * fix SecurityMode being dropped on gateway container Start Signed-off-by: Cory Bennett <cbennett@netflix.com> * bump containerd from 1.5.4 to 1.5.5 Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * go.mod: golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c In preparation of replacing the deprecated github.com/docker/docker/pkg/signal, which uses this version (updating it separately for easier review). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * replace use of deprecated github.com/docker/docker/pkg/signal This package was moved to a separate module in github.com/moby/sys/signal Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Additional tests and cleanup for cache/contenthash This adds a little extra testing around ** patterns, and adds a (currently skipped) test for copying directories under symlinks (moby#2300). It removes an extra call to `filepath.FromSlash` in `shouldIncludePath` and an unused argument to that function. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * all: remove duplicate imports Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the specs-go package import alias to ocispecs ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1 opencontainers /image-spec/specs-go/v1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * hack/dockerfiles: upgrade golangci-lint version to v1.41.1 Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: enable importas and add settings for specs-go package Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * all: unify the go-digest package import alias to digest Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * golangci-lint: add go-digest importas setting Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * Fix IncludePattern/ExcludePattern matching The transformation to rootedPatterns seems very wrong and inconsistent with what the copy logic did. Change it to match the copy logic, and add more testing. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * dockerfile: fix parsing required key without value Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * generated files: use "go install" to install binaries Now that this repository moved to go1.16, we can use 'go install' to install these binaries. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * util/stack: update protoc options to work with newer versions Generating the util/stack protos failed when updating protoc-gen-go to v1.5.2; it looks like this is the only proto that's not generated using protoc-gen-gogo): util/stack/generate.go protoc-gen-go: unable to determine Go import path for "stack.proto" Please specify either: • a "go_package" option in the .proto source file, or • a "M" argument on the command line. See https://developers.google.com/protocol-buffers/docs/reference/go-generated#package for more information. --go_out: protoc-gen-go: Plugin failed with status code 1. util/stack/generate.go:3: running "protoc": exit status 1 Newer protobuf versions expect a go package to be set. Other .proto files in this repository use the bare package name, but with protoc-gen-go v1.5.2, this produces an error (package names must at least have a "/"). In addition to including the option to the .proto file also changes the generated result (`options go_package "<package name>"`). Using the `-go_opt=M<package name>` option on the other hand, didn't change the result (while still on protoc-gen-go v1.3.5), so I used that option instead. protoc-gen-go v1.5.2 also changed the behavior where the generated file is stored, seemingly relative to the `../../vendor` path specified. This coud be fixed either by setting `--go_out=../../`, which was a bit counter-intuitive, or setting the `--go_opt=paths=source_relative` option. The latter also prevented v1.5.2 from storing the file in `utils/stack/github.com/moby/buildkit/utils/stack/` (sigh). Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * add missing ExtraHosts to gateway exec Also adding tests for ExtraHosts and NetMode via gateway exec Signed-off-by: Cory Bennett <cbennett@netflix.com> * add gateway.exec.extrahosts capability Signed-off-by: Cory Bennett <cbennett@netflix.com> * cache: Fix flightcontrol use in computeBlobChain. Previously, the flightcontrol group was being given a key just set to the ref's ID, which meant that concurrent calls using different values of compressionType, createIfNeeded and forceCompression would incorrectly be de-duplicated. The change here splits up the flightcontrol group into a few separate calls and ensures that all the correct input variables are put into the flightcontrol keys. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * solver: include cachemap index in flightcontrol. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * pull: use resolvemode in flightcontrol key. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * util: remove outdated flightcontrol test assertion. The test was making an assertion that is no longer expected to always be true after moby#2195, which purposely made flightcontrol less deterministic. This lead to occasional failures. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * update go to 1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * gomod: update to go1.17 Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Follow links in includedPaths to resolve incorrect caching when source path is behind symlink As discussed in moby#2300, includedPaths does not resolve symlinks when looking up the source path in the prefix tree. If the user requests a path that involves symlinks (for example, /a/foo when a symlink /a -> /b exists), includedPaths will not find it, and will expect nothing to be copied. This does not match the actual copy behavior implemented in fsutil, which will follow symlinks in prefix components of a given path, so it can end up caching an empty result even though the copy will produce a non-empty result, which is quite bad. To fix this, use getFollowLinks to resolve the path before walking it. In the wildcard case, this is done to the non-wildcard prefix of the path (if any), which matches the behavior in fsutil. Fixes the repro case here: https://gist.github.com/aaronlehmann/64054c9a2cff0d27e200cc107bba3d69 Fixes moby#2300 Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * cmd/buildkitd: replace BurntSushi/toml with pelletier/go-toml The BurntSushi/toml project has been deprecated, and the ecosystem is converging on using pelletier/go-toml as the "canonical" replacement. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * control: fix 64bit alignment for buildcount Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Use fixed fileutils matching functions This is important for two reasons: 1) Keeps caching logic consistent with recent fsutil changes to use these functions (also vendored here). 2) Allows us to move forward with removal of the original buggy Matches implementation in moby/moby. Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Add `estargz` compression type Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Refactor cache metadata interface. There are a few goals with this refactor: 1. Remove external access to fields that no longer make sense and/or won't make sense soon due to other potential changes. For example, there can now be multiple blobs associated with a ref (for different compression types), so the fact that you could access the "Blob" field from the Info method on Ref incorrectly implied there was just a single blob for the ref. This is on top of the fact that there is no need for external access to blob digests. 2. Centralize use of cache metadata inside the cache package. Previously, many parts of the code outside the cache package could obtain the bolt storage item for any ref and read/write it directly. This made it hard to understand what fields are used and when. Now, the Metadata method has been removed from the Ref interface and replaced with getters+setters for metadata fields we want to expose outside the package, which makes it much easier to track and understand. Similar changes have been made to the metadata search interface. 3. Use a consistent getter+setter interface for metadata, replacing the mix of interfaces like Metadata(), Size(), Info() and other inconsistencies. Signed-off-by: Erik Sipsma <erik@sipsma.dev> * Use containerd/pkg/seccomp.IsEnabled() This replaces the local SeccompSupported() utility for the implementation in containerd, which performs the same check. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * Compute diff from the upper dir of overlayfs-based snapshotter Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * go.mod: github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 full diff: moby/term@bea5bbe...3f7ff69 updates Azure/go-ansiterm to fix integer overflow on arm Signed-off-by: Sebastiaan van Stijn <github@gone.nl> * go.mod: split the indirect packages After go1.17, all indirect packages are listed in the go.mod file. In addition, has been introduced the ability to list indirect packages separately. Split the indirect packages to make the dependency packages clearer. Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * exporter: support creating blobs with zstd compression Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * update getremote test for zstd Estargz support has been removed from this test as implementation does not guarantee digest stability and only reason it passed were the exceptions in the test via variant map that ignored cases where timing resulted the digest to go wrong. This needs to be addressed in the follow up if we want to keep estargz support. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Add test case for symlink which is not final path component before wildcard Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * hack: allow mounting in workdir in shell Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Handle the case of multiple path component symlinks (including last component) in wildcard prefix Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * Use getFollowLinksWalked Signed-off-by: Aaron Lehmann <alehmann@netflix.com> * bklog: only log tracing ids when span exporter not nil Signed-off-by: Morlay <morlay.null@gmail.com> * Refactor url redacting util Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Clean up old TODOs Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * Move config parsing to a dedicated pkg Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Generate and embed build sources Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * resolver: use different mutext for handlers and hosts hosts mutex is called on initialization, meaning `GetResolver` might block if it is in the middle of auth exchange. This is currently bad in the case where Job initialization needs to register a name before timeout is reached. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * resolver: make sure authorizer is not overwritten on other resolvers Authorizer stores the current session.Group so if it is overwritten for another resolver it means that session might have been dropped and authentication will fail. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * solver: increase timeout for job registration Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * go.mod: sort and move self-managed indirect dependencies to first block Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com> * Fix issues moby#1980 and moby#2198 Signed-off-by: Jonathan Giannuzzi <jonathan@giannuzzi.me> * Add BUILDKIT_SANDBOX_HOSTNAME build-arg Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Fix estargz compression loses the original tar metadata Currently, eStargz compression doesn't preserve the original tar metadata (header bytes and their order). This causes failure of `TestGetRemote` because an uncompressed blob converted from a gzip blob provides different digset against the one converted from eStargz blob even if their original tar (computed by differ) are the same. This commit solves this issue by fixing eStargz to preserve original tar's metadata that is modified by eStargz. Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com> * Enhance ANSI color for progress ui Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Move resolver config to a dedicated package Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * Standard user umask for git process Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * make sure ci runs on version branches Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com> * return an error instead of panicking when failing to get edge Signed-off-by: Maxime Lagresle <maxime@angel.co> * Add support for shm size Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com> * don't cast Value when pipe is errored Signed-off-by: Maxime Lagresle <maxime@angel.co> * Apply Earthly changes to newer buildkit version This commit squashes previous work done in the earthly-main branch 199ad6a into a single commit which is rebased against moby/master branch d429b0b Co-authored-by: Tõnis Tiigi <tonistiigi@gmail.com> Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com> Co-authored-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp> Co-authored-by: Aaron Lehmann <alehmann@netflix.com> Co-authored-by: Cory Bennett <cbennett@netflix.com> Co-authored-by: Justin Chadwell <me@jedevc.com> Co-authored-by: Erik Sipsma <erik@sipsma.dev> Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com> Co-authored-by: Levi Harrison <levisamuelharrison@gmail.com> Co-authored-by: ktock <ktokunaga.mail@gmail.com> Co-authored-by: masibw <masi19bw@gmail.com> Co-authored-by: Morlay <morlay.null@gmail.com> Co-authored-by: CrazyMax <1951866+crazy-max@users.noreply.github.com> Co-authored-by: Claudiu Belu <cbelu@cloudbasesolutions.com> Co-authored-by: Sebastiaan van Stijn <thaJeztah@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sebastiaan van Stijn <github@gone.nl> Co-authored-by: Anders F Björklund <anders.f.bjorklund@gmail.com> Co-authored-by: CrazyMax <github@crazymax.dev> Co-authored-by: Koichi Shiraishi <zchee.io@gmail.com> Co-authored-by: Jonathan Giannuzzi <jonathan@giannuzzi.me> Co-authored-by: Maxime Lagresle <maxime@angel.co>

tonistiigi added the kind/bug label Jul 13, 2021

tonistiigi added this to the v0.9.0 milestone Jul 13, 2021

coryb added a commit to coryb/buildkit that referenced this issue Jul 14, 2021

fix dropped pull progress output due to canceled context

de0019b

fixes moby#2248 Signed-off-by: coryb <cbennett@netflix.com>

coryb mentioned this issue Jul 14, 2021

fix dropped pull progress output due to canceled context #2254

Merged

tonistiigi closed this as completed in #2254 Jul 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[0.9] blob download progress broken #2248

[0.9] blob download progress broken #2248

tonistiigi commented Jul 13, 2021

[0.9] blob download progress broken #2248

[0.9] blob download progress broken #2248

Comments

tonistiigi commented Jul 13, 2021