Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to moby v20.10.11 #281

Merged
2,745 commits merged into from
Feb 21, 2022
Merged

Update to moby v20.10.11 #281

2,745 commits merged into from
Feb 21, 2022

Conversation

robertgzr
Copy link
Contributor

No description provided.

kolyshkin and others added 30 commits February 17, 2021 21:17
Before:

> Generally Necessary:
> - cgroup hierarchy: nonexistent??
>     (see https://github.com/tianon/cgroupfs-mount)

After:

> Generally Necessary:
> - cgroup hierarchy: cgroupv2

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 76b5906)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CONFIG_NF_NAT_IPV4 was removed in kernel commit 3bf195ae6037e310,
which made its way into v5.1-rc1. The functionality is now under
NF_NAT which we already check for.

Make the check for NF_NAT_IPV4 conditional.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit eeb53c1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
CONFIG_NF_NAT_NEEDED was removed in kernel commit 4806e975729f99c7,
which made its way into v5.2-rc1. The functionality is now under
NF_NAT which we already check for.

Make the check for NF_NAT_NEEDED conditional.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 03da411)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Kernel commit 2d1c498072de69e (which made its way into kernel v5.8-rc1)
removed CONFIG_MEMCG_SWAP_ENABLED Kconfig option, making swap accounting
always enabled (unless swapaccount=0 boot option is provided).

Make the check conditional.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 070f9d9)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These config options are removed by kernel commit f382fb0bcef4,
which made its way into kernel v5.0-rc1.

Make the check conditional.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 18e0543)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This parameter was removed by kernel commit 4c145dce260137,
which made its way to kernel v5.3-rc1. Since that commit,
the functionality is built-in (i.e. it is available as long
as CONFIG_XFRM is on).

Make the check conditional.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 06d9020)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This PR was originally proposed by @phillc here: docker-archive/engine#456

Signed-off-by: FreddieOliveira <fredf_oliveira@ufu.br>
(cherry picked from commit 2db5676)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Backport of 2db5676 to the swagger files
used in the documentation

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 240d0b3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Jim Lin <b04705003@ntu.edu.tw>
(cherry picked from commit c9ec21e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Loggers that implement BufSize() (e.g. awslogs) uses the method to
tell Copier about the maximum log line length. However loggerWithCache
and RingBuffer hide the method by wrapping loggers.

As a result, Copier uses its default 16KB limit which breaks log
lines > 16kB even the destinations can handle that.

This change implements BufSize() on loggerWithCache and RingBuffer to
make sure these logger wrappes don't hide the method on the underlying
loggers.

Fixes #41794.

Signed-off-by: Kazuyoshi Kato <katokazu@amazon.com>
(cherry picked from commit bb11365)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This fixes a panic when an admin specifies a custom default runtime,
when a plugin is started the shim config is nil.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 2903863)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When pulling an image by platform, it is possible for the image's
configured platform to not match what was in the manifest list.
The image itself is buggy because either the manifest list is incorrect
or the image config is incorrect. In any case, this is preventing people
from upgrading because many times users do not have control over these
buggy images.

This was not a problem in 19.03 because we did not compare on platform
before. It just assumed if we had the image it was the one we wanted
regardless of platform, which has its own problems.

Example Dockerfile that has this problem:

```Dockerfile
FROM --platform=linux/arm64 k8s.gcr.io/build-image/debian-iptables:buster-v1.3.0
RUN echo hello
```

This fails the first time you try to build after it finishes pulling but
before performing the `RUN` command.
On the second attempt it works because the image is already there and
does not hit the code that errors out on platform mismatch (Actually it
ignores errors if an image is returned at all).

Must be run with the classic builder (DOCKER_BUILDKIT=0).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 3996953)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 7c7e168)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These syscalls (some of which have been in Linux for a while but were
missing from the profile) fall into a few buckets:

 * close_range(2), epoll_pwait2(2) are just extensions of existing "safe
   for everyone" syscalls.

 * The mountv2 API syscalls (fs*(2), move_mount(2), open_tree(2)) are
   all equivalent to aspects of mount(2) and thus go into the
   CAP_SYS_ADMIN category.

 * process_madvise(2) is similar to the other process_*(2) syscalls and
   thus goes in the CAP_SYS_PTRACE category.

Signed-off-by: Aleksa Sarai <asarai@suse.de>
(cherry picked from commit 54eff43)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
…ogger

[20.10 backport] Handle long log messages correctly on SizedLogger
In some cases, in fact many in the wild, an image may have the incorrect
platform on the image config.
This can lead to failures to run an image, particularly when a user
specifies a `--platform`.
Typically what we see in the wild is a manifest list with an an entry
for, as an example, linux/arm64 pointing to an image config that has
linux/amd64 on it.

This change falls back to looking up the manifest list for an image to
see if the manifest list shows the image as the correct one for that
platform.

In order to accomplish this we need to traverse the leases associated
with an image. Each image, if pulled with Docker 20.10, will have the
manifest list stored in the containerd content store with the resource
assigned to a lease keyed on the image ID.
So we look up the lease for the image, then look up the assocated
resources to find the manifest list, then check the manifest list for a
platform match, then ensure that manifest referes to our image config.

This is only used as a fallback when a user specified they want a
particular platform and the image config that we have does not match
that platform.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 4be5453)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Wrap platforms.Only and fallback to our ignore mismatches due to  empty
CPU variants. This just cleans things up and makes the logic re-usable
in other places.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 50f39e7)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
…ntegration_test_fix_subnet_missmatch

[20.10 backport] Update TestDaemonRestartWithLiveRestore: fix docker0 subnet missmatch
[20.10 backport] Update rootlesskit to v0.13.1 to fix handling of IPv6 addresses
[20.10 backport] update runc binary to v1.0.0-rc93
…rns_uid_username_match

[20.10 backport] Fix userns-remap option when username & UID match
…casing

[20.10 backport] api/docs: fix NanoCPUs casing in swagger
…le_panic

[20.10 backport] Fix daemon panic when starting container with invalid device cgroup rule
[20.10 backport] assorted small fixes, docs changes, and contrib
[20.10 backport] replace json.Unmarshal with NewFromJSON in Create
…nconsisent_platform

[20.10 backport] Fix builder inconsistent error on buggy platform
[20.10 backport] builder: ensure libnetwork state file do not leak
…9_plugins_custom_runtime-panic

[20.10 backport] Add shim config for custom runtimes for plugins
[20.10 backport] profiles: seccomp: update to Linux 5.11 syscall list
…d_plat

[20.10] Fallback to manifest list when no platform match
thaJeztah and others added 14 commits November 17, 2021 20:34
[20.10] vendor: update github.com/docker/distribution and github.com/containerd/containerd
imageutil: make mediatype detection more stricter to mitigate CVE-2021-41190.

full diff: moby/buildkit@244e8cd...bc07b2b

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The twelfth patch release for containerd 1.4 contains a few minor bug fixes
and an update to mitigate CVE-2021-41190.

Notable Updates

* Handle ambiguous OCI manifest parsing GHSA-5j5w-g665-5m35
* Update pull to try next mirror for non-404 errors
* Update pull to handle of non-https urls in descriptors

See the changelog for complete list of changes

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Looks like vndr didn't like the replace rule missing a scheme;

    github.com/docker/distribution: Err: exit status 128, out: fatal: repository 'github.com/samuelkarp/docker-distribution' does not exist
    github.com/containerd/containerd: Err: exit status 128, out: fatal: repository 'github.com/moby/containerd' does not exist

While at it, I also replaced the schem for go-immutable-radix, because GitHub
is deprecating the git:// protocol.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Bring mediaType out of reserved status
- specs-go: adding mediaType to the index and manifest structures

full diff: opencontainers/image-spec@v1.0.1...v1.0.2

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit cef0a7c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
[20.10] update containerd binary to v1.4.12
[20.10] vendor: github.com/moby/buildkit v0.8.3-4-gbc07b2b8
…spec

[20.10 backport] vendor: github.com/opencontainers/image-spec v1.0.2
[20.10 backport] info: unset cgroup-related fields when CgroupDriver == none
…lena

* commit '847da184ad5048b27f5bdf9d53d070f731b43180': (1704 commits)
  vendor: github.com/opencontainers/image-spec v1.0.2
  [20.10] fix vendor validation
  [20.10] update containerd binary to v1.4.12
  [20.10] vendor: github.com/moby/buildkit v0.8.3-4-gbc07b2b8
  distribution: validate blob type
  vendor: update github.com/containerd/containerd
  TestBuildUserNamespaceValidateCapabilitiesAreV2: cleanup daemon storage
  testutil: daemon.Cleanup(): cleanup more directories
  info: unset cgroup-related fields when CgroupDriver == none
  [20.10] Update Go to 1.16.10
  vendor: update github.com/docker/distribution
  cmd/dockerd: create panic.log file without readonly flag
  Fix race in TestCreateServiceSecretFileMode, TestCreateServiceConfigFileMode
  Fix racey TestHealthKillContainer
  dockerd-rootless.sh: Fix variable not double quotes cause unexpected behavior
  Remove needless check
  Fix log statement 'failed to exit' timeout accuracy
  docker daemon container stop refactor
  docker kill: fix bug where failed kills didnt fallback to unix kill
  buildkit: normalize build target and local platform
  ...
Signed-off-by: Robert Günzler <robertg@balena.io>
Signed-off-by: Robert Günzler <robertg@balena.io>
Signed-off-by: Robert Günzler <robertg@balena.io>
@ghost
Copy link

ghost commented Jan 5, 2022

An error occurred whilst building your landr site preview:

{
  "methodName": "constructor",
  "name": "Error",
  "message": "Could not find MIME for Buffer <null>",
  "stack": "Error: Could not find MIME for Buffer <null>\n    at Jimp.parseBitmap (/usr/src/app/node_modules/@jimp/core/dist/utils/image-bitmap.js:187:15)\n    at Jimp.parseBitmap (/usr/src/app/node_modules/@jimp/core/dist/index.js:431:32)\n    at new Jimp (/usr/src/app/node_modules/@jimp/core/dist/index.js:384:13)\n    at _construct (/usr/src/app/node_modules/@babel/runtime/helpers/construct.js:19:21)\n    at /usr/src/app/node_modules/@jimp/core/dist/index.js:926:32\n    at new Promise (<anonymous>)\n    at Function.Jimp.read (/usr/src/app/node_modules/@jimp/core/dist/index.js:925:10)\n    at NodeImage._loadByJimp (/usr/src/app/node_modules/node-vibrant/lib/image/node.js:87:21)\n    at NodeImage.load (/usr/src/app/node_modules/node-vibrant/lib/image/node.js:98:25)\n    at Vibrant.getPalette (/usr/src/app/node_modules/node-vibrant/lib/vibrant.js:58:28)\n    at Builder.getPalette (/usr/src/app/node_modules/node-vibrant/lib/builder.js:57:29)\n    at module.exports (/usr/src/app/lib/theme.js:29:33)\n    at Object.exports.run (/usr/src/app/lib/build-runner.js:164:27)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async build (/usr/src/app/bot/index.js:114:19)\n    at async /usr/src/app/bot/index.js:157:25\n    at async Promise.all (index 0)\n    at async middleware (/usr/src/app/node_modules/@octokit/webhooks/dist-node/index.js:355:5)"
}

Signed-off-by: Robert Günzler <robertg@balena.io>
@robertgzr
Copy link
Contributor Author

@balena-ci retest

2 similar comments
@robertgzr
Copy link
Contributor Author

@balena-ci retest

@robertgzr
Copy link
Contributor Author

@balena-ci retest

@robertgzr robertgzr marked this pull request as ready for review February 18, 2022 11:30
@robertgzr
Copy link
Contributor Author

robertgzr commented Feb 18, 2022

alright! ci passing, thanks to:

the last two will most likely make it incompatible with the pre-20.10 version, because of the new target hack/make.sh dynbinary vs hack/make.sh dynbinary-balena and Go 1.16

@lmbarros I don't expect you to review the huge changeset but maybe checkout the branch and run the integration tests yourself, as we don't get those done by CI :)

@robertgzr
Copy link
Contributor Author

I'm relatively confident about this PR though, I opened balena-os/meta-balena#2498 already, which builds on all device types and passes the unmanaged and HUP suites on genericx86-64

Copy link
Contributor

@lmbarros lmbarros left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I focused on a few of the files I am more familiar with, tried a cross-build to arm64 and ran tests. 🚀

PS: Love the new "Working on balenaEngine" instructions! 🙂

@ghost ghost merged commit 50784e9 into master Feb 21, 2022
@robertgzr
Copy link
Contributor Author

@lmbarros cheers o/

Love the new "Working on balenaEngine" instructions

always trying to reduce the amount of "secret" knowledge that only lives in my head haha

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.