Improve buildx test coverage #1857
Labels
Milestone
Comments
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 2, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857.
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 2, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 2, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 2, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 3, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
jsternberg
added a commit
to jsternberg/buildx
that referenced
this issue
Aug 3, 2023
An integration test for `docker buildx version` has been created. The integration test checks that there is one line output, the output is composed of three sections, and that these sections could feasibly be the package path, version, and revision information. The intention of the checks is to find obvious errors in the output like the package path not existing or the version and revision being swapped. It is not intended to assert that these values must be certain values because it is assumed these values may vary depending on the build process for buildx. Related to docker#1857. Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
|
Updated issue description with new cases and ones fixed. Also pinned this issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've now merged #1770 to add support for integration tests 🎉 However, buildx has been around for several years before these tests, so we don't have anywhere near the amount of test coverage that we want. These tests are essential to track regressions, detect changes in behavior as we refactor and build new functionality, and add new modes and drivers.
In general, all new functionality should be added with tests wherever feasible, but we need a plan to work through the existing backlog of code with no tests. This issue is to track what broad groups of tests we need, and how far along we are. This list isn't currently complete -- but as a general rule of thumb, if we document the behavior on https://docs.docker.com, we should be testing it.
Ideally, each sub-point in the list would end up as a PR, though maybe it makes sense to group some of the smaller ones together, or split some of the larger ones up.
(note to maintainers, feel free to add anything that you think is missing 🥳)
The first type of tests that we need are for the top-level
commandspackage - at the time of writing, these currently reside in the top-leveltestspackage. These are designed to test each of the buildx subcommands, such asbuild,bake,inspect, etc.The areas that we need coverage for:
buildx build/buildx bake--platform) tests: build multiplatform #2620--cache-to/--cache-from/--no-cache/--no-cache-filter)--output/--load/--push/--tag) build: handle push/load shorthands for multi exporters #2302--build-arg/--build-context) build: validate build args and labels #2040--metadata-file/--iidfile)--print) test: build print #2423--secret/--ssh) tests: build secret #2381--shm-size) test: build shm-size and ulimit #2279--ulimit) test: build shm-size and ulimit #2279--label) build: validate build args and labels #2040--attest/--provenance/--sbom) build: error on attests on non-multiplatform driver #1988--progress) tests: add basic build progress test #1925BUILDX_EXPERIMENTALenv var,--detachflag) tests: test with buildx experimental #2376buildx imagetoolsbuildx imagetools inspect) imagetools: Allow annotations for OCI image index #1965 tests: add integration test for imagetools create #1978buildx imagetools create) tests: add imagetools tests for copying manifests and indexes #2013--invokeflag--invokeflag (debug-shell,on-error, shell commands)attach/list/help/etc)buildx du/buildx prunebuildx version(Integration test for docker buildx version #1985)buildx create/buildx rm/buildx ls/buildx stop/buildx useThe second type of tests that we need are for the top-level
driverpackage. These drivers allow connecting to buildkit running in different configurations. Each driver supports lots of different types of options, and so we should have tests that test for configuring and setting up buildkit with these different options. Currently, we don't have any tests of this format.The areas that we need coverage for:
dockerdriverdocker-containerdriver (see docs)kubernetesdriver (see docs)remotedriver (see docs)The text was updated successfully, but these errors were encountered: