Regressions in integration tests since the 20.10 upgrade

[lmbarros]

Several of "CLI integration tests" are failing since we upgraded to 20.10 (this PR this PR, and specifically this commit).

We must investigate and fix these failures. I looked at just a few of them, which are related with building images and seem like legit issues we should look into.

Here's a list of failures I get on my laptop when running TIMEOUT=240m make TEST_INTEGRATION_DIR=./integration-cli/ test-integration:

FAIL: TestDockerSuite/TestBuildAddFileNotFound
FAIL: TestDockerSuite/TestBuildAddRemoteFileMTime
FAIL: TestDockerSuite/TestBuildBuildTimeArg
FAIL: TestDockerSuite/TestBuildBuildTimeArgDefaultOverride
FAIL: TestDockerSuite/TestBuildBuildTimeArgDefinitionWithNoEnvInjection
FAIL: TestDockerSuite/TestBuildBuildTimeArgEnv
FAIL: TestDockerSuite/TestBuildBuildTimeArgHistory
FAIL: TestDockerSuite/TestBuildBuildTimeArgOverrideArgDefinedBeforeEnv
FAIL: TestDockerSuite/TestBuildBuildTimeArgOverrideEnvDefinedBeforeArg
FAIL: TestDockerSuite/TestBuildBuildTimeArgUnconsumedArg
FAIL: TestDockerSuite/TestBuildCacheFrom
FAIL: TestDockerSuite/TestBuildCacheFromEqualDiffIDsLength
FAIL: TestDockerSuite/TestBuildClearCmd
FAIL: TestDockerSuite/TestBuildContainerWithCgroupParent
FAIL: TestDockerSuite/TestBuildCopyWildcardCache
FAIL: TestDockerSuite/TestBuildEmptyScratch
FAIL: TestDockerSuite/TestBuildFails
FAIL: TestDockerSuite/TestBuildFromGitWithContext
FAIL: TestDockerSuite/TestBuildFromGitWithF
FAIL: TestDockerSuite/TestBuildFromMixedcaseDockerfile
FAIL: TestDockerSuite/TestBuildFromStdinWithF
FAIL: TestDockerSuite/TestBuildFromURLWithF
FAIL: TestDockerSuite/TestBuildIntermediateTarget
FAIL: TestDockerSuite/TestBuildJSONEmptyRun
FAIL: TestDockerSuite/TestBuildMultiStageArg
FAIL: TestDockerSuite/TestBuildMultiStageCache
FAIL: TestDockerSuite/TestBuildMultiStageCopyFromErrors
FAIL: TestDockerSuite/TestBuildMultiStageCopyFromSyntax
FAIL: TestDockerSuite/TestBuildMultiStageGlobalArg
FAIL: TestDockerSuite/TestBuildMultiStageResetScratch
FAIL: TestDockerSuite/TestBuildMultiStageUnusedArg
FAIL: TestDockerSuite/TestBuildNetContainer
FAIL: TestDockerSuite/TestBuildNetNone
FAIL: TestDockerSuite/TestBuildNoDupOutput
FAIL: TestDockerSuite/TestBuildNotVerboseFailure
FAIL: TestDockerSuite/TestBuildNotVerboseFailureWithNonExistImage
FAIL: TestDockerSuite/TestBuildOnBuildLimitedInheritance
FAIL: TestDockerSuite/TestBuildOnBuildOutput
FAIL: TestDockerSuite/TestBuildRUNErrMsg
FAIL: TestDockerSuite/TestBuildRUNoneJSON
FAIL: TestDockerSuite/TestBuildResourceConstraintsAreUsed
FAIL: TestDockerSuite/TestBuildRm
FAIL: TestDockerSuite/TestBuildSameDockerfileWithAndWithoutCache
FAIL: TestDockerSuite/TestBuildShellInherited
FAIL: TestDockerSuite/TestBuildShellUpdatesConfig
FAIL: TestDockerSuite/TestBuildSpacesWithQuotes
FAIL: TestDockerSuite/TestBuildStartsFromOne
FAIL: TestDockerSuite/TestBuildStderr
FAIL: TestDockerSuite/TestBuildStepsWithProgress
FAIL: TestDockerSuite/TestBuildVerboseOut
FAIL: TestDockerSuite/TestBuildVolumeFileExistsinContainer
FAIL: TestDockerSuite/TestBuildWithFailure
FAIL: TestDockerSuite/TestBuildWithInaccessibleFilesInContext
FAIL: TestDockerSuite/TestBuildWithTabs
FAIL: TestDockerSuite/TestBuildWorkdirCmd
FAIL: TestDockerSuite/TestConflictContainerNetworkAndLinks
FAIL: TestDockerSuite/TestConflictContainerNetworkHostAndLinks
FAIL: TestDockerSuite/TestEventsFilterType
FAIL: TestDockerSuite/TestExecStateCleanup
FAIL: TestDockerSuite/TestImagesFilterSinceAndBefore
FAIL: TestDockerSuite/TestImagesOrderedByCreationDate
FAIL: TestDockerSuite/TestLinksPingLinkedContainers
FAIL: TestDockerSuite/TestPortHostBinding
FAIL: TestDockerSuite/TestPortList
FAIL: TestDockerSuite/TestPsListContainersFilterAncestorImage
FAIL: TestDockerSuite/TestRmiParentImageFail
FAIL: TestDockerSuite/TestRmiTag
FAIL: TestDockerSuite/TestRmiTagWithExistingContainers
FAIL: TestDockerSuite/TestRmiUntagHistoryLayer
FAIL: TestDockerSuite/TestRunTLSVerify
FAIL: TestDockerSuite/TestRunTmpfsMountsOverrideImageVolumes
FAIL: TestDockerSuite/TestRunWithNetAliasOnDefaultNetworks
FAIL: TestDockerSuite/TestUserDefinedNetworkConnectDisconnectAlias
FAIL: TestDockerSuite/TestVolumeCLINoArgs
FAIL: TestDockerRegistrySuite/TestBuildByDigest
FAIL: TestDockerRegistrySuite/TestBuildMultiStageImplicitPull
FAIL: TestDockerRegistrySuite/TestPsListContainersFilterAncestorImageByDigest
FAIL: TestDockerRegistrySuite/TestPushMultipleTags
FAIL: TestDockerSchema1RegistrySuite/TestPullIDStability
FAIL: TestDockerSchema1RegistrySuite/TestPushMultipleTags
FAIL: TestDockerRegistryAuthHtpasswdSuite/TestBuildFromAuthenticatedRegistry
FAIL: TestDockerRegistryAuthHtpasswdSuite/TestBuildWithExternalAuth
FAIL: TestDockerPluginSuite/TestPluginIDPrefix

[zoobot]

@lmbarros Want to divide and conquer? Can one of us start from the bottom and one start from the top or do they need to be run sequentially?

[zoobot]

These tests are passing in Moby, might be worth checking out, or talk about best ways to sync their fixes.

[lmbarros]

Hi @zoobot !

Yup, as far as I know these tests have always passed on Moby. They started failing after we merged 20.10 from Moby into balenaEngine (on this PR -- my links on the issue description were wrong, I corrected it now.)

I cannot look at this right now, need to finish other stuff first. (But feel free, of course!)

[lmbarros]

Internal balena link with some notes about this I wrote some months back. Summary: some of the failures are because our fork is using Buildkit (unlike upstream at the corresponding point in time). Buildkit and non-buildkit error messages are different, so some tests fail because they are expecting to fail with a different error message.