build_and_upload_contract CI job fails

[ethanfrey]

This started with v0.6.0-beta1 which was the first release to include cw1155. However, the issue doesn't seem tied directly to this contract. See https://app.circleci.com/pipelines/github/CosmWasm/cosmwasm-plus/1101/workflows/7db524d2-1c96-4f5c-b216-858c2af3ba08/jobs/19391 (I re-ran this job and got the same results).

It runs normally (in ~8 minutes) to compile all contracts and start optimizing. Then it just hangs for > 10 minutes at this point:

Building contracts/cw721-base ...
   Compiling cw721 v0.6.0-beta1 (/code/packages/cw721)
   Compiling cw721-base v0.6.0-beta1 (/code/contracts/cw721-base)
    Finished release [optimized] target(s) in 26.00s
done.
Optimizing artifacts in workspace...Optimizing ../target/wasm32-unknown-unknown/release/deps/cw1155_base.wasm...done.
Optimizing ../target/wasm32-unknown-unknown/release/deps/cw1_subkeys.wasm...done.
Optimizing ../target/wasm32-unknown-unknown/release/deps/cw1_whitelist.wasm...done.
Optimizing ../target/wasm32-unknown-unknown/release/deps/cw20_atomic_swap.wasm...done.

Too long with no output (exceeded 10m0s): context deadline exceeded

We could increase the timeout but I don't think that is the problem as no job should take > 10 minutes to optimize.

This is what we see when running locally:

Optimizing ../target/wasm32-unknown-unknown/release/deps/cw20_atomic_swap.wasm...done.
Optimizing ../target/wasm32-unknown-unknown/release/deps/cw20_base.wasm...done.
# and much more...

So, it hangs while optimizing cw20_base.wasm?? Maybe some memory/cpu issue in CI? (As it works fine locally)

[maurolacy]

This is that kind of issue that goes away when debugging it (an heisenbug).

Probably some kind of weird bug in CircleCI, related to (remote) docker containers, I/O redirection and such.

Simplest way to "fix it" that I found was to change the shell to /bin/sh. Enabling shell tracing with set -x seems to fix it, too; even if keeping /bin/ash as shell.

Anyway. Will create a pull request in rust-optimizer, and reference it here.

[ethanfrey]

This is that kind of issue that goes away when debugging it (an eisenbug).

Heisenburg? "Observation changes state"?

Simplest way to "fix it" that I found was to change the shell to /bin/sh. Enabling shell tracing with set -x seems to fix it, too; even if keeping /bin/ash as shell.

So odd... Why are you using /bin/ash anyway? I think /bin/sh is a safe fix.

Thank you for digging into this one and getting to the bottom. Such tricky issues

[maurolacy]

This is that kind of issue that goes away when debugging it (an eisenbug).

Heisenburg? "Observation changes state"?

Yes, heisenbug, exactly.

See CosmWasm/optimizer#44 for the "fix".

[maurolacy]

So odd... Why are you using /bin/ash anyway? I think /bin/sh is a safe fix.

To silence some shell syntax warnings, thrown by the shell linter. Nice thing is that changing to /bin/sh still keeps the warnings quiet, because of a directive (# shellcheck shell=dash) in the scripts.

ash is not a bad choice, but sh is the safest / most compatible bet, yes.

[webmaster128]

The beta2 built just succeded after a re-run of the unchanged CI job.

So, it hangs while optimizing cw20_base.wasm??

Yeah, always this one. Is there anything special about this? I wonder why it is so small (just 12KB compared to 200KB for the other contracts). This does not happen if I do (cd contracts/cw3-fixed-multisig && RUSTFLAGS="-C link-arg=-s" cargo build --release --target=wasm32-unknown-unknown --locked).

But look at the paths in the console log: ../target/wasm32-unknown-unknown/release/deps/cw1_subkeys.wasm. This is not the folder we are looking for. In this folder we get a 15KB cw20_base.wasm when compiling (cd contracts/cw20-bonding && RUSTFLAGS="-C link-arg=-s" cargo build --release --target=wasm32-unknown-unknown --locked).

I'm still not sure why this causes the hanging CI job or haning wasmopt, but we should fix our *.wasm search strategy first.

[webmaster128]

This bug was introduced with rust-optimizer:0.11.0 where we traded the use of Rust stable against the unstable --out-dir option.

[ethanfrey]

Huh, is this closed and tested?

#284 seems to have errors. I would reopen this until we have a proper build on cosmwasm-plus ci we can point to

[maurolacy]

But look at the paths in the console log: ../target/wasm32-unknown-unknown/release/deps/cw1_subkeys.wasm. This is not the folder we are looking for.

Good catch, that is not the right path! I fixed it, and now the CI is passing! See https://app.circleci.com/pipelines/github/CosmWasm/cosmwasm-plus/1148/workflows/cb4ac4ca-9063-4af5-88a0-3e293da7465c/jobs/20430

So, I would say, this is the minimal change we need to do to make it work. Will cut v0.11.1 with this fix alone.

[webmaster128]

👍 thanks

Will cut v0.11.1 with this fix alone.

Let's use 0.11.2 to avoid any unnecessary confusion about the different versions and tags.

[ethanfrey]

We need to push 0.11.2 to docker hub and merge that into the cosmwasm-plus CircleCI in order to actually close this issue. I will push this to the docker hub: https://hub.docker.com/r/cosmwasm/rust-optimizer/tags?page=1&ordering=last_updated

Please comment here when the cosmwasm-plus ci pr is ready

[ethanfrey]

I just tagged v0.11.2 with this (and an update to the dockerfile version number) and building and publishing to docker hub

[ethanfrey]

https://hub.docker.com/layers/cosmwasm/rust-optimizer/0.11.2/images/sha256-9ec1aea495785a10d1f4643ff3f096ba15dc3f770611f835ff669ef605f24007?context=explore is on the hub