~/.panacea/wasm/ directory isn't populated when syncing historical blocks from the genesis

[youngjoon-lee]

Background

We've expected that the ~/.panacea/wasm/ directory is also populated when starting a new node by syncing historical blocks from the genesis. But, it was not.
I deployed new Testnet nodes with the same manner above, but the ~/.panacea/wasm/ directory remains empty.

So, when we runs the command: panacead export, the following error occurs.

**panic: Error calling the VM: Cache error: Error opening Wasm file for reading: No such file or directory (os error 2)

goroutine 1 [running]:
github.com/CosmWasm/wasmd/x/wasm/keeper.ExportGenesis.func1(0x1, 0xc005140ca0, 0x20, 0x20, 0xc003fcd4d0, 0x2e, 0x0, 0x0, 0x0, 0x0, ...)
	/home/ubuntu/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.17.0/x/wasm/keeper/genesis.go:90 +0x417
github.com/CosmWasm/wasmd/x/wasm/keeper.Keeper.IterateCodeInfos(0x228d960, 0xc000d7c310, 0x22c5ab8, 0xc000d7c610, 0x22a31f0, 0xc0000f4e10, 0x226e3a0, 0xc000d7c8f0, 0x226b940, 0xc00000c730, ...)
	/home/ubuntu/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.17.0/x/wasm/keeper/keeper.go:732 +0x295
github.com/CosmWasm/wasmd/x/wasm/keeper.ExportGenesis(0x22a9768, 0xc00011e010, 0x22bfe48, 0xc002b9c100, 0x0, 0x0, 0x0, 0x0, 0x3e7143, 0x0, ...)
	/home/ubuntu/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.17.0/x/wasm/keeper/genesis.go:87 +0x288
github.com/CosmWasm/wasmd/x/wasm.AppModule.ExportGenesis(0x22c5ab8, 0xc000d7c610, 0xc000f599b0, 0x226e9e0, 0xc000d2af20, 0x22a9768, 0xc00011e010, 0x22bfe48, 0xc002b9c100, 0x0, ...)
	/home/ubuntu/go/pkg/mod/github.com/!cosm!wasm/wasmd@v0.17.0/x/wasm/module.go:150 +0x7d
github.com/cosmos/cosmos-sdk/types/module.(*Manager).ExportGenesis(0xc000da0150, 0x22a9768, 0xc00011e010, 0x22bfe48, 0xc002b9c100, 0x0, 0x0, 0x0, 0x0, 0x3e7143, ...)
	/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.42.7/types/module/module.go:325 +0x135
github.com/medibloc/panacea-core/v2/app.(*App).ExportAppStateAndValidators(0xc000f59200, 0xc000f4b400, 0x2f8af30, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/ubuntu/panacea-core/app/export.go:33 +0x1e8
github.com/medibloc/panacea-core/v2/cmd/panacead/cmd.appCreator.appExport(0x22b5b08, 0xc00000f1b8, 0x22c5ab8, 0xc000d7c610, 0x22bc398, 0xc000ccd580, 0xc0001231e8, 0x22aa5a0, 0xc000f4b440, 0x22c2390, ...)
	/home/ubuntu/panacea-core/cmd/panacead/cmd/root.go:279 +0x34d
github.com/cosmos/cosmos-sdk/server.ExportCmd.func1(0xc000ddc780, 0x2f8af30, 0x0, 0x0, 0x0, 0x0)
	/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.42.7/server/export.go:71 +0x45e
github.com/spf13/cobra.(*Command).execute(0xc000ddc780, 0x2f8af30, 0x0, 0x0, 0xc000ddc780, 0x2f8af30)
	/home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:852 +0x472
github.com/spf13/cobra.(*Command).ExecuteC(0xc000daa500, 0x0, 0x0, 0xc0000478f0)
	/home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
github.com/spf13/cobra.(*Command).ExecuteContext(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:890
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000daa500, 0xc0000478f0, 0x15, 0x22c5ab8, 0xc000d7c610)
	/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.42.7/server/cmd/execute.go:36 +0x265
main.main()
	/home/ubuntu/panacea-core/cmd/panacead/main.go:13 +0x45

I've seen that some wasm data is included in the genesis.json, but it seems that it's not enough.

Goal

Let's

• find the exact scenarios which reproduce that issue.
• find the reason.
• find the solution.

After a quick googling, I found the similar issue: CosmWasm/wasmd#478. This issue is related to the state sync, but I think it has the core reason is the same.

[gyuguen]

When I checked on the testnet, there was no problem when registering as genesis.json.

# Some genesis.json.(This is genesis.json exported from hygieia-5.)
{
"wasm": {
      "codes": [
        {
          "code_bytes": "AGFzbQEAAAABlAEW ... ",
          "code_id": "1",
          "code_info": {
            "builder": "",
            "code_hash": "ekZZPXevCSSAokBmxN8rBKAv7NAGDmawl1vtynIc3gg=",
            "creator": "panacea136e7cal0p66vrrk5plvtkay5t2j6xwx4zmx6e6",
            "instantiate_config": {
              "address": "",
              "permission": "Everybody"
            },
            "source": ""
          },
          "pinned": false
    }
  ]
}

Save information from hygieia-5 (old)

# /home/ubuntu/.panacea_hygieia_5/wasm
ubuntu@testnet-sentry:~/.panacea_hygieia_5/wasm$ tree
.
└── wasm
    ├── modules
    └── wasm

Save information from hygieia-6 (new)

# /home/ubuntu/.panacea/wasm
ubuntu@testnet-sentry:~/.panacea/wasm$ tree
.
└── wasm
    ├── modules
    │   └── v1
    │       └── 7a46593d77af092480a24066c4df2b04a02fecd0060e66b0975bedca721cde08
    └── wasm
        └── 7a46593d77af092480a24066c4df2b04a02fecd0060e66b0975bedca721cde08

[youngjoon-lee]

Does it mean that the ~/.panacea/wasm directory is populated well when starting a new node from the genesis?

[gyuguen]

Yes, that's right. I didn't copy those binary ones.

[youngjoon-lee]

Aha. Then, we can probably close this issue without any action. I still don't get why this issue happened on the testnet before, but I guess something was messed up when we moved the testnet across different infrastructures.

[gyuguen]

Yes, the problem is not reproduced at the moment, so I think it's okay to close this issue.