tikv panic when to rolling update tikv to v3.0.2 from v2.1.3

[cwen0]

tikv panic log:

thread 'main' panicked at 'invalid auto generated configuration file conf/tikv.toml, err invalid number at line 17', src/config.rs:1453:13
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/libunwind.rs:88
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/gh.neting.cc-1ecc6299db9ec823/backtrace-0.3.29/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:47
   3: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:36
   4: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   5: std::panicking::default_hook
             at src/libstd/panicking.rs:212
   6: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:475
   7: std::panicking::continue_panic_fmt
             at src/libstd/panicking.rs:382
   8: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:337
   9: tikv::config::TiKvConfig::from_file::{{closure}}
             at /home/jenkins/workspace/release_tidb_3.0/tikv/<::std::macros::panic macros>:9
  10: core::result::Result<T,E>::unwrap_or_else
             at /rustc/0e4a56b4b04ea98bb16caada30cb2418dd06e250/src/libcore/result.rs:766
  11: tikv::config::TiKvConfig::from_file
             at /home/jenkins/workspace/release_tidb_3.0/tikv/src/config.rs:1448
  12: tikv_server::main::{{closure}}
             at src/bin/tikv-server.rs:140
  13: core::option::Option<T>::map_or_else
             at /rustc/0e4a56b4b04ea98bb16caada30cb2418dd06e250/src/libcore/option.rs:460
  14: tikv_server::main
             at src/bin/tikv-server.rs:138
  15: std::rt::lang_start::{{closure}}
             at /rustc/0e4a56b4b04ea98bb16caada30cb2418dd06e250/src/libstd/rt.rs:64
  16: main
  17: __libc_start_main
  18: <unknown>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I changed the tidb-ansible version to v3.0.2 and use the old tikv config. I found the following config in the new tikv config file which produced by tidb-ansible. block-cache can not be recognized by tikv and caused tikv panic.

[storage]
block-cache = null   

config in v2.1.3: https://github.com/pingcap/tidb-ansible/blob/v2.1.3/conf/tikv.yml#L84-L100
in v3.0.2: https://github.com/pingcap/tidb-ansible/blob/v3.0.2/conf/tikv.yml#L106-L146

[kennytm]

The problem is because in YAML

x:
  # blah
  #t: 0
  # blah 
  # blah 
  #y: z

is interpreted as

{"x": null}  // unexpected null

where what TiKV expected is an empty dictionary

{"x": {}}    // what we really want

You should provide at least one default item to force the type to be a dictionary.

x:
  # blah
  t: 0
  # blah
  # blah
  #y: z

[cwen0]

@kennytm thanks! I will add a check cmd in tikv to help tidb-ansible identify the diff config. so we can avoid this scene before to upgrade config.