zfs kernel panic & pool lockup - VERIFY3(range_tree_space(rt) + size <= sm->sm_size) failed

[mcr-ksh]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	Bionic (18.04.2) LTS
Linux Kernel	4.15.0-51-generic
Architecture	x86_64
ZFS Version	0.7.5-1ubuntu16.4
SPL Version	0.7.5-1ubuntu1

Describe the problem you're observing

the pool locks up with the following kernel panic. system must be rebooted.

Describe how to reproduce the problem

there are VMs on the pool. Just fire up the VM and the lockup will occur.

Include any warning/errors/backtraces from the system logs

dmesg:

[ 3314.588652] VERIFY3(range_tree_space(rt) + size <= sm->sm_size) failed (16908288 <= 16777216)
[ 3314.588748] PANIC at space_map.c:118:space_map_load()
[ 3314.588797] Showing stack for process 6311
[ 3314.588803] CPU: 10 PID: 6311 Comm: worker Tainted: P           OE    4.15.0-51-generic #55-Ubuntu
[ 3314.588805] Hardware name: Dell Inc. Precision T7600/082WXT, BIOS A12 09/29/2014
[ 3314.588807] Call Trace:
[ 3314.588825]  dump_stack+0x63/0x8b
[ 3314.588844]  spl_dumpstack+0x42/0x50 [spl]
[ 3314.588853]  spl_panic+0xc8/0x110 [spl]
[ 3314.589385]  ? _nv022213rm+0x169/0x1c0 [nvidia]
[ 3314.589392]  ? avl_add+0x67/0x90 [zavl]
[ 3314.589396]  ? avl_add+0x67/0x90 [zavl]
[ 3314.589494]  ? metaslab_rt_add+0x2b/0x60 [zfs]
[ 3314.589563]  ? range_tree_add+0x13b/0x300 [zfs]
[ 3314.589623]  ? dnode_rele+0x39/0x40 [zfs]
[ 3314.589698]  space_map_load+0x3fe/0x4f0 [zfs]
[ 3314.589766]  metaslab_load+0x36/0xf0 [zfs]
[ 3314.589829]  metaslab_activate+0x93/0xc0 [zfs]
[ 3314.589835]  ? _cond_resched+0x19/0x40
[ 3314.589897]  metaslab_alloc+0x496/0x1150 [zfs]
[ 3314.589972]  zio_alloc_zil+0x6a/0x1f0 [zfs]
[ 3314.589977]  ? mutex_lock+0x12/0x40
[ 3314.590049]  ? txg_list_add+0x7b/0xa0 [zfs]
[ 3314.590121]  zil_lwb_write_start+0x1c0/0x370 [zfs]
[ 3314.590190]  zil_commit_writer+0x2d6/0x7c0 [zfs]
[ 3314.590257]  zil_commit.part.14+0x93/0x100 [zfs]
[ 3314.590323]  zil_commit+0x17/0x20 [zfs]
[ 3314.590389]  zfs_fsync+0x77/0xe0 [zfs]
[ 3314.590457]  zpl_fsync+0x68/0xa0 [zfs]
[ 3314.590465]  vfs_fsync_range+0x51/0xb0
[ 3314.590469]  do_fsync+0x3d/0x70
[ 3314.590473]  SyS_fdatasync+0x13/0x20
[ 3314.590480]  do_syscall_64+0x73/0x130
[ 3314.590485]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[ 3314.590489] RIP: 0033:0x7f5b1a9d82e7
[ 3314.590491] RSP: 002b:00007f59337fd8c0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
[ 3314.590496] RAX: ffffffffffffffda RBX: 0000000000000014 RCX: 00007f5b1a9d82e7
[ 3314.590498] RDX: 0000000000000000 RSI: 0000561f0bc35da0 RDI: 0000000000000014
[ 3314.590500] RBP: 0000561f0cfdb0d0 R08: 0000000000000000 R09: 00000000ffffffff
[ 3314.590503] R10: 00007f59337fd900 R11: 0000000000000293 R12: 0000561f0e590750
[ 3314.590505] R13: 0000561f0cec4998 R14: 0000561f0cec4920 R15: 00007f59ca1f57a0

Don't take this for a word, but the problem first occurred one week after swap on ZFS volume was introduced. #7734 not sure if it's related or not. We are currently rolling back all the snapshots to the date, in case something internal/critital was overwritten

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.