0.6.5.3 PANIC at zfs_vfsops.c:426:zfs_space_delta_cb()

[ebyrne242]

This is the second time I've seen this problem. The first time was a week ago, a day after I updated to 0.6.5.3. Previously, 0.6.4.x had been flawless for me.

I search bug reports and saw a couple of other people with similar errors, but they were all blaming zfs send or similar command. I was not doing any zfs/pool related commands, though there may have been normal user-level R/W going on at the time. This is a single-device dev. After the panic, zpool status reports no errors, and I am still able to access the filesytem normally. However, zpool history or zpool scrub just hangs.

It may also be worth noting, I have another pool on the same system on which zpool history does not hang, so I assume the panic only affected a single pool.

[316806.972922] VERIFY3(sa.sa_magic == 0x2F505A) failed (1446016985 == 3100762)
[316806.972943] PANIC at zfs_vfsops.c:426:zfs_space_delta_cb()
[316806.972949] Showing stack for process 2305
[316806.972952] CPU: 1 PID: 2305 Comm: txg_sync Tainted: P O 4.2.3-1.el7.elrepo.x86_64 #1
[316806.972954] Hardware name: Apple Inc. Macmini4,1/Mac-F2208EC8, BIOS MM41.88Z.0042.B00.1004221740 04/22/10
[316806.972957] 0000000000000000 00000000ccdad7d7 ffff88012fbfb7a8 ffffffff816c73b9
[316806.972960] 0000000000000000 ffffffffa01a89bb ffff88012fbfb7b8 ffffffffa000e5b4
[316806.972963] ffff88012fbfb948 ffffffffa000e67b ffff88012fbfb7d8 ffffffff810aab9c
[316806.972966] Call Trace:
[316806.972975] [] dump_stack+0x45/0x57
[316806.972988] [] spl_dumpstack+0x44/0x50 [spl]
[316806.972994] [] spl_panic+0xbb/0xf0 [spl]
[316806.972998] [] ? __enqueue_entity+0x6c/0x70
[316806.973001] [] ? __slab_free+0x11f/0x217
[316806.973063] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
[316806.973069] [] ? __schedule+0x2af/0x880
[316806.973077] [] ? spl_kmem_free+0x2a/0x40 [spl]
[316806.973120] [] zfs_space_delta_cb+0xc2/0x190 [zfs]
[316806.973151] [] dmu_objset_userquota_get_ids+0xdc/0x450 [zfs]
[316806.973183] [] dnode_sync+0xed/0x9a0 [zfs]
[316806.973191] [] ? spl_kmem_free+0x2a/0x40 [spl]
[316806.973220] [] ? dnode_add_ref+0x4c/0x100 [zfs]
[316806.973247] [] dmu_objset_sync_dnodes+0x9b/0xc0 [zfs]
[316806.973275] [] dmu_objset_sync+0x1e2/0x2f0 [zfs]
[316806.973302] [] ? recordsize_changed_cb+0x20/0x20 [zfs]
[316806.973329] [] ? dmu_objset_sync+0x2f0/0x2f0 [zfs]
[316806.973359] [] dsl_dataset_sync+0x52/0xa0 [zfs]
[316806.973390] [] dsl_pool_sync+0x9d/0x3f0 [zfs]
[316806.973423] [] spa_sync+0x379/0xb40 [zfs]
[316806.973458] [] txg_sync_thread+0x3bf/0x620 [zfs]
[316806.973492] [] ? txg_fini+0x290/0x290 [zfs]
[316806.973498] [] thread_generic_wrapper+0x71/0x80 [spl]
[316806.973504] [] ? __thread_exit+0x20/0x20 [spl]
[316806.973508] [] kthread+0xd8/0xf0
[316806.973510] [] ? kthread_create_on_node+0x1b0/0x1b0
[316806.973513] [] ret_from_fork+0x3f/0x70
[316806.973516] [] ? kthread_create_on_node+0x1b0/0x1b0

[ryao]

This might be solved by bc4501f.

It also might be #1303.

[ryao]

This might be a duplicate of #2354.

[tych0]

Hi guys,

I think this is still an issue, I've just hit:

May 24 14:37:16 dev kernel: [1099282.414633] VERIFY3(sa.sa_magic == 0x2F505A) failed (1 == 3100762)
May 24 14:37:16 dev kernel: [1099282.417674] PANIC at zfs_vfsops.c:426:zfs_space_delta_cb()
May 24 14:37:16 dev kernel: [1099282.418507] Showing stack for process 26241
May 24 14:37:16 dev kernel: [1099282.418515] CPU: 2 PID: 26241 Comm: txg_sync Tainted: P           OE   4.4.0-22-generic #39-Ubuntu
May 24 14:37:16 dev kernel: [1099282.418517] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
May 24 14:37:16 dev kernel: [1099282.418518]  0000000000000286 00000000108a2143 ffff88000b1ef838 ffffffff813e9c53
May 24 14:37:16 dev kernel: [1099282.418521]  ffffffffc0737632 00000000000001aa ffff88000b1ef848 ffffffffc0401ce2
May 24 14:37:16 dev kernel: [1099282.418522]  ffff88000b1ef9d0 ffffffffc0401dab ffff88000b1ef8a0 ffffffff00000030
May 24 14:37:16 dev kernel: [1099282.418524] Call Trace:
May 24 14:37:16 dev kernel: [1099282.418545]  [<ffffffff813e9c53>] dump_stack+0x63/0x90
May 24 14:37:16 dev kernel: [1099282.418562]  [<ffffffffc0401ce2>] spl_dumpstack+0x42/0x50 [spl]
May 24 14:37:16 dev kernel: [1099282.418566]  [<ffffffffc0401dab>] spl_panic+0xbb/0xf0 [spl]
May 24 14:37:16 dev kernel: [1099282.418632]  [<ffffffffc0677673>] ? dnode_rele_and_unlock+0x53/0x80 [zfs]
May 24 14:37:16 dev kernel: [1099282.418651]  [<ffffffffc06776d9>] ? dnode_rele+0x39/0x40 [zfs]
May 24 14:37:16 dev kernel: [1099282.418666]  [<ffffffffc065b5e3>] ? dbuf_rele_and_unlock+0x373/0x3e0 [zfs]
May 24 14:37:16 dev kernel: [1099282.418692]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:37:16 dev kernel: [1099282.418696]  [<ffffffffc03ff8c1>] ? taskq_dispatch_ent+0x51/0x120 [spl]
May 24 14:37:16 dev kernel: [1099282.418722]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:37:16 dev kernel: [1099282.418725]  [<ffffffffc03fc84a>] ? spl_kmem_free+0x2a/0x40 [spl]
May 24 14:37:16 dev kernel: [1099282.418751]  [<ffffffffc06e9f03>] zfs_space_delta_cb+0xc3/0x190 [zfs]
May 24 14:37:16 dev kernel: [1099282.418769]  [<ffffffffc066b122>] dmu_objset_userquota_get_ids+0x122/0x410 [zfs]
May 24 14:37:16 dev kernel: [1099282.418787]  [<ffffffffc067a13a>] dnode_sync+0xea/0x8f0 [zfs]
May 24 14:37:16 dev kernel: [1099282.418799]  [<ffffffff81823162>] ? mutex_lock+0x12/0x30
May 24 14:37:16 dev kernel: [1099282.418817]  [<ffffffffc0669eee>] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
May 24 14:37:16 dev kernel: [1099282.418834]  [<ffffffffc066a0ed>] dmu_objset_sync+0x1dd/0x310 [zfs]
May 24 14:37:16 dev kernel: [1099282.418850]  [<ffffffffc06684f0>] ? recordsize_changed_cb+0x20/0x20 [zfs]
May 24 14:37:16 dev kernel: [1099282.418867]  [<ffffffffc066a220>] ? dmu_objset_sync+0x310/0x310 [zfs]
May 24 14:37:16 dev kernel: [1099282.418886]  [<ffffffffc0682a8e>] dsl_dataset_sync+0x4e/0x90 [zfs]
May 24 14:37:16 dev kernel: [1099282.418905]  [<ffffffffc068bc3f>] dsl_pool_sync+0x9f/0x430 [zfs]
May 24 14:37:16 dev kernel: [1099282.418927]  [<ffffffffc06a75b6>] spa_sync+0x366/0xb30 [zfs]
May 24 14:37:16 dev kernel: [1099282.418936]  [<ffffffff810abe52>] ? default_wake_function+0x12/0x20
May 24 14:37:16 dev kernel: [1099282.418960]  [<ffffffffc06b8a4a>] txg_sync_thread+0x3ba/0x630 [zfs]
May 24 14:37:16 dev kernel: [1099282.418969]  [<ffffffff8106428e>] ? kvm_sched_clock_read+0x1e/0x30
May 24 14:37:16 dev kernel: [1099282.418992]  [<ffffffffc06b8690>] ? txg_delay+0x180/0x180 [zfs]
May 24 14:37:16 dev kernel: [1099282.418996]  [<ffffffffc03fee31>] thread_generic_wrapper+0x71/0x80 [spl]
May 24 14:37:16 dev kernel: [1099282.418999]  [<ffffffffc03fedc0>] ? __thread_exit+0x20/0x20 [spl]
May 24 14:37:16 dev kernel: [1099282.419005]  [<ffffffff810a0588>] kthread+0xd8/0xf0
May 24 14:37:16 dev kernel: [1099282.419007]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:37:16 dev kernel: [1099282.419011]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
May 24 14:37:16 dev kernel: [1099282.419012]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:39:41 dev kernel: [1099427.574062] INFO: task txg_sync:26241 blocked for more than 120 seconds.
May 24 14:39:41 dev kernel: [1099427.576010]       Tainted: P           OE   4.4.0-22-generic #39-Ubuntu
May 24 14:39:41 dev kernel: [1099427.577972] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 24 14:39:41 dev kernel: [1099427.580082] txg_sync        D ffff88000b1ef830     0 26241      2 0x00000000
May 24 14:39:41 dev kernel: [1099427.580089]  ffff88000b1ef830 ffff88001290faf8 ffff8800775c8c80 ffff88000b1a2580
May 24 14:39:41 dev kernel: [1099427.580093]  ffff88000b1f0000 00000000000001aa ffffffffc071d900 ffffffffc0737b80
May 24 14:39:41 dev kernel: [1099427.580096]  ffff88001290faf8 ffff88000b1ef848 ffffffff818211f5 ffffffffc0737632
May 24 14:39:41 dev kernel: [1099427.580099] Call Trace:
May 24 14:39:41 dev kernel: [1099427.580111]  [<ffffffff818211f5>] schedule+0x35/0x80
May 24 14:39:41 dev kernel: [1099427.580124]  [<ffffffffc0401dd0>] spl_panic+0xe0/0xf0 [spl]
May 24 14:39:41 dev kernel: [1099427.580174]  [<ffffffffc0677673>] ? dnode_rele_and_unlock+0x53/0x80 [zfs]
May 24 14:39:41 dev kernel: [1099427.580211]  [<ffffffffc06776d9>] ? dnode_rele+0x39/0x40 [zfs]
May 24 14:39:41 dev kernel: [1099427.580242]  [<ffffffffc065b5e3>] ? dbuf_rele_and_unlock+0x373/0x3e0 [zfs]
May 24 14:39:41 dev kernel: [1099427.580322]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:39:41 dev kernel: [1099427.580332]  [<ffffffffc03ff8c1>] ? taskq_dispatch_ent+0x51/0x120 [spl]
May 24 14:39:41 dev kernel: [1099427.580383]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:39:41 dev kernel: [1099427.580390]  [<ffffffffc03fc84a>] ? spl_kmem_free+0x2a/0x40 [spl]
May 24 14:39:41 dev kernel: [1099427.580442]  [<ffffffffc06e9f03>] zfs_space_delta_cb+0xc3/0x190 [zfs]
May 24 14:39:41 dev kernel: [1099427.580478]  [<ffffffffc066b122>] dmu_objset_userquota_get_ids+0x122/0x410 [zfs]
May 24 14:39:41 dev kernel: [1099427.580514]  [<ffffffffc067a13a>] dnode_sync+0xea/0x8f0 [zfs]
May 24 14:39:41 dev kernel: [1099427.580519]  [<ffffffff81823162>] ? mutex_lock+0x12/0x30
May 24 14:39:41 dev kernel: [1099427.580554]  [<ffffffffc0669eee>] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
May 24 14:39:41 dev kernel: [1099427.580587]  [<ffffffffc066a0ed>] dmu_objset_sync+0x1dd/0x310 [zfs]
May 24 14:39:41 dev kernel: [1099427.580620]  [<ffffffffc06684f0>] ? recordsize_changed_cb+0x20/0x20 [zfs]
May 24 14:39:41 dev kernel: [1099427.580653]  [<ffffffffc066a220>] ? dmu_objset_sync+0x310/0x310 [zfs]
May 24 14:39:41 dev kernel: [1099427.580690]  [<ffffffffc0682a8e>] dsl_dataset_sync+0x4e/0x90 [zfs]
May 24 14:39:41 dev kernel: [1099427.580728]  [<ffffffffc068bc3f>] dsl_pool_sync+0x9f/0x430 [zfs]
May 24 14:39:41 dev kernel: [1099427.580773]  [<ffffffffc06a75b6>] spa_sync+0x366/0xb30 [zfs]
May 24 14:39:41 dev kernel: [1099427.580777]  [<ffffffff810abe52>] ? default_wake_function+0x12/0x20
May 24 14:39:41 dev kernel: [1099427.580824]  [<ffffffffc06b8a4a>] txg_sync_thread+0x3ba/0x630 [zfs]
May 24 14:39:41 dev kernel: [1099427.580829]  [<ffffffff8106428e>] ? kvm_sched_clock_read+0x1e/0x30
May 24 14:39:41 dev kernel: [1099427.580874]  [<ffffffffc06b8690>] ? txg_delay+0x180/0x180 [zfs]
May 24 14:39:41 dev kernel: [1099427.580881]  [<ffffffffc03fee31>] thread_generic_wrapper+0x71/0x80 [spl]
May 24 14:39:41 dev kernel: [1099427.580888]  [<ffffffffc03fedc0>] ? __thread_exit+0x20/0x20 [spl]
May 24 14:39:41 dev kernel: [1099427.580891]  [<ffffffff810a0588>] kthread+0xd8/0xf0
May 24 14:39:41 dev kernel: [1099427.580895]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:39:41 dev kernel: [1099427.580898]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
May 24 14:39:41 dev kernel: [1099427.580901]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:39:41 dev kernel: [1099427.580910] INFO: task zfs:1749 blocked for more than 120 seconds.
May 24 14:39:41 dev kernel: [1099427.582886]       Tainted: P           OE   4.4.0-22-generic #39-Ubuntu
May 24 14:39:41 dev kernel: [1099427.584726] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 24 14:39:41 dev kernel: [1099427.587029] zfs             D ffff88002f4c3758     0  1749   1686 0x00000000
May 24 14:39:41 dev kernel: [1099427.587034]  ffff88002f4c3758 0000000000000000 ffff880079900000 ffff88000310cb00
May 24 14:39:41 dev kernel: [1099427.587038]  ffff88002f4c4000 ffff880006061a20 ffff880006061a48 ffff880006061ae0
May 24 14:39:41 dev kernel: [1099427.587040]  0000000000000000 ffff88002f4c3770 ffffffff818211f5 ffff880006061ad8
May 24 14:39:41 dev kernel: [1099427.587043] Call Trace:
May 24 14:39:41 dev kernel: [1099427.587049]  [<ffffffff818211f5>] schedule+0x35/0x80
May 24 14:39:41 dev kernel: [1099427.587059]  [<ffffffffc0403c3b>] cv_wait_common+0x10b/0x140 [spl]
May 24 14:39:41 dev kernel: [1099427.587079]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
May 24 14:39:41 dev kernel: [1099427.587085]  [<ffffffffc0403c85>] __cv_wait+0x15/0x20 [spl]
May 24 14:39:41 dev kernel: [1099427.587135]  [<ffffffffc06b82b5>] txg_wait_synced+0xe5/0x130 [zfs]
May 24 14:39:41 dev kernel: [1099427.587175]  [<ffffffffc0693bf9>] dsl_sync_task+0x179/0x260 [zfs]
May 24 14:39:41 dev kernel: [1099427.587219]  [<ffffffffc0711290>] ? dsl_destroy_head_check_impl+0x130/0x130 [zfs]
May 24 14:39:41 dev kernel: [1099427.587264]  [<ffffffffc0711080>] ? dsl_destroy_head_sync_impl+0x9f0/0x9f0 [zfs]
May 24 14:39:41 dev kernel: [1099427.587306]  [<ffffffffc0711290>] ? dsl_destroy_head_check_impl+0x130/0x130 [zfs]
May 24 14:39:41 dev kernel: [1099427.587348]  [<ffffffffc0711080>] ? dsl_destroy_head_sync_impl+0x9f0/0x9f0 [zfs]
May 24 14:39:41 dev kernel: [1099427.587392]  [<ffffffffc070f71f>] dsl_destroy_head+0xef/0x170 [zfs]
May 24 14:39:41 dev kernel: [1099427.587424]  [<ffffffffc067b4cb>] ? dsl_dataset_disown+0x8b/0xa0 [zfs]
May 24 14:39:41 dev kernel: [1099427.587455]  [<ffffffffc066f88d>] dmu_recv_cleanup_ds.isra.11+0x4d/0x70 [zfs]
May 24 14:39:41 dev kernel: [1099427.587470]  [<ffffffff8120c974>] ? vfs_read+0x114/0x130
May 24 14:39:41 dev kernel: [1099427.587478]  [<ffffffffc0400f66>] ? vn_rdwr+0xc6/0xe0 [spl]
May 24 14:39:41 dev kernel: [1099427.587508]  [<ffffffffc066d962>] ? restore_read+0x112/0x150 [zfs]
May 24 14:39:41 dev kernel: [1099427.587538]  [<ffffffffc067034e>] dmu_recv_stream+0x28e/0xb50 [zfs]
May 24 14:39:41 dev kernel: [1099427.587566]  [<ffffffffc04261e1>] ? nvlist_common.part.87+0x111/0x200 [znvpair]
May 24 14:39:41 dev kernel: [1099427.587575]  [<ffffffffc042655a>] ? nvlist_xpack+0xea/0x120 [znvpair]
May 24 14:39:41 dev kernel: [1099427.587581]  [<ffffffffc03fc84a>] ? spl_kmem_free+0x2a/0x40 [spl]
May 24 14:39:41 dev kernel: [1099427.587588]  [<ffffffffc0426c5e>] ? fnvlist_pack_free+0xe/0x10 [znvpair]
May 24 14:39:41 dev kernel: [1099427.587632]  [<ffffffffc06e1839>] zfs_ioc_recv+0x209/0xc20 [zfs]
May 24 14:39:41 dev kernel: [1099427.587653]  [<ffffffff810abe52>] ? default_wake_function+0x12/0x20
May 24 14:39:41 dev kernel: [1099427.587660]  [<ffffffff810c33b2>] ? __wake_up_common+0x52/0x90
May 24 14:39:41 dev kernel: [1099427.587687]  [<ffffffffc065b796>] ? dmu_buf_rele+0x36/0x40 [zfs]
May 24 14:39:41 dev kernel: [1099427.587725]  [<ffffffffc069d130>] ? rrw_exit+0x60/0x150 [zfs]
May 24 14:39:41 dev kernel: [1099427.587764]  [<ffffffffc06b3a6e>] ? spa_close+0xe/0x10 [zfs]
May 24 14:39:41 dev kernel: [1099427.587799]  [<ffffffffc068ce49>] ? dsl_pool_rele+0x29/0x30 [zfs]
May 24 14:39:41 dev kernel: [1099427.587812]  [<ffffffff811ec609>] ? __kmalloc+0x1b9/0x250
May 24 14:39:41 dev kernel: [1099427.587819]  [<ffffffffc03fc89b>] ? strdup+0x3b/0x60 [spl]
May 24 14:39:41 dev kernel: [1099427.587862]  [<ffffffffc06e0623>] zfsdev_ioctl+0x423/0x4b0 [zfs]
May 24 14:39:41 dev kernel: [1099427.587871]  [<ffffffff8122bca2>] ? m_start+0x22/0x90
May 24 14:39:41 dev kernel: [1099427.587878]  [<ffffffff8122026f>] do_vfs_ioctl+0x29f/0x490
May 24 14:39:41 dev kernel: [1099427.587882]  [<ffffffff8120c318>] ? __vfs_read+0x18/0x40
May 24 14:39:41 dev kernel: [1099427.587884]  [<ffffffff8120c8e6>] ? vfs_read+0x86/0x130
May 24 14:39:41 dev kernel: [1099427.587887]  [<ffffffff812204d9>] SyS_ioctl+0x79/0x90
May 24 14:39:41 dev kernel: [1099427.587892]  [<ffffffff818252f2>] entry_SYSCALL_64_fastpath+0x16/0x71
May 24 14:41:41 dev kernel: [1099547.584522] INFO: task txg_sync:26241 blocked for more than 120 seconds.
May 24 14:41:41 dev kernel: [1099547.586688]       Tainted: P           OE   4.4.0-22-generic #39-Ubuntu
May 24 14:41:41 dev kernel: [1099547.588956] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 24 14:41:41 dev kernel: [1099547.591362] txg_sync        D ffff88000b1ef830     0 26241      2 0x00000000
May 24 14:41:41 dev kernel: [1099547.591369]  ffff88000b1ef830 ffff88001290faf8 ffff8800775c8c80 ffff88000b1a2580
May 24 14:41:41 dev kernel: [1099547.591373]  ffff88000b1f0000 00000000000001aa ffffffffc071d900 ffffffffc0737b80
May 24 14:41:41 dev kernel: [1099547.591376]  ffff88001290faf8 ffff88000b1ef848 ffffffff818211f5 ffffffffc0737632
May 24 14:41:41 dev kernel: [1099547.591380] Call Trace:
May 24 14:41:41 dev kernel: [1099547.591392]  [<ffffffff818211f5>] schedule+0x35/0x80
May 24 14:41:41 dev kernel: [1099547.591403]  [<ffffffffc0401dd0>] spl_panic+0xe0/0xf0 [spl]
May 24 14:41:41 dev kernel: [1099547.591452]  [<ffffffffc0677673>] ? dnode_rele_and_unlock+0x53/0x80 [zfs]
May 24 14:41:41 dev kernel: [1099547.591488]  [<ffffffffc06776d9>] ? dnode_rele+0x39/0x40 [zfs]
May 24 14:41:41 dev kernel: [1099547.591520]  [<ffffffffc065b5e3>] ? dbuf_rele_and_unlock+0x373/0x3e0 [zfs]
May 24 14:41:41 dev kernel: [1099547.591571]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:41:41 dev kernel: [1099547.591579]  [<ffffffffc03ff8c1>] ? taskq_dispatch_ent+0x51/0x120 [spl]
May 24 14:41:41 dev kernel: [1099547.591629]  [<ffffffffc0700050>] ? zio_taskq_member.isra.6+0x90/0x90 [zfs]
May 24 14:41:41 dev kernel: [1099547.591636]  [<ffffffffc03fc84a>] ? spl_kmem_free+0x2a/0x40 [spl]
May 24 14:41:41 dev kernel: [1099547.591687]  [<ffffffffc06e9f03>] zfs_space_delta_cb+0xc3/0x190 [zfs]
May 24 14:41:41 dev kernel: [1099547.591723]  [<ffffffffc066b122>] dmu_objset_userquota_get_ids+0x122/0x410 [zfs]
May 24 14:41:41 dev kernel: [1099547.591759]  [<ffffffffc067a13a>] dnode_sync+0xea/0x8f0 [zfs]
May 24 14:41:41 dev kernel: [1099547.591764]  [<ffffffff81823162>] ? mutex_lock+0x12/0x30
May 24 14:41:41 dev kernel: [1099547.591798]  [<ffffffffc0669eee>] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
May 24 14:41:41 dev kernel: [1099547.591832]  [<ffffffffc066a0ed>] dmu_objset_sync+0x1dd/0x310 [zfs]
May 24 14:41:41 dev kernel: [1099547.591865]  [<ffffffffc06684f0>] ? recordsize_changed_cb+0x20/0x20 [zfs]
May 24 14:41:41 dev kernel: [1099547.591898]  [<ffffffffc066a220>] ? dmu_objset_sync+0x310/0x310 [zfs]
May 24 14:41:41 dev kernel: [1099547.591935]  [<ffffffffc0682a8e>] dsl_dataset_sync+0x4e/0x90 [zfs]
May 24 14:41:41 dev kernel: [1099547.591978]  [<ffffffffc068bc3f>] dsl_pool_sync+0x9f/0x430 [zfs]
May 24 14:41:41 dev kernel: [1099547.592029]  [<ffffffffc06a75b6>] spa_sync+0x366/0xb30 [zfs]
May 24 14:41:41 dev kernel: [1099547.592034]  [<ffffffff810abe52>] ? default_wake_function+0x12/0x20
May 24 14:41:41 dev kernel: [1099547.592081]  [<ffffffffc06b8a4a>] txg_sync_thread+0x3ba/0x630 [zfs]
May 24 14:41:41 dev kernel: [1099547.592086]  [<ffffffff8106428e>] ? kvm_sched_clock_read+0x1e/0x30
May 24 14:41:41 dev kernel: [1099547.592131]  [<ffffffffc06b8690>] ? txg_delay+0x180/0x180 [zfs]
May 24 14:41:41 dev kernel: [1099547.592138]  [<ffffffffc03fee31>] thread_generic_wrapper+0x71/0x80 [spl]
May 24 14:41:41 dev kernel: [1099547.592145]  [<ffffffffc03fedc0>] ? __thread_exit+0x20/0x20 [spl]
May 24 14:41:41 dev kernel: [1099547.592149]  [<ffffffff810a0588>] kthread+0xd8/0xf0
May 24 14:41:41 dev kernel: [1099547.592152]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:41:41 dev kernel: [1099547.592155]  [<ffffffff8182568f>] ret_from_fork+0x3f/0x70
May 24 14:41:41 dev kernel: [1099547.592158]  [<ffffffff810a04b0>] ? kthread_create_on_node+0x1e0/0x1e0
May 24 14:41:41 dev kernel: [1099547.592167] INFO: task zfs:1749 blocked for more than 120 seconds.
May 24 14:41:41 dev kernel: [1099547.594367]       Tainted: P           OE   4.4.0-22-generic #39-Ubuntu
May 24 14:41:41 dev kernel: [1099547.596493] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 24 14:41:41 dev kernel: [1099547.598952] zfs             D ffff88002f4c3758     0  1749   1686 0x00000004
May 24 14:41:41 dev kernel: [1099547.598958]  ffff88002f4c3758 0000000000000000 ffff880079900000 ffff88000310cb00
May 24 14:41:41 dev kernel: [1099547.598961]  ffff88002f4c4000 ffff880006061a20 ffff880006061a48 ffff880006061ae0
May 24 14:41:41 dev kernel: [1099547.598996]  0000000000000000 ffff88002f4c3770 ffffffff818211f5 ffff880006061ad8
May 24 14:41:41 dev kernel: [1099547.598999] Call Trace:
May 24 14:41:41 dev kernel: [1099547.599005]  [<ffffffff818211f5>] schedule+0x35/0x80
May 24 14:41:41 dev kernel: [1099547.599015]  [<ffffffffc0403c3b>] cv_wait_common+0x10b/0x140 [spl]
May 24 14:41:41 dev kernel: [1099547.599021]  [<ffffffff810c3a70>] ? wake_atomic_t_function+0x60/0x60
May 24 14:41:41 dev kernel: [1099547.599027]  [<ffffffffc0403c85>] __cv_wait+0x15/0x20 [spl]
May 24 14:41:41 dev kernel: [1099547.599075]  [<ffffffffc06b82b5>] txg_wait_synced+0xe5/0x130 [zfs]
May 24 14:41:41 dev kernel: [1099547.599114]  [<ffffffffc0693bf9>] dsl_sync_task+0x179/0x260 [zfs]
May 24 14:41:41 dev kernel: [1099547.599163]  [<ffffffffc0711290>] ? dsl_destroy_head_check_impl+0x130/0x130 [zfs]
May 24 14:41:41 dev kernel: [1099547.599211]  [<ffffffffc0711080>] ? dsl_destroy_head_sync_impl+0x9f0/0x9f0 [zfs]
May 24 14:41:41 dev kernel: [1099547.599258]  [<ffffffffc0711290>] ? dsl_destroy_head_check_impl+0x130/0x130 [zfs]
May 24 14:41:41 dev kernel: [1099547.599305]  [<ffffffffc0711080>] ? dsl_destroy_head_sync_impl+0x9f0/0x9f0 [zfs]
May 24 14:41:41 dev kernel: [1099547.599352]  [<ffffffffc070f71f>] dsl_destroy_head+0xef/0x170 [zfs]
May 24 14:41:41 dev kernel: [1099547.599388]  [<ffffffffc067b4cb>] ? dsl_dataset_disown+0x8b/0xa0 [zfs]
May 24 14:41:41 dev kernel: [1099547.599421]  [<ffffffffc066f88d>] dmu_recv_cleanup_ds.isra.11+0x4d/0x70 [zfs]
May 24 14:41:41 dev kernel: [1099547.599426]  [<ffffffff8120c974>] ? vfs_read+0x114/0x130
May 24 14:41:41 dev kernel: [1099547.599433]  [<ffffffffc0400f66>] ? vn_rdwr+0xc6/0xe0 [spl]
May 24 14:41:41 dev kernel: [1099547.599466]  [<ffffffffc066d962>] ? restore_read+0x112/0x150 [zfs]
May 24 14:41:41 dev kernel: [1099547.599499]  [<ffffffffc067034e>] dmu_recv_stream+0x28e/0xb50 [zfs]
May 24 14:41:41 dev kernel: [1099547.599510]  [<ffffffffc04261e1>] ? nvlist_common.part.87+0x111/0x200 [znvpair]
May 24 14:41:41 dev kernel: [1099547.599518]  [<ffffffffc042655a>] ? nvlist_xpack+0xea/0x120 [znvpair]
May 24 14:41:41 dev kernel: [1099547.599525]  [<ffffffffc03fc84a>] ? spl_kmem_free+0x2a/0x40 [spl]
May 24 14:41:41 dev kernel: [1099547.599532]  [<ffffffffc0426c5e>] ? fnvlist_pack_free+0xe/0x10 [znvpair]
May 24 14:41:41 dev kernel: [1099547.599581]  [<ffffffffc06e1839>] zfs_ioc_recv+0x209/0xc20 [zfs]
May 24 14:41:41 dev kernel: [1099547.599584]  [<ffffffff810abe52>] ? default_wake_function+0x12/0x20
May 24 14:41:41 dev kernel: [1099547.599588]  [<ffffffff810c33b2>] ? __wake_up_common+0x52/0x90
May 24 14:41:41 dev kernel: [1099547.599618]  [<ffffffffc065b796>] ? dmu_buf_rele+0x36/0x40 [zfs]
May 24 14:41:41 dev kernel: [1099547.599659]  [<ffffffffc069d130>] ? rrw_exit+0x60/0x150 [zfs]
May 24 14:41:41 dev kernel: [1099547.599701]  [<ffffffffc06b3a6e>] ? spa_close+0xe/0x10 [zfs]
May 24 14:41:41 dev kernel: [1099547.599739]  [<ffffffffc068ce49>] ? dsl_pool_rele+0x29/0x30 [zfs]
May 24 14:41:41 dev kernel: [1099547.599743]  [<ffffffff811ec609>] ? __kmalloc+0x1b9/0x250
May 24 14:41:41 dev kernel: [1099547.599749]  [<ffffffffc03fc89b>] ? strdup+0x3b/0x60 [spl]
May 24 14:41:41 dev kernel: [1099547.599796]  [<ffffffffc06e0623>] zfsdev_ioctl+0x423/0x4b0 [zfs]
May 24 14:41:41 dev kernel: [1099547.599800]  [<ffffffff8122bca2>] ? m_start+0x22/0x90
May 24 14:41:41 dev kernel: [1099547.599803]  [<ffffffff8122026f>] do_vfs_ioctl+0x29f/0x490
May 24 14:41:41 dev kernel: [1099547.599806]  [<ffffffff8120c318>] ? __vfs_read+0x18/0x40
May 24 14:41:41 dev kernel: [1099547.599809]  [<ffffffff8120c8e6>] ? vfs_read+0x86/0x130
May 24 14:41:41 dev kernel: [1099547.599812]  [<ffffffff812204d9>] SyS_ioctl+0x79/0x90
May 24 14:41:41 dev kernel: [1099547.599816]  [<ffffffff818252f2>] entry_SYSCALL_64_fastpath+0x16/0x71

with

$ modinfo zfs
filename:       /lib/modules/4.4.0-22-generic/kernel/zfs/zfs/zfs.ko
version:        0.6.5.6-0ubuntu3
license:        CDDL
author:         OpenZFS on Linux
description:    ZFS
srcversion:     A87B4C0C518AC85BD042501
depends:        spl,znvpair,zunicode,zcommon,zavl
vermagic:       4.4.0-22-generic SMP mod_unload modversions 
parm:           zvol_inhibit_dev:Do not create zvol device nodes (uint)
parm:           zvol_major:Major number for zvol device (uint)
parm:           zvol_max_discard_blocks:Max number of blocks to discard (ulong)
parm:           zvol_prefetch_bytes:Prefetch N bytes at zvol start+end (uint)
parm:           zio_delay_max:Max zio millisec delay before posting event (int)
parm:           zio_requeue_io_start_cut_in_line:Prioritize requeued I/O (int)
parm:           zfs_sync_pass_deferred_free:Defer frees starting in this pass (int)
parm:           zfs_sync_pass_dont_compress:Don't compress starting in this pass (int)
parm:           zfs_sync_pass_rewrite:Rewrite new bps starting in this pass (int)
parm:           zil_replay_disable:Disable intent logging replay (int)
parm:           zfs_nocacheflush:Disable cache flushes (int)
parm:           zil_slog_limit:Max commit bytes to separate log device (ulong)
parm:           zfs_object_mutex_size:Size of znode hold array (uint)
parm:           zfs_read_chunk_size:Bytes to read per chunk (long)
parm:           zfs_immediate_write_sz:Largest data block to write to zil (long)
parm:           zfs_dbgmsg_enable:Enable ZFS debug message log (int)
parm:           zfs_dbgmsg_maxsize:Maximum ZFS debug log size (int)
parm:           zfs_admin_snapshot:Enable mkdir/rmdir/mv in .zfs/snapshot (int)
parm:           zfs_expire_snapshot:Seconds to expire .zfs/snapshot (int)
parm:           zfs_vdev_aggregation_limit:Max vdev I/O aggregation size (int)
parm:           zfs_vdev_read_gap_limit:Aggregate read I/O over gap (int)
parm:           zfs_vdev_write_gap_limit:Aggregate write I/O over gap (int)
parm:           zfs_vdev_max_active:Maximum number of active I/Os per vdev (int)
parm:           zfs_vdev_async_write_active_max_dirty_percent:Async write concurrency max threshold (int)
parm:           zfs_vdev_async_write_active_min_dirty_percent:Async write concurrency min threshold (int)
parm:           zfs_vdev_async_read_max_active:Max active async read I/Os per vdev (int)
parm:           zfs_vdev_async_read_min_active:Min active async read I/Os per vdev (int)
parm:           zfs_vdev_async_write_max_active:Max active async write I/Os per vdev (int)
parm:           zfs_vdev_async_write_min_active:Min active async write I/Os per vdev (int)
parm:           zfs_vdev_scrub_max_active:Max active scrub I/Os per vdev (int)
parm:           zfs_vdev_scrub_min_active:Min active scrub I/Os per vdev (int)
parm:           zfs_vdev_sync_read_max_active:Max active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_read_min_active:Min active sync read I/Os per vdev (int)
parm:           zfs_vdev_sync_write_max_active:Max active sync write I/Os per vdev (int)
parm:           zfs_vdev_sync_write_min_active:Min active sync write I/Os per vdev (int)
parm:           zfs_vdev_mirror_switch_us:Switch mirrors every N usecs (int)
parm:           zfs_vdev_scheduler:I/O scheduler (charp)
parm:           zfs_vdev_cache_max:Inflate reads small than max (int)
parm:           zfs_vdev_cache_size:Total size of the per-disk cache (int)
parm:           zfs_vdev_cache_bshift:Shift size to inflate reads too (int)
parm:           metaslabs_per_vdev:Divide added vdev into approximately (but no more than) this number of metaslabs (int)
parm:           zfs_txg_timeout:Max seconds worth of delta per txg (int)
parm:           zfs_read_history:Historic statistics for the last N reads (int)
parm:           zfs_read_history_hits:Include cache hits in read history (int)
parm:           zfs_txg_history:Historic statistics for the last N txgs (int)
parm:           zfs_flags:Set additional debugging flags (uint)
parm:           zfs_recover:Set to attempt to recover from fatal errors (int)
parm:           zfs_free_leak_on_eio:Set to ignore IO errors during free and permanently leak the space (int)
parm:           zfs_deadman_synctime_ms:Expiration time in milliseconds (ulong)
parm:           zfs_deadman_enabled:Enable deadman timer (int)
parm:           spa_asize_inflation:SPA size estimate multiplication factor (int)
parm:           spa_slop_shift:Reserved free space in pool (int)
parm:           spa_config_path:SPA config file (/etc/zfs/zpool.cache) (charp)
parm:           zfs_autoimport_disable:Disable pool import at module load (int)
parm:           spa_load_verify_maxinflight:Max concurrent traversal I/Os while verifying pool during import -X (int)
parm:           spa_load_verify_metadata:Set to traverse metadata on pool import (int)
parm:           spa_load_verify_data:Set to traverse data on pool import (int)
parm:           zio_taskq_batch_pct:Percentage of CPUs to run an IO worker thread (uint)
parm:           metaslab_aliquot:allocation granularity (a.k.a. stripe size) (ulong)
parm:           metaslab_debug_load:load all metaslabs when pool is first opened (int)
parm:           metaslab_debug_unload:prevent metaslabs from being unloaded (int)
parm:           metaslab_preload_enabled:preload potential metaslabs during reassessment (int)
parm:           zfs_mg_noalloc_threshold:percentage of free space for metaslab group to allow allocation (int)
parm:           zfs_mg_fragmentation_threshold:fragmentation for metaslab group to allow allocation (int)
parm:           zfs_metaslab_fragmentation_threshold:fragmentation for metaslab to allow allocation (int)
parm:           metaslab_fragmentation_factor_enabled:use the fragmentation metric to prefer less fragmented metaslabs (int)
parm:           metaslab_lba_weighting_enabled:prefer metaslabs with lower LBAs (int)
parm:           metaslab_bias_enabled:enable metaslab group biasing (int)
parm:           zfs_zevent_len_max:Max event queue length (int)
parm:           zfs_zevent_cols:Max event column width (int)
parm:           zfs_zevent_console:Log events to the console (int)
parm:           zfs_top_maxinflight:Max I/Os per top-level (int)
parm:           zfs_resilver_delay:Number of ticks to delay resilver (int)
parm:           zfs_scrub_delay:Number of ticks to delay scrub (int)
parm:           zfs_scan_idle:Idle window in clock ticks (int)
parm:           zfs_scan_min_time_ms:Min millisecs to scrub per txg (int)
parm:           zfs_free_min_time_ms:Min millisecs to free per txg (int)
parm:           zfs_resilver_min_time_ms:Min millisecs to resilver per txg (int)
parm:           zfs_no_scrub_io:Set to disable scrub I/O (int)
parm:           zfs_no_scrub_prefetch:Set to disable scrub prefetching (int)
parm:           zfs_free_max_blocks:Max number of blocks freed in one txg (ulong)
parm:           zfs_dirty_data_max_percent:percent of ram can be dirty (int)
parm:           zfs_dirty_data_max_max_percent:zfs_dirty_data_max upper bound as % of RAM (int)
parm:           zfs_delay_min_dirty_percent:transaction delay threshold (int)
parm:           zfs_dirty_data_max:determines the dirty space limit (ulong)
parm:           zfs_dirty_data_max_max:zfs_dirty_data_max upper bound in bytes (ulong)
parm:           zfs_dirty_data_sync:sync txg when this much dirty data (ulong)
parm:           zfs_delay_scale:how quickly delay approaches infinity (ulong)
parm:           zfs_max_recordsize:Max allowed record size (int)
parm:           zfs_prefetch_disable:Disable all ZFS prefetching (int)
parm:           zfetch_max_streams:Max number of streams per zfetch (uint)
parm:           zfetch_min_sec_reap:Min time before stream reclaim (uint)
parm:           zfetch_block_cap:Max number of blocks to fetch at a time (uint)
parm:           zfetch_array_rd_sz:Number of bytes in a array_read (ulong)
parm:           zfs_pd_bytes_max:Max number of bytes to prefetch (int)
parm:           zfs_send_corrupt_data:Allow sending corrupt data (int)
parm:           zfs_mdcomp_disable:Disable meta data compression (int)
parm:           zfs_nopwrite_enabled:Enable NOP writes (int)
parm:           zfs_dedup_prefetch:Enable prefetching dedup-ed blks (int)
parm:           zfs_dbuf_state_index:Calculate arc header index (int)
parm:           zfs_arc_min:Min arc size (ulong)
parm:           zfs_arc_max:Max arc size (ulong)
parm:           zfs_arc_meta_limit:Meta limit for arc size (ulong)
parm:           zfs_arc_meta_min:Min arc metadata (ulong)
parm:           zfs_arc_meta_prune:Meta objects to scan for prune (int)
parm:           zfs_arc_meta_adjust_restarts:Limit number of restarts in arc_adjust_meta (int)
parm:           zfs_arc_meta_strategy:Meta reclaim strategy (int)
parm:           zfs_arc_grow_retry:Seconds before growing arc size (int)
parm:           zfs_arc_p_aggressive_disable:disable aggressive arc_p grow (int)
parm:           zfs_arc_p_dampener_disable:disable arc_p adapt dampener (int)
parm:           zfs_arc_shrink_shift:log2(fraction of arc to reclaim) (int)
parm:           zfs_arc_p_min_shift:arc_c shift to calc min/max arc_p (int)
parm:           zfs_disable_dup_eviction:disable duplicate buffer eviction (int)
parm:           zfs_arc_average_blocksize:Target average block size (int)
parm:           zfs_arc_min_prefetch_lifespan:Min life of prefetch block (int)
parm:           zfs_arc_num_sublists_per_state:Number of sublists used in each of the ARC state lists (int)
parm:           l2arc_write_max:Max write bytes per interval (ulong)
parm:           l2arc_write_boost:Extra write bytes during device warmup (ulong)
parm:           l2arc_headroom:Number of max device writes to precache (ulong)
parm:           l2arc_headroom_boost:Compressed l2arc_headroom multiplier (ulong)
parm:           l2arc_feed_secs:Seconds between L2ARC writing (ulong)
parm:           l2arc_feed_min_ms:Min feed interval in milliseconds (ulong)
parm:           l2arc_noprefetch:Skip caching prefetched buffers (int)
parm:           l2arc_nocompress:Skip compressing L2ARC buffers (int)
parm:           l2arc_feed_again:Turbo L2ARC warmup (int)
parm:           l2arc_norw:No reads during writes (int)
parm:           zfs_arc_lotsfree_percent:System free memory I/O throttle in bytes (int)
parm:           zfs_arc_sys_free:System free memory target size in bytes (ulong)

[dsanader]

Just stambled on this one too with the 0.6.5.8 module.

The error was triggered by new writes (package upgrade).

Growing the underlying partition and the pool resolved the problem (zfs listed 7G available before the growth, it was a small 20G pool).

[EvilBob01]

Happened to me too.
I'm using version: 0.6.5.6-0ubuntu10
It happened during an rsync. I'm running ECC ram and have plenty of space left.

Nov 14 17:17:04 storage kernel: [5597660.460599] VERIFY3(sa.sa_magic == 0x2F505A) failed (1400516255 == 3100762)
Nov 14 17:17:04 storage kernel: [5597660.462589] PANIC at zfs_vfsops.c:426:zfs_space_delta_cb()
Nov 14 17:17:04 storage kernel: [5597660.464465] Showing stack for process 7356
Nov 14 17:17:04 storage kernel: [5597660.464471] CPU: 6 PID: 7356 Comm: txg_sync Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:17:04 storage kernel: [5597660.464473] Hardware name: ASUSTeK COMPUTER INC. Z10PE-D16 Series/Z10PE-D16 Series, BIOS 3104 01/25/2016
Nov 14 17:17:04 storage kernel: [5597660.464476] 0000000000000286 00000000a323b6e2 ffff88104a9a3838 ffffffff813f13b3
Nov 14 17:17:04 storage kernel: [5597660.464482] ffffffffc08f56ba 00000000000001aa ffff88104a9a3848 ffffffffc077ace2
Nov 14 17:17:04 storage kernel: [5597660.464486] ffff88104a9a39d0 ffffffffc077adab ffff88105594fba0 0000000000000030
Nov 14 17:17:04 storage kernel: [5597660.464490] Call Trace:
Nov 14 17:17:04 storage kernel: [5597660.464506] [] dump_stack+0x63/0x90
Nov 14 17:17:04 storage kernel: [5597660.464527] [] spl_dumpstack+0x42/0x50 [spl]
Nov 14 17:17:04 storage kernel: [5597660.464535] [] spl_panic+0xbb/0xf0 [spl]
Nov 14 17:17:04 storage kernel: [5597660.464544] [] ? __slab_free+0x185/0x2c0
Nov 14 17:17:04 storage kernel: [5597660.464594] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464598] [] ? kmem_cache_free+0x1d4/0x1e0
Nov 14 17:17:04 storage kernel: [5597660.464606] [] ? spl_kmem_free+0x2a/0x40 [spl]
Nov 14 17:17:04 storage kernel: [5597660.464672] [] zfs_space_delta_cb+0xc3/0x190 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464712] [] dmu_objset_userquota_get_ids+0x122/0x410 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464754] [] dnode_sync+0xea/0x8f0 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464796] [] ? dbuf_sync_list+0xca/0xf0 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464843] [] ? dnode_sync+0x30f/0x8f0 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464849] [] ? mutex_lock+0x12/0x30
Nov 14 17:17:04 storage kernel: [5597660.464894] [] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464938] [] dmu_objset_sync+0x1dd/0x310 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.464982] [] ? recordsize_changed_cb+0x20/0x20 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465026] [] ? dmu_objset_sync+0x310/0x310 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465075] [] dsl_dataset_sync+0x4e/0x90 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465126] [] dsl_pool_sync+0x9f/0x430 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465184] [] spa_sync+0x366/0xb30 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465191] [] ? default_wake_function+0x12/0x20
Nov 14 17:17:04 storage kernel: [5597660.465253] [] txg_sync_thread+0x3ba/0x630 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465316] [] ? txg_delay+0x180/0x180 [zfs]
Nov 14 17:17:04 storage kernel: [5597660.465326] [] thread_generic_wrapper+0x71/0x80 [spl]
Nov 14 17:17:04 storage kernel: [5597660.465334] [] ? __thread_exit+0x20/0x20 [spl]
Nov 14 17:17:04 storage kernel: [5597660.465339] [] kthread+0xd8/0xf0
Nov 14 17:17:04 storage kernel: [5597660.465343] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:17:04 storage kernel: [5597660.465348] [] ret_from_fork+0x3f/0x70
Nov 14 17:17:04 storage kernel: [5597660.465351] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.872610] INFO: task txg_sync:7356 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.874624] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.876561] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.878494] txg_sync D ffff88104a9a3830 0 7356 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.878502] ffff88104a9a3830 ffff8803c26dbaf8 ffff88105c276e00 ffff88104ad6d280
Nov 14 17:19:22 storage kernel: [5597797.878507] ffff88104a9a4000 00000000000001aa ffffffffc08db900 ffffffffc08f5c08
Nov 14 17:19:22 storage kernel: [5597797.878511] ffff8803c26dbaf8 ffff88104a9a3848 ffffffff81829ec5 ffffffffc08f56ba
Nov 14 17:19:22 storage kernel: [5597797.878515] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.878534] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.878553] [] spl_panic+0xe0/0xf0 [spl]
Nov 14 17:19:22 storage kernel: [5597797.878563] [] ? __slab_free+0x185/0x2c0
Nov 14 17:19:22 storage kernel: [5597797.878619] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878624] [] ? kmem_cache_free+0x1d4/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.878632] [] ? spl_kmem_free+0x2a/0x40 [spl]
Nov 14 17:19:22 storage kernel: [5597797.878699] [] zfs_space_delta_cb+0xc3/0x190 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878739] [] dmu_objset_userquota_get_ids+0x122/0x410 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878782] [] dnode_sync+0xea/0x8f0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878816] [] ? dbuf_sync_list+0xca/0xf0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878857] [] ? dnode_sync+0x30f/0x8f0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878860] [] ? mutex_lock+0x12/0x30
Nov 14 17:19:22 storage kernel: [5597797.878899] [] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878936] [] dmu_objset_sync+0x1dd/0x310 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.878973] [] ? recordsize_changed_cb+0x20/0x20 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879011] [] ? dmu_objset_sync+0x310/0x310 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879054] [] dsl_dataset_sync+0x4e/0x90 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879098] [] dsl_pool_sync+0x9f/0x430 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879150] [] spa_sync+0x366/0xb30 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879156] [] ? default_wake_function+0x12/0x20
Nov 14 17:19:22 storage kernel: [5597797.879214] [] txg_sync_thread+0x3ba/0x630 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879269] [] ? txg_delay+0x180/0x180 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.879277] [] thread_generic_wrapper+0x71/0x80 [spl]
Nov 14 17:19:22 storage kernel: [5597797.879285] [] ? __thread_exit+0x20/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.879290] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.879293] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.879298] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.879301] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.879630] INFO: task nfsd:15383 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.881573] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.883509] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.885452] nfsd D ffff8810349cf6c0 0 15383 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.885457] ffff8810349cf6c0 0000000000000000 ffff88085c6f8000 ffff8810349a44c0
Nov 14 17:19:22 storage kernel: [5597797.885461] ffff8810349d0000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.885465] 0000000000000000 ffff8810349cf6d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.885468] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.885475] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.885497] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.885514] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.885523] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.885584] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885633] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885675] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885754] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885787] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885819] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885851] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885911] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885970] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.885978] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.886035] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.886041] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.886097] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.886105] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.886111] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.886115] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.886136] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886152] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886164] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886176] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886186] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886227] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.886247] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.886257] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886266] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.886271] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.886274] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.886278] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.886281] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.886284] INFO: task nfsd:15384 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.888049] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.889815] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.891576] nfsd D ffff88104735f6c0 0 15384 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.891582] ffff88104735f6c0 0000000000000000 ffff88104ad6c4c0 ffff8810349a3700
Nov 14 17:19:22 storage kernel: [5597797.891586] ffff881047360000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.891589] 0000000000000000 ffff88104735f6d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.891593] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.891598] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.891608] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.891613] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.891621] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.891677] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891716] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891753] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891811] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891843] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891874] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891904] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.891962] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.892018] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.892023] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.892078] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.892083] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.892138] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.892142] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.892146] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.892150] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.892160] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892174] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892186] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892196] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892206] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892225] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.892245] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.892255] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892265] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.892268] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.892272] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.892276] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.892279] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.892282] INFO: task nfsd:15385 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.894130] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.895994] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.897851] nfsd D ffff881037a2f6c0 0 15385 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.897856] ffff881037a2f6c0 0000000000000000 ffff88085c6f8000 ffff8810349a2940
Nov 14 17:19:22 storage kernel: [5597797.897860] ffff881037a30000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.897863] 0000000000000000 ffff881037a2f6d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.897867] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.897872] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.897883] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.897902] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.897916] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.897973] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898018] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898056] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898116] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898149] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898181] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898213] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898282] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898338] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898343] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.898395] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898401] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.898452] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.898457] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.898462] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.898465] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.898475] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898489] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898501] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898511] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898520] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898539] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.898557] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.898568] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898577] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.898582] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.898585] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.898589] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.898592] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.898596] INFO: task nfsd:15386 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.900360] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.902131] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.903931] nfsd D ffff8810473a36c0 0 15386 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.903937] ffff8810473a36c0 0000000000000246 ffff88105c2744c0 ffff8810349a1b80
Nov 14 17:19:22 storage kernel: [5597797.903941] ffff8810473a4000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.903944] 0000000000000000 ffff8810473a36d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.903948] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.903953] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.903964] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.903968] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.903978] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.904031] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904071] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904111] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904169] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904204] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904239] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904274] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904332] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904391] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904404] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.904462] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904481] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.904544] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.904557] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.904565] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.904578] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.904591] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904606] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904618] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904631] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904641] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904661] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.904688] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.904702] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904711] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.904720] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.904725] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.904729] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.904732] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.904737] INFO: task nfsd:15387 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.906542] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.908274] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.910036] nfsd D ffff8810473036c0 0 15387 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.910040] ffff8810473036c0 ffff8810473036e0 ffff880852fe6e00 ffff881034988000
Nov 14 17:19:22 storage kernel: [5597797.910044] ffff881047304000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.910047] 0000000000000000 ffff8810473036d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.910050] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.910056] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.910066] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.910084] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.910098] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.910154] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910196] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910233] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910292] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910323] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910354] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910384] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910440] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910495] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910500] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.910563] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910568] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.910615] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.910620] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.910624] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.910628] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.910637] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910650] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910660] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910671] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910679] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910697] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.910714] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.910723] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910730] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.910734] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.910737] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.910740] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.910743] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.910745] INFO: task nfsd:15388 blocked for more than 120 seconds.
Nov 14 17:19:22 storage kernel: [5597797.912368] Tainted: P O 4.4.0-36-generic #55-Ubuntu
Nov 14 17:19:22 storage kernel: [5597797.913987] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 14 17:19:22 storage kernel: [5597797.915604] nfsd D ffff88103497b6c0 0 15388 2 0x00000000
Nov 14 17:19:22 storage kernel: [5597797.915609] ffff88103497b6c0 0000000000000000 ffff88085c6f9b80 ffff881034988dc0
Nov 14 17:19:22 storage kernel: [5597797.915612] ffff88103497c000 ffff881055bcfa20 ffff881055bcfa48 ffff881055bcfb70
Nov 14 17:19:22 storage kernel: [5597797.915615] 0000000000000000 ffff88103497b6d8 ffffffff81829ec5 ffff881055bcfb68
Nov 14 17:19:22 storage kernel: [5597797.915618] Call Trace:
Nov 14 17:19:22 storage kernel: [5597797.915623] [] schedule+0x35/0x80
Nov 14 17:19:22 storage kernel: [5597797.915632] [] cv_wait_common+0x10b/0x140 [spl]
Nov 14 17:19:22 storage kernel: [5597797.915636] [] ? wake_atomic_t_function+0x60/0x60
Nov 14 17:19:22 storage kernel: [5597797.915643] [] __cv_wait+0x15/0x20 [spl]
Nov 14 17:19:22 storage kernel: [5597797.915692] [] txg_wait_open+0xb8/0x100 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915725] [] dmu_tx_wait+0x38a/0x3a0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915757] [] dmu_tx_assign+0x92/0x4e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915808] [] zfs_write+0x3e2/0xc00 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915835] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915862] [] ? dbuf_read+0x65b/0x840 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915889] [] ? dbuf_rele_and_unlock+0x2dc/0x3e0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915940] [] zpl_write_common_iovec+0x8c/0xe0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915989] [] zpl_iter_write+0xa0/0xd0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.915993] [] do_iter_readv_writev+0x6c/0xa0
Nov 14 17:19:22 storage kernel: [5597797.916042] [] ? zpl_read+0xc0/0xc0 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.916046] [] do_readv_writev+0x18f/0x230
Nov 14 17:19:22 storage kernel: [5597797.916109] [] ? zpl_compat_ioctl+0x40/0x40 [zfs]
Nov 14 17:19:22 storage kernel: [5597797.916112] [] ? security_prepare_creds+0x43/0x60
Nov 14 17:19:22 storage kernel: [5597797.916116] [] ? __wake_up+0x44/0x50
Nov 14 17:19:22 storage kernel: [5597797.916120] [] vfs_writev+0x39/0x50
Nov 14 17:19:22 storage kernel: [5597797.916129] [] nfsd_vfs_write+0xbd/0x380 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916141] [] ? nfs4_preprocess_stateid_op+0x116/0x410 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916151] [] nfsd4_write+0x1a4/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916160] [] nfsd4_proc_compound+0x38a/0x660 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916168] [] nfsd_dispatch+0xb8/0x200 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916184] [] svc_process_common+0x40c/0x650 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.916202] [] svc_process+0x103/0x1c0 [sunrpc]
Nov 14 17:19:22 storage kernel: [5597797.916210] [] nfsd+0xef/0x160 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916219] [] ? nfsd_destroy+0x60/0x60 [nfsd]
Nov 14 17:19:22 storage kernel: [5597797.916222] [] kthread+0xd8/0xf0
Nov 14 17:19:22 storage kernel: [5597797.916225] [] ? kthread_create_on_node+0x1e0/0x1e0
Nov 14 17:19:22 storage kernel: [5597797.916228] [] ret_from_fork+0x3f/0x70
Nov 14 17:19:22 storage kernel: [5597797.916231] [

[dbavatar]

Still seeing this issue in 7.2 in Ubuntu 16.04: 0.7.2-0york1~16.04. Seems odd it's always asserting the unix timestamp to the same integer.

[Tue Oct 10 16:22:01 2017] VERIFY3(sa.sa_magic == 0x2F505A) failed (1507666897 == 3100762)
[Tue Oct 10 16:22:01 2017] PANIC at zfs_vfsops.c:583:zfs_space_delta_cb()
[Tue Oct 10 16:22:01 2017] Showing stack for process 10689
[Tue Oct 10 16:22:01 2017] CPU: 2 PID: 10689 Comm: dp_sync_taskq Tainted: P OE 4.4.0-96-generic #119-Ubuntu
[Tue Oct 10 16:22:01 2017] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77X-D3H, BIOS F18i 01/06/2014
[Tue Oct 10 16:22:01 2017] 0000000000000286 625cea94ec8a17f0 ffff88073ecafaf0 ffffffff813fabd3
[Tue Oct 10 16:22:01 2017] ffffffffc15a6ef1 0000000000000247 ffff88073ecafb00 ffffffffc12618c2
[Tue Oct 10 16:22:01 2017] ffff88073ecafc88 ffffffffc1261995 0000000000000003 ffff880700000030
[Tue Oct 10 16:22:01 2017] Call Trace:
[Tue Oct 10 16:22:01 2017] [] dump_stack+0x63/0x90
[Tue Oct 10 16:22:01 2017] [] spl_dumpstack+0x42/0x50 [spl]
[Tue Oct 10 16:22:01 2017] [] spl_panic+0xc5/0x100 [spl]
[Tue Oct 10 16:22:01 2017] [] ? taskq_dispatch_ent+0x58/0x180 [spl]
[Tue Oct 10 16:22:01 2017] [] ? zio_reexecute+0x380/0x380 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? spa_taskq_dispatch_ent+0x8e/0xc0 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? dbuf_rele+0x10/0x10 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? zio_taskq_dispatch+0x94/0xa0 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? zio_issue_async+0x12/0x20 [zfs]
[Tue Oct 10 16:22:01 2017] [] zfs_space_delta_cb+0xdf/0x1d0 [zfs]
[Tue Oct 10 16:22:01 2017] [] dmu_objset_userquota_get_ids+0x122/0x3c0 [zfs]
[Tue Oct 10 16:22:01 2017] [] dnode_sync+0x103/0x850 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? default_wake_function+0x12/0x20
[Tue Oct 10 16:22:01 2017] [] ? mutex_lock+0x12/0x30
[Tue Oct 10 16:22:01 2017] [] sync_dnodes_task+0x7b/0xb0 [zfs]
[Tue Oct 10 16:22:01 2017] [] taskq_thread+0x255/0x440 [spl]
[Tue Oct 10 16:22:01 2017] [] ? wake_up_q+0x70/0x70
[Tue Oct 10 16:22:01 2017] [] ? taskq_cancel_id+0x130/0x130 [spl]
[Tue Oct 10 16:22:01 2017] [] kthread+0xe5/0x100
[Tue Oct 10 16:22:01 2017] [] ? kthread_create_on_node+0x1e0/0x1e0
[Tue Oct 10 16:22:01 2017] [] ret_from_fork+0x3f/0x70
[Tue Oct 10 16:22:01 2017] [] ? kthread_create_on_node+0x1e0/0x1e0
[Tue Oct 10 16:22:01 2017] VERIFY3(sa.sa_magic == 0x2F505A) failed (1507666898 == 3100762)
[Tue Oct 10 16:22:01 2017] PANIC at zfs_vfsops.c:583:zfs_space_delta_cb()
[Tue Oct 10 16:22:01 2017] Showing stack for process 10688
[Tue Oct 10 16:22:01 2017] CPU: 2 PID: 10688 Comm: dp_sync_taskq Tainted: P OE 4.4.0-96-generic #119-Ubuntu
[Tue Oct 10 16:22:01 2017] Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./Z77X-D3H, BIOS F18i 01/06/2014
[Tue Oct 10 16:22:01 2017] 0000000000000286 0b44cb7f0498c5d3 ffff88072c8bbaf0 ffffffff813fabd3
[Tue Oct 10 16:22:01 2017] ffffffffc15a6ef1 0000000000000247 ffff88072c8bbb00 ffffffffc12618c2
[Tue Oct 10 16:22:01 2017] ffff88072c8bbc88 ffffffffc1261995 0000000000000000 ffff880700000030
[Tue Oct 10 16:22:01 2017] Call Trace:
[Tue Oct 10 16:22:01 2017] [] dump_stack+0x63/0x90
[Tue Oct 10 16:22:01 2017] [] spl_dumpstack+0x42/0x50 [spl]
[Tue Oct 10 16:22:01 2017] [] spl_panic+0xc5/0x100 [spl]
[Tue Oct 10 16:22:01 2017] [] ? set_next_entity+0x9c/0xb0
[Tue Oct 10 16:22:01 2017] [] ? dnode_rele_and_unlock+0x53/0x80 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? dnode_rele+0x39/0x40 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? dbuf_rele_and_unlock+0x426/0x470 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? __schedule+0x3b6/0xa30
[Tue Oct 10 16:22:01 2017] [] ? spl_kmem_free+0x2a/0x40 [spl]
[Tue Oct 10 16:22:01 2017] [] zfs_space_delta_cb+0xdf/0x1d0 [zfs]
[Tue Oct 10 16:22:01 2017] [] dmu_objset_userquota_get_ids+0x122/0x3c0 [zfs]
[Tue Oct 10 16:22:01 2017] [] dnode_sync+0x103/0x850 [zfs]
[Tue Oct 10 16:22:01 2017] [] ? __switch_to+0x1dc/0x5c0
[Tue Oct 10 16:22:01 2017] [] ? mutex_lock+0x12/0x30
[Tue Oct 10 16:22:01 2017] [] sync_dnodes_task+0x7b/0xb0 [zfs]
[Tue Oct 10 16:22:01 2017] [] taskq_thread+0x255/0x440 [spl]
[Tue Oct 10 16:22:01 2017] [] ? wake_up_q+0x70/0x70
[Tue Oct 10 16:22:01 2017] [] ? taskq_cancel_id+0x130/0x130 [spl]
[Tue Oct 10 16:22:01 2017] [] kthread+0xe5/0x100
[Tue Oct 10 16:22:01 2017] [] ? kthread_create_on_node+0x1e0/0x1e0
[Tue Oct 10 16:22:01 2017] [] ret_from_fork+0x3f/0x70
[Tue Oct 10 16:22:01 2017] [] ? kthread_create_on_node+0x1e0/0x1e0

[behlendorf]

Closing as duplicate of #7117.