Deadlock problem (memory starvation issues?) #407
Comments
|
Forgot to add this after the trace : Is there a way to limit ZFS in the memory it can consume for ARC ? (such as the vfs.zfs.arc_max/vfs.zfs.arc_min tunables in FreeBSD) |
|
Thanks for filing the issue. You may have hit a deadlock somehow, it looks like the active txg is blocked for some reason. As for limiting the ARC this can be done through linux module options, see 'modinfo zfs'. |
|
I have the same or similar problem, I tried "modprobe zfs zfs_arc_max=131072" and verified it with "cat /sys/module/zfs/parameters/zfs_arc_max", but it didn't help. Memory usage grows, then it starts swapping, and then the system freezes. What works for me is "echo 3 >/proc/sys/vm/drop_caches", and I also need to stop any heavy io activity or else the echo command blocks and memory usage is still going up. My distribution is Ubuntu 11.10, kernel 3.0.0-12-generic x86_64, zfs-linux 0.6.0.34-0ubuntu1 from the PPA, running on my home computer with 4GB RAM. For ZFS I created 40GB LVM volume to test how it compares to zfs-fuse for my backup jobs. I have a backup server with around 700GB zfs-fuse pool, onto which several machines are backed up daily over the net using rsync. For the test I started backing up one of the machines, a small one, around 15GB and around 300000 files. Initial rsync finished successfully, but further backups freeze my computer unless I "kill -STOP" rsync process once in a while, drop caches, wait around 10-15 sec. for it to work, then "kill -CONT" the process. |
|
Oh, so the zfs_arc_max is in BYTES (doh), I set it too low and it was ignored, as I found in "/proc/spl/kstat/zfs/arcstats". It would be nice if in such case ZFS just used the smallest allowed value and print some warning in the log. I ended up putting "options zfs zfs_arc_max=268435456 zfs_arc_meta_limit=268435456" in /etc/modprobe.d/zfs.conf, and I think I can live with this workaround for a while. I added zfs_arc_meta_limit because I've seen arc_meta_used in arcstats going above 1GB. I don't mind free memory being used for cache, but for some reason this memory is not being reclaimed under pressure. |
|
I believe this is a duplicate of issue #287, however it's hard to say for sure because your stacks aren't 100% complete. |
|
Closing issue, this should be fixed by the following two commits for #287. |
I am running ZFS on Linux on a 3.0.0 kernel on Debian wheezy/sid :
Linux esprit-doujin.doujin-damashii.jp 3.0.0-1-amd64 #1 SMP Sat Aug 27 16:21:11 UTC 2011 x86_64 GNU/Linux
The system would run perfectly fine until memory usage and I/O would crank up (first time happening so I couldn't monitor and keep track of resource usage and pinpoint the exact conditions.)
My root file system is ext4, homes and data partitions are in ZFS. I am using the Ubuntu repository, and the package version 0.6.0.32-0ubuntu1~natty1.
The following trace and phenomon seems to indicate I had used all of my memory and swap, and ended up with a deadlock between ZFS and swapping. Basic services (simple HTTP pages) could still answer, but establishing SSH sessions (even as root, whose home is on ext4fs) was not possible. Existing sessions would lock the very second I tried accessing files on ZFS.
Sep 22 09:34:48 esprit-doujin kernel: [505534.217260] kthreadd D ffff88040f142e20 0 2 0 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.217265] ffff88040f142e20 0000000000000046 0000000000000082 ffff880400000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.217271] ffff88040f1a20c0 0000000000012800 ffff88040f14bfd8 ffff88040f14bfd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.217276] 0000000000012800 ffff88040f142e20 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.217281] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.217296] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.217303] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.217329] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.217348] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.217354] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.217370] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.217374] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.217379] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.217383] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.217388] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.217394] [] ? paravirt_read_tsc+0x5/0x8
Sep 22 09:34:48 esprit-doujin kernel: [505534.217398] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.217403] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.217407] [] ? cpumask_next_and+0x2b/0x3a
Sep 22 09:34:48 esprit-doujin kernel: [505534.217412] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.217418] [] ? copy_process+0x109/0x10eb
Sep 22 09:34:48 esprit-doujin kernel: [505534.217422] [] ? perf_event_task_sched_out+0x48/0x54
Sep 22 09:34:48 esprit-doujin kernel: [505534.217427] [] ? do_fork+0xff/0x263
Sep 22 09:34:48 esprit-doujin kernel: [505534.217431] [] ? kernel_thread+0x7b/0x83
Sep 22 09:34:48 esprit-doujin kernel: [505534.217435] [] ? kthread_worker_fn+0x149/0x149
Sep 22 09:34:48 esprit-doujin kernel: [505534.217439] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.217442] [] ? kthreadd+0xe7/0x125
Sep 22 09:34:48 esprit-doujin kernel: [505534.217446] [] ? finish_task_switch+0x84/0xaf
Sep 22 09:34:48 esprit-doujin kernel: [505534.217449] [] ? kernel_thread_helper+0x4/0x10
Sep 22 09:34:48 esprit-doujin kernel: [505534.217453] [] ? tsk_fork_get_node+0x1a/0x1a
Sep 22 09:34:48 esprit-doujin kernel: [505534.217456] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.219901] kswapd0 D ffff88040c187710 0 29 2 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.219905] ffff88040c187710 0000000000000046 0000000000000086 ffff88040132b550
Sep 22 09:34:48 esprit-doujin kernel: [505534.219910] ffff88040bbc0930 0000000000012800 ffff88040c189fd8 ffff88040c189fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.219915] 0000000000012800 ffff88040c187710 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.219920] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.219924] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.219932] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.219938] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.219961] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.219982] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.219989] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.220007] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.220013] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.220020] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.220027] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.220033] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.220039] [] ? shrink_slab+0x143/0x155
Sep 22 09:34:48 esprit-doujin kernel: [505534.220045] [] ? balance_pgdat+0x28e/0x514
Sep 22 09:34:48 esprit-doujin kernel: [505534.220052] [] ? kswapd+0x2b4/0x2cd
Sep 22 09:34:48 esprit-doujin kernel: [505534.220058] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.220064] [] ? balance_pgdat+0x514/0x514
Sep 22 09:34:48 esprit-doujin kernel: [505534.220070] [] ? balance_pgdat+0x514/0x514
Sep 22 09:34:48 esprit-doujin kernel: [505534.220076] [] ? kthread+0x7a/0x82
Sep 22 09:34:48 esprit-doujin kernel: [505534.220082] [] ? kernel_thread_helper+0x4/0x10
Sep 22 09:34:48 esprit-doujin kernel: [505534.220088] [] ? kthread_worker_fn+0x149/0x149
Sep 22 09:34:48 esprit-doujin kernel: [505534.220093] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.222574] z_wr_iss/0 D ffff8803fedf1790 0 677 2 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.222579] ffff8803fedf1790 0000000000000046 ffff880400c16ea0 ffff880400c16ea0
Sep 22 09:34:48 esprit-doujin kernel: [505534.222583] ffff880400c16ea0 0000000000012800 ffff8803fedf3fd8 ffff8803fedf3fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.222588] 0000000000012800 ffff8803fedf1790 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.222593] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.222596] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.222603] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222607] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.222627] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222645] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222649] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.222665] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222669] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.222673] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.222678] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.222682] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.222687] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.222692] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.222697] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.222702] [] ? kmem_getpages+0x55/0xf0
Sep 22 09:34:48 esprit-doujin kernel: [505534.222706] [] ? fallback_alloc+0x129/0x1c1
Sep 22 09:34:48 esprit-doujin kernel: [505534.222713] [] ? kmem_alloc_debug+0x73/0xcf [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222717] [] ? __kmalloc+0x95/0xf0
Sep 22 09:34:48 esprit-doujin kernel: [505534.222723] [] ? kmem_alloc_debug+0x73/0xcf [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222745] [] ? lzjb_compress+0x43/0x1d5 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222750] [] ? should_resched+0x5/0x24
Sep 22 09:34:48 esprit-doujin kernel: [505534.222754] [] ? _cond_resched+0x9/0x20
Sep 22 09:34:48 esprit-doujin kernel: [505534.222771] [] ? zio_compress_data+0x63/0xa1 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222788] [] ? zio_write_bp_init+0x1bf/0x3e6 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222805] [] ? zio_execute+0xf1/0x119 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222810] [] ? arch_local_irq_save+0x14/0x1d
Sep 22 09:34:48 esprit-doujin kernel: [505534.222817] [] ? taskq_thread+0x1d2/0x32c [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222821] [] ? try_to_wake_up+0x199/0x199
Sep 22 09:34:48 esprit-doujin kernel: [505534.222828] [] ? __taskq_create+0x373/0x373 [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222838] [] ? __taskq_create+0x373/0x373 [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.222842] [] ? kthread+0x7a/0x82
Sep 22 09:34:48 esprit-doujin kernel: [505534.222845] [] ? kernel_thread_helper+0x4/0x10
Sep 22 09:34:48 esprit-doujin kernel: [505534.222849] [] ? kthread_worker_fn+0x149/0x149
Sep 22 09:34:48 esprit-doujin kernel: [505534.222853] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.225397] txg_sync D ffff880400c06e60 0 821 2 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.225401] ffff880400c06e60 0000000000000046 ffff88040126bc58 ffffffff81035861
Sep 22 09:34:48 esprit-doujin kernel: [505534.225406] ffff8803fedf02c0 0000000000012800 ffff8803fb403fd8 ffff8803fb403fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.225411] 0000000000012800 ffff880400c06e60 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.225415] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.225419] [] ? __wake_up_common+0x41/0x78
Sep 22 09:34:48 esprit-doujin kernel: [505534.225423] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.225430] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225433] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.225438] [] ? _cond_resched+0x9/0x20
Sep 22 09:34:48 esprit-doujin kernel: [505534.225454] [] ? zio_wait+0x12b/0x15c [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225475] [] ? dsl_pool_sync+0x2b9/0x422 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225498] [] ? spa_sync+0x48b/0x83a [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225503] [] ? __wake_up+0x35/0x46
Sep 22 09:34:48 esprit-doujin kernel: [505534.225508] [] ? __bitmap_weight+0x36/0x76
Sep 22 09:34:48 esprit-doujin kernel: [505534.225530] [] ? txg_sync_thread+0x1d6/0x331 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225537] [] ? __thread_create+0x129/0x129 [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225556] [] ? txg_thread_wait.clone.2+0x2f/0x2f [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225562] [] ? __thread_create+0x129/0x129 [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225568] [] ? thread_generic_wrapper+0x6a/0x75 [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.225572] [] ? kthread+0x7a/0x82
Sep 22 09:34:48 esprit-doujin kernel: [505534.225576] [] ? kernel_thread_helper+0x4/0x10
Sep 22 09:34:48 esprit-doujin kernel: [505534.225580] [] ? kthread_worker_fn+0x149/0x149
Sep 22 09:34:48 esprit-doujin kernel: [505534.225583] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.228125] mysqld D ffff8800b1dda140 0 31890 1646 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.228130] ffff8800b1dda140 0000000000000082 0000000000000096 ffff880400000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.228135] ffff88040f1f2f60 0000000000012800 ffff88000cb43fd8 ffff88000cb43fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.228140] 0000000000012800 ffff8800b1dda140 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.228144] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.228152] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228155] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.228175] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228194] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228198] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.228214] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228218] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.228223] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.228227] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.228232] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.228237] [] ? paravirt_read_tsc+0x5/0x8
Sep 22 09:34:48 esprit-doujin kernel: [505534.228241] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.228246] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.228251] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.228256] [] ? kmem_getpages+0x55/0xf0
Sep 22 09:34:48 esprit-doujin kernel: [505534.228259] [] ? fallback_alloc+0x129/0x1c1
Sep 22 09:34:48 esprit-doujin kernel: [505534.228264] [] ? kmem_cache_alloc_node_trace+0x8a/0x10f
Sep 22 09:34:48 esprit-doujin kernel: [505534.228279] [] ? __ext4_handle_dirty_metadata+0xe7/0xf0 [ext4]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228285] [] ? __alloc_skb+0x75/0x12d
Sep 22 09:34:48 esprit-doujin kernel: [505534.228289] [] ? sock_alloc_send_pskb+0xb4/0x2d3
Sep 22 09:34:48 esprit-doujin kernel: [505534.228301] [] ? __ext4_journal_stop+0x72/0x78 [ext4]
Sep 22 09:34:48 esprit-doujin kernel: [505534.228306] [] ? cred_to_ucred+0x57/0x5f
Sep 22 09:34:48 esprit-doujin kernel: [505534.228310] [] ? unix_stream_sendmsg+0x119/0x2b0
Sep 22 09:34:48 esprit-doujin kernel: [505534.228315] [] ? sock_aio_write+0xde/0xed
Sep 22 09:34:48 esprit-doujin kernel: [505534.228320] [] ? do_sync_write+0xb1/0xea
Sep 22 09:34:48 esprit-doujin kernel: [505534.228326] [] ? security_file_permission+0x18/0x33
Sep 22 09:34:48 esprit-doujin kernel: [505534.228329] [] ? vfs_write+0xb9/0xf9
Sep 22 09:34:48 esprit-doujin kernel: [505534.228333] [] ? sys_write+0x45/0x6b
Sep 22 09:34:48 esprit-doujin kernel: [505534.228337] [] ? system_call_fastpath+0x16/0x1b
Sep 22 09:34:48 esprit-doujin kernel: [505534.230902] cron D ffff88040b96c040 0 2378 1 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.230906] ffff88040b96c040 0000000000000086 0000000000000082 ffff880400c16ea0
Sep 22 09:34:48 esprit-doujin kernel: [505534.230911] ffff8803fede6280 0000000000012800 ffff8803fca11fd8 ffff8803fca11fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.230915] 0000000000012800 ffff88040b96c040 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.230924] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.230929] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.230939] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.230945] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.230968] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.230988] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.230995] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.231013] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.231019] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.231026] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.231033] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.231039] [] ? cpumask_next_and+0x2b/0x3a
Sep 22 09:34:48 esprit-doujin kernel: [505534.231046] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.231053] [] ? paravirt_read_tsc+0x5/0x8
Sep 22 09:34:48 esprit-doujin kernel: [505534.231059] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.231065] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.231072] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.231080] [] ? copy_process+0x109/0x10eb
Sep 22 09:34:48 esprit-doujin kernel: [505534.231086] [] ? cp_new_stat+0xc7/0xdd
Sep 22 09:34:48 esprit-doujin kernel: [505534.231093] [] ? do_fork+0xff/0x263
Sep 22 09:34:48 esprit-doujin kernel: [505534.231099] [] ? sys_newstat+0x20/0x29
Sep 22 09:34:48 esprit-doujin kernel: [505534.231106] [] ? stub_clone+0x13/0x20
Sep 22 09:34:48 esprit-doujin kernel: [505534.231112] [] ? system_call_fastpath+0x16/0x1b
Sep 22 09:34:48 esprit-doujin kernel: [505534.233725] sshd D ffff88040bc1e040 0 2455 1 0x00000004
Sep 22 09:34:48 esprit-doujin kernel: [505534.233730] ffff88040bc1e040 0000000000000086 0000000000000082 ffff880400000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.233735] ffffffff8160b020 0000000000012800 ffff88040136bfd8 ffff88040136bfd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.233739] 0000000000012800 ffff88040bc1e040 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.233744] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.233751] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.233754] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.233774] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.233792] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.233808] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.233812] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.233816] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.233821] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.233825] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.233830] [] ? paravirt_read_tsc+0x5/0x8
Sep 22 09:34:48 esprit-doujin kernel: [505534.233835] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.233839] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.233844] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.233849] [] ? kmem_cache_alloc+0x86/0xf0
Sep 22 09:34:48 esprit-doujin kernel: [505534.233854] [] ? copy_process+0x109/0x10eb
Sep 22 09:34:48 esprit-doujin kernel: [505534.233857] [] ? kmem_cache_alloc+0x86/0xf0
Sep 22 09:34:48 esprit-doujin kernel: [505534.233861] [] ? get_empty_filp+0x94/0x113
Sep 22 09:34:48 esprit-doujin kernel: [505534.233866] [] ? do_fork+0xff/0x263
Sep 22 09:34:48 esprit-doujin kernel: [505534.233870] [] ? need_resched+0x1a/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.233874] [] ? should_resched+0x5/0x24
Sep 22 09:34:48 esprit-doujin kernel: [505534.233878] [] ? _cond_resched+0x9/0x20
Sep 22 09:34:48 esprit-doujin kernel: [505534.233883] [] ? stub_clone+0x13/0x20
Sep 22 09:34:48 esprit-doujin kernel: [505534.233887] [] ? system_call_fastpath+0x16/0x1b
Sep 22 09:34:48 esprit-doujin kernel: [505534.236572] rtorrent D ffff8800b50cb7d0 0 5447 5446 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.236576] ffff8800b50cb7d0 0000000000000082 0000000000000092 ffff880400c16ea0
Sep 22 09:34:48 esprit-doujin kernel: [505534.236581] ffff8803fede6970 0000000000012800 ffff880347c39fd8 ffff880347c39fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.236586] 0000000000012800 ffff8800b50cb7d0 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.236590] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.236594] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.236601] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.236605] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.236625] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.236642] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.236647] [] ? __pagevec_free+0x5b/0x6a
Sep 22 09:34:48 esprit-doujin kernel: [505534.236663] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.236667] [] ? pageout+0x130/0x232
Sep 22 09:34:48 esprit-doujin kernel: [505534.236671] [] ? shrink_page_list+0x267/0x496
Sep 22 09:34:48 esprit-doujin kernel: [505534.236676] [] ? shrink_inactive_list+0x23a/0x391
Sep 22 09:34:48 esprit-doujin kernel: [505534.236681] [] ? shrink_zone+0x36d/0x46c
Sep 22 09:34:48 esprit-doujin kernel: [505534.236685] [] ? do_try_to_free_pages+0x106/0x315
Sep 22 09:34:48 esprit-doujin kernel: [505534.236690] [] ? try_to_free_pages+0xb6/0xf6
Sep 22 09:34:48 esprit-doujin kernel: [505534.236695] [] ? __alloc_pages_nodemask+0x4e7/0x75c
Sep 22 09:34:48 esprit-doujin kernel: [505534.236701] [] ? alloc_pages_current+0xa5/0xbf
Sep 22 09:34:48 esprit-doujin kernel: [505534.236706] [] ? __do_page_cache_readahead+0x96/0x1c8
Sep 22 09:34:48 esprit-doujin kernel: [505534.236711] [] ? vma_prio_tree_insert+0x20/0x3e
Sep 22 09:34:48 esprit-doujin kernel: [505534.236716] [] ? force_page_cache_readahead+0x62/0x88
Sep 22 09:34:48 esprit-doujin kernel: [505534.236720] [] ? sys_madvise+0x3bd/0x658
Sep 22 09:34:48 esprit-doujin kernel: [505534.236724] [] ? fput+0x1a/0x1a2
Sep 22 09:34:48 esprit-doujin kernel: [505534.236728] [] ? sys_mmap_pgoff+0x136/0x160
Sep 22 09:34:48 esprit-doujin kernel: [505534.236732] [] ? system_call_fastpath+0x16/0x1b
Sep 22 09:34:48 esprit-doujin kernel: [505534.239455] flush-zfs-29 D ffff88040132b550 0 6364 2 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.239460] ffff88040132b550 0000000000000046 0000000000000086 ffff880400c16ea0
Sep 22 09:34:48 esprit-doujin kernel: [505534.239470] ffff88040bbc1020 0000000000012800 ffff88012f247fd8 ffff88012f247fd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.239479] 0000000000012800 ffff88040132b550 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.239489] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.239494] [] ? _raw_spin_lock_irqsave+0x9/0x25
Sep 22 09:34:48 esprit-doujin kernel: [505534.239503] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.239509] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.239530] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.239550] [] ? zfs_putpage+0x110/0x1d4 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.239569] [] ? zpl_putpage+0x22/0x2d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.239575] [] ? write_cache_pages+0x200/0x335
Sep 22 09:34:48 esprit-doujin kernel: [505534.239593] [] ? zpl_readpage+0x4a/0x4a [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.239601] [] ? writeback_single_inode+0xac/0x1b5
Sep 22 09:34:48 esprit-doujin kernel: [505534.239607] [] ? writeback_sb_inodes+0xbf/0x13c
Sep 22 09:34:48 esprit-doujin kernel: [505534.239613] [] ? writeback_inodes_wb+0x161/0x173
Sep 22 09:34:48 esprit-doujin kernel: [505534.239620] [] ? wb_writeback+0x246/0x366
Sep 22 09:34:48 esprit-doujin kernel: [505534.239626] [] ? need_resched+0x1a/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.239632] [] ? schedule+0x5e5/0x5fc
Sep 22 09:34:48 esprit-doujin kernel: [505534.239639] [] ? wb_do_writeback+0x145/0x1a2
Sep 22 09:34:48 esprit-doujin kernel: [505534.239645] [] ? bdi_writeback_thread+0x86/0x20a
Sep 22 09:34:48 esprit-doujin kernel: [505534.239651] [] ? wb_do_writeback+0x1a2/0x1a2
Sep 22 09:34:48 esprit-doujin kernel: [505534.239657] [] ? wb_do_writeback+0x1a2/0x1a2
Sep 22 09:34:48 esprit-doujin kernel: [505534.239662] [] ? kthread+0x7a/0x82
Sep 22 09:34:48 esprit-doujin kernel: [505534.239668] [] ? kernel_thread_helper+0x4/0x10
Sep 22 09:34:48 esprit-doujin kernel: [505534.239674] [] ? kthread_worker_fn+0x149/0x149
Sep 22 09:34:48 esprit-doujin kernel: [505534.239680] [] ? gs_change+0x13/0x13
Sep 22 09:34:48 esprit-doujin kernel: [505534.242475] sftp-server D ffff88040132a080 0 6620 1 0x00000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.242479] ffff88040132a080 0000000000000082 0000000000000092 ffff880400000000
Sep 22 09:34:48 esprit-doujin kernel: [505534.242484] ffff88040f1a20c0 0000000000012800 ffff88038bf9bfd8 ffff88038bf9bfd8
Sep 22 09:34:48 esprit-doujin kernel: [505534.242489] 0000000000012800 ffff88040132a080 0000000000012800 0000000000012800
Sep 22 09:34:48 esprit-doujin kernel: [505534.242493] Call Trace:
Sep 22 09:34:48 esprit-doujin kernel: [505534.242500] [] ? cv_wait_common+0x75/0xbc [spl]
Sep 22 09:34:48 esprit-doujin kernel: [505534.242504] [] ? wake_up_bit+0x23/0x23
Sep 22 09:34:48 esprit-doujin kernel: [505534.242523] [] ? txg_wait_open+0x57/0x8d [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.242541] [] ? zfs_write+0x505/0x9b0 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.242545] [] ? current_fs_time+0x1e/0x24
Sep 22 09:34:48 esprit-doujin kernel: [505534.242550] [] ? should_resched+0x5/0x24
Sep 22 09:34:48 esprit-doujin kernel: [505534.242567] [] ? zpl_write_common+0x51/0x69 [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.242583] [] ? zpl_write+0x59/0x7c [zfs]
Sep 22 09:34:48 esprit-doujin kernel: [505534.242586] [] ? vfs_write+0xa6/0xf9
Sep 22 09:34:48 esprit-doujin kernel: [505534.242589] [] ? generic_file_llseek+0x47/0x52
Sep 22 09:34:48 esprit-doujin kernel: [505534.242593] [] ? sys_write+0x45/0x6b
Sep 22 09:34:48 esprit-doujin kernel: [505534.242597] [] ? system_call_fastpath+0x16/0x1b
The text was updated successfully, but these errors were encountered: