OpenZFS 7504 - kmem_reap hangs spa_sync and administrative tasks

Authored by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Prakash Surya <prakash.surya@delphix.com>
Approved by: Dan McDonald <danmcd@omniti.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-by: Tim Chase <tim@chase2k.com>
Ported-by: George Melikov <mail@gmelikov.ru>

OpenZFS-issue: https://www.illumos.org/issues/7504
OpenZFS-commit: openzfs/openzfs@405a5a0
Closes #5736

module/zfs/arc.c

@@ -4193,7 +4193,6 @@ arc_reclaim_thread(void)
 	mutex_enter(&arc_reclaim_lock);
 	while (!arc_reclaim_thread_exit) {
 		int64_t to_free;
-		int64_t free_memory = arc_available_memory();
 		uint64_t evicted = 0;
 
 		arc_tuning_update();
@@ -4215,6 +4214,14 @@ arc_reclaim_thread(void)
 #endif
 		mutex_exit(&arc_reclaim_lock);
 
+		/*
+		 * We call arc_adjust() before (possibly) calling
+		 * arc_kmem_reap_now(), so that we can wake up
+		 * arc_get_data_buf() sooner.
+		 */
+		evicted = arc_adjust();
+
+		int64_t free_memory = arc_available_memory();
 		if (free_memory < 0) {
 
 			arc_no_grow = B_TRUE;
@@ -4247,8 +4254,6 @@ arc_reclaim_thread(void)
 			arc_no_grow = B_FALSE;
 		}
 
-		evicted = arc_adjust();
-
 		mutex_enter(&arc_reclaim_lock);
 
 		/*