lock contention on z_acl_lock during concurrent file creation

[ahrens]

When many threads are concurrently creating files, I observed that we spend quite a bit of time acquiring the z_acl_lock mutex from zfs_zaccess_aces_check(). In this workload, we are not actually modifying anything protected by the z_acl_lock. We could change this to be a rwlock, and acquire it for RW_READER in these code paths. However, each thread acquiring it for RW_READER must modify the lock to increment/decrement the hold count. This requires moving the cache line to the running CPU, inducing cache line contention.

A fix for this bug could involve using a "fanned out reader-writer lock", i.e. a lock which has several sub-locks on different cache lines, approximately one for each CPU. To acquire this type of lock for reader, only the CPU's assigned sub-lock need be acquired. To acquire this type of lock for writer, all sub-locks must be acquired. Therefore, acquiring the lock for writer is more expensive than with a normal rwlock. The uses of z_acl_lock where the protected values are being modified, and therefore we would need to acquire the lock for writer, appear to not be exercised frequently on common workloads.

Note that the bottlenecks caused by #4636 #4641 #4703 #4804 may need to be fixed before fixing this bug will show a significant improvement. In terms of cost/reward, once the other bottlenecks are addressed, fixing this bug will provide a small performance gain and require a small amount of effort to implement.

Analysis sponsored by Intel Corp.

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.