Hung/Crash for unknown reason #2247
Milestone
Comments
|
I'm sorry to say it does look zfs related. However, based in the stacks this might be a known issue which was addressed in the latest code. |
|
Any guess as to when the next release is expected? |
|
Had another early this morning, don't know if it's the same or not. |
|
Yes, it looks the same. We'll get the next release out once we resolve the remaining outstanding issues associated with it. https://github.com/zfsonlinux/zfs/issues?labels=&milestone=23&page=1&state=open |
|
Any tips on what to do in the meantime, to try to minimize the occurrence of this? |
|
OK, are there any more detailed install instructions somewhere, or is it very straight forward to just run autogen.sh and build from there? Any prerequisites or requirements that I need to be aware of? |
|
Never mind, found this: http://zfsonlinux.org/generic-rpm.html |
|
Hmm, do I perhaps need something more than a168788 ? |
|
Or maybe I shouldn't try to cherry-pick that onto master, since it seems to have been updated with something newer. Onto what should I cherry-pick a168788 ? |
|
@jby That fix is already applied to master. My suggestion would be just to run master, or cherry-pick that back to 0.6.2. Although it may not apply cleanly so just building packages using master is probably easiest. |
|
Thanks, I'll go with master then. |
|
Hmm, This is from very early this morning - running master from tuesday (zfs-0.6.2-259_gde39ec1.el6.x86_64): |
|
Yes, but only a couple of times - and all within the same hour. I built and installed master on April 22 (commit de39ec1) and have had the happen 10 times within an hour early in the morning on October 18. That's it. I've been planning to upgrade to an official release, but since it's been running pretty well since I installed from master I've tried to leave it alone. |
behlendorf
pushed a commit
to openzfs/spl
that referenced
this issue
Dec 19, 2014
It is known that mutexes in Linux are not safe when using them to synchronize the freeing of object in which the mutex is embedded: http://lwn.net/Articles/575477/ The known places in ZFS which are suspected to suffer from the race condition are zio->io_lock and dbuf->db_mtx. * zio uses zio->io_lock and zio->io_cv to synchronize freeing between zio_wait() and zio_done(). * dbuf uses dbuf->db_mtx to protect reference counting. This patch fixes this kind of race by forcing serialization on mutex_exit() with a spin lock, making the mutex safe by sacrificing a bit of performance and memory overhead. This issue most commonly manifests itself as a deadlock in the zio pipeline caused by a process spinning on the damaged mutex. Similar deadlocks have been reported for the dbuf->db_mtx mutex. And it can also cause a NULL dereference or bad paging request under the right circumstances. This issue any many like it are linked off the openzfs/zfs#2523 issue. Specifically this fix resolves at least the following outstanding issues: openzfs/zfs#401 openzfs/zfs#2523 openzfs/zfs#2679 openzfs/zfs#2684 openzfs/zfs#2704 openzfs/zfs#2708 openzfs/zfs#2517 openzfs/zfs#2827 openzfs/zfs#2850 openzfs/zfs#2891 openzfs/zfs#2897 openzfs/zfs#2247 openzfs/zfs#2939 Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Closes #421
|
This issue which is a duplicate of #2523 was resolved by the following commit. Full details can be found in the commit message and related lwn article. openzfs/spl@a3c1eb7 mutex: force serialization on mutex_exit() to fix races |
dajhorn
added a commit
to zfsonlinux/pkg-spl
that referenced
this issue
Dec 20, 2014
Commit: openzfs/zfs@a3c1eb7 From: Chunwei Chen <tuxoko@gmail.com> Date: Fri, 19 Dec 2014 11:31:59 +0800 Subject: mutex: force serialization on mutex_exit() to fix races It is known that mutexes in Linux are not safe when using them to synchronize the freeing of object in which the mutex is embedded: http://lwn.net/Articles/575477/ The known places in ZFS which are suspected to suffer from the race condition are zio->io_lock and dbuf->db_mtx. * zio uses zio->io_lock and zio->io_cv to synchronize freeing between zio_wait() and zio_done(). * dbuf uses dbuf->db_mtx to protect reference counting. This patch fixes this kind of race by forcing serialization on mutex_exit() with a spin lock, making the mutex safe by sacrificing a bit of performance and memory overhead. This issue most commonly manifests itself as a deadlock in the zio pipeline caused by a process spinning on the damaged mutex. Similar deadlocks have been reported for the dbuf->db_mtx mutex. And it can also cause a NULL dereference or bad paging request under the right circumstances. This issue any many like it are linked off the openzfs/zfs#2523 issue. Specifically this fix resolves at least the following outstanding issues: openzfs/zfs#401 openzfs/zfs#2523 openzfs/zfs#2679 openzfs/zfs#2684 openzfs/zfs#2704 openzfs/zfs#2708 openzfs/zfs#2517 openzfs/zfs#2827 openzfs/zfs#2850 openzfs/zfs#2891 openzfs/zfs#2897 openzfs/zfs#2247 openzfs/zfs#2939 Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Backported-by: Darik Horn <dajhorn@vanadac.com> Closes #421 Conflicts: include/sys/mutex.h
behlendorf
pushed a commit
to openzfs/spl
that referenced
this issue
Dec 23, 2014
It is known that mutexes in Linux are not safe when using them to synchronize the freeing of object in which the mutex is embedded: http://lwn.net/Articles/575477/ The known places in ZFS which are suspected to suffer from the race condition are zio->io_lock and dbuf->db_mtx. * zio uses zio->io_lock and zio->io_cv to synchronize freeing between zio_wait() and zio_done(). * dbuf uses dbuf->db_mtx to protect reference counting. This patch fixes this kind of race by forcing serialization on mutex_exit() with a spin lock, making the mutex safe by sacrificing a bit of performance and memory overhead. This issue most commonly manifests itself as a deadlock in the zio pipeline caused by a process spinning on the damaged mutex. Similar deadlocks have been reported for the dbuf->db_mtx mutex. And it can also cause a NULL dereference or bad paging request under the right circumstances. This issue any many like it are linked off the openzfs/zfs#2523 issue. Specifically this fix resolves at least the following outstanding issues: openzfs/zfs#401 openzfs/zfs#2523 openzfs/zfs#2679 openzfs/zfs#2684 openzfs/zfs#2704 openzfs/zfs#2708 openzfs/zfs#2517 openzfs/zfs#2827 openzfs/zfs#2850 openzfs/zfs#2891 openzfs/zfs#2897 openzfs/zfs#2247 openzfs/zfs#2939 Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Closes #421
|
@behlendorf There is a class of issues that looked like the #2523 issue, but are a different problem that affects NUMA systems according to #3091. In reviewing the issues marked fixed by the #2523 fix, this one appears to be distinct. @jby Did you try using a newer version of ZoL that had the patch that fixed #2523 and did the issue go away when you did? |
|
After looking again, the problem is that the stack traces just don't resolve the RIP value. This looks exactly like #3091. |
|
I've been running zfs-0.6.3-1.2.el6.x86_64 for quite a while now, without any problems. |
|
@ryao #3091 definitely is a legitimate issue, however from what I've seen reported thus far it is far less likely to occur that #2523 which has been fixed. Since it's very hard to tell the different between the two issues I suspect most of the existing issues I closer were caused by #2523. However, it's entirely possible a few people saw both problems. However, I thank we can and should track any remaining issues similar to this under #3091. |
ryao
pushed a commit
to ryao/spl
that referenced
this issue
Feb 19, 2015
It is known that mutexes in Linux are not safe when using them to synchronize the freeing of object in which the mutex is embedded: http://lwn.net/Articles/575477/ The known places in ZFS which are suspected to suffer from the race condition are zio->io_lock and dbuf->db_mtx. * zio uses zio->io_lock and zio->io_cv to synchronize freeing between zio_wait() and zio_done(). * dbuf uses dbuf->db_mtx to protect reference counting. This patch fixes this kind of race by forcing serialization on mutex_exit() with a spin lock, making the mutex safe by sacrificing a bit of performance and memory overhead. This issue most commonly manifests itself as a deadlock in the zio pipeline caused by a process spinning on the damaged mutex. Similar deadlocks have been reported for the dbuf->db_mtx mutex. And it can also cause a NULL dereference or bad paging request under the right circumstances. This issue any many like it are linked off the openzfs/zfs#2523 issue. Specifically this fix resolves at least the following outstanding issues: openzfs/zfs#401 openzfs/zfs#2523 openzfs/zfs#2679 openzfs/zfs#2684 openzfs/zfs#2704 openzfs/zfs#2708 openzfs/zfs#2517 openzfs/zfs#2827 openzfs/zfs#2850 openzfs/zfs#2891 openzfs/zfs#2897 openzfs/zfs#2247 openzfs/zfs#2939 Signed-off-by: Chunwei Chen <tuxoko@gmail.com> Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov> Signed-off-by: Richard Yao <ryao@gentoo.org> Closes openzfs#421 Conflicts: include/sys/mutex.h
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
There are a couple of rsync tasks at the end, they might not be interesting but then again they might be. The hung rsync tasks are caused by the previous hangs.
Don't know if it's 100% ZFS related since it starts off with the hung kthreadd.
The text was updated successfully, but these errors were encountered: