PS-9107 : [ERROR] [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.c… #5257
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
satya-bodapati
force-pushed
the
PS-8.0-9107
branch
from
07c0634
to
c34e6ed
Compare
dlenev
requested changes
Mar 18, 2024
|
Testing was done using the script provided in PS-9107. Mtr testcase is not possible because of nature of ibuf randomness. |
…c:3833:ib::fatal triggered thread https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
satya-bodapati
force-pushed
the
PS-8.0-9107
branch
from
9181c17
to
b45c4fa
Compare
|
Thanks for the review @dlenev ! |
satya-bodapati
added a commit
to satya-bodapati/percona-server
that referenced
this pull request
Mar 21, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted. (cherry picked from commit cff6f3d)
satya-bodapati
added a commit
to satya-bodapati/percona-server
that referenced
this pull request
Mar 21, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted. (cherry picked from commit cff6f3d)
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 29, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted. (cherry picked from commit cff6f3d)
oleksandr-kachan
pushed a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
Apr 29, 2024
…ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted. (cherry picked from commit cff6f3d)
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
May 29, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
oleksandr-kachan
added a commit
to oleksandr-kachan/percona-server
that referenced
this pull request
May 29, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257)" This reverts commit ccd2e08.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
May 31, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
Jun 4, 2024
…c:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
Jun 5, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
Jun 10, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
Jun 12, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
VarunNagaraju
pushed a commit
to VarunNagaraju/percona-server
that referenced
this pull request
Jun 12, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
inikep
pushed a commit
that referenced
this pull request
Sep 25, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Sep 25, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
inikep
pushed a commit
to inikep/percona-server
that referenced
this pull request
Oct 28, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (percona#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
inikep
pushed a commit
that referenced
this pull request
Oct 30, 2024
…ailure: ibuf0ibuf.cc:3833:ib::fatal triggered thread (#5257) https://perconadev.atlassian.net/browse/PS-9107 Problem: -------- Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage). Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts. In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34) At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing. And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example. This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries. Fix: ---- ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different. fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags() Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not. ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
…c:3833:ib::fatal triggered thread
https://perconadev.atlassian.net/browse/PS-9107
Problem:
Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage).
Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts.
In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34)
At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing.
And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example.
This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries.
Fix:
ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different.
fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags()
Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not.
ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page(). We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted. Additional state of 'being deleted' is handled as well.