Innovation/trunk: PS-9107 : [ERROR] [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.c…

[satya-bodapati]

…c:3833:ib::fatal triggered thread (#5257)

https://perconadev.atlassian.net/browse/PS-9107

Problem:

Two threads are trying to delete the same ibuf rec. The first one succeeds with optimistic delete and this means the record is completely gone (added to page garbage).

Now second thread, cannot do optimistic. So it tries to do pessmistic and wants to position the cursor on the record deleted by the first thread. It cannot. The record is gone. And ibuf asserts.

In one of the innodb master thread active tasks (srv_master_do_active_tasks), innodb tries to merge/contract ibuf in background. So it does this by reading actual pages from disk. So it leaves the job to IO threads. On IO read, we apply pending change buffer entries. This thread (on certain conditions), can decide to a sync read vs async read. In our case, it is an async read. Master thread submits a request to read the page. Lets say space_id:page_no (978:34)

At around same time, an insert/delete into secondary indexes wants to buffer a change but the ibuf has reached to its max size, so it initiates a ibuf_contract(). See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor at a random record and it can happen that it sees the same space_id:page_no as the IO thread is processing.

And just when this tablespace records are about to be processed, the tablespace is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the actual read of secondary index pages. Hey, the tablespace is gone! whats the point in bringing those pages to buffer pool. Hence it decides to delete all ibuf records belonging to space_id (978) in our example.

This leads to the case where two threads can simultaneously process the ibuf records of the same page. IO thread reading a secondary index page and contract ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task) user threads doing ibuf_contract, they process entries belonging to a different tablespace. And when they see that tabelspace is dropped), they try to delete ibuf entries.

Fix:

ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0, the way tablespaces are dropped is a bit different.

fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed. And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags()

Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because the window between the actual deletion of the tablespace object and the user drop tablespace command is bigger. In 8.0, space object is still available, so flags is NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr. We will use this call to determine if tablespace is dropped or not.

ibuf code saw the tablespace as deleted by using fil_space_acquire() in ibuf_merge_or_delete_for_page().

We will use the same call in ibuf_restore_pos() to determine if the tablespace is being deleted.

(cherry picked from commit cff6f3d)

[dlenev]

LGTM!