PS-9107 : [ERROR] [MY-013183] [InnoDB] Assertion failure: ibuf0ibuf.c…

…c:3833:ib::fatal triggered thread (percona#5257)

https://perconadev.atlassian.net/browse/PS-9107

Problem:
--------
Two threads are trying to delete the same ibuf rec. The first one succeeds with
optimistic delete and this means the record is completely gone (added to page garbage).

Now second thread, cannot do optimistic. So it tries to do pessmistic and wants
to position the cursor on the record deleted by the first thread. It cannot.
The record is gone. And ibuf asserts.

In one of the innodb master thread active tasks (srv_master_do_active_tasks),
innodb tries to merge/contract ibuf in background. So it does this by reading
actual pages from disk. So it leaves the job to IO threads. On IO read,
we apply pending change buffer entries. This thread (on certain conditions),
can decide to a sync read vs async read. In our case, it is an async read.
Master thread submits a request to read the page. Lets say space_id:page_no (978:34)

At around same time, an insert/delete into secondary indexes wants to buffer a
change but the ibuf has reached to its max size, so it initiates a ibuf_contract().
See ibuf_insert()-> calling ibuf_contract()->ibuf_merge_pages. This opens a cursor
at a random record and it can happen that it sees the same space_id:page_no as the
IO thread is processing.

And just when this tablespace records are about to be processed, the tablespace
is dropped. So the ibuf_merge_pages() decides it is no longer necessary to do the
actual read of secondary index pages. Hey, the tablespace is gone! whats the point
in bringing those pages to buffer pool. Hence it decides to delete all ibuf records
belonging to space_id (978) in our example.

This leads to the case where two threads can simultaneously process the ibuf
records of the same page. IO thread  reading a secondary index page and contract
ibuf belonging to this space_id,page_no (this read is initiated by innodb master ibuf merge task)
user threads doing ibuf_contract, they process entries belonging to a different tablespace.
And when they see that tabelspace is dropped), they try to delete ibuf entries.

Fix:
----
ibuf restore pos is designed to handle the “dropped tablespace” already, but in 8.0,
the way tablespaces are dropped is a bit different.

fil_space_get_flags(space) === ULINT32_UNDEFINED happens when space object is freed.
And on nullptr, ULINT32_UNDEFINED is returned by fil_space_get_flags()

Technically the problem can happen in 5.7 too, but it is more obvious in 8.0 because
the window between the actual deletion of the tablespace object and the user drop
tablespace command is bigger. In 8.0, space object is still available, so flags is
NOT ULINT32_UNDEFINED. At the time of crash, only stop_new_ops is set to true. If this
fil_space_t->stop_new_ops is true, fil_space_acquire() returns nullptr.
We will use this call to determine if tablespace is dropped or not.

ibuf code saw the tablespace as deleted by using fil_space_acquire() in
ibuf_merge_or_delete_for_page().

We will use the same call in ibuf_restore_pos() to determine if the tablespace is being
deleted.

storage/innobase/ibuf/ibuf0ibuf.cc

@@ -38,6 +38,7 @@ this program; if not, write to the Free Software Foundation, Inc.,
 #include "sync0sync.h"
 
 #include "my_dbug.h"
+#include "scope_guard.h"
 
 #if defined UNIV_DEBUG || defined UNIV_IBUF_DEBUG
 bool srv_ibuf_disable_background_merge;
@@ -3812,7 +3813,14 @@ static bool ibuf_restore_pos(
     return true;
   }
 
-  if (fil_space_get_flags(space) == UINT32_UNDEFINED) {
+  /* Check if the tablespace is dropped. */
+  fil_space_t *sp = fil_space_acquire_silent(space);
+
+  auto guard = create_scope_guard([&]() {
+    if (sp != nullptr) fil_space_release(sp);
+  });
+
+  if (sp == nullptr || sp->flags == UINT32_UNDEFINED) {
     /* The tablespace has been dropped.  It is possible
     that another thread has deleted the insert buffer
     entry.  Do not complain. */