Graceful stop is ignored when running SQL tests via xk6-sql plugin

[alex-dubrouski]

Brief summary

We are using K6 with xk6-sql plugin to load test our local TiDB cluster. On top of regular load tests we run basic chaos engineering like cluster failover and node restart during the test. We noticed that TiDB cluster disruptions cause K6 process to hang for a long time.

k6 version

v0.53.0

OS

Linux x86-64

Docker version and image (if applicable)

No response

Steps to reproduce the problem

When running simple select test and manually restarting one of the cluster nodes:

./k6 run -e executor="ramping-arrival-rate" -e preAllocatedVUs=10000 -e gracefulStop=30s -e stages="10s:5,30s:10000,4m:10000,10s:5" -e K6_DNS="ttl=0,select=roundRobin,policy=preferIPv6" simple-range.js

this test is expected to finish in 4mins 50sec, but it actually takes:

running (19m19.8s), 00000/10000 VUs, 2597641 complete and 2502 interrupted iterations
open_model ✓ [======================================] 02502/10000 VUs  4m50s  00114.01 iters/s

Please note running (19m19.8s), traffic correctly drops to 0 after 4mins 50sec, but process keeps sitting idle for over 15 more minutes
I built K6 locally from the trunk with debug symbols and profiled the app when it hangs with perf and bcc-tools:
Offcpu (idle descheduled threads)

Offwake (re-scheduled threads and threads that woke them up combined in one stack)

and perf:

# ........  .......  .................  ......................................
#
    13.89%  k6       k6                 [.] runtime.checkTimers
    13.69%  k6       k6                 [.] runtime.findRunnable
    13.65%  k6       [kernel.kallsyms]  [k] __perf_event_task_sched_in
    13.52%  k6       k6                 [.] runtime.checkdead
    10.98%  k6       k6                 [.] runtime/internal/syscall.EpollWait
    10.66%  k6       [kernel.kallsyms]  [k] __schedule
    10.60%  k6       [kernel.kallsyms]  [k] touch_atime
     3.35%  k6       k6                 [.] runtime.runtimer
     2.78%  k6       [kernel.kallsyms]  [k] fput_many
     1.82%  k6       k6                 [.] runtime.slicebytetostring
     1.50%  k6       [kernel.kallsyms]  [k] _raw_read_lock_irqsave
     1.23%  k6       k6                 [.] runtime.retake
     1.19%  k6       [kernel.kallsyms]  [k] perf_event_groups_first
     0.34%  k6       [kernel.kallsyms]  [k] syscall_return_via_sysret
     0.23%  k6       k6                 [.] runtime.runOneTimer
     0.22%  k6       k6                 [.] runtime.stealWork
     0.18%  k6       [kernel.kallsyms]  [k] htab_map_update_elem
     0.15%  k6       [kernel.kallsyms]  [k] finish_task_switch.isra.0
     0.03%  k6       [kernel.kallsyms]  [k] native_write_msr
     0.00%  k6       [kernel.kallsyms]  [k] native_apic_msr_write

Test script is very simple it just randomly selects data range from a set of configured tables:

import sql from 'k6/x/sql';
import { check } from 'k6';
import * as db_config from './db-config.js';
import * as common from './common.js';

export const options = common.options;

export default function() {
    const randomId = common.getRandomId();
    const randomTable = common.getRandomTable();

    const results = sql.query(db_config.db, 'SELECT c FROM ' + randomTable + ' WHERE id BETWEEN ? AND ?',
        randomId, randomId + common.rangeSize);

    check(results, {
        'is length >= 1': (r) => r.length >= 1,
    });
}

export function teardown() {
    db_config.db.close();
}

Expected behaviour

K6 process supposed to exit after configured graceful stop

Actual behaviour

K6 process keeps sitting idle for a prolonged amount of time