1105 (HY000): cannot set read timestamp to a future time

[emchristiansen]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

I'm running v7.1.1 on K8s using the provided operator.

I configure read replicas like this to enable local reads:

mysql \
    --host basic-tidb.tidb-cluster.svc.cluster.local \
    --port 4000 \
    --user root \
    --password=<pw> \
    --execute "set global tidb_replica_read = 'closest-replicas';"

I execute select statements which end with the following string to enable stale reads:

AS OF TIMESTAMP TIDB_BOUNDED_STALENESS(NOW() - INTERVAL 60 SECOND, NOW())

3. What did you see instead (Required)

My reads sometimes fail with the error 1105 (HY000): cannot set read timestamp to a future time.
Also, this typically happens on just one of my three physical nodes - perhaps that node has a clock skew issue which is surfacing this bug.

Note, this only started to happen when I upgraded to v7.1.1 from v6.5.2.
I don't believe my code changed in any important way in the interim.

[zyguan]

May be related to #44215 , but I'm not sure why the issue didn't happend in v6.5.2, the fix hasn't been backported to both v7.1.1 (#45760 not merged) and v6.5.2 (#44281 merged but for >=v6.5.3).

[emchristiansen]

If clock skew indeed surfaced this bug, perhaps we're only seeing this now due to clock skew's cumulative nature?

But FWIW the affected node is running an NTP synchronization daemon (whatever the default is for Ubuntu 22.04), so that's evidence against the clock skew explanation.

Another bit of evidence: I adjusted the timestamp bound upper limit from NOW() to NOW() - 1 SECOND and that appears to have fixed things.