[3.4] backports #11990 #12933

[fuweid]

It is trying to backport two improvements into release 3.4.

• Reduce memory when Range for total or with limit count
  • mvcc: reduce count-only range overhead #11771
  • mvcc: push down RangeOptions.limit argv into index tree to reduce memory overhead #11990
  • etcdserver: Fix invalid count returned on Range with Limit #13060
• Reduce expensive deepCopy when create ConcurrentReadTx
  • server: set multiple concurrentReadTx instances share one txReadBuffer #12933
  • Fixes looks like the etcd 3.4 performance drops #11783: When there is only read requests, the throughput will be limited by go-runtime gc.

Click to check rw-heatmap benchmark result

Use https://github.com/fuweid/etcd/tree/tools-use-imshow-instead-of-tripcolor to render heatmap

backports-based-on-v3.4.23.csv
v3.4.23.csv

Click to check range not-found key benchmark result

Based on steps provided by #11783, the script is like

   cd /data
    export CLIENT_PORT="23790"
    export BACKEND_SIZE="$((20 * 1024 * 1024 * 1024))"
    /root/etcd/bin/etcd --quota-backend-bytes=${BACKEND_SIZE} \
      --log-level 'error' \
      --listen-client-urls http://0.0.0.0:${CLIENT_PORT} \
      --advertise-client-urls http://127.0.0.1:${CLIENT_PORT} \
      &>/dev/null &
    sleep 5

    /root/etcd/bin/tools/benchmark --endpoints http://127.0.0.1:${CLIENT_PORT} \
       --clients 1000 --conns 100 \
       put --key-size 8 --val-size 256 --total 100000

    sleep 2

    /root/etcd/bin/tools/benchmark --endpoints http://127.0.0.1:${CLIENT_PORT} \
       --clients 1000 --conns 100 \
       range 12345678123456781234567812345678 --consistency=l --total 100000

• v3.4.23: Requests/sec: 26169.3610
• Backport: Requests/sec: 127041.7200

Hope it is acceptable!

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[fuweid]

ping @ahrtr @ptabor @serathius ~

The CI fails because of exiting flaky cases. When the lease is not found, the grpcproxy closes connection instead of returning error. So the case got curl: (52) Empty reply from server

[ahrtr]

Usually we only backport bug fixes, but this PR backports features. Especially release-3.4 has more than 3 years of history, stability is even more important.

Note: backporting multiple features in one PR is not good to me. I prefer to backport only one change in one PR.

[fuweid]

@ahrtr Thanks for the comment.

I will split it into several prs and seek the comments from yours~.
Before that, I am working on fixing the existing flaky cases in v3.4 release.