tests/robustness: enhance compact failpoint

[fuweid]

If the cluster serves requests slowly, the database has few revision number and then Compact won't trigger BatchCommit. Add a loop to check the last revision is big enough to trigger panic.

REF: #14691 (comment)

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

[serathius]

Instead of using this decorator and doing name matching, can you add a flag?

var (
	CompactBeforeCommitBatchPanic            Failpoint = goPanicFailpoint{"compactBeforeCommitBatch", triggerCompact{waitForRevision: true}, AnyMember}
	CompactAfterCommitBatchPanic             Failpoint = goPanicFailpoint{"compactAfterCommitBatch", triggerCompact{waitForRevision: true}, AnyMember}
)

type triggerCompact struct {
	waitForRevision bool
}

	
func (t triggerCompact) Trigger(_ *testing.T, ctx context.Context, member e2e.EtcdProcess, _ *e2e.EtcdProcessCluster) error {
	cc, err := clientv3.New(clientv3.Config{
		Endpoints:            member.EndpointsGRPC(),
		Logger:               zap.NewNop(),
		DialKeepAliveTime:    10 * time.Second,
		DialKeepAliveTimeout: 100 * time.Millisecond,
	})
	if err != nil {
		return fmt.Errorf("failed creating client: %w", err)
	}
	defer cc.Close()
	if t.waitForRevision {
		for {
			resp, err := cc.Get(ctx, "/")
			if err != nil {
				return err
			}
			if resp.Header.Revision > member.Cfg.CompactionBatchLimit {
				break
			}
			time.Sleep(50 * time.Millisecond)
		}	
	}

	_, err = cc.Compact(ctx, rev)
	if err != nil && !strings.Contains(err.Error(), "error reading from server: EOF") {
		return err
	}
	return nil
}

[serathius]

Idea is good, just need to improve implementation.

[serathius]

waitForRevision was a working name. Something like multiBatchCompaction would be better as we are basically waiting for revision to be big enough so compaction needs to be done in multiple batches.

[fuweid]

multiBatchCompaction looks better and more accurate. let me update it.