storage: local raft log truncation

[tbg]

We can now truncate the Raft logs locally without going through quorum (since we unreplicated the TruncatedState). We can use this to make performance-optimized truncation decisions. For example, on followers we may chose to not keep any Raft log, while on leaders we may want to simplify the heuristics used by the Raft log queue today and make them local decisions, too.

Jira issue: CRDB-4513

Epic CRDB-39898

[nvanbenschoten]

cc. @irfansharif

[nvanbenschoten]

@tbg I think this would be a good project for @sumeerbhola to start on, with the eventual goal for him to help move us towards #38322, which requires local storage engine compaction state to feed back in to local log truncation decisions.

I know we've talked about this issue in person, but do you mind spelling out the goals and work items of this issue? Is the eventual goal to remove the raftLogQueue? Or do we just want to get rid of TruncateLogRequest?

@irfansharif should also be kept in the loop here, as this has a small amount of overlap with his current work on #7807.

[tbg]

Good idea. Here are my high level thoughts:

• (experimentally) figure out a reasonable local truncation criterion for followers. Is it best to truncate when the log reaches a fixed length or does it need to be smarter than that? I like length as opposed over size because we've found that correctly tracking the size is a bad idea, and we originally tracked size mostly because we had stability problems related to large raft logs, which are no longer an issue (though Raft will always use the log to catch up followers, which can be much slower than sending a snapshot after a certain point).
• put raft log queue in charge of local truncation on raft leader only (i.e. it stops proposing to raft), perhaps actually remove the queue but keep a little bit of per-replica state and trigger investigation of a truncateDecision when it's useful (strawman idea: start with follower heuristics, but every time a truncation is not carried out because of leader constraints, relax the heuristics until it hits some ceiling, addressing storage: raftLogQueue locking causes contention on large machines #36251). In case it's not clear, the raft leader can't freely truncate or it will cause tons of snapshots, so the heuristics we have to avoid that pretty much have to remain active on it.
• make sure log is truncated on quiesce (addressing storage: significant space remains in RaftLog after dropping table #26339) on all replicas

The nice thing about the strawman is that it would unify the code on follower vs leader. We'd have something like this in a single place somewhere in the raft handling loop:

	aboveThresh := r.mu.threshLen > logLen
	newFirstIdx := r.mu.lastIndex
	if aboveThresh {
		if isRaftLeader {
			decision := newTruncateDecision(r.raftStatusLocked())
			if decision.ShouldTruncate() {
				if decision.NumTruncated() > (2*r.mu.threshLen)/3 {
					r.mu.threshLen = max(r.mu.threshLen, defaultThreshLen)
				} else {
					r.mu.threshLen = min(2*r.mu.threshLen, maxThreshLen)
				}
				newFirstIdx := decision.NewFirstIndex()
			} else {
				shouldTruncate = false
			}
		}
	}

(This is just a starting point, it might look pretty different in practice).

[sumeerbhola]

@tbg basic question "put raft log queue in charge of local truncation on raft leader only (i.e. it stops proposing to raft)" -- who will truncate on followers?
And can you point me to the source files I should read to understand how all this works now?

[tbg]

The followers need to learn how to truncate locally according to a hopefully much simpler heuristic (i.e. when the log reaches X entries, blindly truncate).

I'm about to run out for my meetings, but here are a few pointers to get you started:

raft_log_queue.go
cmd_truncate_log (evaluates the truncation above raft)
handleTruncatedStateBelowRaft and callers (carry out the truncation below raft)
all usages of RaftTruncatedState, and also the TruncatedStateType (the legacy type is the one we can't truncate locally with, so this has to continue going through raft unfortunately, and will make it unlikely that you'll get to remove the queue, we need long-running migrations to really migrate over)

[tbg]

Just to update this, we have migrated out of the replicated truncated state, so in principle we can do local raft log truncations now.

[sumeerbhola]

We now have loosely-coupled raft log truncation, where the raft leader continues to propose the truncation, and the followers queue up these truncations as pending truncations, and apply them when the guaranteed-durable state in their state machine is past the truncation point. The kv.raft_log.loosely_coupled_truncation.enabled cluster setting defaults to true.
Remaining work:

• Remove the cluster setting for v22.2, i.e., no strongly coupled raft log truncation.
• Do local raft log truncation for quiesced replicas. This was originally included in kvserver: loosely couple raft log truncation #76215, but was left out of the final version. This truncation could drop all raft log entries.
• Consider doing local raft log truncation in the general case. This is tricky since we do want followers to be able to server snapshots to other replicas.

[sumeerbhola]

loosely-coupled raft log truncation was disabled due to the performance regression explained in #78412 (comment)

[tbg]

Next steps: #78412 (comment)

Basically this issue is blocked on having two actual separate engines. As is, with logs and state sharing an engine, when the state flushes we also flushed the log, so the truncation will come too late.

So in effect we do this issue when we actually do two engines (we can't keep using strongly coupled truncations with two engines)