Bug Report: vreplication error can fail to persist, with insufficient logging

[shlomi-noach]

Overview of the Issue

It's possible for vreplication's controller to hit an unrecoverable error, e.g. attempting to write an invalid data type / value to the target table. So far so good. The controller logs the error and also attempts to persist it in _vt.vreplication.message:

vitess/go/vt/vttablet/tabletmanager/vreplication/controller.go

Lines 271 to 277 in 0babfe2

	if isUnrecoverableError(err) || !ct.lastWorkflowError.ShouldRetry() {
	log.Errorf("vreplication stream %d going into error state due to %+v", ct.id, err)
	if errSetState := vr.setState(binlogplayer.BlpError, err.Error()); errSetState != nil {
	return err // yes, err and not errSetState.
	}
	return nil // this will cause vreplicate to quit the workflow
	}

The persisting action itself can fail, which is still accounted for in the above. However, there is no clear indication that the action failed. The controller does return an error, but the message column is still populated with whatever previous value it contained (e.g. Picked source tablet...).