[Transform] Finetune Schedule to be less noisy on retry and retry slower

[hendrikmuhs]

reduce amount of log and audits if the same failure happens in a row
and change the minimum wait time for retrying to 5s

This finetunes changes done in #84657 which makes retrying independent of frequency, which as a result triggers retry faster than before with the side-effect of producing more noise. For this change I investigated the log/audit behavior and adjusted it.

Notes:

• the change contains mostly test changes, the important code changes are:
  • set MIN_DELAY_MILLIS to 5s (from 1s)
  • dedup logs/audits based on exception class
    • the 1st (retry-able) failure gets logged/audited, consequent failures only if they differ from the last one. The last retry is logged as well
    • exceptions are usually wrapped in a SearchPhaseExecutionException, so it is important to unwrap this as a 1st step. Just taking the message from the unwrapped class would still produce a lot of noise, because the messages can slightly differ although they have the same root cause. Using the class name of the unwrapped exception seems like a pragmatic way to dedup them.
    • I moved the last error into the context, note: there is no need to persist this as we do not stop the transform after such a failure, but keep the error in-memory only
• Testing:
  • because testing this change as a system test isn't possible with the current infrastructure, I created mock module tests, see TransformIndexerFailureHandlingTests
• Log/Audit message:
  • "Transform encountered an exception: [{Exception}]; Will automatically retry [1/10]"

relates #84657

(marked as non-issue as this is an addition to #84657)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[droberts195]

If would be good to distinguish the case where we don't see the message at all to the case where we see it too many times. Otherwise, if we ever get this test failure we won't know which scenario was to blame.

In fact, just reading further the error message in MultipleSeenAuditExpectation is ideal, so maybe just delete this class and always use MultipleSeenAuditExpectation with expected count 1.

[hendrikmuhs]

I realize that I changed the way it worked, before it just flipped a bool, so it was "seen at least once", with the count I changed it to "exactly once". However the tests pass and "exactly once" is probably what we want anyway.

I will merge the 2 classes and keep the constructor that defaults to "seen once" as convenience. That way it does not trigger further refactorings.

[droberts195]

LGTM