Add WAL directory tailer #1
Conversation
tail/tail.go
Outdated
| type tailer struct { | ||
| ctx context.Context | ||
| dir string | ||
| next int |
There was a problem hiding this comment.
can you call this nextNoun, for documentation?
There was a problem hiding this comment.
I meant as a placeholder. I'm not sure what noun you're tracking there.
There was a problem hiding this comment.
Ah, yes – changed it to nextSegment.
| // Tail the prommetheus/tsdb write ahead log in the given directory. Checkpoints | ||
| // are read before reading any WAL segments. | ||
| // Tailing may fail if we are racing with the DB itself in deleting obsolete checkpoints | ||
| // and segments. The caller should implement relevant logic to retry in those cases. |
There was a problem hiding this comment.
Is the relevant logic to call this method again? Is that the only failure mode, and if not, should we add a retriable error?
There was a problem hiding this comment.
We could add a special error. Many things could fail of course – anything along file opens/reads/etc. It's hard (and probably sometimes impossible) to tell which ones a permanent failures.
I'd just go with the caller retrying with a backoff strategy. Since none of this writes data, we shouldn't be messing anything up.
| @@ -0,0 +1,112 @@ | |||
| package tail | |||
tail/tail.go
Outdated
| if err == tsdb.ErrNotFound { | ||
| // Next segment doesn't exist yet. We'll probably just have to | ||
| // wait for more data to be written. | ||
| time.Sleep(time.Second) |
There was a problem hiding this comment.
Did you intend to pass backoff instead of time.Second?
Signed-off-by: Fabian Reinartz <freinartz@google.com>
@jkohen this should do the job but let me know whether we need some additional insight like current
(segment, offset)or similar.