Skip to content

Commit

Permalink
roachprod: use exit-on-error:false for crdb log cfg
Browse files Browse the repository at this point in the history
See cockroachdb#62763.

We seem to frequently miss the runtime errors resulting from
out-of-memory conditions in the stderr logs. We don't understand
exactly why yet, but it is very likely that with `exit-on-error`
(which is true by default) we are hitting errors outputting to
the sink which then kill the process before the runtime errors
bubble up.

While we develop a proper fix, avoid the problematic configuration
on roachprod clusters, which notably includes roachtests.

Release note: None
  • Loading branch information
tbg committed Apr 13, 2021
1 parent 92a8ad8 commit 1105337
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion pkg/cmd/roachprod/install/cockroach.go
Original file line number Diff line number Diff line change
Expand Up @@ -457,7 +457,8 @@ func (h *crdbInstallHelper) generateStartArgs(
}

logDir := h.c.Impl.LogDir(h.c, nodes[nodeIdx])
args = append(args, "--log-dir="+logDir)
// Specify exit-on-error=false to work around #62763.
args = append(args, `--log "file-defaults: {dir: '`+logDir+`', exit-on-error: false}"`)

if vers.AtLeast(version.MustParse("v1.1.0")) {
cache := 25
Expand Down

0 comments on commit 1105337

Please sign in to comment.