ingest/ledgerbackend: Restart Stellar-Core when it's context is cancelled

[bartekn]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot, or all or doc if the changes are broad or impact many
  packages.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

After refactoring in #4088 (and as a result of my wrong comment: #4088 (comment)) the CaptiveCoreBackend.isPrepared method returned true if stellarCoreRunner process was shutdown without calling close() - so in case of binary update but also in case of Stellar-Core crash.

This commit fixes this bug by checking if stellarCoreRunner context was cancelled (meaning Stellar-Core is closed or closing but not as a result of close call). I also removed isClose method because it was simply checking close variable. All test changes are only adding an extra context() call mocks because isPrepared now calls it.

[Shaptic]

Could we simplify this to include the above condition, instead? In other words,

func (c *CaptiveStellarCore) isClosed() bool {
	return c.closed || c.stellarCoreRunner == nil || c.stellarCoreRunner.context().Err() != nil
}

Basically, is there a situation in which the instance would not be closed and the runner wouldn't exist? Intuitively seems like no, but I'm not strong on this part of the codebase.

[bartekn]

Adding extra conditions to isClosed won't work. In #4088 we wanted to separate two cases:

• CaptiveCoreBackend closed - means no usable, you need to create a new instance.
• stellarCoreRunner closed - means it just needs to be restarted by calling PrepareRange.

[Shaptic]

Got it, thanks for clarifying! LGTM 👍