CBG-3957 add reason in all_dbs for why db is offline

[torcolvin]

• fix issue where DatabaseContext.RequiresResync was only updated when
  creating a Database. Update this value after resync completes

I think I'd rather put requires_resync field to match GET /db/ output, but I thought there was a reason that we didn't do that.

Made the fields public only to assert on them in rest.adminapitest package.

Pre-review checklist

• Removed debug logging (fmt.Print, log.Print, ...)
• Logging sensitive data? Make sure it's tagged (e.g. base.UD(docID), base.MD(dbName))
• Updated relevant information in the API specifications (such as endpoint descriptions, schemas, ...) in docs/api

Integration Tests

• GSI=true,xattrs=true https://jenkins.sgwdev.com/job/SyncGateway-Integration/2467/

[github-actions]

Redocly previews

• Admin API
• Public API
• Metric API
• Diagnostic API

[adamcfraser]

We should only do this work if we successfully updated SyncInfo for any collections (on line 223) - otherwise can skip this (which would include any resync done with regenerate_sequences=false)

[adamcfraser]

Actually, can we just remove any collections that we successfully updated on line 223 from db.RequireResync, instead of recomputing the set?

[torcolvin]

This is much easier to reason through, addressed.

[adamcfraser]

I feel like we shouldn't be using InitSyncInfo here - I don't think there's any situation we'd want to be creating SyncInfo if for whatever reason it doesn't exist. But see comment above about maybe just updating db.RequireResync based on line 223.

[torcolvin]

Addressed with a refactor, I didn't like this either.

[adamcfraser]

We're only in one state at a time, so I think a simple 'reason' property is preferable. (e.g. in future if we want to provide a reason for state=Resyncing, we can reuse). What do you think?

[torcolvin]

I think I'd prefer to do create a parameter that is RequireResync to match the /db/ endpoint, and there isn't ambiguity.

I think the status field offers a lot of flexibility, but it comes at some downsides, which is that it is not always populated. The reason that reason it might not be populated is opaque, so if the user calls /db/_offline or sets start_offline: true should the reason be offline in that case?

Capella doesn't use the /db/ require_resync field, and implementing this take the place of that field.

[adamcfraser]

Since require_resync can potentially be very large, I don't think it's necessarily appropriate for a list REST endpoint. I think it makes sense to keep the _all_dbs response relatively lightweight (to support frequent polling), and keep require_resync available on the /db/ endpoint. That gives consumers the option to fetch the data based on the reason in the list view, but not be committed to it on every polling interval.