Add a construct in between a Watch and an Informer

[shawkins]

The fabric8 built-in watch support is similar to a retrywatcher in the go client - until http gone it will maintain the watch automatically. Because the watch is restarted at an exact resourceVersion, there are no duplicate add events.

The next option is to use an informer, but in some circumstances there may be a concern about the size of an informer cache. Either we need to offer additional cache customization features - similar to #3472 - or introduce a further enhanced watch construct that goes beyond the retry to handle http gone. Under the covers this would look like an informer that caches only the resource key and version, but the user would still use the Watcher interface.

[shawkins]

The only issue is that there is no way currently to retrieve just a list of the identifiers and resource versions to process the removals after a "relist". You would still end up pulling the entire list into memory, albeit for a small amount of time.

About the only thing I can think of is to implement the relist using limit/continue to keep the memory footprint controllable.

[shawkins]

Ostensibly this work is being done for operator scaling. The work in #3587 adds the underlying mechanics for batch fetching. There is also some discussion on alternative approaches, such as sharding - #3753 (comment) - that don't involve further changes to the informer framework.

I should add that while possible, a sharding approach seems fairly invasive / difficult - you'd need a good hashing function wrt the number of statefulset replicas, transitively set the necessary index label to dependent resources, and create/deletes would need special handling. Presuming that you are using sharding to avoid a single large process you could not simply use an informer in the coordinating operator to keep the index label up-to-date - you'd have to use a periodic paginated reconciliation (possibly with a watch). If you ever needed to modify the replica count (or need to be defensive to any modification), the index can be changed - which would be seen as a delete event by informers looking for that label value. So there would need to be an additional lookup or other mechanism to establish the difference between a changed index, and a full deletion.