[Feat]: for better UX, stream blobs as you fetch when on-demand sync is enabled

[rchincha]

Is your feature request related to a problem? Please describe.

Currently, we fetch blobs and store locally and then serve. It could cause some clients to error out. So basically, on-demand is mainly a trigger. A better UX for clients would be to stream to them as the blobs are fetched.

Describe the solution you'd like

For this feature, we may have to enhance or replace the containers/image library entirely and use our own custom library.

You may need a pattern like so (or a combo):
https://pkg.go.dev/io#MultiWriter
https://pkg.go.dev/io#TeeReader

Also, a SyncWatcher so that we pull only once and fan-out to pending clients

Describe alternatives you've considered

No response

Additional context

No response

[rchincha]

@liangyuanpeng ^
https://liangyuanpeng.com/post/new-container-image-registry-of-zot-from-cncf/

[rchincha]

A blob that is sync'ed is in three possible states

1. Found, so not need to trigger sync on client pulls
2. NotFound, so need to trigger sync and add a callback (and store a http.Response ref)
3. Partial, sync is still in progress and a client is requesting the blob so it must be added to the list maintained in 2.

Keep in mind locking and memory usage (channels, buffers, etc cost us memory)

[rchincha]

main...rchincha:zot:issue-1395
^ just some quick pseudo-code

[eusebiu-constantin-petu-dbk]

Hello,

I encountered some issues with fsnotify package and with containers/image(used by sync) package while working on this:
so we create a temp directory (uuid.New()) where containers/image will copy images before commiting them to zot's imagestore.

1st issue with fsnotify is that it doesn't support watching dirs recursively, trying to overcome this by adding subdirs to watchlist whenever a CREATE event comes in, will result in races (a CREATE event could be triggered before adding a subdir to watchlist)

2nd issue with containers/image is that they first write to a temp blob file in the form of "oci-put-blob3507476942" before renaming it to the actual blob path.

so I can not identify blobs being written by digest.

basically in sync: we write a blob in a temp file, then move it to a temp directory, then move it to imagestore, trying to keep all these actions synchronized would be really hard.

[elee1766]

see: https://pkg.go.dev/github.com/mailgun/multibuf?utm_source=godoc#WriterOnce

[rchincha]

For sync feature, add a "stream: true" to enable this ux.