fix(server): transcodes failing due to storage migration happening simultaneously #3071
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
| @@ -163,9 +164,15 @@ export class JobService { | |||
| await this.jobRepository.queue({ name: JobName.CLASSIFY_IMAGE, data: item.data }); | |||
| await this.jobRepository.queue({ name: JobName.ENCODE_CLIP, data: item.data }); | |||
| await this.jobRepository.queue({ name: JobName.RECOGNIZE_FACES, data: item.data }); | |||
| if (item.data.source !== 'upload') { | |||
There was a problem hiding this comment.
What is the reason for this check here?
There was a problem hiding this comment.
Videos are currently only automatically queued for transcoding on upload, not when any other job is run. This check is to maintain that behavior. The asset check in this section also seems specific to uploading.
|
|
||
| const [asset] = await this.assetRepository.getByIds([item.data.id]); | ||
| if (asset) { | ||
| if (asset.type === AssetType.VIDEO) { |
There was a problem hiding this comment.
Can you give a more explanation on why moving the job here would help with race conditions?
There was a problem hiding this comment.
Currently metadata extraction and transcoding are queued at the same time. Since metadata extraction is followed by migration, this can cause transcoding to fail. But since thumbnail generation is queued only after migration, there's no longer any risk of a race condition.
|
I tried that, but it inflated the number of queued assets for transcoding in the dashboard. You could have 1000 assets waiting one second and 0 the next if the queue only had images. |
Description
Schedules the video conversion job to be enqueued only after storage migration to avoid a
File not founderror while transcoding. Specifically, it's queued after JPEG thumbnail generation is complete. The reason why this particular job is chosen as a prerequisite is mainly because it already queries for the asset itself on completion, which means the result of this query can be used to check if the asset is a video and only queue for transcoding if so.An earlier iteration queued all uploaded assets after the storage migration job, but this was misleading from a UX perspective as the number of queued assets for transcoding could be much higher than the number actually transcoded.
Fixes #3047
How Has This Been Tested?
I made a new account and uploaded about a thousand assets. Where before there would sometimes be a "file not found" error in the past, I never got this error with this PR; all videos were encoded successfully.