RangeError: Maximum call stack size exceeded at ThumbnailGenerationJob

[mrtnbr]

Describe the bug
Indexing jobs runs and fills database (tried both mariadb and sqlite). Then the next job Thumbnail generation hangs at 'Loading files from db'. See below for log with error.

If I start browsing the gallery, thumbnails are generated.

Photo/video (optional) that causes the bug

n/a

Screenshots (optional)

n/a

Server logs (optional)

[Typeconfig] Processing cli and ENV inputs: {
"PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HOSTNAME": "pigallery2",
"TZ": "Europe/Amsterdam",
"NODE_ENV": "debug",
"NODE_VERSION": "14.17.0",
"YARN_VERSION": "1.22.5",
"PI_DOCKER": true,
"HOME": "/root",
"VIPSHOME": "/target"
}

7/28/2021, 12:43:46 PM[INFO_][JOB] Running job : Thumbnail Generation
7/28/2021, 12:43:46 PM[SILLY][FileJob] Loading files from db
7/28/2021, 12:43:46 PM[DEBUG] POST /api/admin/jobs/scheduled/Thumbnail%20Generation/start 200 18ms
7/28/2021, 12:43:46 PM[VERBS] GET /api/admin/jobs/scheduled/progress 200 2ms
7/28/2021, 12:43:51 PM[ERROR][JOB] RangeError: Maximum call stack size exceeded
at ThumbnailGenerationJob. (/app/src/backend/model/jobs/jobs/FileJob.js:138:28)
at Generator.next ()
at fulfilled (/app/src/backend/model/jobs/jobs/FileJob.js:5:58)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
7/28/2021, 12:43:55 PM[VERBS] GET /api/admin/jobs/scheduled/progress 200 3ms

Environment (please complete the following information):

• OS: docker on debian
• Browser Firefox

Used app version:

• docker bpatrik/pigallery2:latest-debian-buster

[desertwitch]

I have been able to replicate the problem and find the bug in src/backend/model/jobs/jobs/FileJob.ts, which stems from pushing the complete array of images (retrieved from the database) into the fileQueue at once, rather than pushing element by element into the fileQueue.

In databases with more than 125.052 files (such as above), this results in a too-large array being pushed into the fileQueue at once and a RangeError occurring. This can be avoided by pushing element by element into the fileQueue, for example by using a for-loop to iterate over the elements & push them individually.

See here for a more detailed description of this particular problem: https://anchortagdev.com/range-error-maximum-call-stack-size-exceeded-error-using-spread-operator-in-node-js-javascript/

_loadAllMediaFilesFromDB() - Line 139 - CURRENT, broken > 125.052

this.fileQueue.push(...(result .map(f => path.join(ProjectPath.ImageFolder, f.directory.path, f.directory.name, f.name))));

_loadAllMediaFilesFromDB() - Line 139 - PROPOSED, working > 125.052

for (const item of result) { this.fileQueue.push(path.join(ProjectPath.ImageFolder, item.directory.path, item.directory.name, item.name)); }

Weirdly, I've noticed that the issue-causing push-map approach is being used in multiple locations throughout the file, while - at the same time - the proposed working approach is also used in another location of the file. Look, in src/backend/model/jobs/jobs/FileJob.ts the function loadADirectoryFromDisk() correctly implements such a for-loop on Line 101 and then proceeds to use the issue-causing push-map approach on Line 105 and Line 109.

I propose a fix to this particular issue with larger datasets by always going the element-by-element approach with a for-loop in such cases (where lots or all of the data is selected for a task), rather than the single-push-all approach currently implemented with the push-map code on Lines 105, 109 and 139.

Thank you for your hard work on this amazing project!
Veronika