Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix crashes in thumbnails generation #342

Merged
merged 1 commit into from
Oct 25, 2021
Merged

Fix crashes in thumbnails generation #342

merged 1 commit into from
Oct 25, 2021

Conversation

uraid
Copy link
Contributor

@uraid uraid commented Sep 7, 2021

While generating thumbnails for a big old photos library (~12k photos), encountered two distinct crashes in the thumbnail processor:

2021-09-03 13:26:20,750 INFO Finished thumbnail processing batch
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/srv/photonix/photos/management/commands/thumbnail_processor.py", line 24, in worker
generate_thumbnails_for_photo(task.subject_id, task)
File "/srv/photonix/photos/utils/thumbnails.py", line 37, in generate_thumbnails_for_photo
get_thumbnail(photo=photo, width=thumbnail[0], height=thumbnail[1], crop=thumbnail[2], quality=thumbnail[3], force_regenerate=True, force_accurate=thumbnail[5])
File "/srv/photonix/photos/utils/thumbnails.py", line 100, in get_thumbnail
im = srgbResize(im, (width, height), crop, Image.BICUBIC)
File "/srv/photonix/photos/utils/thumbnails.py", line 196, in srgbResize
arr = np.array(im, dtype=np.float32) / 255.0
TypeError: float() argument must be a string or a number, not 'JpegImageFile'

And also:

Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/srv/photonix/photos/management/commands/thumbnail_processor.py", line 24, in worker
generate_thumbnails_for_photo(task.subject_id, task)
File "/srv/photonix/photos/utils/thumbnails.py", line 37, in generate_thumbnails_for_photo
get_thumbnail(photo=photo, width=thumbnail[0], height=thumbnail[1], crop=thumbnail[2], quality=thumbnail[3], force_regenerate=True, force_accurate=thumbnail[5])
File "/srv/photonix/photos/utils/thumbnails.py", line 94, in get_thumbnail
im = im.rotate(-90, expand=True)
File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2024, in rotate
return self.transpose(ROTATE_270)
File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2524, in transpose
self.load()
File "/usr/local/lib/python3.8/site-packages/PIL/ImageFile.py", line 274, in load
raise_oserror(err_code)
File "/usr/local/lib/python3.8/site-packages/PIL/ImageFile.py", line 67, in raise_oserror
raise OSError(message + " when reading image file")
OSError: broken data stream when reading image file

Ran the thumbnail_processor a few times until I was left with 2 problematic images that generated those errors.
Each time a rescan of the images happened, another 2 tasks for those images were duplicated, I also had to manually run the thumbnail_processor, it seems to not restart like the object and face detection.

After seeing the 2 images files, it seems they are corrupted from a recovery attempt I've made some years ago (lost some data).
After searching for a solution and contemplating how to fix this, found this thread: python-pillow/Pillow#3185
This flag will let Pillow read files even if they're corrupted/truncated (no EOI marker 0xffd9), so photonix can continue processing gracefully.

One can argue that maybe it's best to catch those exceptions and make:

  1. Special handling in DB for those, to mark as 'completed with error' (e.g. 'ce').
  2. A new error inside the UI for user notification.
    But different formats can have different exceptions for those corrupted images, so requires more testing and fiddling before implementing something like this.

@nscendoni
Copy link

I had the same issue and the patch worked for me.
Thanks

@damianmoore damianmoore merged commit f18ccdd into photonixapp:master Oct 25, 2021
@damianmoore
Copy link
Collaborator

Thanks for this @uraid and apologies for the delay in reviewing it. It's merged now and will be in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants