-
Notifications
You must be signed in to change notification settings - Fork 687
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improvements in image upload functionality #3057
Improvements in image upload functionality #3057
Conversation
This looks good on first inspection, note you have a few linting errors (once this is fixed then the full staging tests will run):
Note you can add |
I'll correct the linting issues. May I ask what does staging-test-with-rebase do? |
The
We also had a |
@@ -32,6 +32,9 @@ <h2>{{ gettext('Logo Image') }}</h2> | |||
{{ form.logo(id="logo-upload") }} | |||
<br> | |||
</p> | |||
<h5> | |||
{{gettext('Recommended size: 500px * 450px')}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Two things here:
- We use
{{ gettext(...) }}
(with spaces) stylistically to improve readability. - Is there a reason to suggest images that are 500x450 pixels but we thumbnail to 500x500 pixels max?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- Thanks for letting me know. I'll adjust that.
Pillow
suggests giving size as a 2-tuple. Thethumbnail
method automatically adjusts the lower dimension according to aspect ratio. 500 is the upper limit.
If a user uploads square image with edge length greater than 500px, the end result will be 500px by 500px image, since aspect ratio is preserved. It will be resized to 500px by 450px only when the initial aspect ratio was 10:9.
I think the size 500px by 450px comes from the size of default SecureDrop logo. Maybe we can recommend a square image like 500px by 500px.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with either, but what we suggest and what we thumbnail to should be the same number just to avoid unexpected behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I changed the maximum possible height of thumbnail to 450px. If default logo can be changed to 500 * 500, we can start recommending 500 * 500
securedrop/tests/test_journalist.py
Outdated
form = journalist_app.forms.LogoForm( | ||
logo=(StringIO('imagedata'), 'test.png') | ||
logo=(BytesIO(base64.decodestring | ||
("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQ" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the other reviewers, this was something I gave @aydwi out of band, and that is definitely a 1x1 white pixel.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes thanks @heartsucker. I added a comment to clarify that to anyone reading the code as well. Also, I think including an image whose size is greater than 500px can also help test the resize function, although the base64 string will be very large.
I would also change this: class LogoForm(FlaskForm):
logo = FileField(validators=[
FileRequired(message=gettext('File required.')),
FileAllowed(['jpg', 'png', 'jpeg'],
message=gettext('Upload images only.'))
]) To have the error message I know there's no ticket for this, but it would really make this PR better :D That said, this isn't a blocker on merging. |
securedrop/journalist_app/admin.py
Outdated
@@ -34,6 +36,11 @@ def manage_config(): | |||
static_filepath = os.path.join(config.SECUREDROP_ROOT, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should save this to a temp file, then once we've successfully resized it move it to the old location. Right now if we throw an error thumbnailing it, we'll have a broken image.
It should go:
- Save to tmp
- Thumbnail that overwrites tmp
- Move tmp to
static/i/logo.png
usingos.rename
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Made this change
securedrop/journalist_app/admin.py
Outdated
@@ -34,6 +36,11 @@ def manage_config(): | |||
static_filepath = os.path.join(config.SECUREDROP_ROOT, | |||
"static/i/logo.png") | |||
f.save(static_filepath) | |||
|
|||
with Image.open(static_filepath) as im: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If someone gives us a PNG or JPG that Pillow doesn't understand, the use will get the generic 500 error page. We should try/catch this to and flash
something like "We couldn't process that image. Try another one."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not write any exception handling here because I think Flask handles the type of images, and only JPG/JPEG/PNG make through. But that is a good point, Pillow
may encounter something unexpected. I'll update the code.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm rather sure that the FileAllowed
validator only checks the extension, and it's possible someone has a valid PNG that Pillow doesn't understand. We should try to catch these cases.
Also, why is the lint failing again? 😟 |
It looks like we're having
@kushaldas Some help here? |
@heartsucker If I change the message to 'You can only upload files ending in .png, .jpg, or .jpeg', should I also make changes to |
The lint error is not the last line of the output because we intentionally run all lint checks even if one of them fail. The error at https://circleci.com/gh/freedomofpress/securedrop/8936?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link is: flake8 --exclude='config.py,.venv/' ./securedrop/tests/test_journalist.py:754:17: E128 continuation line under-indented for visual indent ./securedrop/tests/test_journalist.py:755:17: E128 continuation line under-indented for visual indent |
@aydwi No, you don't need to do that. That will be taken care of by a different process. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few more notes.
Also, something we could consider to ease the logic of the logo change is to add an endpoint called sd_logo
that does something lke
if path.exists(path.join(current_app.static_folder, 'i', 'custom_logo.png')):
return redirect(url_for('static', filename='i/custom_logo.png'))
else:
return redirect(url_for('static', filename='i/logo.png'))
This will give us a safe fallback in most error cases.
securedrop/journalist_app/admin.py
Outdated
imcopy = im.copy() | ||
imcopy.thumbnail((500, 450), resample=3) | ||
imcopy.save(temp_static_filepath, "PNG") | ||
if os.path.exists(static_filepath): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this guard is unnecessary since we're already catching the IOError
. Also if that path doesn't exist either because static/i/
is missing then we have big problems, or if static/i/logo.png
is missing, then we should just copy it anyway.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
or if static/i/logo.png is missing, then we should just copy it anyway.
I think I read once that the path to which we are moving should exist before os.rename()
is called.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That wouldn't make sense for the API to do that because you'd always have to create a new file at the destination before doing a move. Also:
heartsucker@pythagoras:~$ mkdir /tmp/wat
heartsucker@pythagoras:~$ cd !$
cd /tmp/wat
heartsucker@pythagoras:/tmp/wat$ touch foo
heartsucker@pythagoras:/tmp/wat$ python2 -c 'import os; os.rename("foo", "bar")'
heartsucker@pythagoras:/tmp/wat$ ls
bar
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remove the unnecessary guard then
securedrop/journalist_app/admin.py
Outdated
os.rename(temp_static_filepath, static_filepath) | ||
flash(gettext("Image updated."), "logo-success") | ||
|
||
except IOError: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would change this to except Exception
because I'm fairly sure that Image
raises Pillow specific errors on all sorts of things, and we want to alert the user in all cases.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll update it
securedrop/journalist_app/admin.py
Outdated
flash(gettext("Image updated."), "logo-success") | ||
|
||
except IOError: | ||
with open(static_filepath, 'w') as img_file: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This with
block is unnecessary becuse os.rename
is effectively atomic, so I think it's acceptable to assume that we never end up with a broken logo.png
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since f.save(static_filepath)
always writes logo.png, this with
block replaces it with current image if uploaded image is broken. Without this restoration, only a message will be flashed, but the broken image will remain saved as logo.png
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, which is why I'm suggesting that we have an endpoint that selects and uses something called custom_logo.png
so we don't end up with a broken logo.png
. If the app terminates while the write in that with
block happens, we'll have a broken logo. You're more IO than necessary, and that could be a problem. Also, the advantage of having and endpoint that selects custom_logo.png
is that if someone runs the Ansible playbooks, they won't overwrite the uploaded image.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Before moving on to logo endpoint approach, I want to discuss my current approach and what purpose the two with
blocks are serving.
Case 1-
User tries to upload logo for the first time
The image is a valid png file - A.png
The default SecureDrop logo is backed up
f.save
writes A.png
In the try
block, no exception occurs while working with image, images is resized and os.rename()
executes
Thus the resized image is written at static_filepath
Case 2-
User takes a non-image file B.png (by renaming any other type of file to png) and tries to upload
The current logo is backed up by first with
block
f.save
writes B.png (which is a valid file, but not a valid image file)
Pillow raises IOError
in the try
block while trying to read the file, and os.rename()
does not execute
In the except
block, second with
overwrites static_filepath
with previously backed up image, which is not broken
Thus, the broken image is discarded, and a warning is shown
If the with
blocks are removed, the non-image file will be saved. So I think they are not unnecessary in the current approach.
securedrop/journalist_app/admin.py
Outdated
@@ -33,9 +35,30 @@ def manage_config(): | |||
f = form.logo.data | |||
static_filepath = os.path.join(config.SECUREDROP_ROOT, | |||
"static/i/logo.png") | |||
temp_static_filepath = os.path.join(config.SECUREDROP_ROOT, | |||
"static/i/temp_logo.png") | |||
with open(static_filepath) as img_file: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As noted in other comments, I don't think we need to read this in. os.rename
should just do the right thing on its own.
securedrop/journalist_app/admin.py
Outdated
@@ -33,9 +35,30 @@ def manage_config(): | |||
f = form.logo.data | |||
static_filepath = os.path.join(config.SECUREDROP_ROOT, | |||
"static/i/logo.png") | |||
temp_static_filepath = os.path.join(config.SECUREDROP_ROOT, | |||
"static/i/temp_logo.png") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we should save temp files in ./static
. I think we can use the SECUREDROP_DATA_ROOT
and make sure there's a tmp
in there. This would provide us with safe place for temp files.
@heartsucker I'll try to incorporate these changes. Can I know why "staging-test-with-rebase" is failing? |
@aydwi its failing during app installation
i don't think uppercase should matter but maybe pip interprets the requirements file differently than the cli tool? i dunno thats my first guess is to try setting it |
@msheiny I built the requirements file with I added |
huh @aydwi .... that cant be it then.. if pip-tools is handling it, should be fine.... hrmmm |
|
https://circleci.com/gh/freedomofpress/securedrop/8969?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link shows in the Run Debian builds part which is green / success: TASK [build-securedrop-app-code-deb-pkg : Create pip wheel archive for Debian package requirements.] changed: [build] TASK [build-securedrop-app-code-deb-pkg : Install dependencies for building translations] *** changed: [build] => (item=[u'gnupg2', u'haveged', u'python', u'python-pip', u'secure-delete', u'sqlite', u'apparmor-utils', u'redis-server', u'supervisor', u'libjpeg-dev']) But Create pip wheel archive for Debian package does - name: Create pip wheel archive for Debian package requirements. command: > pip wheel -r {{ securedrop_code_filtered }}/requirements/securedrop-app-code-requirements.txt -w {{ securedrop_app_code_deb_dir }}/var/securedrop/wheelhouse tags: pip which should fail to populate the wheelhouse with pillow ... because the dependency it needs (libjpeg-dev) is installed after. So it looks like pip wheel does not fail when it should and we have wheelhouse that does not contain pillow, reason why it fails to install from the package (because the package does not download from the net, it installs from locally available wheelhouse exclusively). |
I tried manually and failed to reproduce with $ cd securedrop $ ./bin/dev-shell bash $ sudo bash # apt-get remove --purge libjpeg-dev # pip uninstall pillow # pip wheel -r requirements/securedrop-app-code-requirements.txt -w /tmp/w # ls /tmp/w/Pillow* Pillow-5.0.0-cp27-none-linux_x86_64.whl Trying |
you need to add The headers or library files could not be found for jpeg, a required dependency when compiling Pillow from source. Please see the install instructions at: https://pillow.readthedocs.io/en/latest/installation.html Traceback (most recent call last): File "", line 1, in File "/tmp/pip_build_root/pillow/setup.py", line 804, in raise RequiredDependencyException(msg) __main__.RequiredDependencyException: The headers or library files could not be found for jpeg, a required dependency when compiling Pillow from source. Please see the install instructions at: https://pillow.readthedocs.io/en/latest/installation.html ---------------------------------------- Failed building wheel for pillow Running setup.py bdist_wheel for psutil Destination directory: /tmp/z Running setup.py bdist_wheel for pycryptodomex Destination directory: /tmp/z Running setup.py bdist_wheel for scrypt Destination directory: /tmp/z Running setup.py bdist_wheel for sqlalchemy Destination directory: /tmp/z Running setup.py bdist_wheel for webassets Destination directory: /tmp/z Running setup.py bdist_wheel for wtforms Destination directory: /tmp/z Successfully built cssmin flask-assets gnupg itsdangerous jsmin markupsafe psutil pycryptodomex scrypt sqlalchemy webassets wtforms Failed to build pillow but it does not trigger an error because pip wheel never returns on error when it fails, it just outputs and error message. |
@dachary Okay |
For the record #3065 will fail the build in a more debuggable way in the future. |
@heartsucker We wouldn't want to revert to in-place resizing of logo. Is there a way permissions can be granted? Anyways I'm working on a way to handle this, will share soon. |
@aydwi In chat, @dachary corrected me that I have it backwards. The logo needs to be owned by
When we do this, we don't have to ever touch the original |
I'm working on it. Before that, I'll push another method for review. |
Codecov Report
@@ Coverage Diff @@
## develop #3057 +/- ##
===========================================
- Coverage 86.38% 84.13% -2.25%
===========================================
Files 34 34
Lines 2122 2042 -80
Branches 233 223 -10
===========================================
- Hits 1833 1718 -115
- Misses 234 271 +37
+ Partials 55 53 -2
Continue to review full report at Codecov.
|
In case you don't know about it @aydwi github does not notify when you push new commits. You should explicitly ping @heartsucker and let him know you have something new addressing his comments. If you already knew just ignore me :-P |
securedrop/journalist_app/admin.py
Outdated
try: | ||
with Image.open(static_filepath) as im: | ||
with Image.open(f) as im: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure why you're copying the image and than saving the copy. The original is still being overwritten so we can still end up in a place where we have a broken image.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was trying to work with a copy of image instead of modifying it in place. That is unnecessary though, I'll remove it.
@aydwi a gentle reminder about the need to rebase this pull request |
@dachary Sure. I'll rebase it with develop 🙂 |
@heartsucker Pushed the changes to implement the endpoint and squashed the commits |
@aydwi you need another rebase because changes to tests were merged in the past 24h. Good practice :-) |
@dachary Will do |
Added blank lines Fix linting issues Made requested changes Linting build: pip wheel must fail when it does not build a python module build: re-create the builder docker image when Dockerfile changes It takes longer but works all the time. Uploading an image to a docker repository to speed-up the build requires an additional manual step and access to the docker repository. A possible optimization (would be a few minutes faster) could be implemented by storing docker layers, similar to what is done for the Dockerfile used when testing. Try to fix failing CI build Changes in methodology Added an endpoint to handle custom logo image Add logo endpoint to _insecure_views Fix the error in the previous commit
@aydwi you should also clean up the commit message in this PR. |
@kushaldas Sure |
I'm going to merge this because it is a strict improvement over the current behavior. However, I am going to open up some followup tickets to tighten this up a little bit (for example, if we want to save in |
Status
Ready for review
Description of Changes
This PR-
Fixes Admin logo image upload: Resize images server-side if they are too large #2807 by resizing the uploaded logo to a maximum width of 500px, or a maximum height of 450px (whichever is higher) while maintaining the original aspect ratio. Maximum limit is chosen in accordance with the recommended size by SecureDrop docs. Images with dimensions below this constraint are not resized.
Fixes Image uploaded by journalist is always saved as a PNG file #3043 by converting JPG/JPEG image files to PNG before saving them. Thus no matter which format the Admin chooses to upload, the final image is always a PNG file.
Possibly fixes Recommend dimensions for logo(s) uploads #2801 by adding a message at the logo upload screen informing the user about the recommended image size as per the docs.
Dependencies Introduced
Python
Pillow
, installed viapip
libjpeg-dev
, installed viaapt
Testing
After successfully uploading, image size can be verified by checking the dimensions of
securedrop/securedrop/static/i/custom_logo.png
.After successfully uploading a JPG/JPEG image, it can be verified that
securedrop/securedrop/static/i/
always containscustom_logo.png
Journalist app should display the recommended size on the upload page.
Try to upload a non-image file with an extension like JPG/JPEG/PNG. A warning message should appear that image cannot be processed.
Checklist
If you made changes to the app code: