Long-delayed shot creation on Imgur and Giphy

[IuliaCristescu4]

[Affected versions]:

• Firefox 55.0b11 build1 (20170720101430)

[Affected platforms]:

• Windows 10 x64
• Mac OS X 10.11.6
• Ubuntu 16.04 x64

[Steps to reproduce]:

1. Launch Firefox with a clean profile.
2. Enable Firefox Screenshots (if it isn't already) - navigate to about:config and set the extensions.screenshots.system-disabled pref to false.
3. Navigate to http://imgur.com/ (or to https://giphy.com) and scroll down on the page to load more content.
4. Click the Screenshots button from the main browser toolbar.
5. Click the Save full page button.

• notice the time until the shot is created (displayed) in the preview page (and in My Shots page)

[Expected result]:
The time until the shot is created is unnoticeable (reasonable).

[Actual result]:
There are encountered very big delays until the shot creation (from 2 to 40 minutes).

[Note]:
This isn't an e10s issue.

[SoftVision-PaulOiegas]

@IuliaCristescu4 Is this reproducing constantly or totally random? I am unable to reproduce this on latest Nightly build (20170723030206). Maybe I'm doing something wrong. I have tried loading lots of content and then used either full page or visible area and in each case it took less than 10-15 seconds (first time was longer).

Also, couldn't this be related to how the website is built? Giphy website is slowly loading for me. I imagine we cannot make Screenshots work for every website on internet, especially for those that are struggling to work.
Also, I hardly think that someone will make a screenshot of a website that has an individual link for every image that it holds.

[johngruen]

@IuliaCristescu4 I am also unable to reproduce this with screenshots 10.8 on osX and windows 10.

[johngruen]

@IuliaCristescu4 i am also unable to reproduce this with 10.5

[lizzard]

I'm seeing this take a long time, not sure which version of screenshots but I'm using the June 23 Nightly and MacOS 10.12.5. I tried doing a full page screenshot on imgur.com and got a message the first time that Screenshots had "gone haywire". The second try worked after a few minutes. The 3rd try gave me a grey screen with "Creating Imgur" in the title bar, for a minute or two (definitely over one minute) and then a pop up notification that says Out of Order! Sorry, we couldn't save your shot.

[emilghittasv]

Hi @johngruen !

Somehow I managed to reproduce the issue (reported by Iulia) on Firefox 55.0b12 (Firefox Screenshots v 10.8.0) on Windows 10 64bit.

[Please Note]
-It seems that this issue is not that easily to reproduce as it was in Firefox Screenshots v 10.5.0.

-In order to reproduce this issue I followed the steps provided in the description.

-I think that this issue occurs because the tab is not loading when the screenshot is processing. I think that there is a workaround in solving this issue by accessing the "My shots" page (After accessing my shots page, it seems that the screenshot successfully loads).

Here is a screencast of this issue: https://drive.google.com/open?id=0B94iuIVMr-TEWGNjSnY1QXJqWDg

[ianb]

A couple things going on:

• We have a 25mb request body limit on the server. This includes the image, base64-encoded. There's a 5000px height/width limit on those images as well. This is big enough for a text-heavy full page, but with an image-heavy full page (like imgur or giphy) it can exceed that size.
• Some timings for a full page imgur upload, without invoking infinite scroll:
  • canvas.drawWindow is fast, 150ms
  • canvas.toDataURL is slower, 1800ms
  • data URL is about 23mb
  • JSON wrapper is only marginally larger
  • It takes about 1500ms to send the image/JSON from the content worker to the background worker
  • It takes 36s to upload the shot to screenshots.firefox.com
  • I don't have a time here, but screenshots.firefox.com in turn decodes the image and uploads it to S3, then re-fetches it to display the page, and this takes a considerable amount of additional time

If you hit the 25mb limit, then everything is run, but after the upload completes it is rejected by the server with a 500 response. So we fail very late.

For download (instead of upload) we still have the toDataURL overhead, and the communication overhead, but then the download completes fine.

If I invoke infinite scroll on imgur (getting us to the 5000px limit) some of the timings change:

• toDataURL is 3500ms
• data URL size is 48mb
• communication time is 2800ms
• upload fails so I don't get a good upload timing

[SoftVision-PaulOiegas]

Is the number of people using Screenshots on imgur and giphy so big, that we want to remove the functionality for the release? If yes, can't we figure out a solution? We're talking about 1-2 websites not the entire web.

I think that "Save Full Page" is one of the key features of Screenshots over other screen capturing applications (could be the only advantage). What else could make a user prefer Screenshots instead of the app he is used to already? Maybe the fact that you don't need a second app installed, but that doesn't seem to be such a problem for normal users that usually have tons installed...

By removing this for launch, we could end up with a dull feature that will not be embraced by the Firefox users. First impression matters for a lot of users.

[emilghittasv]

I totally agree with @SoftVision-PaulOiegas that we shouldn't remove the functionality for the release.

I wanted to add that the "Save full Page" option isn't available only on Firefox Screenshots. I saw that "Nimbus Screen Capture" (Add-on) has the same option (and they have the same issues as per #2007 regarding "save full page" on popular websites).

I believe that we need to focus on fixing the bugs that could become annoying for the users. I don't believe that we should necessarily focus on bringing something new above bringing something better (especially if it is about something that will be enabled by default).

[fzzzy]

I think the 5000px limit is too surprising and not documented anywhere. I think it is a safer approach to ship without "full page" and then add it back in later, perhaps without the 5000px limit, a better implementation that perhaps leverages the code path the dev tools uses to make full page screenshots, and a progress bar to help with the long wait times.

[luser]

We have a 25mb request body limit on the server. This includes the image, base64-encoded. There's a 5000px height/width limit on those images as well. This is big enough for a text-heavy full page, but with an image-heavy full page (like imgur or giphy) it can exceed that size.

It kinda sucks that we wound up with a screenshot API that produces a data URL instead of a Blob because we implemented Chrome's API. I wonder if we'd be better off making a new API, or extending this API to provide a Blob instead?

[SoftVision-CosminMuntean]

I have retested this issue on latest Nightly 57.0a1 build with Screenshots 18.0.0 on stage server and I think the behavior is a little bit changed.

After using the "Save Full page" option on Imgur, you are redirected to the "Saving your shot..." page and takes ~15-30 s until the shot is saved. Afterwards, you are redirected to the save shot preview page, but the image is not displayed until the page is refreshed.
Here is a screen recording with the issue: https://goo.gl/hxJUYr

[jaredhirsch]

@SoftVision-CosminMuntean Would you mind trying again on latest dev? #3513 should significantly reduce the lag when uploading full page images.

[SoftVision-CosminMuntean]

@SoftVision-CosminMuntean Would you mind trying again on latest dev? #3513 should significantly reduce the lag when uploading full page images.

I have tried to retest this issue on dev server with the following methods:
1. Install Screenshots from: https://screenshots.dev.mozaws.net/homepage/install-test-local.html
Screenshots is successfully installed, but I don't see any changes when saving a full page on Imgur or other large websites. The shot still has the ".png" extension and has ~9 Mb.
Probably this page is not updated with the latest Screenshots dev version?

2. Custom build created from Github repository
In order to create a Screenshots build, I have used the following command:
git clone https://github.com/mozilla-services/screenshots && cd screenshots/ && npm install && SCREENSHOTS_BACKEND=https://screenshots.dev.mozaws.net make zip

After I installed the add-on, I get an error every time when I try to save a shot.
However, the Screenshots custom build created yesterday seem to work fine. Only the build created today it's broken.
Please let me know if I'm doing something wrong.

Here is a screenshot with browser console errors:

Here is a screen recording with the issue:

[jaredhirsch]

Hmm. The server version does seem a little behind, but the deployed version (at https://screenshots.dev.mozaws.net/__version__) does include the jpeg commit.

I'm able to install the addon from the dev server on Nightly and create full-page jpegs: https://screenshots.dev.mozaws.net/hJeW7lpA9cGHPSl3/imgur.com

@SoftVision-CosminMuntean Mind trying again?

[SoftVision-CosminMuntean]

I have retested this issue using the Screenshots on dev server and the behavior has clearly improved. The large shots are correctly saved as ".jpeg" and the lag is significantly reduced when uploading a large shot. 💯

But, sometimes when a large selection is saved, you are correctly redirected to the saved shot page, but the image is not displayed until the page is refreshed. This is very intermittend and probably is a different issue. However, no errors are displayed in browser console.

[ghost]

Thanks Cosmin. Closing this then. This is a big win, thanks everyone!