If a file download fails with 503, don't abort the whole sync

[guruz]

The failing file will fail with 503 Service Unavailable, the 2 other concurrently syncing files will get Operation Cancelled.

We could actually continue to sync the other 2 files.

HOWEVER what would this mean if all other files (more than the 3 currently syncing) lead to a 503? We'd probably not want to get an endless 503 sync..

[dragotin]

Where does the 503 come from?

The definition of 503 is: If the 503 is returned to a PROPFIND on root, it means that the server is in maintenance, and the whole sync goes offline.

If the 503 is returned on any operation on an individual file or (more imnportant) folder, it means that the folder is not available current, and is IGNORED. That behavior is important for external storages that can go away because of their own problems. The server will return 503 for them in that case, to avoid the client remove all their contents in the dir.

[guruz]

@dragotin See bug subject, file download :-)

[guruz]

Related #5187

[moscicki]

Yes, we also see this and it affects our production services. One GET 503 will block the entire synchronization for all unrelated files as well.

[guruz]

@moscicki (semi-related) How long are your files usually 503?

[ckamm]

Mmh, looks like the client needs more careful backoff then. The other ask for 503s was "please give the server some breathing room when you see it" (for the overload or maintenance case). I guess we could keep going after the first and only abort when a certain number of errors have accumulated.

@moscicki If I understand you correctly you're using 503s on a resource level and they can be persistent? Is this related to the overloading with "503 storage unavailable"?

[guruz]

I guess we could keep going after the first and only abort when a certain number of errors have accumulated.

But what's the number?
Indeed maybe it would be better for @moscicki to mark the whole directory as "503 storage unavailable" as this won't abort the sync?

[moscicki]

Cannot do that. We get 503 when a file to be downloaded is "busy" (similar to "locked"). A problem is that sometime files may stay a long time in such state.

…

On Wed, Nov 21, 2018 at 6:14 PM Markus Goetz ***@***.***> wrote: I guess we could keep going after the first and only abort when a certain number of errors have accumulated. But what's the number? Indeed maybe it would be better for @moscicki <https://github.com/moscicki> to mark the whole directory as "503 storage unavailable" as this won't abort the sync? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5088 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAl9jUFuQTC0cSJohG-IcqxASRbUMfUjks5uxYnfgaJpZM4JXJDQ> .

-- --- Best regards, Kuba

[guruz]

@butonic @DeepDiver1975 @PVince81 @ogoffart Didn't we somewhere say we could support sending a Retry-After header (optional) on 503 GET in case we think the file will come back up soon and should not abort the whole sync?

[PVince81]

AFAIK the server returns 503 either when the whole server is not available "Service not available" or whenever the underlying external storage is temporarily unavailable "Storage not available".

We have some logic that detects "unknown errors" in the storage and maps them to "Storage not available" so it is possible that in some specific yet undiscovered scenarios, legit storage specific errors that should actually be mapped to other errors.

As for "Retry-After", we could add that in the StorageNotAvailableException on Sabre level.

[guruz]

@moscicki Would this be OK for you? CERNBox could send Retry-After, then we treat the 503 differently?

[moscicki]

So if I understand correctly if a file GET would return 503 + Retry-After then the sync would not be aborted but a GET retried after the specified amount of time? Without a Retry-After header the sync would be aborted as it is now.

Is this interpretation correct?

[moscicki]

Ping.

Also, is there any contradiction of what we discuss here with this issue: #3932 ?

[guruz]

It looks like the Retry-After suggestion is indeed #3932 (CC @butonic )

@ogoffart has implemented something else though in https://github.com/owncloud/client/pull/6906/files#diff-7e6f8d2579afb992a2e4ed25a154935bR78 ... so as long as the 503 doesnt look like the maintenance-mode-503 the sync will go on.
@ogoffart when will the next sync be triggered?

[labkode]

@guruz what is the status of this?

[michaelstingl]

@guruz what is the status of this?

@labkode It should be improved in the upcoming 2.6.0. Please verify with the 2.6.0-alpha1 and provide feedback:
https://github.com/owncloud/client/releases/tag/v2.6.0-alpha1

[HanaGemela]

Why have I got the message twice? Logs shared with @ckamm and @guruz

Also the Not synced protocol shows two entries - one without file name

[HanaGemela]

Not really sure how to recreate but sometimes the message looks like this. Logs shared

[HanaGemela]

I've tried with server in maintenance mode. While in maintenance mode, I've unchecked one folder in selective sync list and disabled VFS for another account. After turning off the maintenance mode, the client crashed 48a065f1-c5e8-4541-b427-9355b200dd62
Not sure if it is related to this issue or not. I've found the scenario with maintenance mode in the top level issue. In case it is not related, I'll raise a new issue

[ckamm]

@HanaGemela Thanks for the logs. EDIT: Made a PR against the error duplication.

The crash is unrelated and worth its own issue if you can reproduce it. EDIT: The crash on sentry looks like an unspecific error in the Qt network stack unfortunately: https://sentry.io/organizations/owncloud/issues/1131537840/events/48a065f1c5e84541b4279355b200dd62/?project=79001&statsPeriod=14d&utc=true

[HanaGemela]

@ckamm Great, now I see the error message only once.

Still some issues here:

1. Sometimes the error is e:http:response_header as already described above
2. When syncing a folder that is unavailable, the rest of the files are not synced
3. When the folder becomes available, the client doesn't recover - folder is still shown as unavailable. I believe this is a server issue though

[ckamm]

@HanaGemela

1. "Undefined variable: http_response_header" is an error message the server sends. I don't know why it's intermittent.
2. A 503 during discovery should mean "ignore that folder" and the sync should continue with other files. A 503 during propagation is normally a normal error where other files will be synced after. The exception is if the 503 reply contains ">Sabre\DAV\Exception\ServiceUnavailable<" because then we assume maintenance mode has been switched on and abort the sync.

[HanaGemela]

@ckamm then it's broken, the sync should continue but it doesn't

[ckamm]

@HanaGemela How do you set up the 503 reply for your tests? I need it to reproduce your specific case.

[HanaGemela]

@ckamm Start two servers:

docker run -e OWNCLOUD_SHARING_FEDERATION_ALLOW_HTTP_FALLBACK=true -p 8080:8080 owncloud/server
docker run -e OWNCLOUD_SHARING_FEDERATION_ALLOW_HTTP_FALLBACK=true -p 8081:8080 owncloud/server

Then:

1. User1 shares a folder with User2 from another server (federated share)
2. Turn off the first server 
3. Check the account of User2

[ckamm]

Thanks, that helped. An unavilable federated share gets reported as:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\ServiceUnavailable</s:exception>
  <s:message>Storage is temporarily not available</s:message>
</d:error>

but the error handling code assumes this ServiceUnavailable means maintenance mode and considers it a fatal error.

[ckamm]

@HanaGemela I've added a PR for an unreliable workaround in 2.6.0 - a reliable fix will need to be deferred to 2.6.1. (it currently relies on the s:message string, which is a user-facing translatable string; a correct solution would trigger a follow-up request to status.php to check for maintenance mode)

[HanaGemela]

Works as expected on 2.6.0rc1 (build 12411), mac 10.14.6