MySQL timeouts sanitising big transfers

[jtweed]

Looks like Archivematica is holding open database connections whilst it tries to do jobs.

This means when you have a transfer that contains a lot of files, by the time it tries to write the results of the job, the connection has timed out and it fails the job.

Example:

Transfer: EXHIBITION ARCHIVE
Task: 45485279-8855-459c-9df3-02eb22310b94

archivematicaClient.py: ERROR 2019-01-31 19:56:55,721 archivematica.common:retryOnFailure:373: Failed to complete transaction "Write task results" after 10 retries
archivematicaClient.py: ERROR 2019-01-31 19:56:55,721 archivematica.mcp.client:execute_command:231: Exception while processing task sanitizeobjectnames_v0.0: (2006, 'MySQL server has gone away')

[helenst]

Related: archivematica/Issues#313

[alexwlchan]

I saw a live repro:

archivematicaClient.py: ERROR 2019-03-20 17:05:30,038 archivematica.common:retryOnFailure:316: Failed to complete transaction "Write task results" after 10 retries
archivematicaClient.py: ERROR 2019-03-20 17:05:30,038 archivematica.mcp.client:execute_command:231: Exception while processing task sanitizeobjectnames_v0.0: (2006, 'MySQL server has gone away')
Traceback (most recent call last):
  File "/src/MCPClient/lib/archivematicaClient.py", line 224, in execute_command
    retryOnFailure("Write task results", write_task_results_callback)
  File "/src/archivematicaCommon/lib/databaseFunctions.py", line 317, in retryOnFailure
    raise e
OperationalError: (2006, 'MySQL server has gone away')

Googling suggests this can be caused if you have a timeout or you ask SQL to execute an unusually large query. I’ll drop in some more tracing tomorrow and see what I can find.

[jtweed]

Closing to tidy up the backlog. Will reopen as a new issue if we see it again with our actual workflow.