Nextcloud 20 updates to 21 despite unsupported MySQL 5.x

[caugner]

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Steps to reproduce

1. See MySQL-related warning in Administration / Overview:
2. See available update to 20.0.9 on same page:
3. Notice that the Updater wants to update to Nextcloud 21 anyway:

Expected behaviour

1. Updater performs major update to Nextcloud 21 despite Update notification & Administration Overview advertising patch update to 20.0.9 #26469 - The Updater should update to the same version as advised in the Administration Overview.
2. The Updater should not update to a major version that is incompatible with the current environment.

Actual behaviour

The Updater updates to the next major version, ignoring the incompatible MySQL version.

Server configuration

Operating system: Ubuntu

Web server: Apache 2.4

Database: 5.7.28-nmm1-log

PHP version: 7.4

Nextcloud version: 20.0.8

Updated from an older Nextcloud/ownCloud or fresh install: ~ ownCloud 5

Where did you install Nextcloud from: Updater?

Signing status:

Signing status

No errors have been found.

List of activated apps:

App list

n/a

Nextcloud configuration:

Config report

n/a

Are you using external storage, if yes which one: No

Are you using encryption: No / Yes (one instance each)

Are you using an external user-backend, if yes which one: No

Client configuration

Browser: Firefox Browser Nightly 89.0a1 (2021-04-08)

Operating system: Windows 10

Logs

Web server error log

Web server error log

(on demand)

Nextcloud log (data/nextcloud.log)

Nextcloud log

(on demand)

Browser log

Browser log

(on demand)

[quizilkend]

I experience this as well. Right now my instance is not reachable.

I am using MariaDB mysql Ver 15.1 Distrib 10.3.27-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2 and I'm not sure which version would fit NC21.

[caugner]

I am using MariaDB

@quizilkend As per the System requirements, Nextcloud 21 supports MySQL 8.0+ or MariaDB 10.2+ (recommended), so you should be fine.

[quizilkend]

@caugner: Thanks for your reply. Now I have no clue why my instance is not up and running.

The updater ran flawlessly but occ upgrade hanged silently until the kernel (OOM-killer) killed it.

Do you have an idea how to further investigate this, to better locate the problem?

[caugner]

@quizilkend Please make sure to clear your browser cache (CTRL+F5). If the problem persists, I would recommend to have a look at the updater.log file in your Nextcloud data directory and open a new bug issue, in which you post the relevant sections from that logfile (i.e. all lines from today).

[caugner]

FYI The good news is that Nextcloud 21 still seems to generally work with MySQL 5.7, even though it's not supported anymore.

[andrimont]

FYI The good news is that Nextcloud 21 still seems to generally work with MySQL 5.7, even though it's not supported anymore.

@caugner So with -MariaDB-1:10.4.13, I should be fine. But I encounter the same issue as you reported on this thread.
Did you succeed to install the "stable" 21.0.1 ?
What is your advice when the install goes wrongly from 20.0.8 to 20.0.9 (announced) but goes to 21.0.1 and then crashes ?

[mind-overflow]

FYI The good news is that Nextcloud 21 still seems to generally work with MySQL 5.7, even though it's not supported anymore.

@caugner So with -MariaDB-1:10.4.13, I should be fine. But I encounter the same issue as you reported on this thread.
Did you succeed to install the "stable" 21.0.1 ?
What is your advice when the install goes wrongly from 20.0.8 to 20.0.9 (announced) but goes to 21.0.1 and then crashes ?

I had literally the same issue. I have written here, on the Nextcloud forums. The update failed and CPU/RAM usage skyrocketed until my server completely died. I'm still in the process of trying to recover access to it.

[daCaPo]

Upgrade here went well, but I agree that the updater must not update when running with an officially unsuported database (mysql 5.7.33 in my case).

Furthermore, the settings page now shows a red warning because the PHP memory limit is below 512 MB. An update should not be allowed without explicit warning if the requirements of a new release would not be fulfilled.

[caugner]

Furthermore, the settings page now shows a red warning because the PHP memory limit is below 512 MB.

This warning is not new to Nextcloud 21. However every update overwrites the .htaccess file, so if you manually added a directive there to increase the memory limit, you will have to add it again. It's a feature, not a bug, to increase stability.

[andrimont]

Upgrade here went well, but I agree that the updater must not update when running with an officially unsuported database (mysql 5.7.33 in my case).

Did you succeed with 21.0.1 ?
In my case I used backup and I installed manually 20.0.9. It is roughly ok ; but there is some side effects.
I still do not get the point of the issue with 20.0.9. > 21.0.1…
I am very afraid with the install process now

[daCaPo]

Did you succeed with 21.0.1 ?

As I can tell so far, yes.

I am very afraid with the install process now

So am I, running two more instances whicht I won't touch before this issue is solved.

[mind-overflow]

Ok. I unfortunately have no great way to prove this, but here's what I did to recover from the resource-hogging disaster, after forcefully shutting down the machine.

I disabled apache2 and php7.4-fpm services, to prevent the cronjob from restarting automatically (it actually did, and I had to kill it via htop - but after stopping the services, it stopped respawning).

Then, I restored 20.0.8 from the /data/updater-xxxxxxx/backups/ folder. I used rsync

rsync -Aaxv /var/www/nextcloud/data/updater-xxxxxxx/backups/nextcloud-20.0.8.1-xxxxxxxx /var/www/nextcloud/

(be careful with the correct folder names).

Then, without touching the database, I restarted everything. I visited the Nextcloud URL on the browser, and I logged in. After logging in, I got an internal server error.

I looked into the Nextcloud log (/var/www/nextcloud/data/nextcloud.log), and I saw an error about missing database tables.

My intuition was that, probably, the database was updated to the 21.0.1 format. So, to avoid destroying the database, I did this:

• I went into the nextcloud/updater/ directory.
• I ran the updater manually.

sudo -u www-data php updater.phar

It went fairly smoothly, downloaded a bunch of stuff and moved it in place. Then it asked

Should the "occ upgrade" command be executed? [Y/n]

I said "yes". The update process started, and it showed a bunch of "repair steps". Nice.
At the end, there was a final step, where the server died on me last time:

2021-04-10T09:25:32+00:00 Starting code integrity check...

At this point, I launched a parallel SSH terminal and launched htop, to monitor resources usage.

The Nextcloud cron.php script was launcher once, and taking up about 6GBs of RAM. After 3/4 minutes, it was spawned ANOTHER time (so I had two contemporarily running jobs), and started taking up RAM and CPU. I killed both of them, and RAM usage dropped to 2GB. The update script did not fail however, saying that "integrity check" was passed. (I doubt it, as I literally killed the script that was checking for it - but it doesn't mean it failed either). After that, Nextcloud was responsive again, on latest version 21.0.1.

2021-04-10T09:30:30+00:00 Finished code integrity check
2021-04-10T09:30:31+00:00 Update successful
2021-04-10T09:30:31+00:00 Maintenance mode is kept active
2021-04-10T09:30:31+00:00 Resetting log level

Skipping integrity check is not a good thing nor a good practice, but it is what it is. It never happened to me in the past that the integrity check would take up so many resources. I am getting no errors in the logs for the moment, so hopefully everything is fine.

[szaimen]

taking up about 6GBs of RAM. After 3/4 minutes, it was spawned ANOTHER time (so I had two contemporarily running jobs), and started taking up RAM and CPU

Did you see this workaround for high RAM and CPU usage?
#25761 (comment)

[b-pfl]

FYI The good news is that Nextcloud 21 still seems to generally work with MySQL 5.7, even though it's not supported anymore.

Some details about (in-)compatibility are given in #23210. It seems that there are some reasons for not using / recommending MySQL 5.7 (issues with Mail, @ChristophWurst can these be fixed in a different way than just upgrading MySQL?). I hope that @nickvergessen or @ChristophWurst can clarify the situation and help out.

What is clearly needed is a reliable statement about the use of (or problems with) MySQL 5.7 with NC21.

I see two options:

1. If it turns out that NC21 should still work with MySQL 5.7, the warning in NC20 about MySQL should be removed and we hopefully are on the same side. In that case, the auto updated behavior NC20.0.8 -> NC21.0.1 seems fine.
2. If NC21 should not be used with MySQL 5.7, then the updater MAY NOT update to NC21.

Obviously, case 1 would be the most appreciated one!

[nickvergessen]

[mind-overflow]

Did you see this workaround for high RAM and CPU usage?
#25761 (comment)

thank you. this was literally the solution i needed.

[szaimen]

I think the underlying problem was fixed with nextcloud-releases/updater_server#558

[caugner]

@szaimen Afaik that PR only fixed the inconsistency between the advertised update target and the actual update target. It didn't add a check to prevent an update if the target version doesn't support the current environment (here: MySQL 5.6).

[NicoHood]

I am using Nextcloud 23.0.x, but I am running and old mariadb 10.1.x which should be not supported anymore. However the installation still works. Can anyone tell me what could be broken and if I should better reinstall the whole nextcloud or just keep it running until my provider updates the database version?

[szaimen]

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+