-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[NC 21] OCC Update error: allowed memory size exhausted #25742
Comments
I am also affected by this. I run Arch Linux. I even tried a fresh install (but old data directory). The web interface works as expected. |
I also run Arch Linux, and in the web interface, of O go to the page admin/overview, it says, among other things |
debian 10 here, same issue: tried with php 7.0 7.3 and 8.0 no joy, this is a nextcloud issue not an operative system one... I upgraded from 21.0.0 RC2 to 21.0.0 and the issue presented itself on the upgrade at the end of the update process (that is the first time occ is run AFTER the files changed from rc2 to stable), this suggests me the problem might be somewhere in the changelog between the rc2 and the production ? |
As a matter of fact, there are whole bunch of things that don't work with web interface. I can't update, enable, disable, remove or download applications. |
As it might be relevant, here is some info about the php cli configuration: php -i
|
I can reproduce this bug with Nextcloud stable21 and php v7.4 on Arch Linux. Another finding: The occ command works fine if the instance is not yet installed. After installing (e.g. using web installer or occ install command) it becomes unusable. |
Currently I suspect the upgrade process somehow corrupted the Mysql database? InnoDB: Page [...] log sequence number [...] is in the future!
If I stop the database, occ runs until connection timeout to Mysql server. So my guess is Occ crashes after connecting to the database ... |
Okay maybe it is something different. Nevertheless I put this at line 133 in
Which is giving me this stack trace showing some kind of iteration with arrays including the PHP environment and configuration array?! https://onny.project-insanity.org/files/stacktrace_short.txt Information which might be useful in this case:
|
This is what I get from logging mysql queries:
Nothing more ... Edit: oc_appconfig has a lot of keys so I don't know where to look at. In case some value in there is causing the issue ... |
I hit OOM when upgrading to latest nextcloud by running occ upgrade. |
I second this. I have been testing using the stable21 branch last week and it worked fine until a couple days ago. The regression was introduced recently. Maybe bisecting from 21RC2 to the current version is a good starting point. |
Same issue here. Arch Linux ARM, but with Postgres backend, so it might not be related to mysql at all.
Disabling all extensions except for php -i
|
Even with |
Same problem on FreeBSD 12.2, php 7.3. Upgrades to NC 21 RC 1 and NC 21 RC 2 went through without problems. Now it's just the main release NC 21.0.0:
and a few more reports on the forum: |
Also having this problem on FreeBSD 12.2, with php 7.4. Downgrading (using FreeBSD binary packages built myself) restores functionality, the DB looks unmodified by the botched update. |
I had exactly the same issue after I upgraded from NC 20.0.7 to 21.0.0 release. The cron.php also did not run. CentOS 7 - postgreSQL 13 - php 7.4 or 8.0 (did not matter) I "solved" the problem by downgrading to NC 21.0.0 RC2 |
@andreaslutsch your solution worked for me (21.0.0, ArchLinux). For reference for others, you can enable the option by adding the parameter |
@onny Do you think, this issue is fixed? For me it is a regression and thus a bug when software changes it behavior and is crashing, when there is something missing in the environment. At least this should be documented, if apc is a hard requirement (and checked by the application when starting and throwing an error message). Enabling apc is in this case just a "workaround". |
The apc.enable_cli workaround does not work here (21.0.0, Alpine Linux 3.13, PHP 7.4.15). |
@waja it looks like a regression for me because it worked without this option before and it's only recommended for the cronjob. |
Similar issue here on Ubuntu server 20.10, php 7.4, MariaDB 10.3, whatever amount of memory I allocate to my Nextcloud instance gets exhausted. Thanks for your help |
Just in case: it is Edit: added version info
|
I ran git bisect and the commit a745d58 seems to be introducing this regression. |
Thanks for your reply. EDIT: my mistake it worked with apc.enable_cli=1 |
Was there any comments from maintainers about fixing this or apcu is mandatory now? |
Set your log_level to 2 in you config. That’s all in need to do. |
That may be so, but a bug that is so dramatic that it has the potential to tear down not quite cleanly set up servers is still a huge problem in my eyes. On servers that are not only running nextcloud, this can have far-reaching consequences for other services. This will happen to many and in my opinion is so critical that it should block the further rollout of Nextcloud 21 until the problem is fixed. |
The changeset which has caused this issue is known. Someone made a critical modification after the testing of the last release candidate 2 for Nextcloud 21 was finished short before Nextcloud 21.0.0 was released. This is not a good strategy. If you install Nextcloud 21RC2 then you don't have this bug. |
This is not true, i reported this bug during the beta phase and again after RC1 was released. |
TheRaven500: You are talking about NC21.0.1 RC1 NC21.0.0 RC 2 does NOT have this bug. NC21.0.0 Release introduced it. Your report (26154) you are linking to was created on 16. March 2021 and is a duplicate of this one which was added on 22. Februar 2021. The bug was introduced short BEFORE NC21.0.0 was released and AFTER NC21.0.0 RC2 was out for testing. I am still using Nextcloud 21.0.0 RC2 and I don't have this bug. |
Just updated to NC 21.0.1. Managed to consume 24 GB RAM and then 24 GB of SWAP. Caught it just in time to stop the server crashing. PHP is limited to 18 GB on my system, APCu is already enabled, log level is already at 2. Considering the comments reporting this issue, and just how crippling it is, I don't understand why it's still available for people to run the update. |
@RealSnoopyDog: |
Yes. But not "may". Talking about software and bugs is not guessing, it is based on facts. And everything was written here already (not by me, I only repeated what was found by others). |
Hi Ubuntu 20.04.2 LTS here. I had the crash as well, |
I'm getting a notification that 21.0.2 is available for upgrade. Can I assume that as this bug is still open, it's likely to try and cripple my server all over again? |
In my particular case it eats all memory and forces me to kill the php process. I did disable the accessibility app and set loglevel to 2, but when i start the update process again, it resets loglevel back to 0 and screws up again. |
Have you checked your APC/Redis cache setting, libraries and server? |
nextcloud instance itself runs in its own container by itself. Redis and the database are in a different container. The nextcloud container is the one that has the memory leak. Haven't checked libraries - do you have any specific ones in mind (fyi, i'm not experienced with php)? Since it's a personal cloud server that i've already backed up, i tried this workaround #25742 (comment) and so far everything seems to be in order. |
In my case changing the nextcloud loglevel absolutely nothing. I removed the php-redis package/library by mistake, which led to nextcloud being unable to use said caching service. Reinstalling that library fixed it. Removing it broke it again. Switching to memcache by installing the daemon and php libraries and changing the nextcloud config appropriately fixed it, removing the libraries/package broke it again. And sure, it's nextcloud that blows up, not any other service. So it absolutely makes sense that the nexcloud container is the one that dies, if you are using a containerized setup. That is why I recommended to check your cache settings and your installed php libraries in your nextcloud container. |
Installing php7.4-xml and php8.0-xml fixed it for me. |
This is not a fix for the real problem, it is a workaround (like all other hints) |
- fixes 'allowed memory size exhausted' during nextcloud upgrades/occ execution - nextcloud/server#25742 - nextcloud/server#25770
I haven't had this problem until NC 22 Beta 3. Now I have the same error. Removing solves APCu resolves the problem but is not in line with the recommendations in the adminstration manual. |
…rades/occ execution: - enable acpu cache/accelerator for php CLI - disable accessibility app - nextcloud/server#25742 - nextcloud/server#25770
…rades/occ execution: - enable acpu cache/accelerator for php CLI - disable accessibility app - nextcloud/server#25742 - nextcloud/server#25770
Posting this here in case it helps anyone else... You can check the PHP memory limit using the command line like this: php -r "phpinfo();" | grep memory_limit
memory_limit => 128M => 128M With the above defaults I was unable to upgrade Nextcloud: php updater/updater.phar --no-backup
...
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 167968632 bytes) in phar:///home/cloud/sites/nextcloud/updater/updater.phar/lib/Updater.php on line 637 However after creating a
And then running the update using this ini file: php --php-ini ~/.php.ini updater/updater.phar --no-backup There isn't a problem with the updater. As far as I can tell you don't need to enable the OPCache or APCu for the CLI, the upgrade works for me with these settings: php -r "phpinfo();" | grep enable_cli
apc.enable_cli => Off => Off
opcache.enable_cli => Off => Off |
sudo --u www-data php --define memory_limit=512M /var/www/html/occ files:scan --all |
How to use GitHub
Steps to reproduce
php ./occ
in Nextcloud root directory./occ
crashesExpected behaviour
Occ should work normally, for example printing available commands
Actual behaviour
Occ crashes with the following error:
To be more precise, this also happens running php cli against other files of Nextcloud, for example
index.php
orconsole.php
.Surprisingly Nextcloud is running fine through php-fpm and web server. It was even able to perform the upgrade to v21 just fine.
Another user commented on the ArchLinux bug tracker:
Possibly not only related to ArchLinux setups. An other user reported the exact issue here, which was triggered on a Ubuntu installation.
Server configuration
Operating system: Archlinux x64
Web server: Caddy 2
Database: Mysql
PHP version: 8.0.2
Nextcloud version: (see Nextcloud admin page) 21.0.0
Updated from an older Nextcloud/ownCloud or fresh install: Yes
Where did you install Nextcloud from: Package repository
Signing status:
Signing status
List of activated apps:
App list
Nextcloud configuration:
Config report
Are you using external storage, if yes which one: None, had configured SFTP mount point
Are you using encryption: No
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Latest Firefox
Operating system: NixOS
Nextcloud log (data/nextcloud.log)
Nextcloud log
The text was updated successfully, but these errors were encountered: