Set higher open files limit in init script #8819

nineinchnick · 2021-08-06T11:46:57Z

#5066 added mention of ulimit to Trino's requirements. These recommended values should be used by default when installing Trino using the RPM package.

The reason to set these in the init script, not in a /etc/security/limits.d/99-trino.conf file is that it's a simpler change and at some point in time Trino might migrate from a SysV script to a SystemD unit file, which would ignore limits from /etc/security since those are only used by PAM (the pam_limits.so module).

Defining these limits before sourcing the env.sh file allows users to adjust them further if required.

hashhar

Good change % verification.

hashhar · 2021-08-06T12:04:03Z

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino

@@ -37,6 +37,9 @@ CONFIGURATION=(
    --server-log-file "${SERVER_LOG_FILE:-/var/log/trino/server.log}"
 )

+ulimit -Hn 131072
+ulimit -Sn 131072


I expect these to fail (i.e. not take effect) unless one of /etc/security/limits.conf or /etc/systemd/user.conf or /etc/systemd/system.conf has a value higher than this allowed for the Trino user.

I'll try and verify in a VM.

EDIT: Correction. It's /etc/security/limits.conf and /etc/systemd/user.conf. They need to have higher values to allow ulimit to succeed otherwise it fails with an error. To check values you can use sysctl -a | grep 'files.nr_open'.

I think root can execute them regardless of what's in /etc/security/limits.conf.

BTW I checked CentOS packages and found only one that defines limits in /etc/security/limits.d, which is libvma. But it's also the single package that has a SysV init script, not a SystemD unit file.

This only works because CentOS (since 7) uses SystemD and systemd sets the default hard limit to a high enough value that your command doesn't fail. This is not an assumption that is safe to make.

See https://www.freedesktop.org/software/systemd/man/systemd-system.conf.html#DefaultLimitCPU=.

And since we are setting the values we should also be setting nproc too either way.

My suggestion is to test whether we get a meaningful error message at startup.
We don't want the default state to be a situation where out of the box it's not visible why Trino fails to startup.

Try this:

# CentOS 7 [root@centos hashhar]# sysctl -a | grep 'fs.nr_open' fs.nr_open = 1048576 [root@centos hashhar]# ulimit -Hn 1048577 bash: ulimit: open files: cannot modify limit: Operation not permitted [root@centos hashhar]# sysctl -w fs.nr_open=1048579 [root@centos hashhar]# ulimit -Hn 1048579 [root@centos hashhar]# ulimit -Hn 1048579

~~Also the soft limit needs to be increased too to actually allow any running process to make use of the increased hard limit.~~ (EDIT: I see you set both).

I found https://unix.stackexchange.com/questions/447583/ulimit-vs-file-max very helpful.

nproc at some point was dropped from the docs PR I linked. I'm not sure why, none of the comments there refer to that part. So we need an agreement for the right value and also put it in docs. Sounds like a separate issue.

I'm not sure what's the ask here. To set these to the least of 131072 and current fs.nr_open?

I've made changes here to only raise the limit, never lower it, and never raise it above fs.nr_open.

hashhar · 2021-08-09T08:13:57Z

I'm starting to think the better solution is to only verify the ulimits are not insanely low (e.g. maybe values from

trino/testing/trino-product-tests-launcher/src/main/java/io/trino/tests/product/launcher/env/Environment.java

Lines 404 to 411 in 55230fb

    
           private static List<Ulimit> standardUlimits() 
        
           { 
        
               return ImmutableList.of( 
        
                       // Number of open file descriptors 
        
                       new Ulimit("nofile", 65535L, 65535L), 
        
                       // Number of processes 
        
                       new Ulimit("nproc", 8096L, 8096L)); 
        
           }

) and throw an error with more information if not.

The correct place to do this seems to be the launcher since that's not limited to just the RPM and will work regardless of how people are running Trino (Docker, tarball or RPM etc.).

cc: @trinodb/maintainers Please chime in with opinions.

For context the issue is that most OSs default ulimits are very low (1024/4096) which can cause the ServiceLoader to fail but silently swallow too many open files and instead fail with a generic looking undebuggable error.

2021-08-05T09:42:06.396Z        ERROR   main    io.trino.server.Server  No service providers of type io.trino.spi.Plugin
java.lang.IllegalStateException: No service providers of type io.trino.spi.Plugin
        at com.google.common.base.Preconditions.checkState(Preconditions.java:591)
        at io.trino.server.PluginManager.loadPlugin(PluginManager.java:139)
        at io.trino.server.PluginManager.loadPlugin(PluginManager.java:129)
        at io.trino.server.ServerPluginsProvider.loadPlugins(ServerPluginsProvider.java:48)
        at io.trino.server.PluginManager.loadPlugins(PluginManager.java:110)
        at io.trino.server.Server.doStart(Server.java:122)
        at io.trino.server.Server.lambda$start$0(Server.java:77)
        at io.trino.$gen.Trino_360_e_0____20210805_094147_1.run(Unknown Source)
        at io.trino.server.Server.start(Server.java:77)

mosabua · 2021-08-09T15:27:33Z

I think we should not modify the system, but instead in the launcher verify everything is sufficient and if not - report error and fail start up. We dont automatically install / upgrade / downgrade Python or Java either. And that verification should be compatible so that it works with tarball, rpm and on containers and is independent of the Linux distro.

This change downgrades the ulimit for systems that might have higher values for some good reason ..

nineinchnick · 2021-08-09T16:22:49Z

This change downgrades the ulimit for systems that might have higher values for some good reason ..

@mosabua I've made changes here to only raise the limit, never lower it, and never raise it above fs.nr_open.

mosabua · 2021-08-11T18:48:10Z

Looks good now.. ultimately I still think this should go into the validation in the launcher script. Could be a follow up task and then we remove this here again.

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino

findepi · 2021-08-23T12:07:27Z

cc @losipiuk

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino

findepi · 2021-08-23T15:04:08Z

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino

+if max_limit=$(cat /proc/sys/fs/nr_open) && [ "$files_limit" -gt "$max_limit" ]; then
+    files_limit=$max_limit
+fi
+ulimit -Hn "$files_limit"


nit: When the current_limit is gt what we want, we should not invoke ulimit at all.
Avoid calling ulimit unnecessarily, as it could, in theory, fail.

We'd have to either:

conditionally call ulimit -Hn and always call ulimit -Sn

also read the current value of ulimit -S and conditionally call both ulimit -Hn and ulimit -Sn

always call ulimit -n - I just checked the manual, and when neither -H nor -S is specified, it sets both limits.

To keep things simple, I'd assume ulimit is declarative and always call it. It's a bash built-in, so we don't even start another process here. I changed this to be a single call.

hashhar

@findepi i dont agree about doing this in RPM. This problem is general and affects every deployment mechanism like tarball, RPM and Docker.

I think the better solution is for the launcher to verify the limits are at-least some sane value.

hashhar · 2021-08-27T09:49:05Z

Turns out we already have validation for this (thanks to Piotr for pointing to it) in place at

trino/core/trino-main/src/main/java/io/trino/server/TrinoSystemRequirements.java

Lines 126 to 131 in 66f7d23

    
           if (maxFileDescriptorCount.getAsLong() < MIN_FILE_DESCRIPTORS) { 
        
               failRequirement("Trino requires at least %s file descriptors (found %s)", MIN_FILE_DESCRIPTORS, maxFileDescriptorCount.getAsLong()); 
        
           } 
        
           if (maxFileDescriptorCount.getAsLong() < RECOMMENDED_FILE_DESCRIPTORS) { 
        
               warnRequirement("Current OS file descriptor limit is %s. Trino recommends at least %s", maxFileDescriptorCount.getAsLong(), RECOMMENDED_FILE_DESCRIPTORS); 
        
           }

which runs as the first thing during Server startup and gives useful error messages.

I think we can close this PR now.

findepi · 2021-08-27T12:39:36Z

I think we can close this PR now.

The added value of the change is that RPM 'just works' instead of having user to update things. I don't feel strongly about this though.

nineinchnick · 2021-12-16T14:17:43Z

@hashhar @findepi this is good to go.

hashhar

Looks good % comment + a justification in commit msg.

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino

hashhar

Thanks

Make sure the init script installed by the RPM package is setting the maximum number of open files as recommended in the documentation. If the value is too low, a warning would be logged. But if there's a large number of connectors present, the server would fail to start. Any limits set in `/etc/security/limits.conf` would override these defaults.

cla-bot bot added the cla-signed label Aug 6, 2021

hashhar reviewed Aug 6, 2021

View reviewed changes

nineinchnick force-pushed the ulimit-in-rpm branch from db05794 to e2a446c Compare August 9, 2021 16:19

nineinchnick force-pushed the ulimit-in-rpm branch 2 times, most recently from 7741416 to a478157 Compare August 9, 2021 16:24

findepi reviewed Aug 23, 2021

View reviewed changes

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino Outdated Show resolved Hide resolved

findepi reviewed Aug 23, 2021

View reviewed changes

findepi approved these changes Aug 23, 2021

View reviewed changes

nineinchnick force-pushed the ulimit-in-rpm branch from a478157 to d820ca2 Compare August 24, 2021 08:30

findepi approved these changes Aug 24, 2021

View reviewed changes

hashhar requested changes Aug 27, 2021

View reviewed changes

hashhar approved these changes Aug 27, 2021

View reviewed changes

hashhar reviewed Dec 16, 2021

View reviewed changes

core/trino-server-rpm/src/main/resources/dist/etc/init.d/trino Show resolved Hide resolved

nineinchnick force-pushed the ulimit-in-rpm branch 2 times, most recently from 0d1a71d to e7342b8 Compare December 17, 2021 10:22

hashhar approved these changes Dec 17, 2021

View reviewed changes

nineinchnick force-pushed the ulimit-in-rpm branch from e7342b8 to 5adb595 Compare February 1, 2022 13:45

github-actions bot added the docs label Feb 1, 2022

hashhar merged commit 11688a4 into trinodb:master Feb 1, 2022

mosabua mentioned this pull request Feb 2, 2022

Add Trino 370 release notes #10793

Merged

github-actions bot added this to the 370 milestone Feb 2, 2022

hashhar mentioned this pull request Feb 2, 2022

Release notes for 370 #10794

Closed

nineinchnick deleted the ulimit-in-rpm branch November 2, 2022 08:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set higher open files limit in init script #8819

Set higher open files limit in init script #8819

nineinchnick commented Aug 6, 2021

hashhar left a comment

hashhar Aug 6, 2021 •

edited

Loading

nineinchnick Aug 6, 2021

nineinchnick Aug 6, 2021

hashhar Aug 6, 2021 •

edited

Loading

hashhar Aug 6, 2021 •

edited

Loading

nineinchnick Aug 6, 2021 •

edited

Loading

nineinchnick Aug 9, 2021

hashhar commented Aug 9, 2021 •

edited

Loading

mosabua commented Aug 9, 2021

nineinchnick commented Aug 9, 2021

mosabua commented Aug 11, 2021

findepi commented Aug 23, 2021

findepi Aug 23, 2021

nineinchnick Aug 24, 2021

hashhar left a comment

hashhar commented Aug 27, 2021

findepi commented Aug 27, 2021

nineinchnick commented Dec 16, 2021

hashhar left a comment

hashhar left a comment

Set higher open files limit in init script #8819

Set higher open files limit in init script #8819

Conversation

nineinchnick commented Aug 6, 2021

hashhar left a comment

Choose a reason for hiding this comment

hashhar Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

nineinchnick Aug 6, 2021

Choose a reason for hiding this comment

nineinchnick Aug 6, 2021

Choose a reason for hiding this comment

hashhar Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

hashhar Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

nineinchnick Aug 6, 2021 • edited Loading

Choose a reason for hiding this comment

nineinchnick Aug 9, 2021

Choose a reason for hiding this comment

hashhar commented Aug 9, 2021 • edited Loading

mosabua commented Aug 9, 2021

nineinchnick commented Aug 9, 2021

mosabua commented Aug 11, 2021

findepi commented Aug 23, 2021

findepi Aug 23, 2021

Choose a reason for hiding this comment

nineinchnick Aug 24, 2021

Choose a reason for hiding this comment

hashhar left a comment

Choose a reason for hiding this comment

hashhar commented Aug 27, 2021

findepi commented Aug 27, 2021

nineinchnick commented Dec 16, 2021

hashhar left a comment

Choose a reason for hiding this comment

hashhar left a comment

Choose a reason for hiding this comment

hashhar Aug 6, 2021 •

edited

Loading

hashhar Aug 6, 2021 •

edited

Loading

hashhar Aug 6, 2021 •

edited

Loading

nineinchnick Aug 6, 2021 •

edited

Loading

hashhar commented Aug 9, 2021 •

edited

Loading