Do we have a memory leak ?

[OlivierBondu]

I was trying to play with the GridIn repository out-of-the-box, and I currently have > 40 % of jobs failing (with the default TTBar sample) with exit code 50660: Application terminated by wrapper because using too much RAM (RSS)

[blinkseb]

Is the upper limit still 2 GB? We may have an issue here indeed as python + pyroot tends to use a lot more memory than plain ROOT.

I don't think there's a leak somewhere (at least not in the python code) and C++ code is pretty trivial (only getter). I'll try to investigate a bit but I'm not sure there's more miracle solution here.

I cc @swertz and @BrieucF as they also have issues with pyroot and memory 😅

[OlivierBondu]

yep, it seems it's still 2GB by default (well, 2000MB) according to https://twiki.cern.ch/twiki/bin/view/CMSPublic/CRAB3ConfigurationFile

ok, so we have to work around by splitting into more jobs ?

[blinkseb]

I guess yes. We can also ask for more RAM during job creating. It may restrict the number of available site, but it's still better than having hundreds of jobs per tasks...

[OlivierBondu]

Good to know: the max RAM available seems to be 2.5GB [1]

The failure rate seems lower if we launch one file per job instead of 5 files per job
but this is quite non optimal...

It still does look like a memory leak :(

[1] Warning: task requests 3000 memory but only 2500 is guaranteed to be available. Jobs may not find a site where to run and stay idle forever.