Trouble with Memory allocation

[Thom-Teears]

Hello,
I would like to use sdmTMB to create abundance indices in multiple regions using CPUE data. The model is specified as shown below, the dataset has been subsampled to ~500,000 observations, and the mesh has 970 knots and has boundaries for land masses. I get the following error when trying to fit the model:

Error in MakeADFunObject(data, parameters, reportenv, ADreport = ADreport, :
Memory allocation fail in function 'MakeADFunObject'

 I am trying to reduce the knots but I wonder if there were some other tricks to help the model with memory allocation. Any help would be greatly appreciated.

Thanks,
Thom

fit<- sdmTMB(
Response_variable ~ FLAG + s(HBF.ratio, k = 3),
time = "Year",
data = DG,
mesh = mesh,
#spatial_varying = ~ 0 + ENSO,
spatial = "on",
spatiotemporal = "iid",
family = delta_gamma(link1 = "logit", link2 = "log"),
control = sdmTMBcontrol(newton_loops = 3)
)

[seananderson]

That's a fairly large dataset, although I imagine some version of a model should work. A few thoughts:

• A delta model will be larger than a non-delta model since it doubles the number of fixed and random effects. I'm not sure what your response is, but perhaps you could use the Tweedie (continuous) or negative binomial (counts)?
• This wont effect memory use at this stage, but you are very unlikely to need 3 Newton optimization attempts. 1 should do it if you need it at all.
• The size of the mesh will have a major impact. 970 knots is definitely on the larger side. Can you try a less complex mesh at least for now? As long as the model fits well and the estimated range is considerably larger than the triangle length you should be fine.
• A model without spatiotemporal random fields will be much smaller and faster to fit. Often just the spatial fields get you most of the way there. Maybe start there and assess whether you need the spatiotemporal fields by looking at residuals?
• A model with a polynomial instead of a penalized smoother will be a little less complex, but this effect is likely minimal.
• If you are ultimately creating indices for multiple regions, you could consider fitting the multiple regions separately.
• Get more RAM? :)

[Thom-Teears]

Thanks for the quick response and great suggestions. I will give them a try. Many thanks.

[ericward-noaa]

I'm working on a huge sdmTMB model for some fishery dependent data -- we're using knots similar to what you're implementing, and it's a fairly large raw dataset (millions of observations). One additional thing I've been exploring is rather than model the raw observations, rasterizing the raw locations to ~ grid cells (1 or 2km) has a huge effect on resolution / speed

[Thom-Teears]

That sounds like an excellent idea. I'll give it a try as well. Thank you so much for the suggestion.

[Thom-Teears]

I got the warning below. I was wondering if it was possible to run the model longer (i.e., with more newton loops) to improve convergence without restarting the model from the beginning. Thanks.

Warning message:
The model may not have converged. Maximum final gradient: 0.0128090032260388.

[Lewis-Barnett-NOAA]

Yes Thom, see the function run_extra_optimization()

…

On Tue, Jun 20, 2023 at 10:41 PM Thom Teears ***@***.***> wrote: I got the warning below. I was wondering if it was possible to run the model longer (i.e., with more newton loops) to improve convergence without restarting the model from the beginning. Thanks. Warning message: The model may not have converged. Maximum final gradient: 0.0128090032260388. — Reply to this email directly, view it on GitHub <#181 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKMJP4KWB4PDJICB2CFRF3XMKCPPANCNFSM6AAAAAAVDTSSRM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Lewis Barnett, PhD (he/him/his) Research Fish Biologist NOAA Fisheries, Alaska Fisheries Science Center 7600 Sand Point Way NE, Bldg 4 Seattle, Washington 98115 Google Voice: (206) 526-4111

[Thom-Teears]

Oh, excellent. Thanks very much.