Failure to produce a Grid with Spline due to a MemoryError

[AlfredManda]

I am using data with 50m x 50 m data and the data is huge. However, I trimmed to 1/10th its original size but the error was still appearing. Here is the error: MemoryError: Unable to allocate 60.6 GiB for an array with shape (90150, 90150) and data type float64. How do I deal with error?

[leouieda]

👋🏽 Hi @AlfredManda, I'm afraid you only have 2 options here:

1. Buy more RAM (you'll need at least 60Gb for what you want to do)
2. Use a different interpolation method. The Verde Spline is good for sparse data of moderate size (a few 10,000s). If you're interpolating between grids, use the built-in interpolation in xarray instead. Even then, anything other than nearest neighbours will be very limited for larger grids like yours.

The next version of Verde will include a fast nearest neighbours interpolator which would be great for making Lidar DEMs for example. If you want to use it, you'll have to install the development version of Verde from GitHub. See https://www.fatiando.org/verde/dev/api/generated/verde.KNeighbors.html#verde.KNeighbors and https://www.fatiando.org/verde/dev/install.html

Hope this helps!

[leouieda]

@AlfredManda if this resolves your questions could you please "Mark as answer"? That makes it easier for other people to find it.

By the way, we've been trying to get a memory efficient spline going for a while but it's a tough problem to solve.

[AlfredManda]

Hi,

Sorry for the late reply this worked as an answer, however, I thought I had marked it as an answer.

[mdtanker]

@leouieda does the dtype affect the memory usage much? If we knew our data was compatible with a more limited dtype and we were able to change the dtype for the jacobian to something smaller, do you think it would make much difference?