save.dataset - problem with MTG #2914

pstruzik · 2024-09-29T11:54:50Z

Describe the bug
When I process MTG FCI data,, I cannot save composite to file. I,m getting message "Memory protection violation (memory dump)".

To Reproduce

# Your code here
from satpy.scene import Scene
from satpy.resample import get_area_def
from satpy import find_files_and_readers

dirscr = "/home/piotr/EMCinput/MTG/"
fnames=[dirstr + '/*.nc']
files = find_files_and_readers(base_dir=dirstr, reader='fci_l1c_nc')
scn = Scene(filenames=files)
composite = 'natural_color'
area = 'PS_PL'
scn.load([composite])
scn_resampled = scn.resample(area, resampler='nearest', radius_of_influence=5000)
scn_resampled.save_dataset(composite,'./MTG.png')

Expected behavior
File MTG.png saved to actual folder

Actual results
message "Memory protection violation (memory dump)"

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

OS: Ubuntu Linux
Satpy Version: 0.51, 0.44 (both produce the same message)
PyResample Version:
Readers and writers dependencies (when relevant): [run from satpy.utils import check_satpy; check_satpy()]

Additional context
The same code, work perfect on Windows 10 environment.

The text was updated successfully, but these errors were encountered:

mraspaud · 2024-09-29T12:14:15Z

Thanks for the feedback! I never saw such an error unfortunately...
Do the two environments you mentioned have the same amount of memory?

Other reason I could think of is a broken environment...

pstruzik · 2024-09-29T12:26:45Z

On Windows is 16 GB RAM. On Linux is 32 GB RAM. On Linux, I process operationally MSG data without any problems (presented on https://sat365.pl).
I have created two virtual environments with satpy 0.44 (used operationally) and newest satpy 0.51. The result is the same.
Without the line:
scn_resampled.save_dataset(composite,'./MTG.png')
no errors are reported.

Best regards,
Piotr

pnuu · 2024-09-29T12:33:34Z

Even if this doesn't seem to be the case of memory running out, I suggest to try these two things to test are

resampler="gradient_search (requires shapely)
save to geotiff

This way both the resampler and writer are switched, so might help narrow down the source of the error. Both are more efficient than the ones now used.

pstruzik · 2024-09-29T12:43:25Z

I have added:
import shapely

and changed code:

composite = 'natural_color'
area = 'PS_PL'
scn.load([composite])
resampler="gradient_search"
scn_resampled = scn.resample(area, resampler='nearest', radius_of_influence=5000)
scn_resampled.save_dataset(composite,'./MTG.tif')

Result is the same:
(Pytrol10) piotr@piotr-HCAR5000-MI:$ python3.10 Test_pytroll_MTG.py
dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240928172415_IDPFI_OPE_20240928172007_20240928172924_N__C_0105_0000
20240928 1724 MTI1
NatCol_20240928_1724_Eu.jpg
Naruszenie ochrony pamięci (zrzut pamięci)
(Pytrol10) piotr@piotr-HCAR5000-MI:$ python3 Test_pytroll_MTG.py
dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240928172415_IDPFI_OPE_20240928172007_20240928172924_N__C_0105_0000
20240928 1724 MTI1
NatCol_20240928_1724_Eu.jpg
Naruszenie ochrony pamięci (zrzut pamięci)
(Pytrol10) piotr@piotr-HCAR5000-MI:~$ ^C

pstruzik · 2024-09-29T12:48:20Z

The whole code is:

from satpy.scene import Scene
from satpy.resample import get_area_def
from satpy import find_files_and_readers
from datetime import datetime
from PIL import Image
import glob
import warnings
import os
import shutil, sys
import aggdraw
from pydecorate import DecoratorAGG
import hdf5plugin
import pathlib, os
from pathlib import Path
import shapely

#with dask.config.set("array.chunk-size": "32MiB")

warnings.filterwarnings('ignore')
srcdir = "/home/piotr/EMCinput/MTG/"
imgdir = "/home/piotr/EMCdata/images/MTG/"

dir = max(pathlib.Path(srcdir).glob('*/'), key=os.path.getmtime)
#print ("sciezka:")
#print (dir)

dirstr = os.path.abspath (dir)
print ("dirstr=")
print (dirstr)

nazwa = os.path.basename (dir)
#print (nazwa)

Dataa = nazwa [74:82]
Rok= nazwa [74:78]
Mies= nazwa [78:80]
Dzien= nazwa [80:82]
Czas = nazwa [82:86]
Godz= nazwa [82:84]
Min= nazwa [84:86]
msg = nazwa [32:36]
print (Dataa, Czas, msg)

fnames=[dirstr + '/*.nc']
files = find_files_and_readers(base_dir=dirstr, reader='fci_l1c_nc')
scn = Scene(filenames=files)
#print(scn.available_dataset_names())

#Natural Color

plikout = 'NatCol_' + Dataa + '_' + Czas + '_Eu.jpg'
print (plikout)

composite = 'natural_color'
area = 'PS_PL'
scn.load([composite])
resampler="gradient_search"
scn_resampled = scn.resample(area, resampler='nearest', radius_of_influence=5000)
scn_resampled.save_dataset(composite,'./MTG.tif')

pstruzik · 2024-09-29T12:51:03Z

Sorry, two lines corrected:
#resampler="gradient_search"
scn_resampled = scn.resample(area, resampler='gradient_search', radius_of_influence=5000)

strandgren · 2024-09-29T14:31:58Z

Just as a sanity check, what's inside dirstr?

pstruzik · 2024-09-29T14:53:25Z

In dirstr are files dowloaded from EUMETSAT datastore by command:
./eumdac download -c EO:EUM:DAT:0662 --download-coverage Q4 --limit 1 -o /home/piotr/EMCinput/MTG

ls -l:
-rw-rw-r-- 1 piotr piotr 24951868 wrz 29 14:17 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173016_IDPFI_OPE_20240928172643_20240928172729_N__C_0105_0029.nc
-rw-rw-r-- 1 piotr piotr 22551508 wrz 29 14:19 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173020_IDPFI_OPE_20240928172701_20240928172746_N__C_0105_0030.nc
-rw-rw-r-- 1 piotr piotr 21806372 wrz 29 14:22 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173054_IDPFI_OPE_20240928172711_20240928172755_N__C_0105_0031.nc
-rw-rw-r-- 1 piotr piotr 21288420 wrz 29 14:22 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173056_IDPFI_OPE_20240928172729_20240928172803_N__C_0105_0032.nc
-rw-rw-r-- 1 piotr piotr 21646881 wrz 29 14:23 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173112_IDPFI_OPE_20240928172737_20240928172819_N__C_0105_0033.nc
-rw-rw-r-- 1 piotr piotr 20478130 wrz 29 14:23 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173112_IDPFI_OPE_20240928172811_20240928172849_N__C_0105_0035.nc
-rw-rw-r-- 1 piotr piotr 21265837 wrz 29 14:23 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173113_IDPFI_OPE_20240928172755_20240928172835_N__C_0105_0034.nc
-rw-rw-r-- 1 piotr piotr 18701634 wrz 29 14:24 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173113_IDPFI_OPE_20240928172819_20240928172856_N__C_0105_0036.nc
-rw-rw-r-- 1 piotr piotr 16716521 wrz 29 14:24 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173113_IDPFI_OPE_20240928172835_20240928172908_N__C_0105_0037.nc
-rw-rw-r-- 1 piotr piotr 14045073 wrz 29 14:24 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173120_IDPFI_OPE_20240928172849_20240928172914_N__C_0105_0038.nc
-rw-rw-r-- 1 piotr piotr 10728092 wrz 29 14:26 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173125_IDPFI_OPE_20240928172856_20240928172919_N__C_0105_0039.nc
-rw-rw-r-- 1 piotr piotr 4486674 wrz 29 14:26 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240928173231_IDPFI_OPE_20240928172908_20240928172924_N__C_0105_0040.nc
-rw-rw-r-- 1 piotr piotr 1401598 wrz 29 14:26 W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-TRAIL---NC4E_C_EUMT_20240928172415_IDPFI_OPE_20240928172007_20240928172924_N__C_0105_0041.nc

pstruzik · 2024-09-29T15:12:46Z

In fact, this is to late time to produce 'Natural_color". The 'airmass' composition gives the same message. From those files, with the same script (added deco and coasts), I produced:

pstruzik · 2024-09-29T15:16:39Z

Presented images were produced on Windows-10 by the same script !

pstruzik · 2024-09-29T15:30:58Z

Regarding dirstr, I was to fast with answer. Application eumdac produce the subfolder:
W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240928172415_IDPFI_OPE_20240928172007_20240928172924_N__C_0105_0000

Inside this subfolder are FCI chunks 29-41 listed in my earlier message. My script finds the latest subfolder in /home/piotr/EMCinput/MTG and this path is in dirstr.

mraspaud · 2024-09-30T06:34:29Z

So I don't understand the error message copied a bit higher up, but at that point it sounds really strange. I think it's worth the try recreating the environment from scratch.

pstruzik · 2024-09-30T06:58:51Z

When, I have unblocked warnings, I have got such warning:
:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

mraspaud · 2024-09-30T07:36:06Z

Ok, make sure you have the latest numpy, and that the right numpy is user if you have multiple installations of it (eg conda environment vs linux installation)

pstruzik · 2024-09-30T12:41:27Z

I was playing with script and data and found interesting result. I have deleted all chunks without one - chunk number 38:
W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--CHK-BODY---NC4E_C_EUMT_20240930120044_IDPFI_OPE_20240930115849_20240930115919_N__C_0072_0038.nc

The result was correct (originally Tif, converted to jpg):

pstruzik · 2024-09-30T12:43:15Z

Execution with all warnings:

(Pytroll) piotr@piotr-HCAR5000-MI:~$ python3 Test_pytroll_MTG.py
dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240930115223_IDPFI_OPE_20240930115007_20240930115924_N__C_0072_0000
20240930 1152 MTI1
<frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
NatCol_20240930_1152_Eu.jpg
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/future/geometry/_subset.py:103: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  return area_to_cover.boundary(frequency=boundary_shape, force_clockwise=True)
/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/resample.py:1397: DeprecationWarning: `GradientSearchResampler` is deprecated, please use `create_gradient_search_resampler` instead.
  resampler_instance = resampler_class(source_area, destination_area)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:155: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  x, y = self.area_to_contain.get_edge_bbox_in_projection_coordinates(frequency=10)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:183: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  poly_to_crop = Polygon(zip(*self.area_to_crop.get_edge_bbox_in_projection_coordinates(frequency=10)))
(Pytroll) piotr@piotr-HCAR5000-MI:~$

Edited by @pnuu

pstruzik · 2024-09-30T12:44:56Z

dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240930115223_IDPFI_OPE_20240930115007_20240930115924_N__C_0072_0000
20240930 1152 MTI1
<frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
NatCol_20240930_1152_Eu.jpg
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/future/geometry/_subset.py:103: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  return area_to_cover.boundary(frequency=boundary_shape, force_clockwise=True)
/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/resample.py:1397: DeprecationWarning: `GradientSearchResampler` is deprecated, please use `create_gradient_search_resampler` instead.
  resampler_instance = resampler_class(source_area, destination_area)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:155: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  x, y = self.area_to_contain.get_edge_bbox_in_projection_coordinates(frequency=10)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:183: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  poly_to_crop = Polygon(zip(*self.area_to_crop.get_edge_bbox_in_projection_coordinates(frequency=10)))
(Pytroll) piotr@piotr-HCAR5000-MI:~$

Edited by @pnuu

pstruzik · 2024-09-30T12:46:15Z

I don't know, why text is crossout.

pstruzik · 2024-09-30T12:46:53Z

Latest contetn of the script:

from satpy.scene import Scene
from satpy.resample import get_area_def
from satpy import find_files_and_readers
from datetime import datetime
from PIL import Image
import glob
import warnings
import os
import shutil, sys
import aggdraw
from pydecorate import DecoratorAGG
import hdf5plugin
import pathlib, os
from pathlib import Path
import shapely
from pyresample import geometry

#with dask.config.set("array.chunk-size": "32MiB")
  
warnings.filterwarnings('default')
srcdir = "/home/piotr/EMCinput/MTG/"
imgdir = "/home/piotr/EMCdata/images/MTG/"

dir = max(pathlib.Path(srcdir).glob('*/'), key=os.path.getmtime)
#print ("sciezka:")
#print (dir)

dirstr = os.path.abspath (dir)
print ("dirstr=")
print (dirstr)

nazwa = os.path.basename (dir)
#print (nazwa)

Dataa = nazwa [74:82]
Rok= nazwa [74:78]
Mies= nazwa [78:80]
Dzien= nazwa [80:82]
Czas = nazwa [82:86]
Godz= nazwa [82:84]
Min= nazwa [84:86]
msg = nazwa [32:36]
print (Dataa, Czas, msg)

fnames=[dirstr + '/*.nc']
files = find_files_and_readers(base_dir=dirstr, reader='fci_l1c_nc')
scn = Scene(filenames=files)
#print(scn.available_dataset_names())


#Natural Color
     
plikout = 'NatCol_' + Dataa + '_' + Czas + '_Eu.jpg'
print (plikout)

composite = 'natural_color'
#area = 'PS_PL'
area = 'euro1'
scn.load([composite])
#scn_resampled = scn.resample(area, resampler='nearest', radius_of_influence=5000)
scn_resampled = scn.resample(area, resampler='gradient_search', radius_of_influence=5000)
scn_resampled.save_dataset(composite,'./MTG.tif')

Edited by @pnuu

pnuu · 2024-09-30T13:00:25Z

I don't know, why text is crossout.

Enclose your code in a code block so the autoformatting doesn't mess it up. That is, add three ` characters before and after the code so it ends up looking like this:

print(foo)

pstruzik · 2024-09-30T13:04:35Z

Thanks for help.

from satpy.scene import Scene
from satpy.resample import get_area_def
from satpy import find_files_and_readers
from datetime import datetime
from PIL import Image
import glob
import warnings
import os
import shutil, sys
import aggdraw
from pydecorate import DecoratorAGG
import hdf5plugin
import pathlib, os
from pathlib import Path
import shapely
from pyresample import geometry

#with dask.config.set("array.chunk-size": "32MiB")
  
warnings.filterwarnings('default')
srcdir = "/home/piotr/EMCinput/MTG/"
imgdir = "/home/piotr/EMCdata/images/MTG/"

dir = max(pathlib.Path(srcdir).glob('*/'), key=os.path.getmtime)
#print ("sciezka:")
#print (dir)

dirstr = os.path.abspath (dir)
print ("dirstr=")
print (dirstr)

nazwa = os.path.basename (dir)
#print (nazwa)

Dataa = nazwa [74:82]
Rok= nazwa [74:78]
Mies= nazwa [78:80]
Dzien= nazwa [80:82]
Czas = nazwa [82:86]
Godz= nazwa [82:84]
Min= nazwa [84:86]
msg = nazwa [32:36]
print (Dataa, Czas, msg)

fnames=[dirstr + '/*.nc']
files = find_files_and_readers(base_dir=dirstr, reader='fci_l1c_nc')
scn = Scene(filenames=files)
#print(scn.available_dataset_names())


#Natural Color
     
plikout = 'NatCol_' + Dataa + '_' + Czas + '_Eu.jpg'
print (plikout)

composite = 'natural_color'
#area = 'PS_PL'
area = 'euro1'
scn.load([composite])
#scn_resampled = scn.resample(area, resampler='nearest', radius_of_influence=5000)
scn_resampled = scn.resample(area, resampler='gradient_search', radius_of_influence=5000)
scn_resampled.save_dataset(composite,'./MTG.tif')

pnuu · 2024-09-30T13:05:56Z

I modified the last three entries to format the code/output properly.

The only thing I see is what @mraspaud mentioned about Numpy. It could be the version is propably different to the one which the gradient search extension was compiled with and raises the warning. This shouldn't affect the code exectution though.

Could one of the other segments be corrupted causing a problem in the NetCDF library?

pstruzik · 2024-09-30T13:08:31Z

And execution:

(Pytroll) piotr@piotr-HCAR5000-MI:~$ python3 Test_pytroll_MTG.py
dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240930115223_IDPFI_OPE_20240930115007_20240930115924_N__C_0072_0000
20240930 1152 MTI1
<frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
NatCol_20240930_1152_Eu.jpg
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/future/geometry/_subset.py:103: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  return area_to_cover.boundary(frequency=boundary_shape, force_clockwise=True)
/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/resample.py:1397: DeprecationWarning: `GradientSearchResampler` is deprecated, please use `create_gradient_search_resampler` instead.
  resampler_instance = resampler_class(source_area, destination_area)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:155: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  x, y = self.area_to_contain.get_edge_bbox_in_projection_coordinates(frequency=10)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:183: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  poly_to_crop = Polygon(zip(*self.area_to_crop.get_edge_bbox_in_projection_coordinates(frequency=10)))
(Pytroll) piotr@piotr-HCAR5000-MI:~$

pstruzik · 2024-09-30T13:09:15Z

Pytrol10) piotr@piotr-HCAR5000-MI:~$ pip install numpy
Requirement already satisfied: numpy in ./Pytrol10/lib/python3.10/site-packages (2.1.1)

pstruzik · 2024-09-30T13:12:06Z

Sorry, last message was from the second environment where I tried Python 10.
In first environment, I have Python 12.3

(Pytroll) piotr@piotr-HCAR5000-MI:~$ pip install numpy
Requirement already satisfied: numpy in ./Pytroll/lib/python3.12/site-packages (2.0.0)

and different version ov numpy. The processing results are the same.

pstruzik · 2024-09-30T13:14:37Z

On Windows, where everythig is runnig perfectly, I have numpy 1.26.4.

mraspaud · 2024-09-30T13:18:42Z

Can you try changing the numpy version to 1.x to see if it helps?

pstruzik · 2024-09-30T13:22:50Z

Installed numpy 1.26.4.
Execution:

(Pytroll) piotr@piotr-HCAR5000-MI:~$ python3 Test_pytroll_MTG.py
dirstr=
/home/piotr/EMCinput/MTG/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD--x-x---x_C_EUMT_20240930130234_IDPFI_OPE_20240930130007_20240930130924_N__C_0079_0000
20240930 1302 MTI1
<frozen importlib._bootstrap>:488: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject
NatCol_20240930_1302_Eu.jpg
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/future/geometry/_subset.py:103: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  return area_to_cover.boundary(frequency=boundary_shape, force_clockwise=True)
/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/resample.py:1397: DeprecationWarning: `GradientSearchResampler` is deprecated, please use `create_gradient_search_resampler` instead.
  resampler_instance = resampler_class(source_area, destination_area)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:155: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  x, y = self.area_to_contain.get_edge_bbox_in_projection_coordinates(frequency=10)
/home/piotr/Pytroll/lib/python3.12/site-packages/pyresample/slicer.py:183: PendingDeprecationWarning: The `frequency` argument is pending deprecation, use `vertices_per_side` instead
  poly_to_crop = Polygon(zip(*self.area_to_crop.get_edge_bbox_in_projection_coordinates(frequency=10)))
Traceback (most recent call last):
  File "/home/piotr/Test_pytroll_MTG.py", line 62, in <module>
    scn_resampled.save_dataset(composite,'./MTG.tif')
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/scene.py", line 1221, in save_dataset
    return writer.save_dataset(self[dataset_id],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/writers/__init__.py", line 885, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/satpy/writers/geotiff.py", line 278, in save_image
    return img.save(filename, fformat='tif', driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/trollimage/xrimage.py", line 248, in save
    return self.rio_save(filename, fformat=fformat, driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/trollimage/xrimage.py", line 432, in rio_save
    res = da.store(*to_store)
          ^^^^^^^^^^^^^^^^^^^
  File "/home/piotr/Pytroll/lib/python3.12/site-packages/dask/array/core.py", line 1229, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
Naruszenie ochrony pamięci (zrzut pamięci)
(Pytroll) piotr@piotr-HCAR5000-MI:~$

No result. Last message in Polish is again "Memory protection violated (memory dump).

mraspaud · 2024-09-30T14:32:02Z

ok, so I'm still worried about the numpy binary compatibility. Would it be possible for you to install pyresample from source and rebuild the cython extension to see if it helps?

djhoese · 2024-09-30T14:37:04Z

Not that the numpy warning shouldn't be dealt with in the long run, but we used to get that all the time. I'm less concerned about that. I'm wondering if this memory protection violation is actually just coming from running out of memory on your machine. I haven't read this entire thread so sorry if this has already been discussed.

mraspaud · 2024-09-30T15:46:24Z

@djhoese to summarize, the linux with 32GB memory doesn't work, while the windows with 16GB works.

mraspaud · 2024-09-30T15:48:26Z

@pstruzik You showed it was working with only one segment. Have you tried running all segments one by one to see if one of them was corrupted? If not, how many segments to you need to run into the error?

pstruzik · 2024-10-01T07:41:00Z

Problem was solved (;-).
I have installed the same versions od numpy and netCDF4 as on Windows:
numpy 1.26.4
netCDF4 1.6.3

Result:

mraspaud · 2024-10-01T09:34:58Z

Great that you got it to work!
So you think it's the numpy version which is the problem, or the netcdf4?

pstruzik · 2024-10-01T09:38:11Z

First, I installed netCDF4 1.6.3 but was conflict with numpy. After installation of numpy 1.26.4 everything was OK. I am testing now different composition and areas. No problems until now !

mraspaud · 2024-10-01T09:39:23Z

Do you know what versions of numpy and netcdf4 you had when it crashed?

pstruzik · 2024-10-01T09:41:50Z

netCDF4 1.7.1
numpy 2.1.1

mraspaud · 2024-10-01T09:42:28Z

Thanks, we will try this locally.

pstruzik · 2024-10-01T09:43:26Z

Thanks for constructive discussion.

mraspaud · 2024-10-01T21:18:36Z

Someone in the community tried with the netcdf 1.7.1 and numpy 2.1.1 and didn't have any problems... So maybe it's something else. Anyway, as it seems to work for you now, I'm closing this issue for now.

mraspaud closed this as completed Oct 1, 2024

save.dataset - problem with MTG #2914

save.dataset - problem with MTG #2914

Comments

pstruzik commented Sep 29, 2024

mraspaud commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pnuu commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pstruzik commented Sep 29, 2024

strandgren commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pstruzik commented Sep 29, 2024

pstruzik commented Sep 29, 2024

mraspaud commented Sep 30, 2024

pstruzik commented Sep 30, 2024

mraspaud commented Sep 30, 2024

pstruzik commented Sep 30, 2024

pstruzik commented Sep 30, 2024 • edited by pnuu Loading

pstruzik commented Sep 30, 2024 • edited by pnuu Loading

pstruzik commented Sep 30, 2024

pstruzik commented Sep 30, 2024 • edited by pnuu Loading

pnuu commented Sep 30, 2024

pstruzik commented Sep 30, 2024

pnuu commented Sep 30, 2024

pstruzik commented Sep 30, 2024

pstruzik commented Sep 30, 2024

pstruzik commented Sep 30, 2024

pstruzik commented Sep 30, 2024

mraspaud commented Sep 30, 2024

pstruzik commented Sep 30, 2024

mraspaud commented Sep 30, 2024

djhoese commented Sep 30, 2024

mraspaud commented Sep 30, 2024

mraspaud commented Sep 30, 2024

pstruzik commented Oct 1, 2024

mraspaud commented Oct 1, 2024

pstruzik commented Oct 1, 2024

mraspaud commented Oct 1, 2024

pstruzik commented Oct 1, 2024

mraspaud commented Oct 1, 2024

pstruzik commented Oct 1, 2024

mraspaud commented Oct 1, 2024

pstruzik commented Sep 30, 2024 •

edited by pnuu

Loading

pstruzik commented Sep 30, 2024 •

edited by pnuu

Loading

pstruzik commented Sep 30, 2024 •

edited by pnuu

Loading