Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RuntimeError while compositing after resampling datasets #2909

Closed
BigShuiTai opened this issue Sep 25, 2024 · 4 comments
Closed

RuntimeError while compositing after resampling datasets #2909

BigShuiTai opened this issue Sep 25, 2024 · 4 comments

Comments

@BigShuiTai
Copy link

BigShuiTai commented Sep 25, 2024 •
edited
Loading

Describe the bug
I got RuntimeError: NetCDF: Not a valid ID while executing scn.save_dataset, and this error only occurred in MTG data

To Reproduce

import glob
import numpy as np
from satpy.scene import Scene
from satpy.utils import debug_on; debug_on()

def main():
    files = glob.glob('./MTG_1200Z/W_XX-EUMETSAT-Darmstadt,IMG+SAT,MTI1+FCI-1C-RRAD-FDHSI-FD*.nc')
    files.sort()
    scn = Scene(files, reader='fci_l1c_nc')
    scn.load(['true_color'])
    scn = scn.resample(scn.finest_area(), resampler='native') # Succeed if I delete this code, same to other composites
    scn.save_dataset('true_color', filename=f'{satellite_name}_true_color.png')

if __name__ == '__main__':
    main()

Expected behavior
Save a true color image.

Actual results

[DEBUG: 2024-09-25 18:10:49 : satpy.readers.yaml_reader] Requested orientation for Dataset None is 'native' (default). No flipping is applied.
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Computing sun zenith angles.
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2024-09-25 18:10:49 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2024-09-25 18:10:49 : h5py._conv] Creating converter from 5 to 7
[DEBUG: 2024-09-25 18:10:49 : h5py._conv] Creating converter from 7 to 5
[DEBUG: 2024-09-25 18:10:49 : h5py._conv] Creating converter from 5 to 7
[INFO: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'rayleigh_only' for 'vis_06'
[INFO: 2024-09-25 18:10:49 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] LUT filename: /root/.local/share/pyspectral/rayleigh_only/rayleigh_lut_us-standard.h5
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Filename: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[WARNING: 2024-09-25 18:10:49 : pyspectral.rsr_reader] rsr data may not be up to date: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[INFO: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Will download from internet...
[INFO: 2024-09-25 18:10:49 : pyspectral.utils] Download RSR files and store in directory /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] RSR URL: https://zenodo.org/records/13833977/files/pyspectral_rsr_data.tgz
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] Destination = /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org:443
[ERROR: 2024-09-25 18:10:49 : pyspectral.rayleigh] No spectral responses for this platform and sensor: MTG-I1 fci
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rayleigh.py", line 306, in _get_rsr_wavelength_from_band_name
    rsr = RelativeSpectralResponse(platform_name, sensor)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 136, in __init__
    self._check_consistent_input()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 149, in _check_consistent_input
    self._get_filename()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 181, in _get_filename
    download_rsr()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 360, in download_rsr
    _download_tarball_and_extract(HTTP_PYSPECTRAL_RSR, filename, dest_dir)
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 410, in _download_tarball_and_extract
    response = requests.get(tarball_url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Could not get the reflectance correction using band name: vis_06
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Will try use the wavelength, however, this may be ambiguous!
[WARNING: 2024-09-25 18:10:49 : pyspectral.rayleigh] A wavelength is provided instead of band name - disregard the relative spectral responses and assume it is the effective wavelength: 0.640000 (micro meter)
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] Band name: 0.64  Effective wavelength: 0.640000um
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Computing sun zenith angles.
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Apply the standard sun-zenith correction [1/cos(sunz)]
[INFO: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'rayleigh_only' for 'vis_05'
[INFO: 2024-09-25 18:10:49 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] LUT filename: /root/.local/share/pyspectral/rayleigh_only/rayleigh_lut_us-standard.h5
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Filename: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[WARNING: 2024-09-25 18:10:49 : pyspectral.rsr_reader] rsr data may not be up to date: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[INFO: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Will download from internet...
[INFO: 2024-09-25 18:10:49 : pyspectral.utils] Download RSR files and store in directory /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] RSR URL: https://zenodo.org/records/13833977/files/pyspectral_rsr_data.tgz
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] Destination = /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org:443
[ERROR: 2024-09-25 18:10:49 : pyspectral.rayleigh] No spectral responses for this platform and sensor: MTG-I1 fci
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rayleigh.py", line 306, in _get_rsr_wavelength_from_band_name
    rsr = RelativeSpectralResponse(platform_name, sensor)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 136, in __init__
    self._check_consistent_input()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 149, in _check_consistent_input
    self._get_filename()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 181, in _get_filename
    download_rsr()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 360, in download_rsr
    _download_tarball_and_extract(HTTP_PYSPECTRAL_RSR, filename, dest_dir)
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 410, in _download_tarball_and_extract
    response = requests.get(tarball_url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Could not get the reflectance correction using band name: vis_05
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Will try use the wavelength, however, this may be ambiguous!
[WARNING: 2024-09-25 18:10:49 : pyspectral.rayleigh] A wavelength is provided instead of band name - disregard the relative spectral responses and assume it is the effective wavelength: 0.510000 (micro meter)
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] Band name: 0.51  Effective wavelength: 0.510000um
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Computing sun zenith angles.
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Apply the standard sun-zenith correction [1/cos(sunz)]
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Applying sun zen correction
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Computing sun zenith angles.
[DEBUG: 2024-09-25 18:10:49 : satpy.modifiers.geometry] Apply the standard sun-zenith correction [1/cos(sunz)]
[INFO: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Removing Rayleigh scattering with atmosphere 'us-standard' and aerosol type 'rayleigh_only' for 'vis_04'
[INFO: 2024-09-25 18:10:49 : pyspectral.rayleigh] Atmosphere chosen: us-standard
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] LUT filename: /root/.local/share/pyspectral/rayleigh_only/rayleigh_lut_us-standard.h5
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Filename: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[WARNING: 2024-09-25 18:10:49 : pyspectral.rsr_reader] rsr data may not be up to date: /root/.local/share/pyspectral/rsr_fci_MTG-I1.h5
[INFO: 2024-09-25 18:10:49 : pyspectral.rsr_reader] Will download from internet...
[INFO: 2024-09-25 18:10:49 : pyspectral.utils] Download RSR files and store in directory /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] RSR URL: https://zenodo.org/records/13833977/files/pyspectral_rsr_data.tgz
[DEBUG: 2024-09-25 18:10:49 : pyspectral.utils] Destination = /root/.local/share/pyspectral
[DEBUG: 2024-09-25 18:10:49 : urllib3.connectionpool] Starting new HTTPS connection (1): zenodo.org:443
[ERROR: 2024-09-25 18:10:49 : pyspectral.rayleigh] No spectral responses for this platform and sensor: MTG-I1 fci
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rayleigh.py", line 306, in _get_rsr_wavelength_from_band_name
    rsr = RelativeSpectralResponse(platform_name, sensor)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 136, in __init__
    self._check_consistent_input()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 149, in _check_consistent_input
    self._get_filename()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/rsr_reader.py", line 181, in _get_filename
    download_rsr()
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 360, in download_rsr
    _download_tarball_and_extract(HTTP_PYSPECTRAL_RSR, filename, dest_dir)
  File "/usr/local/lib/python3.11/dist-packages/pyspectral/utils.py", line 410, in _download_tarball_and_extract
    response = requests.get(tarball_url)
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/requests/adapters.py", line 517, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Could not get the reflectance correction using band name: vis_04
[WARNING: 2024-09-25 18:10:49 : satpy.modifiers.atmosphere] Will try use the wavelength, however, this may be ambiguous!
[WARNING: 2024-09-25 18:10:49 : pyspectral.rayleigh] A wavelength is provided instead of band name - disregard the relative spectral responses and assume it is the effective wavelength: 0.444000 (micro meter)
[DEBUG: 2024-09-25 18:10:49 : pyspectral.rayleigh] Band name: 0.444  Effective wavelength: 0.444000um
/usr/local/lib/python3.11/dist-packages/satpy/composites/spectral.py:68: UserWarning: 'green_nocorr' is a deprecated composite. Use the equivalent 'hybrid_green_nocorr' instead.
  return super().__call__((new_channel,), **attrs)
[DEBUG: 2024-09-25 18:10:49 : satpy.composites] Computing sun zenith angles.
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/fci.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Data for DataID(name='true_color_cira', resolution=1000) will be enhanced with options:
        [{'name': 'color', 'method': <function color_conversion at 0x7a8c4c15c860>}, {'name': 'alog_stretch', 'method': <function alog_stretch at 0x7a8c4c297f60>}, {'name': 'stretch', 'method': <function stretch at 0x7a8c4c23c9a0>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0.24, 0.24, 0.24], 'max_stretch': [1.03, 1.03, 1.03]}}]
[DEBUG: 2024-09-25 18:10:50 : satpy.enhancements] Applying the alog-stretch
[DEBUG: 2024-09-25 18:10:50 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0.24, 0.24, 0.24], 'max_stretch': [1.03, 1.03, 1.03]}
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/fci.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Data for DataID(name='true_color_nocorr', resolution=1000) will be enhanced with options:
        [{'name': 'alog_stretch', 'method': <function alog_stretch at 0x7a8c4c297f60>}, {'name': 'stretch', 'method': <function stretch at 0x7a8c4c23c9a0>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0.24, 0.24, 0.24], 'max_stretch': [1.03, 1.03, 1.03]}}]
[DEBUG: 2024-09-25 18:10:50 : satpy.enhancements] Applying the alog-stretch
[DEBUG: 2024-09-25 18:10:50 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0.24, 0.24, 0.24], 'max_stretch': [1.03, 1.03, 1.03]}
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_08', wavelength=WavelengthRange(min=0.815, central=0.865, max=0.915, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=())
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_06', wavelength=WavelengthRange(min=0.59, central=0.64, max=0.69, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_05', wavelength=WavelengthRange(min=0.47, central=0.51, max=0.55, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_08', wavelength=WavelengthRange(min=0.815, central=0.865, max=0.915, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='green', resolution=1000)
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected',))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='vis_04', wavelength=WavelengthRange(min=0.384, central=0.444, max=0.504, unit='µm'), resolution=1000, calibration=<1>, modifiers=('sunz_corrected', 'rayleigh_corrected'))
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='true_color_cira', resolution=1000)
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='green_nocorr', resolution=1000)
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Unloading dataset: DataID(name='true_color_nocorr', resolution=1000)
[DEBUG: 2024-09-25 18:10:50 : satpy.scene] Resampling DataID(name='true_color_hybrid', resolution=1000)
[DEBUG: 2024-09-25 18:10:50 : pyresample.geometry] Projections for data and slice areas are identical: PROJCRS["unknown",BASEGEOGCRS["unknown",DATUM["unknown",ELLIPSOID["WGS 84",6378137,298.25722356301,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8901]]],CONVERSION["unknown",METHOD["Geostationary Satellite (Sweep Y)"],PARAMETER["Longitude of natural origin",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8802]],PARAMETER["Satellite Height",35786400,LENGTHUNIT["metre",1,ID["EPSG",9001]]],PARAMETER["False easting",0,LENGTHUNIT["metre",1],ID["EPSG",8806]],PARAMETER["False northing",0,LENGTHUNIT["metre",1],ID["EPSG",8807]]],CS[Cartesian,2],AXIS["(E)",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["(N)",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]]
MTG-I1 2024/09/24 12:00:00Z
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Reading ['/usr/local/lib/python3.11/dist-packages/satpy/etc/writers/simple_image.yaml']
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/generic.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Adding enhancement configuration from file: /usr/local/lib/python3.11/dist-packages/satpy/etc/enhancements/fci.yaml
[DEBUG: 2024-09-25 18:10:50 : satpy.writers] Data for DataID(name='true_color_hybrid', resolution=1000) will be enhanced with options:
        [{'name': 'stretch', 'method': <function stretch at 0x7a8c4c23c9a0>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0.0, 0.0, 0.0], 'max_stretch': [1.0, 1.0, 1.0]}}]
[DEBUG: 2024-09-25 18:10:50 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0.0, 0.0, 0.0], 'max_stretch': [1.0, 1.0, 1.0]}
[DEBUG: 2024-09-25 18:10:50 : satpy.writers.simple_image] Saving to image: MTG-I1_true_color.png
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing BlpImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing BmpImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing BufrStubImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing CurImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing DcxImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing DdsImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing EpsImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing FitsImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing FitsStubImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing FliImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing FpxImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing FtexImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing GbrImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing GifImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing GribStubImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing Hdf5StubImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing IcnsImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing IcoImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing ImImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing ImtImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing IptcImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing JpegImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing Jpeg2KImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing McIdasImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing MicImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing MpegImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing MpoImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing MspImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PalmImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PcdImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PcxImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PdfImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PixarImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PngImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PpmImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing PsdImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing SgiImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing SpiderImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing SunImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing TgaImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing TiffImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing WebPImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing WmfImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing XbmImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing XpmImagePlugin
[DEBUG: 2024-09-25 18:10:50 : PIL.Image] Importing XVThumbImagePlugin
Traceback (most recent call last):
  File "/www/wwwroot/test/test_mtg.py", line 21, in <module>
    main()
  File "/www/wwwroot/test/test_mtg.py", line 16, in main
    scn.save_dataset('true_color_hybrid', filename=f'{satellite_name}_true_color.png')
  File "/usr/local/lib/python3.11/dist-packages/satpy/scene.py", line 1234, in save_dataset
    return writer.save_dataset(self[dataset_id],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/satpy/writers/__init__.py", line 885, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/satpy/writers/simple_image.py", line 67, in save_image
    return img.save(filename, compute=compute, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/trollimage/xrimage.py", line 255, in save
    return self.pil_save(filename, fformat, fill_value,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/trollimage/xrimage.py", line 471, in pil_save
    return delay.compute()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/base.py", line 314, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/base.py", line 599, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/threaded.py", line 89, in get
    results = get_async(
              ^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/usr/local/lib/python3.11/dist-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/usr/local/lib/python3.11/dist-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/optimization.py", line 990, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/dask/array/core.py", line 120, in getter
    c = a[b]
        ~^^^
  File "src/netCDF4/_netCDF4.pyx", line 4916, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 4519, in netCDF4._netCDF4.Variable.shape.__get__
  File "src/netCDF4/_netCDF4.pyx", line 4467, in netCDF4._netCDF4.Variable._getdims
  File "src/netCDF4/_netCDF4.pyx", line 2014, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Not a valid ID

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

  • OS: Ubuntu 24.04
  • Satpy Version: 0.51.0
  • PyResample Version: 1.30.0
@pnuu
Copy link
Member

pnuu commented Sep 25, 2024

Did you notice this error?

requests.exceptions.SSLError: HTTPSConnectionPool(host='zenodo.org', port=443): Max retries exceeded with url: /records/13833977/files/pyspectral_rsr_data.tgz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')))

I'm pretty certain that the NetCDF access error is coming because the download failed but for some reason the processing still continued. When Rayleigh corrector is then trying to access the file when computation is triggered in save_datasets() the file doesn't exist and the error is raised.

Also, you should not be doing this as root user...

@pnuu
Copy link
Member

pnuu commented Sep 25, 2024 •
edited
Loading

Ah, now I see. Do not re-use the scn variable for the resampled Scene. Instead do this:

    resampled_scn = scn.resample(scn.finest_area(), resampler='native') # Succeed if I delete this code, same to other composites
    resampled_scn.save_dataset('true_color', filename=f'{satellite_name}_true_color.png')

EDIT:

The explanation is that by overwriting scn you are losing the filehandlers and thus access to the data.

@BigShuiTai
Copy link
Author

Ah, now I see. Do not re-use the scn variable for the resampled Scene. Instead do this:

    resampled_scn = scn.resample(scn.finest_area(), resampler='native') # Succeed if I delete this code, same to other composites
    resampled_scn.save_dataset('true_color', filename=f'{satellite_name}_true_color.png')

EDIT:

The explanation is that by overwriting scn you are losing the filehandlers and thus access to the data.

I succeed after changing the variable name. Very thanks to your advice!

@pnuu
Copy link
Member

pnuu commented Sep 25, 2024

Ok, closing as it now works 👍

@pnuu pnuu closed this as completed Sep 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pnuu @BigShuiTai