Relax nanosecond datetime restriction in CF time decoding

doc/api.rst

@@ -52,6 +52,7 @@ Creating a dataset
 
    Dataset
    decode_cf
+   CFDatetimeCoder
 
 Attributes
 ----------

doc/internals/index.rst

@@ -26,3 +26,4 @@ The pages in this section are intended for:
    how-to-add-new-backend
    how-to-create-custom-index
    zarr-encoding-spec
+   time-coding

doc/user-guide/time-series.rst

@@ -21,20 +21,31 @@ core functionality.
 Creating datetime64 data
 ------------------------
 
-Xarray uses the numpy dtypes ``datetime64[ns]`` and ``timedelta64[ns]`` to
-represent datetime data, which offer vectorized (if sometimes buggy) operations
-with numpy and smooth integration with pandas.
+Xarray uses the numpy dtypes ``datetime64[unit]`` and ``timedelta64[unit]``
+(where unit is anything of "s", "ms", "us" and "ns") to represent datetime
+data, which offer vectorized operations with numpy and smooth integration with pandas.
 
 To convert to or create regular arrays of ``datetime64`` data, we recommend
 using :py:func:`pandas.to_datetime` and :py:func:`pandas.date_range`:
 
 .. ipython:: python
 
     pd.to_datetime(["2000-01-01", "2000-02-02"])
+    pd.DatetimeIndex(
+        ["2000-01-01 00:00:00", "2000-02-02 00:00:00"], dtype="datetime64[s]"
+    )
     pd.date_range("2000-01-01", periods=365)
+    pd.date_range("2000-01-01", periods=365, unit="s")
+
+.. note::
+    Care has to be taken to create the output with the wanted resolution.
+    For :py:func:`pandas.date_range` the ``unit``-kwarg has to be specified
+    and for :py:func:`pandas.to_datetime` the selection of the resolution
+    isn't possible at all. For that :py:class:`pd.DatetimeIndex` can be used
+    directly.
 
 Alternatively, you can supply arrays of Python ``datetime`` objects. These get
-converted automatically when used as arguments in xarray objects:
+converted automatically when used as arguments in xarray objects (with us-resolution):
 
 .. ipython:: python
 
@@ -51,7 +62,7 @@ attribute like ``'days since 2000-01-01'``).
 .. note::
 
    When decoding/encoding datetimes for non-standard calendars or for dates
-   before year 1678 or after year 2262, xarray uses the `cftime`_ library.
+   before 1582-10-15, xarray uses the `cftime`_ library.
    It was previously packaged with the ``netcdf4-python`` package under the
    name ``netcdftime`` but is now distributed separately. ``cftime`` is an
    :ref:`optional dependency<installing>` of xarray.
@@ -66,17 +77,15 @@ You can manual decode arrays in this form by passing a dataset to
 
     attrs = {"units": "hours since 2000-01-01"}
     ds = xr.Dataset({"time": ("time", [0, 1, 2, 3], attrs)})
+    # Default decoding to 'ns'-resolution
     xr.decode_cf(ds)
+    # Decoding to 's'-resolution
+    coder = xr.CFDatetimeCoder(time_unit="s")
+    xr.decode_cf(ds, decode_times=coder)
 
-One unfortunate limitation of using ``datetime64[ns]`` is that it limits the
-native representation of dates to those that fall between the years 1678 and
-2262. When a netCDF file contains dates outside of these bounds, dates will be
-returned as arrays of :py:class:`cftime.datetime` objects and a :py:class:`~xarray.CFTimeIndex`
-will be used for indexing.  :py:class:`~xarray.CFTimeIndex` enables a subset of
-the indexing functionality of a :py:class:`pandas.DatetimeIndex` and is only
-fully compatible with the standalone version of ``cftime`` (not the version
-packaged with earlier versions ``netCDF4``).  See :ref:`CFTimeIndex` for more
-information.
+From xarray TODO: version the resolution of the dates can be tuned between "s", "ms", "us" and "ns". One limitation of using ``datetime64[ns]`` is that it limits the native representation of dates to those that fall between the years 1678 and 2262, which gets increased significantly with lower resolutions. When a netCDF file contains dates outside of these bounds (or dates < 1582-10-15), dates will be returned as arrays of :py:class:`cftime.datetime` objects and a :py:class:`~xarray.CFTimeIndex` will be used for indexing.
+:py:class:`~xarray.CFTimeIndex` enables a subset of the indexing functionality of a :py:class:`pandas.DatetimeIndex`.
+See :ref:`CFTimeIndex` for more information.
 
 Datetime indexing
 -----------------

doc/user-guide/weather-climate.rst

@@ -10,7 +10,7 @@ Weather and climate data
 
     import xarray as xr
 
-Xarray can leverage metadata that follows the `Climate and Forecast (CF) conventions`_ if present. Examples include :ref:`automatic labelling of plots<plotting>` with descriptive names and units if proper metadata is present and support for non-standard calendars used in climate science through the ``cftime`` module(Explained in the :ref:`CFTimeIndex` section). There are also a number of :ref:`geosciences-focused projects that build on xarray<ecosystem>`.
+Xarray can leverage metadata that follows the `Climate and Forecast (CF) conventions`_ if present. Examples include :ref:`automatic labelling of plots<plotting>` with descriptive names and units if proper metadata is present and support for non-standard calendars used in climate science through the ``cftime`` module (explained in the :ref:`CFTimeIndex` section). There are also a number of :ref:`geosciences-focused projects that build on xarray<ecosystem>`.
 
 .. _Climate and Forecast (CF) conventions: https://cfconventions.org
 
@@ -64,8 +64,7 @@ Through the standalone ``cftime`` library and a custom subclass of
 :py:class:`pandas.Index`, xarray supports a subset of the indexing
 functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for
 dates from non-standard calendars commonly used in climate science or dates
-using a standard calendar, but outside the `nanosecond-precision range`_
-(approximately between years 1678 and 2262).
+using a standard calendar, but outside the `precision range`_ and dates prior 1582-10-15.
 
 .. note::
 
@@ -75,18 +74,14 @@ using a standard calendar, but outside the `nanosecond-precision range`_
    any of the following are true:
 
    - The dates are from a non-standard calendar
-   - Any dates are outside the nanosecond-precision range.
+   - Any dates are outside the nanosecond-precision range (prior xarray version 2024.11)
+   - Any dates are outside the time span limited by the resolution (from xarray version v2024.11)
 
    Otherwise pandas-compatible dates from a standard calendar will be
-   represented with the ``np.datetime64[ns]`` data type, enabling the use of a
-   :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[ns]``
-   and their full set of associated features.
+   represented with the ``np.datetime64[unit]`` data type (where unit can be any of ["s", "ms", "us", "ns"], enabling the use of a :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[unit]`` and their full set of associated features.
 
    As of pandas version 2.0.0, pandas supports non-nanosecond precision datetime
-   values.  For the time being, xarray still automatically casts datetime values
-   to nanosecond-precision for backwards compatibility with older pandas
-   versions; however, this is something we would like to relax going forward.
-   See :issue:`7493` for more discussion.
+   values. From xarray version 2024.11 the relaxed non-nanosecond precision datetime values will be used.
 
 For example, you can create a DataArray indexed by a time
 coordinate with dates from a no-leap calendar and a
@@ -115,7 +110,7 @@ instance, we can create the same dates and DataArray we created above using:
 Mirroring pandas' method with the same name, :py:meth:`~xarray.infer_freq` allows one to
 infer the sampling frequency of a :py:class:`~xarray.CFTimeIndex` or a 1-D
 :py:class:`~xarray.DataArray` containing cftime objects. It also works transparently with
-``np.datetime64[ns]`` and ``np.timedelta64[ns]`` data.
+``np.datetime64`` and ``np.timedelta64`` data (with "s", "ms", "us" or "ns" resolution).
 
 .. ipython:: python
 
@@ -137,7 +132,7 @@ Conversion between non-standard calendar and to/from pandas DatetimeIndexes is
 facilitated with the :py:meth:`xarray.Dataset.convert_calendar` method (also available as
 :py:meth:`xarray.DataArray.convert_calendar`). Here, like elsewhere in xarray, the ``use_cftime``
 argument controls which datetime backend is used in the output. The default (``None``) is to
-use ``pandas`` when possible, i.e. when the calendar is standard and dates are within 1678 and 2262.
+use ``pandas`` when possible, i.e. when the calendar is standard and dates starting with 1582-10-15.
 
 .. ipython:: python
 
@@ -241,6 +236,6 @@ For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports:
 
     da.resample(time="81min", closed="right", label="right", offset="3min").mean()
 
-.. _nanosecond-precision range: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timestamp-limitations
+.. _precision range: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#timestamp-limitations
 .. _ISO 8601 standard: https://en.wikipedia.org/wiki/ISO_8601
 .. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#partial-string-indexing

doc/whats-new.rst

@@ -24,7 +24,8 @@ New Features
 - Better support wrapping additional array types (e.g. ``cupy`` or ``jax``) by calling generalized
   duck array operations throughout more xarray methods. (:issue:`7848`, :pull:`9798`).
   By `Sam Levang <https://github.com/slevang>`_.
-
+- Relax nanosecond datetime restriction in CF time decoding (:issue:`7493`, :pull:`9618`).
+  By `Kai Mühlbauer <https://github.com/kmuehlbauer>`_.
 
 Breaking changes
 ~~~~~~~~~~~~~~~~
@@ -37,7 +38,9 @@ Breaking changes
 
 Deprecations
 ~~~~~~~~~~~~
-
+- Time decoding related kwarg ``use_cftime`` is deprecated. Use keyword argument
+  ``decode_times=CFDatetimeCoder(use_cftime=True)`` in the respective functions
+  instead.
 
 Bug fixes
 ~~~~~~~~~

xarray/__init__.py

@@ -15,6 +15,7 @@
 from xarray.coding.cftime_offsets import cftime_range, date_range, date_range_like
 from xarray.coding.cftimeindex import CFTimeIndex
 from xarray.coding.frequencies import infer_freq
+from xarray.coding.times import CFDatetimeCoder
 from xarray.conventions import SerializationWarning, decode_cf
 from xarray.core.alignment import align, broadcast
 from xarray.core.combine import combine_by_coords, combine_nested
@@ -113,6 +114,7 @@
     "where",
     "zeros_like",
     # Classes
+    "CFDatetimeCoder",
     "CFTimeIndex",
     "Context",
     "Coordinates",

xarray/backends/api.py

@@ -33,6 +33,7 @@
     _normalize_path,
 )
 from xarray.backends.locks import _get_scheduler
+from xarray.coding.times import CFDatetimeCoder
 from xarray.core import indexing
 from xarray.core.combine import (
     _infer_concat_order_from_positions,
@@ -481,7 +482,10 @@ def open_dataset(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -543,9 +547,9 @@ def open_dataset(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -569,6 +573,8 @@ def open_dataset(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -698,7 +704,10 @@ def open_dataarray(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | None = None,
-    decode_times: bool | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | None = None,
     use_cftime: bool | None = None,
     concat_characters: bool | None = None,
@@ -761,9 +770,11 @@ def open_dataarray(
         `missing_value` attribute contains multiple values a warning will be
         issued and all array values matching one of the multiple values will
         be replaced by NA. This keyword may not be supported by all the backends.
-    decode_times : bool, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
+        Pass a mapping, e.g. ``{"my_variable": False}``,
+        to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
     decode_timedelta : bool, optional
         If True, decode variables and coordinates with time units in
@@ -781,6 +792,8 @@ def open_dataarray(
         represented using ``np.datetime64[ns]`` objects.  If False, always
         decode times to ``np.datetime64[ns]`` objects; if this is not possible
         raise an error. This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -903,7 +916,10 @@ def open_datatree(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -961,9 +977,9 @@ def open_datatree(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -987,6 +1003,8 @@ def open_datatree(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and
@@ -1118,7 +1136,10 @@ def open_groups(
     cache: bool | None = None,
     decode_cf: bool | None = None,
     mask_and_scale: bool | Mapping[str, bool] | None = None,
-    decode_times: bool | Mapping[str, bool] | None = None,
+    decode_times: bool
+    | CFDatetimeCoder
+    | Mapping[str, bool | CFDatetimeCoder]
+    | None = None,
     decode_timedelta: bool | Mapping[str, bool] | None = None,
     use_cftime: bool | Mapping[str, bool] | None = None,
     concat_characters: bool | Mapping[str, bool] | None = None,
@@ -1180,9 +1201,9 @@ def open_groups(
         be replaced by NA. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
-    decode_times : bool or dict-like, optional
+    decode_times : bool, CFDatetimeCoder or dict-like, optional
         If True, decode times encoded in the standard NetCDF datetime format
-        into datetime objects. Otherwise, leave them encoded as numbers.
+        into datetime objects. Otherwise, use CFDatetimeCoder or leave them encoded as numbers.
         Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
@@ -1206,6 +1227,8 @@ def open_groups(
         raise an error. Pass a mapping, e.g. ``{"my_variable": False}``,
         to toggle this feature per-variable individually.
         This keyword may not be supported by all the backends.
+        Usage of 'use_cftime' as kwarg is deprecated. Please initialize it
+        with CFDatetimeCoder and 'decode_times' kwarg.
     concat_characters : bool or dict-like, optional
         If True, concatenate along the last dimension of character arrays to
         form string arrays. Dimensions will only be concatenated over (and

xarray/coding/cftime_offsets.py

@@ -64,7 +64,7 @@
 from xarray.core.pdcompat import (
     NoDefault,
     count_not_none,
-    nanosecond_precision_timestamp,
+    default_precision_timestamp,
     no_default,
 )
 from xarray.core.utils import attempt_import, emit_user_level_warning
@@ -77,14 +77,6 @@
 T_FreqStr = TypeVar("T_FreqStr", str, None)
 
 
-def _nanosecond_precision_timestamp(*args, **kwargs):
-    # As of pandas version 3.0, pd.to_datetime(Timestamp(...)) will try to
-    # infer the appropriate datetime precision. Until xarray supports
-    # non-nanosecond precision times, we will use this constructor wrapper to
-    # explicitly create nanosecond-precision Timestamp objects.
-    return pd.Timestamp(*args, **kwargs).as_unit("ns")
-
-
 def get_date_type(calendar, use_cftime=True):
     """Return the cftime date type for a given calendar name."""
     if TYPE_CHECKING:
@@ -93,7 +85,7 @@ def get_date_type(calendar, use_cftime=True):
         cftime = attempt_import("cftime")
 
     if _is_standard_calendar(calendar) and not use_cftime:
-        return _nanosecond_precision_timestamp
+        return default_precision_timestamp
 
     calendars = {
         "noleap": cftime.DatetimeNoLeap,
@@ -1479,10 +1471,8 @@ def date_range_like(source, calendar, use_cftime=None):
     if is_np_datetime_like(source.dtype):
         # We want to use datetime fields (datetime64 object don't have them)
         source_calendar = "standard"
-        # TODO: the strict enforcement of nanosecond precision Timestamps can be
-        # relaxed when addressing GitHub issue #7493.
-        source_start = nanosecond_precision_timestamp(source_start)
-        source_end = nanosecond_precision_timestamp(source_end)
+        source_start = default_precision_timestamp(source_start)
+        source_end = default_precision_timestamp(source_end)
     else:
         if isinstance(source, CFTimeIndex):
             source_calendar = source.calendar