diff --git a/doc/index.rst b/doc/index.rst index dbe911011cd..1d3bb110ddb 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -52,6 +52,7 @@ Documentation * :doc:`reshaping` * :doc:`combining` * :doc:`time-series` +* :doc:`weather-climate` * :doc:`pandas` * :doc:`io` * :doc:`dask` @@ -70,6 +71,7 @@ Documentation reshaping combining time-series + weather-climate pandas io dask diff --git a/doc/io.rst b/doc/io.rst index 0dc5181f9b8..51c747189da 100644 --- a/doc/io.rst +++ b/doc/io.rst @@ -1,11 +1,11 @@ .. _io: -Serialization and IO -==================== +Reading and writing files +========================= xarray supports direct serialization and IO to several file formats, from simple :ref:`io.pickle` files to the more flexible :ref:`io.netcdf` -format. +format (recommended). .. ipython:: python :suppress: @@ -739,11 +739,14 @@ options are listed on the PseudoNetCDF page. .. _PseudoNetCDF: http://github.com/barronh/PseudoNetCDF -Formats supported by Pandas ---------------------------- +CSV and other formats supported by Pandas +----------------------------------------- For more options (tabular formats and CSV files in particular), consider exporting your objects to pandas and using its broad range of `IO tools`_. +For CSV files, one might also consider `xarray_extras`_. + +.. _xarray_extras: https://xarray-extras.readthedocs.io/en/latest/api/csv.html .. _IO tools: http://pandas.pydata.org/pandas-docs/stable/io.html diff --git a/doc/plotting.rst b/doc/plotting.rst index a705c683594..c8f568e516f 100644 --- a/doc/plotting.rst +++ b/doc/plotting.rst @@ -39,6 +39,10 @@ For more extensive plotting applications consider the following projects: data structures for building even complex visualizations easily." Includes native support for xarray objects. +- `hvplot `_: ``hvplot`` makes it very easy to produce + dynamic plots (backed by ``Holoviews`` or ``Geoviews``) by adding a ``hvplot`` + accessor to DataArrays. + - `Cartopy `_: Provides cartographic tools. diff --git a/doc/related-projects.rst b/doc/related-projects.rst index c89e324ff7c..e899022e5d4 100644 --- a/doc/related-projects.rst +++ b/doc/related-projects.rst @@ -13,6 +13,7 @@ Geosciences - `aospy `_: Automated analysis and management of gridded climate data. - `infinite-diff `_: xarray-based finite-differencing, focused on gridded climate/meterology data - `marc_analysis `_: Analysis package for CESM/MARC experiments and output. +- `MetPy `_: A collection of tools in Python for reading, visualizing, and performing calculations with weather data. - `MPAS-Analysis `_: Analysis for simulations produced with Model for Prediction Across Scales (MPAS) components and the Accelerated Climate Model for Energy (ACME). - `OGGM `_: Open Global Glacier Model - `Oocgcm `_: Analysis of large gridded geophysical datasets diff --git a/doc/time-series.rst b/doc/time-series.rst index 3249dad2ec6..53efcd45ba2 100644 --- a/doc/time-series.rst +++ b/doc/time-series.rst @@ -212,140 +212,3 @@ Data that has indices outside of the given ``tolerance`` are set to ``NaN``. For more examples of using grouped operations on a time dimension, see :ref:`toy weather data`. - - -.. _CFTimeIndex: - -Non-standard calendars and dates outside the Timestamp-valid range ------------------------------------------------------------------- - -Through the standalone ``cftime`` library and a custom subclass of -:py:class:`pandas.Index`, xarray supports a subset of the indexing -functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for -dates from non-standard calendars commonly used in climate science or dates -using a standard calendar, but outside the `Timestamp-valid range`_ -(approximately between years 1678 and 2262). - -.. note:: - - As of xarray version 0.11, by default, :py:class:`cftime.datetime` objects - will be used to represent times (either in indexes, as a - :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if - any of the following are true: - - - The dates are from a non-standard calendar - - Any dates are outside the Timestamp-valid range. - - Otherwise pandas-compatible dates from a standard calendar will be - represented with the ``np.datetime64[ns]`` data type, enabling the use of a - :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[ns]`` - and their full set of associated features. - -For example, you can create a DataArray indexed by a time -coordinate with dates from a no-leap calendar and a -:py:class:`~xarray.CFTimeIndex` will automatically be used: - -.. ipython:: python - - from itertools import product - from cftime import DatetimeNoLeap - dates = [DatetimeNoLeap(year, month, 1) for year, month in - product(range(1, 3), range(1, 13))] - da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo') - -xarray also includes a :py:func:`~xarray.cftime_range` function, which enables -creating a :py:class:`~xarray.CFTimeIndex` with regularly-spaced dates. For -instance, we can create the same dates and DataArray we created above using: - -.. ipython:: python - - dates = xr.cftime_range(start='0001', periods=24, freq='MS', calendar='noleap') - da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo') - -For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports: - -- `Partial datetime string indexing`_ using strictly `ISO 8601-format`_ partial - datetime strings: - -.. ipython:: python - - da.sel(time='0001') - da.sel(time=slice('0001-05', '0002-02')) - -- Access of basic datetime components via the ``dt`` accessor (in this case - just "year", "month", "day", "hour", "minute", "second", "microsecond", - "season", "dayofyear", and "dayofweek"): - -.. ipython:: python - - da.time.dt.year - da.time.dt.month - da.time.dt.season - da.time.dt.dayofyear - da.time.dt.dayofweek - -- Group-by operations based on datetime accessor attributes (e.g. by month of - the year): - -.. ipython:: python - - da.groupby('time.month').sum() - -- Interpolation using :py:class:`cftime.datetime` objects: - -.. ipython:: python - - da.interp(time=[DatetimeNoLeap(1, 1, 15), DatetimeNoLeap(1, 2, 15)]) - -- Interpolation using datetime strings: - -.. ipython:: python - - da.interp(time=['0001-01-15', '0001-02-15']) - -- Differentiation: - -.. ipython:: python - - da.differentiate('time') - -- Serialization: - -.. ipython:: python - - da.to_netcdf('example-no-leap.nc') - xr.open_dataset('example-no-leap.nc') - -- And resampling along the time dimension for data indexed by a :py:class:`~xarray.CFTimeIndex`: - -.. ipython:: python - - da.resample(time='81T', closed='right', label='right', base=3).mean() - -.. note:: - - - For some use-cases it may still be useful to convert from - a :py:class:`~xarray.CFTimeIndex` to a :py:class:`pandas.DatetimeIndex`, - despite the difference in calendar types. The recommended way of doing this - is to use the built-in :py:meth:`~xarray.CFTimeIndex.to_datetimeindex` - method: - - .. ipython:: python - :okwarning: - - modern_times = xr.cftime_range('2000', periods=24, freq='MS', calendar='noleap') - da = xr.DataArray(range(24), [('time', modern_times)]) - da - datetimeindex = da.indexes['time'].to_datetimeindex() - da['time'] = datetimeindex - - However in this case one should use caution to only perform operations which - do not depend on differences between dates (e.g. differentiation, - interpolation, or upsampling with resample), as these could introduce subtle - and silent errors due to the difference in calendar types between the dates - encoded in your data and the dates stored in memory. - -.. _Timestamp-valid range: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#timestamp-limitations -.. _ISO 8601-format: https://en.wikipedia.org/wiki/ISO_8601 -.. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#partial-string-indexing diff --git a/doc/weather-climate.rst b/doc/weather-climate.rst new file mode 100644 index 00000000000..1950ba62ffb --- /dev/null +++ b/doc/weather-climate.rst @@ -0,0 +1,160 @@ +.. _weather-climate: + +Weather and climate data +======================== + +.. ipython:: python + :suppress: + + import xarray as xr + +``xarray`` can leverage metadata that follows the `Climate and Forecast (CF) conventions`_ if present. Examples include automatic labelling of plots with descriptive names and units if proper metadata is present (see :ref:`plotting`) and support for non-standard calendars used in climate science through the ``cftime`` module (see :ref:`CFTimeIndex`). There are also a number of geosciences-focused projects that build on xarray (see :ref:`related-projects`). + +.. _Climate and Forecast (CF) conventions: http://cfconventions.org + +.. _metpy_accessor: + +CF-compliant coordinate variables +--------------------------------- + +`MetPy`_ adds a ``metpy`` accessor that allows accessing coordinates with appropriate CF metadata using generic names ``x``, ``y``, ``vertical`` and ``time``. There is also a `cartopy_crs` attribute that provides projection information, parsed from the appropriate CF metadata, as a `Cartopy`_ projection object. See `their documentation`_ for more information. + +.. _`MetPy`: https://unidata.github.io/MetPy/dev/index.html +.. _`their documentation`: https://unidata.github.io/MetPy/dev/tutorials/xarray_tutorial.html#coordinates +.. _`Cartopy`: https://scitools.org.uk/cartopy/docs/latest/crs/projections.html + +.. _CFTimeIndex: + +Non-standard calendars and dates outside the Timestamp-valid range +------------------------------------------------------------------ + +Through the standalone ``cftime`` library and a custom subclass of +:py:class:`pandas.Index`, xarray supports a subset of the indexing +functionality enabled through the standard :py:class:`pandas.DatetimeIndex` for +dates from non-standard calendars commonly used in climate science or dates +using a standard calendar, but outside the `Timestamp-valid range`_ +(approximately between years 1678 and 2262). + +.. note:: + + As of xarray version 0.11, by default, :py:class:`cftime.datetime` objects + will be used to represent times (either in indexes, as a + :py:class:`~xarray.CFTimeIndex`, or in data arrays with dtype object) if + any of the following are true: + + - The dates are from a non-standard calendar + - Any dates are outside the Timestamp-valid range. + + Otherwise pandas-compatible dates from a standard calendar will be + represented with the ``np.datetime64[ns]`` data type, enabling the use of a + :py:class:`pandas.DatetimeIndex` or arrays with dtype ``np.datetime64[ns]`` + and their full set of associated features. + +For example, you can create a DataArray indexed by a time +coordinate with dates from a no-leap calendar and a +:py:class:`~xarray.CFTimeIndex` will automatically be used: + +.. ipython:: python + + from itertools import product + from cftime import DatetimeNoLeap + dates = [DatetimeNoLeap(year, month, 1) for year, month in + product(range(1, 3), range(1, 13))] + da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo') + +xarray also includes a :py:func:`~xarray.cftime_range` function, which enables +creating a :py:class:`~xarray.CFTimeIndex` with regularly-spaced dates. For +instance, we can create the same dates and DataArray we created above using: + +.. ipython:: python + + dates = xr.cftime_range(start='0001', periods=24, freq='MS', calendar='noleap') + da = xr.DataArray(np.arange(24), coords=[dates], dims=['time'], name='foo') + +For data indexed by a :py:class:`~xarray.CFTimeIndex` xarray currently supports: + +- `Partial datetime string indexing`_ using strictly `ISO 8601-format`_ partial + datetime strings: + +.. ipython:: python + + da.sel(time='0001') + da.sel(time=slice('0001-05', '0002-02')) + +- Access of basic datetime components via the ``dt`` accessor (in this case + just "year", "month", "day", "hour", "minute", "second", "microsecond", + "season", "dayofyear", and "dayofweek"): + +.. ipython:: python + + da.time.dt.year + da.time.dt.month + da.time.dt.season + da.time.dt.dayofyear + da.time.dt.dayofweek + +- Group-by operations based on datetime accessor attributes (e.g. by month of + the year): + +.. ipython:: python + + da.groupby('time.month').sum() + +- Interpolation using :py:class:`cftime.datetime` objects: + +.. ipython:: python + + da.interp(time=[DatetimeNoLeap(1, 1, 15), DatetimeNoLeap(1, 2, 15)]) + +- Interpolation using datetime strings: + +.. ipython:: python + + da.interp(time=['0001-01-15', '0001-02-15']) + +- Differentiation: + +.. ipython:: python + + da.differentiate('time') + +- Serialization: + +.. ipython:: python + + da.to_netcdf('example-no-leap.nc') + xr.open_dataset('example-no-leap.nc') + +- And resampling along the time dimension for data indexed by a :py:class:`~xarray.CFTimeIndex`: + +.. ipython:: python + + da.resample(time='81T', closed='right', label='right', base=3).mean() + +.. note:: + + + For some use-cases it may still be useful to convert from + a :py:class:`~xarray.CFTimeIndex` to a :py:class:`pandas.DatetimeIndex`, + despite the difference in calendar types. The recommended way of doing this + is to use the built-in :py:meth:`~xarray.CFTimeIndex.to_datetimeindex` + method: + + .. ipython:: python + :okwarning: + + modern_times = xr.cftime_range('2000', periods=24, freq='MS', calendar='noleap') + da = xr.DataArray(range(24), [('time', modern_times)]) + da + datetimeindex = da.indexes['time'].to_datetimeindex() + da['time'] = datetimeindex + + However in this case one should use caution to only perform operations which + do not depend on differences between dates (e.g. differentiation, + interpolation, or upsampling with resample), as these could introduce subtle + and silent errors due to the difference in calendar types between the dates + encoded in your data and the dates stored in memory. + +.. _Timestamp-valid range: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#timestamp-limitations +.. _ISO 8601-format: https://en.wikipedia.org/wiki/ISO_8601 +.. _partial datetime string indexing: https://pandas.pydata.org/pandas-docs/stable/timeseries.html#partial-string-indexing diff --git a/doc/whats-new.rst b/doc/whats-new.rst index 9ef2960ad76..b1363897094 100644 --- a/doc/whats-new.rst +++ b/doc/whats-new.rst @@ -100,7 +100,7 @@ Bug fixes from higher frequencies to lower frequencies. Datapoints outside the bounds of the original time coordinate are now filled with NaN (:issue:`2197`). By `Spencer Clark `_. -- Line plots with the `x` argument set to a non-dimensional coord now plot the correct data for 1D DataArrays. +- Line plots with the ``x`` argument set to a non-dimensional coord now plot the correct data for 1D DataArrays. (:issue:`27251). By `Tom Nicholas `_. - Subtracting a scalar ``cftime.datetime`` object from a :py:class:`CFTimeIndex` now results in a :py:class:`pandas.TimedeltaIndex` diff --git a/xarray/backends/api.py b/xarray/backends/api.py index 61efcfdedf2..36baa9071c0 100644 --- a/xarray/backends/api.py +++ b/xarray/backends/api.py @@ -247,6 +247,13 @@ def open_dataset(filename_or_obj, group=None, decode_cf=True, dataset : Dataset The newly created dataset. + Notes + ----- + ``open_dataset`` opens the file with read-only access. When you modify + values of a Dataset, even one linked to files on disk, only the in-memory + copy you are manipulating in xarray is modified: the original file on disk + is never touched. + See Also -------- open_mfdataset @@ -597,6 +604,13 @@ def open_mfdataset(paths, chunks=None, concat_dim=_CONCAT_DIM_DEFAULT, ------- xarray.Dataset + Notes + ----- + ``open_mfdataset`` opens files with read-only access. When you modify values + of a Dataset, even one linked to files on disk, only the in-memory copy you + are manipulating in xarray is modified: the original file on disk is never + touched. + See Also -------- auto_combine diff --git a/xarray/core/dataarray.py b/xarray/core/dataarray.py index 96b42f19555..e7e12ae3da4 100644 --- a/xarray/core/dataarray.py +++ b/xarray/core/dataarray.py @@ -1385,8 +1385,9 @@ def transpose(self, *dims): Notes ----- - Although this operation returns a view of this array's data, it is - not lazy -- the data will be fully loaded. + This operation returns a view of this array's data. It is + lazy for dask-backed DataArrays but not for numpy-backed DataArrays + -- the data will be fully loaded. See Also -------- @@ -2437,10 +2438,10 @@ def integrate(self, dim, datetime_unit=None): ---------- dim: str, or a sequence of str Coordinate(s) used for the integration. - datetime_unit - Can be specify the unit if datetime coordinate is used. One of - {'Y', 'M', 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns', 'ps', 'fs', - 'as'} + datetime_unit: str, optional + Can be used to specify the unit if datetime coordinate is used. + One of {'Y', 'M', 'W', 'D', 'h', 'm', 's', 'ms', 'us', 'ns', + 'ps', 'fs', 'as'} Returns ------- diff --git a/xarray/core/dataset.py b/xarray/core/dataset.py index f3e6cac1c5b..12c5d139fdc 100644 --- a/xarray/core/dataset.py +++ b/xarray/core/dataset.py @@ -2848,8 +2848,9 @@ def transpose(self, *dims): Notes ----- - Although this operation returns a view of each array's data, it - is not lazy -- the data will be fully loaded into memory. + This operation returns a view of each array's data. It is + lazy for dask-backed DataArrays but not for numpy-backed DataArrays + -- the data will be fully loaded into memory. See Also -------- diff --git a/xarray/core/variable.py b/xarray/core/variable.py index b675317d83d..433f4a05e1f 100644 --- a/xarray/core/variable.py +++ b/xarray/core/variable.py @@ -1133,8 +1133,8 @@ def transpose(self, *dims): Notes ----- - Although this operation returns a view of this variable's data, it is - not lazy -- the data will be fully loaded. + This operation returns a view of this variable's data. It is + lazy for dask-backed Variables but not for numpy-backed Variables. See Also --------