Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Area definition html representation for Jupyter notebooks #450

Merged
merged 64 commits into from
Dec 13, 2023
Merged
Show file tree
Hide file tree
Changes from 60 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
924627c
feat: basic implementation of html repr
BENR0 Jul 28, 2022
af2e9f5
fix: css wrapper width and plotting to inline svg
Jul 29, 2022
e314e70
fix: set_global error
Jul 29, 2022
c875136
refactor: add docstring make header optional
Aug 1, 2022
1844e66
add: extent description
Aug 1, 2022
1f8bc79
refactor: collapsible sections
Aug 1, 2022
9e59637
add: globe svg. Not working yet
Aug 1, 2022
e73913b
fix: indentation of details and formatting of properties list
Aug 2, 2022
fb0b407
fix: globe icon
Aug 2, 2022
af779f3
add: globe icon for map collapsible
Aug 2, 2022
225e86e
add: docstrings
Aug 2, 2022
f2b0509
fix: list display
Aug 2, 2022
7baad8d
fix: plot showing a second time in jupyter lab
Aug 2, 2022
45fb3b3
fix: comment
Aug 2, 2022
fd1c358
Merge branch 'main' into feat_html_repr
BENR0 Aug 5, 2022
b596a08
reafactor: plotting function
BENR0 Aug 8, 2022
fab8e37
refactor: rename formatting_html
BENR0 Aug 8, 2022
9d2fb5d
fix: static files missing in manifest
Aug 10, 2022
131afd0
refactor: area plotting function to be able to plot SwathDefinitions
Aug 16, 2022
9cdaf7e
feat: add initial html representation to SwathDefinition
Aug 16, 2022
d0d8230
refactor: add xarray style like view of lon/lat to swath definition
Aug 16, 2022
affeda9
feat: add polygon plotting variant to swath plotting
Aug 16, 2022
42eabaa
refactor: add satellite icon
Sep 16, 2022
f88ab85
refactor: add fallback if cartopy is not installed
BENR0 Nov 15, 2022
e0da759
Merge branch 'main' into feat_html_repr
BENR0 Nov 15, 2022
1c03707
feat: add function to generate rst list of area definitions
BENR0 Nov 15, 2022
6e6fd05
refactor: rename areadefinition argument to area
BENR0 Nov 15, 2022
710482d
reafactor: remove plot from SwathDefinition html repr
BENR0 Nov 18, 2022
2d39013
refactor: make loading css and html in area_repr call optional
BENR0 Dec 5, 2022
98f83bd
refactor: load static files only for first area
BENR0 Dec 5, 2022
8f3858a
Merge branch 'main' into feat_html_repr
BENR0 May 22, 2023
cceb48d
refactor: display of lons/lats in case xarray is not installed
BENR0 May 25, 2023
abc2b57
fix: remove tests for missing xarray
BENR0 May 25, 2023
f7acc6c
refactor: resolution calculation
BENR0 May 25, 2023
41d3930
refactor: remove dask array test
BENR0 May 25, 2023
212e14d
refactor: remove else: pass
Jun 20, 2023
b8bcc8b
fix: static files being included in all rows instead of only the first.
BENR0 Jun 21, 2023
8112895
fix: doc strings missing in __init__
BENR0 Jun 21, 2023
810eb86
Merge branch 'main' into feat_html_repr
BENR0 Jun 21, 2023
7b4af63
fix: SwathDefinition not inheriting __str__ and _repr_
Jun 22, 2023
d44a65a
Add: tests for static files inclusion, SwathDefinition attrs section
Jun 22, 2023
bf47bdc
refactor: make showing area plot default instead of svg
Jun 22, 2023
67f9df2
fix: flake8
Jun 22, 2023
fc7ebb4
fix: resolution rounding in numpy case
Jun 26, 2023
88acf1e
refactor: remove unnecessary if/else
Jul 10, 2023
f95a2a4
Merge branch 'main' into feat_html_repr
BENR0 Jul 11, 2023
858d25d
doc: fix docstring in test to match whats tested.
Sep 27, 2023
32c3d38
test: add test for generation of area def rst list.
Sep 28, 2023
35877f7
test: add test for generation of area def rst list.
Sep 28, 2023
1e49766
fix: remove mocker attribute in test function.
Sep 28, 2023
79c8ff3
fix: accidental commit of test_areas_yaml.
Sep 28, 2023
55669d6
Merge branch 'main' into feat_html_repr
Sep 28, 2023
89b6663
refactor: plot_area_def function.
Oct 9, 2023
99dc7ad
test: add dedicated tests for plot_area_def.
Oct 9, 2023
2019450
Update copyright year
BENR0 Oct 11, 2023
f9a3e5b
refactor: change docstrings in favour of type annotations.
BENR0 Oct 11, 2023
e99dabe
refactor: cartopy import.
BENR0 Oct 11, 2023
05dc307
Merge branch 'main' into feat_html_repr
BENR0 Dec 11, 2023
809057d
refactor: make overall width smaller.
Dec 12, 2023
22b483a
Merge branch 'feat_html_repr' of github.com:BENR0/pyresample into fea…
Dec 12, 2023
7d6a9b8
fix: mock called with assertions.
BENR0 Dec 12, 2023
d70f0ea
fix: missed one.
BENR0 Dec 12, 2023
bf32af3
refactor: import future annotations.
BENR0 Dec 13, 2023
5191248
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 13, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,7 @@ include pyresample/ewa/_fornav_templates.*
include pyresample/py.typed
include versioneer.py
include pyresample/version.py
include pyresample/static/html/*
include pyresample/static/css/*
include README.md
recursive-include pyresample *.pyx
360 changes: 360 additions & 0 deletions pyresample/_formatting_html.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,360 @@
# Copyright (C) 2023 Pyresample developers
#
# This program is free software: you can redistribute it and/or modify it under
# the terms of the GNU Lesser General Public License as published by the Free
# Software Foundation, either version 3 of the License, or (at your option) any
# later version.
#
# This program is distributed in the hope that it will be useful, but WITHOUT
# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
# FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more
# details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""Functions for html representation of area definition."""

import uuid
from functools import lru_cache
from html import escape
from importlib.resources import read_binary
from typing import Literal, Optional, Union

import numpy as np

import pyresample.geometry as geom

try:
import cartopy
except ModuleNotFoundError:
cartopy = None

Check warning on line 30 in pyresample/_formatting_html.py

View check run for this annotation

Codecov / codecov/patch

pyresample/_formatting_html.py#L29-L30

Added lines #L29 - L30 were not covered by tests

try:
import xarray as xr
from xarray.core.formatting_html import _obj_repr, datavar_section
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to use a private function (_obj_repr)? How does it differ from repr(xarr_obj)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately yes. repr(xarr_obj) produces the string representation which is also used as a fallback/ in the non notebook case. I could just "copy" the function into pyresample and make it non private?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And your usage differs from the public array_repr and dataset_repr in that same formatting_html module in xarray (these functions use _obj_repr)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes because I want to be able to customize the header and the displayed sections (I only want to display the data variables from the whole xarray.Dataset representation). Obviously that is a (personal) design choice which I am happy to talk about. I just tried to reuse as much as was already available but we could also implement something of our own.

Just to give you an idea of the differences (top: _obj_repr, bottom: dataset_repr):
repr_w_obj_repr
repr_w_dataset_repr

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very interesting. Yeah I see what you mean. I was going to say maybe we copy that functionality here, but then noticed it is loading the static icon SVGs so that gets awkward too. I guess that function is small too but it feels weird to copy them.

@mraspaud @pnuu any opinions on this for what "feels" right from the above screenshots? The bottom has so much extra "cruft" for things that aren't used, but maybe that is OK since we are just completely depending on upstream xarray's "private" functions. Or... @BENR0 what about a PR to xarray (in the long term) that let's you exclude empty sections? That would still include the Attributes but maybe that's a good thing? I could see it being beneficial to users to see the extra metadata hanging around the swath definition.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a private function/method from another libraries feels wrong I have to say. I agree that the bottom is maybe too verbose, but having the dimension size explicitly here I think is nice actually.

xarray = True
except ModuleNotFoundError:
xarray = False

Check warning on line 37 in pyresample/_formatting_html.py

View check run for this annotation

Codecov / codecov/patch

pyresample/_formatting_html.py#L36-L37

Added lines #L36 - L37 were not covered by tests


STATIC_FILES = (
("pyresample.static.html", "icons_svg_inline.html"),
("pyresample.static.css", "style.css"),
)


@lru_cache(None)
def _load_static_files():
Comment on lines +48 to +49
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason for this not to be 1?

"""Lazily load the resource files into memory the first time they are needed."""
return [
read_binary(package, resource).decode("utf-8")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea if read_binary suffers from the same performance issues that pkg_resources utilities do when used in an environment with a lot of packages installed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I don't really know. This is probably something that I would move inline in a refactoring step similar to pytroll/satpy#2171. That would also get rid of the directories which I think are not very nice.

for package, resource in STATIC_FILES
]


def _icon(icon_name):
# icon_name should be defined in pyresample/static/html/icon-svg-inline.html
return (
"<svg class='icon pyresample-{0}'>"
"<use xlink:href='#{0}'>"
"</use>"
"</svg>".format(icon_name)
)


def plot_area_def(area: Union['AreaDefinition', 'SwathDefinition'], # noqa F821
feature_res: Optional[str] = "110m",
fmt: Optional[Literal["svg", "png", None]] = None) -> Union[str, None]:
"""Plot area.

Args:
area : Area/Swath to plot.
feature_res :
Resolution of the features added to the map. Argument is handed over
to `scale` parameter in cartopy.feature.
fmt : Output format of the plot. The output is the string representation of
the respective format xml for svg and base64 for png. Either svg or png.
If None (default) plot is just shown.

Returns:
svg or png image as string.
"""
import base64
from io import BytesIO, StringIO

import matplotlib.pyplot as plt

if isinstance(area, geom.AreaDefinition):
crs = area.to_cartopy_crs()
fig, ax = plt.subplots(subplot_kw=dict(projection=crs))
elif isinstance(area, geom.SwathDefinition):
from shapely.geometry.polygon import Polygon

lx, ly = area.get_edge_lonlats()

crs = cartopy.crs.Mercator()
fig, ax = plt.subplots(subplot_kw=dict(projection=crs))

poly = Polygon(list(zip(lx[::-1], ly[::-1]))) # make lat/lon counterclockwise for shapely
ax.add_geometries([poly], crs=cartopy.crs.CRS(area.crs), facecolor="none", edgecolor="red")
bounds = poly.buffer(5).bounds
ax.set_extent([bounds[0], bounds[2], bounds[1], bounds[3]], crs=cartopy.crs.CRS(area.crs))

coastlines = cartopy.feature.NaturalEarthFeature(category="physical",
name="coastline",
scale=feature_res,
linewidth=1,
facecolor="never")
borders = cartopy.feature.NaturalEarthFeature(category="cultural",
name="admin_0_boundary_lines_land", # noqa E114
scale=feature_res,
edgecolor="black",
facecolor="never") # noqa E1>
ocean = cartopy.feature.OCEAN

ax.add_feature(borders)
ax.add_feature(coastlines)
ax.add_feature(ocean, color="lightgrey")

plt.tight_layout(pad=0)

if fmt == "svg":
svg_str = StringIO()
plt.savefig(svg_str, format="svg", bbox_inches="tight")
plt.close()
return svg_str.getvalue()

elif fmt == "png":
png_str = BytesIO()
plt.savefig(png_str, format="png", bbox_inches="tight")
img_str = f"<img src='data:image/png;base64, {base64.encodebytes(png_str.getvalue()).decode('utf-8')}'/>"
plt.close()
return img_str

else:
plt.show()


def collapsible_section(name: str, inline_details: Optional[str] = "", details: Optional[str] = "",
enabled: Optional[bool] = True, collapsed: Optional[bool] = False,
icon: Optional[str] = None) -> str:
"""Create a collapsible section.

Args:
name : Name of the section
inline_details : Information to show when section is collapsed. Default nothing.
details : Details to show when section is expanded.
enabled : Is collapsing enabled. Default True.
collapsed Is the section collapsed on first show. Default False.
icon : Icon to use for collapsible section.

Returns:
Html div structure for collapsible section.

"""
# "unique" id to expand/collapse the section
data_id = "section-" + str(uuid.uuid4())

enabled = "" if enabled else "disabled"
collapsed = "" if collapsed else "checked"
tip = " title='Expand/collapse section'" if enabled else ""

if icon is None:
icon = _icon("icon-database")

Check warning on line 163 in pyresample/_formatting_html.py

View check run for this annotation

Codecov / codecov/patch

pyresample/_formatting_html.py#L163

Added line #L163 was not covered by tests

return ("<div class='pyresample-area-section-item'>"
f"<input id='{data_id}' class='pyresample-area-section-in' "
f"type='checkbox' {enabled} {collapsed}>"
f"<label for='{data_id}' {tip}>{icon} {name}</label>"
f"<div class='pyresample-area-section-preview'>{inline_details}</div>"
f"<div class='pyresample-area-section-details'>{details}</div>"
"</div>"
)


def map_section(area: Union['AreaDefinition', 'SwathDefinition']) -> str: # noqa F821
"""Create html for map section.

Args:
area : AreaDefinition or SwathDefinition.

Returns:
Html with collapsible section with a cartopy plot.

"""
map_icon = _icon("icon-globe")

if cartopy:
coll = collapsible_section("Map", details=plot_area_def(area, fmt="svg"), collapsed=True, icon=map_icon)
else:
coll = collapsible_section("Map",
details="Note: If cartopy is installed a display of the area can be seen here",
collapsed=True, icon=map_icon)

return f"{coll}"


def proj_area_attrs_section(area: 'AreaDefinition') -> str: # noqa F821
"""Create html for attribute section based on an area Area.

Args:
area : Area definition.

Returns:
Html with collapsible section of attributes of Area.

"""
resolution_str = "/".join([str(round(x, 1)) for x in area.resolution])
proj_dict = area.proj_dict
proj_str = "{{{}}}".format(", ".join(["'%s': '%s'" % (str(k), str(proj_dict[k])) for k in
sorted(proj_dict.keys())]))
area_units = proj_dict.get("units", "")

attrs_icon = _icon("icon-file-text2")

area_attrs = ("<dl>"
f"<dt>Area name</dt><dd>{area.area_id}</dd>"
f"<dt>Description</dt><dd>{area.description}</dd>"
f"<dt>Projection</dt><dd>{proj_str}</dd>"
f"<dt>Width/Height</dt><dd>{area.width}/{area.height} Pixel</dd>"
f"<dt>Resolution x/y (SSP)</dt><dd>{resolution_str} {area_units}</dd>"
f"<dt>Extent (ll_x, ll_y, ur_x, ur_y)</dt>"
f"<dd>{tuple(round(x, 4) for x in area.area_extent)}</dd>"
"</dl>"
)

coll = collapsible_section("Properties", details=area_attrs, icon=attrs_icon)

return f"{coll}"


def swath_area_attrs_section(area: 'SwathDefinition') -> str: # noqa F821
"""Create html for attribute section based on SwathDefinition.

Args:
area : Swath definition.

Returns:
Html with collapsible section of swath attributes.

Todo:
- Improve resolution estimation from lat/lon arrays. Maybe use CoordinateDefinition.geocentric_resolution?
djhoese marked this conversation as resolved.
Show resolved Hide resolved

"""
if isinstance(area.lons, np.ndarray) & isinstance(area.lats, np.ndarray):
# Calculate and estimated resolution from lats/lons in meter
area_name = "Arbitrary Swath"
resolution_y = np.mean(area.lats[0:-1, :] - area.lats[1::, :])
resolution_x = np.mean(area.lons[:, 1::] - area.lons[:, 0:-1])
resolution = np.mean(np.array([resolution_x, resolution_y]))
resolution = np.round(40075000 * resolution / 360, 1)
resolution_str = f"{resolution}/{resolution}"
area_units = "m"
else:
lon_attrs = area.lons.attrs
lat_attrs = area.lats.attrs

# use resolution from lat/lons dataarray attributes -> are these always set? -> Maybe try/except?
area_name = f"{lon_attrs.get('sensor')} swath"
resolution_str = "/".join([str(round(x.get("resolution"), 1)) for x in [lat_attrs, lon_attrs]])
area_units = "m"

height, width = area.lons.shape

attrs_icon = _icon("icon-file-text2")

area_attrs = ("<dl>"
# f"<dt>Area name</dt><dd>{area_name}</dd>"
f"<dt>Description</dt><dd>{area_name}</dd>"
f"<dt>Width/Height</dt><dd>{width}/{height} Pixel</dd>"
f"<dt>Resolution x/y (SSP)</dt><dd>{resolution_str} {area_units}</dd>"
"</dl>"
)

if xarray and not isinstance(area.lons, np.ndarray):
ds_dict = {i.attrs['name']: i.rename(i.attrs['name']) for i in [area.lons, area.lats]}
dss = xr.merge(ds_dict.values())

area_attrs += _obj_repr(dss, header_components=[""], sections=[datavar_section(dss.data_vars)])
else:
with np.printoptions(threshold=50):
lons = f"{area.lons}".replace("\n", "<br>")
lats = f"{area.lats}".replace("\n", "<br>")
area_attrs += ("<div class='xr-wrap', style='display:none'>"
"<div class='xr-header'></div>"
"<ul class='xr-sections'>"
"<li class='xr-section-item'>"
"<div class='xr-section-details', style='display:contents'>" # noqa E127
"<ul class='xr-var-list'>" # noqa E127
"<li class='xr-var-item'>" # noqa E127
"<div class='xr-var-name'>" # noqa E127
"<span>Longitude</span>"
"</div>"
f"<div class=xr-var-preview xr-preview>{lons}</div>"
"</li>"
"<li class='xr-var-item'>"
"<div class='xr-var-name'>"
"<span>Latitude</span>"
"</div>"
f"<div class=xr-var-preview xr-preview>{lats}</div>"
"</li>"
"</ul>"
"</div>"
"</li>"
"</ul>"
"</div>"
"</div>"
)

coll = collapsible_section("Properties", details=area_attrs, icon=attrs_icon)

return f"{coll}"


def area_repr(area: Union['AreaDefinition', 'SwathDefinition'], include_header: Optional[bool] = True, # noqa F821
include_static_files: Optional[bool] = True):
"""Return html repr of an AreaDefinition.

Args:
area : Area definition.
include_header : If true a header with object type will be included in
the html. This is mainly intented for display in Jupyter Notebooks. For the
display in the overview of area definitions for the Satpy documentation this
should be set to false.
include_static_files : Load and include css and html needed for representation.

Returns:
Html.

"""
if include_static_files:
icons_svg, css_style = _load_static_files()
html = f"{icons_svg}<style>{css_style}</style>"
else:
html = ""

obj_type = f"pyresample.{type(area).__name__}"
header = ("<div class='pyresample-header'>"
"<div class='pyresample-obj-type'>"
f"{escape(obj_type)}"
"</div>"
"</div>"
)

html += (f"<pre class='pyresample-text-repr-fallback'>{escape(repr(area))}</pre>"
"<div class='pyresample-wrap' style='display:none'>"
)

if include_header:
html += f"{header}"

html += "<div class='pyresample-area-sections'>"
if isinstance(area, geom.AreaDefinition):
html += proj_area_attrs_section(area)
html += map_section(area)
elif isinstance(area, geom.SwathDefinition):
html += swath_area_attrs_section(area)

Check warning on line 356 in pyresample/_formatting_html.py

View check run for this annotation

Codecov / codecov/patch

pyresample/_formatting_html.py#L355-L356

Added lines #L355 - L356 were not covered by tests

html += "</div>"

return html
Loading
Loading