Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 #1347

hoxbro · 2024-06-27T17:17:16Z

A minor compatibility change to support dask_geopandas.dask_expr.

Removed world example as that dataset is no longer available. Haven't looked at the surrounding text.

I also removed the numpy2 feature, as the original test now pulls it in.

codecov · 2024-06-27T20:14:27Z

Codecov Report

Attention: Patch coverage is 97.36842% with 1 line in your changes missing coverage. Please review.

Project coverage is 90.05%. Comparing base (73c3819) to head (0387dbb).

Files	Patch %	Lines
datashader/core.py	91.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1347      +/-   ##
==========================================
- Coverage   90.22%   90.05%   -0.17%     
==========================================
  Files          92       92              
  Lines       18605    18626      +21     
==========================================
- Hits        16786    16774      -12     
- Misses       1819     1852      +33

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ng error

droumis · 2024-06-27T22:09:38Z

started reviewing the notebook, but got stuck debugging the following:

ddf = dd.from_pandas(sgeodf, npartitions=2).pack_partitions(npartitions=100).persist()

tf.shade(cvs.polygons(ddf, geometry='geometry', agg=ds.mean('population')), cmap=cc.kg)

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_core.py:467, in Expr.__getattr__(self, key)
    466 try:
--> 467     return object.__getattribute__(self, key)
    468 except AttributeError as err:

AttributeError: 'FromPandas' object has no attribute 'pack_partitions'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_collection.py:619, in FrameBase.__getattr__(self, key)
    616 try:
    617     # Fall back to `expr` API
    618     # (Making sure to convert to/from Expr)
--> 619     val = getattr(self.expr, key)
    620     if callable(val):

File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_core.py:488, in Expr.__getattr__(self, key)
    487 link = "https://github.com/dask-contrib/dask-expr/blob/main/README.md#api-coverage"
--> 488 raise AttributeError(
    489     f"{err}\n\n"
    490     "This often means that you are attempting to use an unsupported "
    491     f"API function. Current API coverage is documented here: {link}."
    492 )

AttributeError: 'FromPandas' object has no attribute 'pack_partitions'

This often means that you are attempting to use an unsupported API function. Current API coverage is documented here: https://github.com/dask-contrib/dask-expr/blob/main/README.md#api-coverage.

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
Cell In[36], line 1
----> 1 ddf = dd.from_pandas(sgeodf, npartitions=2).pack_partitions(npartitions=100).persist()
      3 tf.shade(cvs.polygons(ddf, geometry='geometry', agg=ds.mean('population')), cmap=cc.kg)

File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_collection.py:3049, in DataFrame.__getattr__(self, key)
   3046     raise err
   3047 except AttributeError:
   3048     # Fall back to `BaseFrame.__getattr__`
-> 3049     return super().__getattr__(key)

File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_collection.py:625, in FrameBase.__getattr__(self, key)
    622     return val
    623 except AttributeError:
    624     # Raise original error
--> 625     raise err

File ~/opt/miniconda3/envs/datashader-dev/lib/python3.12/site-packages/dask_expr/_collection.py:614, in FrameBase.__getattr__(self, key)
    611 def __getattr__(self, key):
    612     try:
    613         # Prioritize `FrameBase` attributes
--> 614         return object.__getattribute__(self, key)
    615     except AttributeError as err:
    616         try:
    617             # Fall back to `expr` API
    618             # (Making sure to convert to/from Expr)

AttributeError: 'DataFrame' object has no attribute 'pack_partitions'

hoxbro · 2024-06-28T08:05:20Z

started reviewing the notebook, but got stuck debugging the following:

Forgot to tell you, but spatialpandas does not work with dask-expr. So you have to disable it: dask config set dataframe.query-planning False or system environment variable export DASK_DATAFRAME__QUERY_PLANNING=False

droumis · 2024-06-28T23:43:33Z

@jbednar ~~Does this match your understanding of interrelationships for Datashader Polygon world?~~ diagram updated.. see later comments in thread

graph TD;
    subgraph SpatialPandas
        A[Polygon]:::spatialpandas -->|used in| E[PolygonArray]:::spatialpandas
        C[MultiPolygon]:::spatialpandas -->|used in| G[MultiPolygonArray]:::spatialpandas
        E -->|used in| F[GeoDataFrame]:::spatialpandas
        G -->|used in| F
        E -->|useable in| N[Pandas DataFrame with Geometry]:::pandas
        G -->|useable in| N
        A -->|converts to| B[shapely.geometry.Polygon]:::shapely
        C -->|converts to| D[shapely.geometry.MultiPolygon]:::shapely
        B -->|converts to| A
        D -->|converts to| C
    end

    subgraph GeoPandas
        H[GeoSeries]:::geopandas -->|converts to| I[spatialpandas.GeoSeries]:::spatialpandas
        H -->|converts to| J[GeometryArray]:::geopandas
    end

    subgraph Dask
        F -->|used in| L[DataFrame with Geometry]:::dask
        I -->|used in| L
    end

    subgraph Datashader
        M[Datashader]:::datashader
        F -->|usable by| M
        L -->|usable by| M
        J -->|usable by| M
        E -->|usable by| M
        G -->|usable by| M
        N -->|usable by| M
    end

    classDef spatialpandas fill:#7aa9e6,stroke:#000,stroke-width:1px;
    classDef shapely fill:#ff7f0e,stroke:#000,stroke-width:1px;
    classDef geopandas fill:#2ca02c,stroke:#000,stroke-width:1px;
    classDef dask fill:#d62728,stroke:#000,stroke-width:1px;
    classDef datashader fill:#9467bd,stroke:#000,stroke-width:1px;
    classDef pandas fill:#bcbd22,stroke:#000,stroke-width:1px;

droumis · 2024-06-28T23:49:13Z

Changed some of the text to make it make more sense (to me at least). I also replaced the dataset with one that shows something more recognizable (USA! USA!)

…le test" This reverts commit e7ac6a3.

tox.ini

droumis · 2024-07-02T14:01:42Z

version 2: (NOT LATEST)

graph TD;
    subgraph SpatialPandas
        A[Polygon]:::spatialpandas -->|used in| E[PolygonArray]:::spatialpandas
        B[MultiPolygon]:::spatialpandas -->|used in| F[MultiPolygonArray]:::spatialpandas
        E -->|used in| G[GeoDataFrame or GeoSeries]:::spatialpandas
        F -->|used in| G
    end

    subgraph Shapely
        H[Polygon]:::shapely <-->|converts to| A
        I[MultiPolygon]:::shapely <-->|converts to| B
    end

    subgraph GeoPandas
        J[GeoDataFrame or GeoSeries]:::geopandas
    end

    subgraph Pandas
        K[DataFrame or Series]:::pandas
    end

    subgraph spatialpandas.dask
        L[DaskGeoDataFrame or DaskGeoSeries]:::dask
    end

    subgraph Datashader
        M{Datashader}:::datashader
    end

    F -->|usable in| K
    E -->|usable in| K
    
    G -->|converts to| L

    G <-->|converts to| J

    G -->|usable by| M
    L -->|usable by| M
    J -->|usable by| M
    K -->|usable by| M

    classDef spatialpandas fill:#4e79a7,stroke:#000,stroke-width:1px;
    classDef shapely fill:#f28e2b,stroke:#000,stroke-width:1px;
    classDef geopandas fill:#59a14f,stroke:#000,stroke-width:1px;
    classDef dask fill:#76b7b2,stroke:#000,stroke-width:1px;
    classDef datashader fill:#b07aa1,stroke:#000,stroke-width:1px;
    classDef pandas fill:#edc948,stroke:#000,stroke-width:1px

droumis · 2024-07-02T16:23:51Z

V3 .. fixing subgraph titles overlapping with arrows with some trickery

graph TD;
    subgraph SP[ ]
        A[Polygon]:::spatialpandas -->|used in| E[PolygonArray]:::spatialpandas
        B[MultiPolygon]:::spatialpandas -->|used in| F[MultiPolygonArray]:::spatialpandas
        E -->|used in| G[GeoDataFrame or<div></div> GeoSeries]:::spatialpandas
        spatialpandas(SpatialPandas):::spatialpandas_title
        F -->|used in| G
    end

    subgraph SH[ ]
        shapely(Shapely):::shapely_title
        H[Polygon]:::shapely -->|converts to| A
        I[MultiPolygon]:::shapely -->|converts to| B
    end

    subgraph GP[ ]
        geopandas(GeoPandas):::geopandas_title
        J[GeoDataFrame or<div></div> GeoSeries]:::geopandas
    end

    subgraph PD[ ]
        pandas(Pandas):::pandas_title
        K[DataFrame or<div></div> Series]:::pandas
    end

    subgraph SPD[ ]
        spatialpandas.dask(SpatialPandas.Dask):::dask_title
        L[DaskGeoDataFrame or<div></div> DaskGeoSeries]:::dask
    end

    M(Datashader):::datashader_title

    F -->|usable in| K
    E -->|usable in| K
    
    G -->|converts to| L

    G <-->|converts to| J

    G -->|usable by| M
    L -->|usable by| M
    J -->|usable by| M
    K -->|usable by| M

    classDef spatialpandas fill:#4e79a7,stroke:#000,stroke-width:0px,color:black;
    classDef shapely fill:#f28e2b,stroke:#000,stroke-width:0px,color:black;
    classDef geopandas fill:#59a14f,stroke:#000,stroke-width:0px,color:black;
    classDef dask fill:#76b7b2,stroke:#000,stroke-width:0px,color:black;
    classDef datashader fill:#b07aa1,stroke:#000,stroke-width:0px,color:black;
    classDef pandas fill:#edc948,stroke:#000,stroke-width:0px,color:black;
    classDef spatialpandas_title fill:#fff,stroke:#4e79a7,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef shapely_title fill:#fff,stroke:#f28e2b,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef geopandas_title fill:#fff,stroke:#59a14f,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef dask_title fill:#fff,stroke:#76b7b2,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef pandas_title fill:#fff,stroke:#edc948,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef datashader_title fill:#fff,stroke:#b07aa1,stroke-width:7px,color:black,fill-opacity:.9,font-weight: bold;
    classDef subgraph_style fill:#fff;

    style SP fill:grey,stroke:#fff,stroke-width:0px
    style SH fill:grey,stroke:#fff,stroke-width:0px
    style GP fill:grey,stroke:#fff,stroke-width:0px
    style PD fill:grey,stroke:#fff,stroke-width:0px
    style SPD fill:grey,stroke:#fff,stroke-width:0px

jbednar · 2024-07-02T19:31:31Z

@droumis , that looks good! Are the double-headed arrows that point in to the Shapely box intentional? I would have thought that Datashader has no interest in converting things to Shapely.

droumis · 2024-07-02T20:00:53Z

They were intentional because the notebook mentions the to_shapely method. But I agree that it's a bit of a distraction so I've now removed those lines going back toward shapely.

droumis · 2024-07-02T20:01:27Z

@hoxbro, I think we can merge now

jbednar

Looks good, thanks!

examples/conftest.py

examples/user_guide/8_Polygons.ipynb

Co-authored-by: James A. Bednar <jbednar@continuum.io>

hoxbro added 3 commits June 27, 2024 19:16

Compability with geopandas 1 and dask-geopandas 0.4.0

7214c6c

Add dask-expr to geopandas test

c9bf0dc

Clean up

d3f4ef8

hoxbro changed the title ~~Compability with geopandas 1 and dask-geopandas 0.4.0~~ Compability with geopandas 1.0 and dask-geopandas 0.4.0 Jun 27, 2024

hoxbro added 5 commits June 27, 2024 19:38

Handle previous versions

95fa388

Simplify dask logic

caf093e

Update logic

bfb6e5d

Remove world reference from polygon example

b951215

Update with skipif

b180e12

droumis added 2 commits June 27, 2024 13:55

install sp from pyviz dev to get at least 0.4.11 to avoid numpy warni…

8652d3d

…ng error

remove mention of .from_geopandas

f6b1394

Lint

36081fb

hoxbro force-pushed the geopandas1 branch from bcdb806 to 02b27bc Compare June 28, 2024 15:07

Remove numpy2 from CI as it is released

32e8be0

hoxbro force-pushed the geopandas1 branch from 02b27bc to 32e8be0 Compare June 28, 2024 15:19

mermaid and text revision and data replace

48ae3d3

mermaid fix

a01e123

hoxbro added 8 commits June 29, 2024 08:48

Fix lint

4b9a9c7

Move install instructions to the bottom

a0185f2

Update gitignore

25a1fb6

Reduce number of time commands

f9a021d

Try to remove mermaid to see if that causes the failing example test

e7ac6a3

Revert "Try to remove mermaid to see if that causes the failing examp…

7ab5bb5

…le test" This reverts commit e7ac6a3.

debug

5fbda94

Debug 2

432c078

hoxbro force-pushed the geopandas1 branch from cb3e9ba to 66b22e9 Compare July 1, 2024 13:32

hoxbro commented Jul 1, 2024

View reviewed changes

tox.ini Outdated Show resolved Hide resolved

hoxbro added 3 commits July 1, 2024 16:26

Update tox.ini

1dac254

Try docs build

712864c

Remove myst_parser for now

5cd5080

hoxbro requested a review from jbednar July 1, 2024 17:04

droumis added 2 commits July 2, 2024 09:55

update mermaid

5071272

run precommit

c0b52f0

droumis added 2 commits July 2, 2024 11:25

fix mermaid subgraph labels

f1bb970

black text

bdf926f

jbednar changed the title ~~Compability with geopandas 1.0 and dask-geopandas 0.4.0~~ Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 Jul 2, 2024

remove lines to shapely

d34ac47

style mermaid

522a910

jbednar approved these changes Jul 2, 2024

View reviewed changes

droumis and others added 9 commits July 2, 2024 16:27

Update examples/user_guide/8_Polygons.ipynb

7b448d7

Co-authored-by: James A. Bednar <jbednar@continuum.io>

Update examples/user_guide/8_Polygons.ipynb

b65bbee

Co-authored-by: James A. Bednar <jbednar@continuum.io>

Update examples/user_guide/8_Polygons.ipynb

a427fb5

Co-authored-by: James A. Bednar <jbednar@continuum.io>

Update examples/user_guide/8_Polygons.ipynb

4913057

Co-authored-by: James A. Bednar <jbednar@continuum.io>

Update examples/user_guide/8_Polygons.ipynb

36c9670

Co-authored-by: James A. Bednar <jbednar@continuum.io>

condition dask-expr workaround

0efa95f

Upload download_data.py

cdf41d1

fix: Ubuntu exit 143 (#1348)

0387dbb

adjust config dask for sp wording

d9b5a8e

hoxbro merged commit 93e88ad into main Jul 3, 2024
13 checks passed

hoxbro deleted the geopandas1 branch July 3, 2024 15:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 #1347

Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 #1347

hoxbro commented Jun 27, 2024 •

edited

Loading

codecov bot commented Jun 27, 2024 •

edited

Loading

droumis commented Jun 27, 2024

hoxbro commented Jun 28, 2024

droumis commented Jun 28, 2024 •

edited

Loading

droumis commented Jun 28, 2024

droumis commented Jul 2, 2024 •

edited

Loading

droumis commented Jul 2, 2024 •

edited

Loading

jbednar commented Jul 2, 2024

droumis commented Jul 2, 2024

droumis commented Jul 2, 2024

jbednar left a comment

Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 #1347

Compatibility with geopandas 1.0 and dask-geopandas 0.4.0 #1347

Conversation

hoxbro commented Jun 27, 2024 • edited Loading

codecov bot commented Jun 27, 2024 • edited Loading

Codecov Report

droumis commented Jun 27, 2024

hoxbro commented Jun 28, 2024

droumis commented Jun 28, 2024 • edited Loading

droumis commented Jun 28, 2024

droumis commented Jul 2, 2024 • edited Loading

droumis commented Jul 2, 2024 • edited Loading

jbednar commented Jul 2, 2024

droumis commented Jul 2, 2024

droumis commented Jul 2, 2024

jbednar left a comment

Choose a reason for hiding this comment

hoxbro commented Jun 27, 2024 •

edited

Loading

codecov bot commented Jun 27, 2024 •

edited

Loading

droumis commented Jun 28, 2024 •

edited

Loading

droumis commented Jul 2, 2024 •

edited

Loading

droumis commented Jul 2, 2024 •

edited

Loading