Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Grid.open_file method support non xarray formats including Shapefile/Geojson #808

Merged
merged 65 commits into from
Sep 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
39a16fd
o start on #635
rajeeja Jan 10, 2024
d554571
o Fix conflicts and merge main
rajeeja Jan 11, 2024
6d4e6da
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Jan 23, 2024
6c8ce7a
o Intermittent work
Feb 22, 2024
fc58f4b
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 5, 2024
04729ec
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja May 6, 2024
efdf46e
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 8, 2024
bea0e81
o pre-commit changes Ruff
rajeeja May 8, 2024
a627b30
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 15, 2024
f59f9a6
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 18, 2024
2a3dba0
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 20, 2024
2aaabcd
o Read polygon and multipolygon
rajeeja May 31, 2024
7d3875c
Merge branch 'main' into rajeeja/io_shapefile
rajeeja May 31, 2024
939b3b4
o Add a test mesh file
rajeeja May 31, 2024
fb10484
o Add geopandas dependency
rajeeja Jun 3, 2024
fa3b73f
o Fix tests, TODO: Fix API and make open_grid work.
rajeeja Jun 3, 2024
a503c51
Possible that np.intp could have different ranges on 32-bit versus 64…
rajeeja Jun 4, 2024
1353d2b
trying np.int16
rajeeja Jun 4, 2024
66ccc8b
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Jul 9, 2024
a7b8328
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja Jul 9, 2024
9664430
o Make a small example, fix bug with max size for connectivity arrray…
rajeeja Jul 10, 2024
1f12543
o Fix setting the datatype for empty connectivity array to a np.int16…
rajeeja Jul 11, 2024
b5fe8ed
o Setting the dtype to int32 as leaving empty defaults to float
rajeeja Jul 11, 2024
65901dd
o Use np.intp for connectivity init
rajeeja Jul 11, 2024
0ec8002
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Jul 17, 2024
00fc5fa
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Jul 18, 2024
07cfae7
o Check to see if np.fulll instead of pad works
rajeeja Jul 19, 2024
c05cfda
o Add test for multipolygon case and also test a file
rajeeja Jul 20, 2024
5ac3e08
o Fix comments
rajeeja Jul 22, 2024
54daaa1
o Fix comments
rajeeja Jul 22, 2024
e97e76f
o Rearrange code
rajeeja Jul 24, 2024
34decd2
Merge branch 'main' into rajeeja/io_shapefile
philipc2 Aug 2, 2024
258948a
Merge branch 'main' into rajeeja/io_shapefile
philipc2 Aug 5, 2024
2deb7fd
Merge branch 'main' into rajeeja/io_shapefile
philipc2 Aug 12, 2024
547405f
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Aug 21, 2024
92de0ec
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja Aug 21, 2024
112dcc7
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Aug 23, 2024
2dd18d0
o Move to geopandas from shapefile
rajeeja Aug 26, 2024
f1534ad
o Add geopandas reader to doc
rajeeja Aug 28, 2024
9387e9d
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Aug 30, 2024
952240b
Update test/test_geopandas.py
rajeeja Sep 3, 2024
1e3a6ac
o Resolve functions as per Orhan's feedback
rajeeja Sep 3, 2024
38d33f0
o Add readme for meshfiles
rajeeja Sep 3, 2024
9845d1b
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Sep 3, 2024
fa4fcba
o Remove commented lines as logic is to read via Grid object
rajeeja Sep 4, 2024
93d5dc7
o Using lists instead of np.append as list is an amortized O(1), np.…
rajeeja Sep 5, 2024
e451aff
o missed precommit
rajeeja Sep 5, 2024
fc7744b
Merge branch 'main' into rajeeja/io_shapefile
erogluorhan Sep 10, 2024
363ec1d
o change grid.py for a making front end more general
rajeeja Sep 10, 2024
6101a63
Merge branch 'main' into rajeeja/io_shapefile
erogluorhan Sep 10, 2024
793d697
Merge branch 'main' into rajeeja/io_shapefile
philipc2 Sep 13, 2024
c439bbe
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Sep 13, 2024
c591083
o Use from_file class method for shapefile/geojson etc. from geopanda…
rajeeja Sep 13, 2024
d7a9778
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja Sep 13, 2024
35ec0b6
o Fix doc
rajeeja Sep 13, 2024
57ea2f5
Merge branch 'main' into rajeeja/io_shapefile
philipc2 Sep 14, 2024
f383a96
o Call backend xarray instead of NetCDF, fix doc
rajeeja Sep 14, 2024
88f1647
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja Sep 14, 2024
a1fc10e
Update uxarray/grid/grid.py
rajeeja Sep 14, 2024
16ca582
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Sep 17, 2024
120e741
Merge branch 'rajeeja/io_shapefile' of github.com:UXARRAY/uxarray int…
rajeeja Sep 17, 2024
a8ba608
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Sep 18, 2024
f4d0ac4
Update uxarray/core/api.py
rajeeja Sep 19, 2024
9f97dfa
Merge branch 'main' into rajeeja/io_shapefile
rajeeja Sep 19, 2024
610e029
undo previous commit and add only precommit fix
rajeeja Sep 19, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/asv.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ dependencies:
- scipy
- shapely
- spatialpandas
- geopandas
- xarray
- asv
- pip:
Expand Down
1 change: 1 addition & 0 deletions ci/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ dependencies:
- pandas
- geocat-datafiles
- spatialpandas
- geopandas
- pip:
- antimeridian
- git+https://github.com/philipc2/pyfma.git # fix until https://github.com/nschloe/pyfma/pull/17 is merged
1 change: 1 addition & 0 deletions ci/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ dependencies:
- scipy
- shapely
- spatialpandas
- geopandas
- xarray
- asv
- pip:
Expand Down
1 change: 1 addition & 0 deletions ci/install-upstream.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ python -m pip install \
pandas \
scikit-learn \
scipy \
geopandas \
xarray

# install rest from source
Expand Down
9 changes: 7 additions & 2 deletions docs/internal_api/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -286,12 +286,17 @@ SCRIP
io._scrip._encode_scrip


Shapefile
GeoPandas
---------
.. autosummary::
:toctree: generated/

io._shapefile._read_shpfile
io._geopandas._read_geodataframe
io._geopandas._read_geopandas
io._geopandas._encode_geopandas
io._geopandas._standardize_geopandas
io._geopandas._is_geopandas
io._geopandas._validate_minimum_geopandas

Vertices
--------
Expand Down
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ dependencies = [
"scipy",
"shapely",
"spatialpandas",
"geopandas",
"xarray",
]
# minimal dependencies end
Expand Down
15 changes: 15 additions & 0 deletions test/meshfiles/geojson/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
GeoJSON is a format for encoding a variety of geographic data structures, including points, lines, and polygons.

This repository contains "sample_chicago_buildings" GeoJSON file featuring a selection of buildings in Chicago. The file includes:

Number of polygons: 10
Polygon 1: 26 sides
Polygon 2: 36 sides
Polygon 3: 29 sides
Polygon 4: 10 sides
Polygon 5: 30 sides
Polygon 6: 8 sides
Polygon 7: 7 sides
Polygon 8: 9 sides
Polygon 9: 7 sides
Polygon 10: 19 sides
17 changes: 17 additions & 0 deletions test/meshfiles/geojson/sample_chicago_buildings.geojson

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions test/meshfiles/shp/5poly/5poly.cpg
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
ISO-8859-1
Binary file added test/meshfiles/shp/5poly/5poly.dbf
Binary file not shown.
Binary file added test/meshfiles/shp/5poly/5poly.shp
Binary file not shown.
Binary file added test/meshfiles/shp/5poly/5poly.shx
Binary file not shown.
15 changes: 15 additions & 0 deletions test/meshfiles/shp/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
### There are 3 sub folders each containing different shapefile examples

1. **5poly** : A very simple file containing only 5 polygons with different number of sides
2. **cb_2018_us_national_20m**: Real word shapefile containing the 2018 Census Bureau US map at 20m resolution
3. **multipoly**: Test file containing multipolygons.

### Files Included in the Shapefile Dataset

1. **.shp** - **Shape Format**: Contains the geometric data (polygons) for the features in the dataset. The first 100 bytes contain metadata about the file.

2. **.shx** - **Shape Index Format**: Contains the index data for the .shp file, allowing for quick access to the geometric data.

3. **.dbf** - **Attribute Format**: Contains attribute data for the features in the .shp file. This is a dBase file that stores information in a tabular format, where each row corresponds to a feature in the .shp file.

4. **.cpg** - **Code Page Format**: Contains character encoding information for the .dbf file, ensuring that text data is correctly interpreted. Eg. ISO-8859-1
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
UTF-8
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
GEOGCS["GCS_North_American_1983",DATUM["D_North_American_1983",SPHEROID["GRS_1980",6378137,298.257222101]],PRIMEM["Greenwich",0],UNIT["Degree",0.017453292519943295]]
Binary file not shown.
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--This file contains all the Entity and Attribute Information--><gfc:FC_FeatureCatalogue xmlns:gmx="http://www.isotc211.org/2005/gmx"
xmlns:gco="http://www.isotc211.org/2005/gco"
xmlns:gmd="http://www.isotc211.org/2005/gmd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns:gml="http://www.opengis.net/gml/3.2"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:gfc="http://www.isotc211.org/2005/gfc"
xsi:schemaLocation="https://www.ngdc.noaa.gov/metadata/published/xsd/schema/gfc/featureCataloging.xsd http://www.isotc211.org/2005/gfc">
<gmx:name>
<gco:CharacterString>Feature Catalog for the 2018 United States 1:20,000,000 Cartographic Boundary File</gco:CharacterString>
</gmx:name>
<gmx:scope>
<gco:CharacterString>The United States at a scale of 1:20,000,000</gco:CharacterString>
</gmx:scope>
<gmx:versionNumber>
<gco:CharacterString>cb_2018_nation_20m</gco:CharacterString>
</gmx:versionNumber>
<gmx:versionDate>
<gco:Date>2019-05</gco:Date>
</gmx:versionDate>
<gmx:language>
<gco:CharacterString>eng</gco:CharacterString>
</gmx:language>
<gmx:characterSet>
<gmd:MD_CharacterSetCode codeList="http://www.isotc211.org/2005/resources/Codelist/gmxCodelists.xml#MD_CharacterSetCode"
codeListValue="utf8"
codeSpace="004">utf8
</gmd:MD_CharacterSetCode>
</gmx:characterSet>
<gfc:producer xlink:href="https://www.ngdc.noaa.gov/docucomp/1df27e57-4768-42de-909b-52f530601fba"
xlink:title="U.S Department of Commerce, U.S Census Bureau, Geographic Customer Services Branch"/>
<gfc:featureType>
<gfc:FC_FeatureType>
<gfc:typeName>
<gco:LocalName>cb_2018_us_nation_20m.shp</gco:LocalName>
</gfc:typeName>
<gfc:definition>
<gco:CharacterString>Current Nation (national)</gco:CharacterString>
</gfc:definition>
<gfc:isAbstract>
<gco:Boolean>false</gco:Boolean>
</gfc:isAbstract>
<gfc:featureCatalogue uuidref="2018_nation_20m.ea.iso.xml"/>
<gfc:carrierOfCharacteristics>
<gfc:FC_FeatureAttribute>
<gfc:memberName>
<gco:LocalName>AFFGEOID</gco:LocalName>
</gfc:memberName>
<gfc:definition>
<gco:CharacterString>American FactFinder summary level code + geovariant code + '00US'</gco:CharacterString>
</gfc:definition>
<gfc:cardinality gco:nilReason="unknown"/>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
<gfc:listedValue>
<gfc:FC_ListedValue>
<gfc:label>
<gco:CharacterString>American FactFinder geographic identifier</gco:CharacterString>
</gfc:label>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
</gfc:FC_ListedValue>
</gfc:listedValue>
</gfc:FC_FeatureAttribute>
</gfc:carrierOfCharacteristics>
<gfc:carrierOfCharacteristics>
<gfc:FC_FeatureAttribute>
<gfc:memberName>
<gco:LocalName>GEOID</gco:LocalName>
</gfc:memberName>
<gfc:definition>
<gco:CharacterString>Nation identifier</gco:CharacterString>
</gfc:definition>
<gfc:cardinality gco:nilReason="unknown"/>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
<gfc:listedValue>
<gfc:FC_ListedValue>
<gfc:label>
<gco:CharacterString>US</gco:CharacterString>
</gfc:label>
<gfc:definition>
<gco:CharacterString>United States</gco:CharacterString>
</gfc:definition>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
</gfc:FC_ListedValue>
</gfc:listedValue>
</gfc:FC_FeatureAttribute>
</gfc:carrierOfCharacteristics>
<gfc:carrierOfCharacteristics>
<gfc:FC_FeatureAttribute>
<gfc:memberName>
<gco:LocalName>NAME</gco:LocalName>
</gfc:memberName>
<gfc:definition>
<gco:CharacterString>Nation name</gco:CharacterString>
</gfc:definition>
<gfc:cardinality gco:nilReason="unknown"/>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
<gfc:listedValue>
<gfc:FC_ListedValue>
<gfc:label>
<gco:CharacterString>United States</gco:CharacterString>
</gfc:label>
<gfc:definition>
<gco:CharacterString>Nation</gco:CharacterString>
</gfc:definition>
<gfc:definitionReference xlink:title="U.S. Census Bureau"
xlink:href="http://www.ngdc.noaa.gov/docucomp/eb139e38-ee29-4d59-b157-5e874d4420c4"/>
</gfc:FC_ListedValue>
</gfc:listedValue>
</gfc:FC_FeatureAttribute>
</gfc:carrierOfCharacteristics>
</gfc:FC_FeatureType>
</gfc:featureType>
</gfc:FC_FeatureCatalogue>
Loading
Loading