Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table to IFeatureLayer #243

Open
wants to merge 45 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
fd7eeec
1st draft of an IFeatureLayer constructor from table
mathieu17g Oct 12, 2021
163e9b8
Fixed IFeatureLayer(table) for tables not supporting view on row (e.g…
mathieu17g Oct 12, 2021
9e0e900
Test table to `IFeatureLayer` conversion with a`missing`, mixed geome…
mathieu17g Oct 12, 2021
91e3a10
Fixed handling of type Any in a column and modified error messages
mathieu17g Oct 12, 2021
a79ec71
Fixed conversion to OGRFieldType error handing
mathieu17g Oct 12, 2021
843155d
JuliaFormatter formating
mathieu17g Oct 12, 2021
d8849e4
Handling of `Tables.schema` returning `nothing` or `Schema{names, no…
mathieu17g Oct 12, 2021
a5090a0
Added basic conversion to layer documentation in "Tables interface" s…
mathieu17g Oct 12, 2021
848fe0e
Completed doc on conversion to layer with an example of writing to a …
mathieu17g Oct 12, 2021
0bd76de
Added in doc an example of writing to the GML more capable driver
mathieu17g Oct 12, 2021
2045d55
Fixed a typo in table docs
mathieu17g Oct 12, 2021
a6277af
Added docstring with example to IFeatureLayer(table)
mathieu17g Oct 12, 2021
5035a0e
Formatted with JuliaFormatter
mathieu17g Oct 12, 2021
541ff9b
Fixed a typo on name option for IFeatureLayer(table)
mathieu17g Oct 12, 2021
1b6ab33
Added `nothing` values handling (cf. PR #238):
mathieu17g Oct 12, 2021
7fa434c
`_fromtable` function code readability enhancements:
mathieu17g Oct 12, 2021
8f8b1bd
Fixed a typo in `_fromtable` column types conversion error message
mathieu17g Oct 12, 2021
0e12703
Corrections following @yeesian review
mathieu17g Oct 13, 2021
6454e09
Fixed arg type in `ctv_toWKT` signature ("Table to layer conversion" …
mathieu17g Oct 13, 2021
8f07154
Added GeoInterface geometries handling to table to layer conversion
mathieu17g Oct 16, 2021
a4bd378
Fixed typo in GeoInterface to IGeometry conversion and added tests on it
mathieu17g Oct 16, 2021
7166097
Added tests on GeoInterface geometries types to `OGRwkbGeometryType`
mathieu17g Oct 16, 2021
33c3a76
Formatting and cleaning
mathieu17g Oct 16, 2021
0b1b9d9
Added WKT and WKB parsing options in table to layer conversion
mathieu17g Oct 21, 2021
f8da022
Enhance test coverage by pruning handling of `Vector{String}` convers…
mathieu17g Oct 22, 2021
c0e4da6
Corrected a typo in `_fromtable`
mathieu17g Oct 22, 2021
77d561d
Extended test coverage by testing individually GeoInterface, WKT, WKB…
mathieu17g Oct 22, 2021
a33bc88
Merge branch 'master' into IFeatureLayer_from_table
mathieu17g Oct 28, 2021
b08919b
Folliowing @visr and @yeesian remarks:
mathieu17g Oct 28, 2021
d41cbc2
Fixed test_tables.jl to use `layer_name` kwarg instead of `name` to c…
mathieu17g Oct 28, 2021
1ff0196
Added geometry column specification kwarg `geom_cols`, without any te…
mathieu17g Oct 29, 2021
125a076
Added `fieldtypes` kwarg option to table to layer conversion, without…
mathieu17g Oct 30, 2021
9e90dc1
Added tests on `geomcols` kwarg and corrected issues revealed while a…
mathieu17g Oct 31, 2021
f57a867
A few issue fixes while adding test to `fieldtypes` kwarg option.
mathieu17g Nov 1, 2021
8911988
Added test on `fieldtypes` kwarg and fixed issues raised by those tests
mathieu17g Nov 1, 2021
fbb4618
Added error message test in `@test_throws` tests for `fieldtypes` and…
mathieu17g Nov 2, 2021
1d39170
Merge branch 'master' into IFeatureLayer_from_table
mathieu17g Nov 3, 2021
5cd1f16
Modified conversion from `GeoInterface.AbstracGeometry` to `IGeometry`
mathieu17g Nov 3, 2021
d08a47c
Exclude columns in `fieldtypes` from WKT/WKB parsing in `_infergeomet…
mathieu17g Nov 3, 2021
ad14efd
Typo in the previous commit `spfieldtypes` intead of `fieldtypes`
mathieu17g Nov 3, 2021
0c61b85
Deleted 2 comments in `_fromtable(sch::Tables.Schema{names,types}, ..…
mathieu17g Nov 3, 2021
60df39e
Dropped `fieldtypes` kwarg in src and tests
mathieu17g Nov 27, 2021
a197973
Merge branch 'master' into IFeatureLayer_from_table
mathieu17g Nov 27, 2021
6ae870c
Refixed `ogrerr` macro
mathieu17g Nov 27, 2021
e8560e4
Dropped `geomcols` kwarg and WKT/WKB parsing in src/ and test/
mathieu17g Nov 28, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 4 additions & 6 deletions benchmark/remotefiles.jl
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,10 @@ julia> open(filepath/filename) do f
end
```
"""
remotefiles = [
(
"data/road.zip",
"058bdc549d0fc5bfb6deaef138e48758ca79ae20df79c2fb4c40cb878f48bfd8",
),
]
remotefiles = [(
"data/road.zip",
"058bdc549d0fc5bfb6deaef138e48758ca79ae20df79c2fb4c40cb878f48bfd8",
)]

function verify(path::AbstractString, hash::AbstractString)
@assert occursin(r"^[0-9a-f]{64}$", hash)
Expand Down
59 changes: 51 additions & 8 deletions docs/src/tables.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,71 @@
# Tabular Interface

```@setup tables
using ArchGDAL, DataFrames
using ArchGDAL; AG = ArchGDAL
using DataFrames
```

ArchGDAL now brings in greater flexibilty in terms of vector data handling via the
[Tables.jl](https://github.com/JuliaData/Tables.jl) API. In general, tables are modelled based on feature layers and support multiple geometries per layer. Namely, the layer(s) of a dataset can be converted to DataFrame(s) to perform miscellaneous spatial operations.

## Conversion to table

Here is a quick example based on the
[`data/point.geojson`](https://github.com/yeesian/ArchGDALDatasets/blob/307f8f0e584a39a050c042849004e6a2bd674f99/data/point.geojson)
dataset:

```@example tables
dataset = ArchGDAL.read("data/point.geojson")

DataFrames.DataFrame(ArchGDAL.getlayer(dataset, 0))
```@repl tables
ds = AG.read("data/point.geojson")
DataFrame(AG.getlayer(ds, 0))
```

To illustrate multiple geometries, here is a second example based on the
[`data/multi_geom.csv`](https://github.com/yeesian/ArchGDALDatasets/blob/master/data/multi_geom.csv)
dataset:

```@example tables
dataset1 = ArchGDAL.read("data/multi_geom.csv", options = ["GEOM_POSSIBLE_NAMES=point,linestring", "KEEP_GEOM_COLUMNS=NO"])
```@repl tables
ds = AG.read("data/multi_geom.csv", options = ["GEOM_POSSIBLE_NAMES=point,linestring", "KEEP_GEOM_COLUMNS=NO"])
DataFrame(AG.getlayer(ds, 0))
```
## Conversion to layer
A table-like source implementing Tables.jl interface can be converted to a layer, provided that:
- Source contains at least one geometry column
- Geometry columns are recognized by their element type being a subtype of `Union{IGeometry, Nothing, Missing}`
- Non geometry columns must contain types handled by GDAL/OGR (e.g. not `Int128` nor composite type)

**Note**: As geometries and fields are stored separately in GDAL features, the backward conversion of the layer won't have the same column ordering. Geometry columns will be the first columns.

DataFrames.DataFrame(ArchGDAL.getlayer(dataset1, 0))
```@repl tables
df = DataFrame([
:point => [AG.createpoint(30, 10), missing],
:mixedgeom => [AG.createpoint(5, 10), AG.createlinestring([(30.0, 10.0), (10.0, 30.0)])],
:id => ["5.1", "5.2"],
:zoom => [1.0, 2],
:location => [missing, "New Delhi"],
])
layer = AG.IFeatureLayer(df)
```

The layer, converted from a source implementing the Tables.jl interface, will be in a [memory](https://gdal.org/drivers/vector/memory.html) dataset.
Hence you can:
- Add other layers to it
- Copy it to a dataset with another driver
- Write it to a file
### Example of writing with ESRI Shapefile driver
```@repl tables
ds = AG.write(layer.ownedby, "test.shp", driver=AG.getdriver("ESRI Shapefile"))
DataFrame(AG.getlayer(AG.read("test.shp"), 0))
rm.(["test.shp", "test.shx", "test.dbf"]) # hide
```
!!! warning
As OGR ESRI Shapefile driver [does not support multi geometries](https://gdal.org/development/rfc/rfc41_multiple_geometry_fields.html#drivers), the second geometry has been dropped
!!! warning
As OGR ESRI Shapefile driver does not support nullable fields, the `missing` location has been replaced by `""`
### Example of writing with GML driver
Using the GML 3.2.1 more capable driver/format, you can write more information to the file
```@repl tables
ds = AG.write(layer.ownedby, "test.gml", driver=AG.getdriver("GML"), options=["FORMAT=GML3.2"])
DataFrame(AG.getlayer(AG.read("test.gml", options=["EXPOSE_GML_ID=NO"]), 0))
rm.(["test.gml", "test.xsd"]) # hide
```
**Note:** [OGR GML driver](https://gdal.org/drivers/vector/gml.html#open-options) option **EXPOSE\_GML\_ID=NO** avoids to read the `gml_id` field, mandatory in GML 3.x format and automatically created by the OGR GML driver
13 changes: 8 additions & 5 deletions src/base/display.jl
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,8 @@ function Base.show(io::IO, layer::AbstractFeatureLayer)::Nothing
end
if ngeomdisplay == 1 # only support printing of a single geom column
for f in layer
geomwkt = toWKT(getgeom(f))
geom = getgeom(f)
geomwkt = geom.ptr != C_NULL ? toWKT(geom) : "NULL Geometry"
length(geomwkt) > 25 && (geomwkt = "$(geomwkt[1:20])...)")
newdisplay = "$display, $geomwkt"
if length(newdisplay) > 75
Expand Down Expand Up @@ -251,26 +252,28 @@ function Base.show(io::IO, spref::AbstractSpatialRef)::Nothing
return nothing
end

function Base.show(io::IO, geom::AbstractGeometry)::Nothing
function Base.show(io::IO, geom::AbstractGeometry, prefix::String)::Nothing
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be named as _show or something, or does it need to have the Base. prefix?

if geom.ptr == C_NULL
print(io, "NULL Geometry")
print(io, "NULL $(prefix)Geometry")
return nothing
end
compact = get(io, :compact, false)

if !compact
print(io, "Geometry: ")
print(io, "$(prefix)Geometry: ")
geomwkt = toWKT(geom)
if length(geomwkt) > 60
print(io, "$(geomwkt[1:50]) ... $(geomwkt[(end - 4):end])")
else
print(io, "$geomwkt")
end
else
print(io, "Geometry: $(getgeomtype(geom))")
print(io, "$(prefix)Geometry: $(getgeomtype(geom))")
end
return nothing
end
Base.show(io::IO, geom::IGeometry) = show(io, geom, "I")
Base.show(io::IO, geom::Geometry) = show(io, geom, "")

function Base.show(io::IO, ct::ColorTable)::Nothing
if ct.ptr == C_NULL
Expand Down
33 changes: 33 additions & 0 deletions src/ogr/geometry.jl
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ function unsafe_fromWKB(data)::Geometry
return Geometry(geom[])
end

convert(::Type{IGeometry}, data::Vector{UInt8}) = fromWKB(data)

"""
fromWKT(data::Vector{String})

Expand Down Expand Up @@ -74,6 +76,8 @@ fromWKT(data::String, args...)::IGeometry = fromWKT([data], args...)
unsafe_fromWKT(data::String, args...)::Geometry =
unsafe_fromWKT([data], args...)

convert(::Type{IGeometry}, s::String) = fromWKT(s)

"""
Destroy geometry object.

Expand Down Expand Up @@ -1636,3 +1640,32 @@ for (f, rt) in ((:create, :IGeometry), (:unsafe_create, :Geometry))
end
end
end

# Conversion from GeoInterface geometry
function convert(::Type{IGeometry}, g::GeoInterface.AbstractPoint)
return createpoint(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractMultiPoint)
return createmultipoint(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractLineString)
return createlinestring(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractMultiLineString)
return createmultilinestring(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractPolygon)
return createpolygon(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractMultiPolygon)
return createmultipolygon(GeoInterface.coordinates(g))
end
function convert(::Type{IGeometry}, g::GeoInterface.AbstractGeometryCollection)
ag_geom = creategeom(wkbGeometryCollection)
for gi_subgeom in GeoInterface.geometries(g)
ag_subgeom = convert(IGeometry, gi_subgeom)
result = GDAL.ogr_g_addgeometry(ag_geom.ptr, ag_subgeom.ptr)
@ogrerr result "Failed to add $ag_subgeom"
end
return ag_geom
end
Loading