HANA database driver

[mrylov]

Dear GDAL/OGR community,

We, the SAP HANA Spatial Team, would like to add HANA support to the GDAL library. SAP HANA is an in-memory database with an OGC-compliant spatial engine. A free community edition of SAP HANA is available here.

Our previous attempt to get an approval from the PSC was not successful. For some reason, the PSC Members were rather inactive, except Even, and the decision has not been made. Therefore, we decided to take a chance with a pull request, hoping to find a community reviewer who will be interested to do the review.

Content

This pull request includes:

• Implementation of the driver
• Driver documentation
• Python tests
• CI integration using a HANA Cloud instance

Requirements

The HANA driver requires the open-source odbc-cpp-wrapper library that is going to be used only during the compilation phase and linked statically. Thus, end users will get only one dynamic/shared library of the HANA driver.

License

The HANA driver is licensed under an MIT/X style license.

Responsibilities

We promise to keep tracking of new issues/bugs, CI runs, API changes, documentation
improvements related to the HANA driver and make sure that they are fixed/implemented promptly.
There will be a contact person at SAP responsible for the maintenance.

[rouault]

• A "HANA" line, after the MapML one, should be added in frmts/drivers.ini so that when the driver is built as a plugin, it is loaded in the same order as when it is compiled in the GDAL core library

• I gave this a local try, and I get i"Invalid read", "Conditional jump or move depends on uninitialised value" and a memory leak when ingesting a shapefile with ogr2ogr

$ valgrind --leak-check=full ogr2ogr -update  "HANA:$HANA_CONNECTION_STR;ENCRYPT=YES;SSL_VALIDATE_CERTIFICATE=false;CHAR_AS_UTF8=1;SCHEMA=gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A" autotest/ogr/data/poly.shp -overwrite --debug on -lco OVERWRITE=YES
==815050== Memcheck, a memory error detector
==815050== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==815050== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==815050== Command: ogr2ogr -update HANA:DRIVER=/usr/sap/hdbclient/libodbcHDB.so;HOST=917df316-4e01-4a10-be54-eac1b6ab15fb.hana.prod-us10.hanacloud.ondemand.com;PORT=443;USER=GDALCI;PASSWORD=Z2!.S-bN]vh62\\G;ENCRYPT=YES;SSL_VALIDATE_CERTIFICATE=false;CHAR_AS_UTF8=1;SCHEMA=gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A autotest/ogr/data/poly.shp -overwrite --debug on -lco OVERWRITE=YES
==815050== 
[snip]
==815050== Invalid read of size 1
==815050==    at 0x483F85C: strcasecmp (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==815050==    by 0x664224F: SetupTargetLayer::Setup(OGRLayer*, char const*, GDALVectorTranslateOptions*, long long&) (ogr2ogr_lib.cpp:4177)
==815050==    by 0x663EB6D: GDALVectorTranslate (ogr2ogr_lib.cpp:3190)
==815050==    by 0x10D344: main (ogr2ogr_bin.cpp:414)
==815050==  Address 0x1328e130 is 16 bytes inside a block of size 120 free'd
==815050==    at 0x483CFBF: operator delete(void*) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==815050==    by 0x111D57F3: __gnu_cxx::new_allocator<OGRHANA::AttributeColumnDescription>::deallocate(OGRHANA::AttributeColumnDescription*, unsigned long) (new_allocator.h:128)
==815050==    by 0x111D372B: std::allocator_traits<std::allocator<OGRHANA::AttributeColumnDescription> >::deallocate(std::allocator<OGRHANA::AttributeColumnDescription>&, OGRHANA::AttributeColumnDescription*, unsigned long) (alloc_traits.h:469)
==815050==    by 0x111D1877: std::_Vector_base<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::_M_deallocate(OGRHANA::AttributeColumnDescription*, unsigned long) (stl_vector.h:351)
==815050==    by 0x111D35A0: void std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::_M_realloc_insert<OGRHANA::AttributeColumnDescription const&>(__gnu_cxx::__normal_iterator<OGRHANA::AttributeColumnDescription*, std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> > >, OGRHANA::AttributeColumnDescription const&) (vector.tcc:500)
==815050==    by 0x111D174B: std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::push_back(OGRHANA::AttributeColumnDescription const&) (stl_vector.h:1195)
==815050==    by 0x111DFFF1: OGRHANA::OGRHanaTableLayer::CreateField(OGRFieldDefn*, int) (ogrhanatablelayer.cpp:1393)
==815050==    by 0x6642A29: SetupTargetLayer::Setup(OGRLayer*, char const*, GDALVectorTranslateOptions*, long long&) (ogr2ogr_lib.cpp:4273)
==815050==    by 0x663EB6D: GDALVectorTranslate (ogr2ogr_lib.cpp:3190)
==815050==    by 0x10D344: main (ogr2ogr_bin.cpp:414)
==815050==  Block was alloc'd at
==815050==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==815050==    by 0x111D7F71: __gnu_cxx::new_allocator<OGRHANA::AttributeColumnDescription>::allocate(unsigned long, void const*) (new_allocator.h:114)
==815050==    by 0x111D73BF: std::allocator_traits<std::allocator<OGRHANA::AttributeColumnDescription> >::allocate(std::allocator<OGRHANA::AttributeColumnDescription>&, unsigned long) (alloc_traits.h:443)
==815050==    by 0x111D56EB: std::_Vector_base<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::_M_allocate(unsigned long) (stl_vector.h:343)
==815050==    by 0x111D33F8: void std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::_M_realloc_insert<OGRHANA::AttributeColumnDescription const&>(__gnu_cxx::__normal_iterator<OGRHANA::AttributeColumnDescription*, std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> > >, OGRHANA::AttributeColumnDescription const&) (vector.tcc:440)
==815050==    by 0x111D174B: std::vector<OGRHANA::AttributeColumnDescription, std::allocator<OGRHANA::AttributeColumnDescription> >::push_back(OGRHANA::AttributeColumnDescription const&) (stl_vector.h:1195)
==815050==    by 0x111CE53B: OGRHANA::OGRHanaLayer::ReadFeatureDefinition(CPLString const&, CPLString const&, CPLString const&, char const*) (ogrhanalayer.cpp:723)
==815050==    by 0x111DB6EF: OGRHANA::OGRHanaTableLayer::ReadTableDefinition() (ogrhanatablelayer.cpp:421)
==815050==    by 0x111DE880: OGRHANA::OGRHanaTableLayer::Initialize(char const*, char const*) (ogrhanatablelayer.cpp:1009)
==815050==    by 0x111BAF62: OGRHanaDataSource::ICreateLayer(char const*, OGRSpatialReference*, OGRwkbGeometryType, char**) (ogrhanadatasource.cpp:1623)
==815050==    by 0x64BAEA2: GDALDataset::CreateLayer(char const*, OGRSpatialReference*, OGRwkbGeometryType, char**) (gdaldataset.cpp:4626)
==815050==    by 0x6640F19: SetupTargetLayer::Setup(OGRLayer*, char const*, GDALVectorTranslateOptions*, long long&) (ogr2ogr_lib.cpp:3848)
==815050== 
HANA: Prepare statement INSERT INTO "gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A"."POLY" ("AREA", "EAS_ID", "PRFEDEA", "OGR_GEOMETRY") VALUES(?, ?, ?, ST_GeomFromWKB(? , 27700)).
HANA: Prepare statement SELECT CURRENT_IDENTITY_VALUE() "current identity value" FROM "gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A"."POLY".
GDALVectorTranslate: 10 features written in layer 'POLY'
Shape: 10 features read on layer 'poly'.
GDAL: GDALClose(autotest/ogr/data/poly.shp, this=0x10f369e0)
==815050== Conditional jump or move depends on uninitialised value(s)
==815050==    at 0x111DDD02: OGRHANA::OGRHanaTableLayer::HasPendingFeatures() const (ogrhanatablelayer.cpp:854)
==815050==    by 0x111DDC2D: OGRHANA::OGRHanaTableLayer::FlushPendingFeatures() (ogrhanatablelayer.cpp:839)
==815050==    by 0x111DB5CD: OGRHANA::OGRHanaTableLayer::~OGRHanaTableLayer() (ogrhanatablelayer.cpp:412)
==815050==    by 0x111DB689: OGRHANA::OGRHanaTableLayer::~OGRHanaTableLayer() (ogrhanatablelayer.cpp:413)
==815050==    by 0x587786F: std::default_delete<OGRLayer>::operator()(OGRLayer*) const (unique_ptr.h:81)
==815050==    by 0x5876BC7: std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >::~unique_ptr() (unique_ptr.h:292)
==815050==    by 0x59E34FC: void std::_Destroy<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> > >(std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*) (stl_construct.h:98)
==815050==    by 0x59E2AF4: void std::_Destroy_aux<false>::__destroy<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*>(std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*, std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*) (stl_construct.h:108)
==815050==    by 0x59E204B: void std::_Destroy<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*>(std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*, std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*) (stl_construct.h:137)
==815050==    by 0x59E0FDA: void std::_Destroy<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*, std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> > >(std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*, std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*, std::allocator<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> > >&) (stl_construct.h:206)
==815050==    by 0x5D59726: std::vector<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >, std::allocator<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> > > >::_M_erase_at_end(std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >*) (stl_vector.h:1793)
==815050==    by 0x5D59665: std::vector<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> >, std::allocator<std::unique_ptr<OGRLayer, std::default_delete<OGRLayer> > > >::clear() (stl_vector.h:1496)
==815050== 
GDAL: GDALClose(HANA:DRIVER=/usr/sap/hdbclient/libodbcHDB.so;HOST=917df316-4e01-4a10-be54-eac1b6ab15fb.hana.prod-us10.hanacloud.ondemand.com;PORT=443;USER=GDALCI;PASSWORD=Z2!.S-bN]vh62\G;ENCRYPT=YES;SSL_VALIDATE_CERTIFICATE=false;CHAR_AS_UTF8=1;SCHEMA=gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A, this=0x10f443c0)
GDAL: In GDALDestroy - unloading GDAL shared library.
==815050== 
==815050== HEAP SUMMARY:
==815050==     in use at exit: 51,856 bytes in 203 blocks
==815050==   total heap usage: 80,753 allocs, 80,550 frees, 18,611,539 bytes allocated
[snip]
==815050== 867 (792 direct, 75 indirect) bytes in 9 blocks are definitely lost in loss record 162 of 176
==815050==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==815050==    by 0x111CBDFD: OGRHANA::(anonymous namespace)::CreateFieldDefn(OGRHANA::AttributeColumnDescription const&) (ogrhanalayer.cpp:183)
==815050==    by 0x111CE37E: OGRHANA::OGRHanaLayer::ReadFeatureDefinition(CPLString const&, CPLString const&, CPLString const&, char const*) (ogrhanalayer.cpp:705)
==815050==    by 0x111DB6EF: OGRHANA::OGRHanaTableLayer::ReadTableDefinition() (ogrhanatablelayer.cpp:421)
==815050==    by 0x111DE880: OGRHANA::OGRHanaTableLayer::Initialize(char const*, char const*) (ogrhanatablelayer.cpp:1009)
==815050==    by 0x111B8E63: OGRHanaDataSource::InitializeLayers(char const*, char const*)::{lambda(char const*, bool)#1}::operator()(char const*, bool) const (ogrhanadatasource.cpp:1260)
==815050==    by 0x111B915E: OGRHanaDataSource::InitializeLayers(char const*, char const*) (ogrhanadatasource.cpp:1274)
==815050==    by 0x111B572F: OGRHanaDataSource::Open(char const*, char**, int) (ogrhanadatasource.cpp:695)
==815050==    by 0x111C5F4A: OGRHanaDriverOpen(GDALOpenInfo*) (ogrhanadriver.cpp:56)
==815050==    by 0x64B90D1: GDALOpenEx (gdaldataset.cpp:3511)
==815050==    by 0x663B59B: GDALVectorTranslate (ogr2ogr_lib.cpp:2243)
==815050==    by 0x10D344: main (ogr2ogr_bin.cpp:414)
==815050== 
==815050== LEAK SUMMARY:
==815050==    definitely lost: 880 bytes in 10 blocks

• I also get errors when reading a layer:

$ valgrind --leak-check=full ogrinfo   "HANA:$HANA_CONNECTION_STR;ENCRYPT=YES;SSL_VALIDATE_CERTIFICATE=false;CHAR_AS_UTF8=1;SCHEMA=gdal_test_20220317_9E6F323070478CCF1700AD12B99BE78A" poly -al  --debug on
[...]
==815290== Conditional jump or move depends on uninitialised value(s)
==815290==    at 0x1181E89A: checkODBCNumericIndicator (GenericOutputConverter.hpp:66)
==815290==    by 0x1181E89A: lttc::enable_if<((((81u)==((Communication::Protocol::DataTypeCodeEnum)81))||((81u)==((Communication::Protocol::DataTypeCodeEnum)82)))||((81u)==((Communication::Protocol::DataTypeCodeEnum)76)))&&((18)==((SQLDBC_HostType)18)), SQLDBC_Retcode>::type SQLDBC::Conversion::convertDatabaseToHostValue<81u, 18>(SQLDBC::Conversion::DatabaseValue const&, SQLDBC::Conversion::HostValue&, SQLDBC::Conversion::ConversionOptions const&) (FixedOutputConverter.cpp:62)
==815290==    by 0x117FB925: SQLDBC::Conversion::DataConversion::convertDatabaseToHostValue(Communication::Protocol::DataTypeCodeEnum, SQLDBC::Conversion::DatabaseValue const&, SQLDBC_HostType, SQLDBC::Conversion::HostValue&, SQLDBC::Conversion::ConversionOptions const&) (DataConversion.cpp:310)
==815290==    by 0x1150077C: SQLDBC_Retcode SQLDBC::ResultSet::readAndConvertValue<SQLDBC::HeapResultSetPart>(SQLDBC::HeapResultSetPart*, unsigned int, Communication::Protocol::DataTypeCodeEnum, SQLDBC::Conversion::DatabaseValue&, long long&, SQLDBC_HostType, SQLDBC::Conversion::HostValue&, SQLDBC::Conversion::ConversionOptions const&, SQLDBC::Error&) (ResultSet.cpp:1914)
==815290==    by 0x115184CE: SQLDBC::ResultSet::readAndConvertValue(int, Communication::Protocol::DataTypeCodeEnum, SQLDBC::Conversion::DatabaseValue&, long long&, SQLDBC_HostType, SQLDBC::Conversion::HostValue&, SQLDBC::Conversion::ConversionOptions const&, SQLDBC::Error&) (ResultSet.cpp:2266)
==815290==    by 0x11522E1C: SQLDBC::RowSet::getObject(int, void*, long long, SQLDBC_HostType, long long*, bool, long long*, long, bool) (RowSet.cpp:156)
==815290==    by 0x113ABE57: SQLDBC::SQLDBC_RowSet::getObject(int, SQLDBC_HostType, void*, long long*, long long, bool) (SQLDBC.cpp:3517)
==815290==    by 0x11363E61: SQLGetData (ODBC_Statement.cpp:2715)
==815290==    by 0x73FA2C1: SQLGetData (SQLGetData.c:474)
==815290==    by 0x112925CE: odbc::ResultSet::getDecimal(unsigned short) (in /home/even/install-odbc-cpp-wrapper/lib/libodbccpp.so)
==815290==    by 0x111CD909: OGRHANA::OGRHanaLayer::ReadFeature() (ogrhanalayer.cpp:557)
==815290==    by 0x111CCD54: OGRHANA::OGRHanaLayer::GetNextFeatureInternal() (ogrhanalayer.cpp:367)
==815290==    by 0x111CF347: OGRHANA::OGRHanaLayer::GetNextFeature() (ogrhanalayer.cpp:869)
==815290== 
==815290== Invalid read of size 2
==815290==    at 0x6F03605: from_utf16_loop (loop.c:336)
==815290==    by 0x6F03605: gconv (skeleton.c:620)
==815290==    by 0x714CC7E: __gconv (gconv.c:78)
==815290==    by 0x714C56A: iconv (iconv.c:52)
==815290==    by 0x52B5033: CPLRecodeFromWCharIconv(wchar_t const*, char const*, char const*) (cpl_recode_iconv.cpp:273)
==815290==    by 0x51A202C: CPLRecodeFromWChar (cpl_recode.cpp:204)
==815290==    by 0x111CAF7D: OGRHANA::OGRHanaFeatureWriter::SetFieldValue(int, char16_t const*) (ogrhanafeaturewriter.cpp:158)
==815290==    by 0x111CDC35: OGRHANA::OGRHanaLayer::ReadFeature() (ogrhanalayer.cpp:597)
==815290==    by 0x111CCD54: OGRHANA::OGRHanaLayer::GetNextFeatureInternal() (ogrhanalayer.cpp:367)
==815290==    by 0x111CF347: OGRHANA::OGRHanaLayer::GetNextFeature() (ogrhanalayer.cpp:869)
==815290==    by 0x647F998: OGRLayer::FeatureIterator::FeatureIterator(OGRLayer*, bool) (ogrlayer.cpp:4304)
==815290==    by 0x647FBAB: OGRLayer::begin() (ogrlayer.cpp:4356)
==815290==    by 0x116C4F: begin(OGRLayer*) (ogrsf_frmts.h:287)
==815290==  Address 0x12ed28b2 is 0 bytes after a block of size 18 alloc'd
==815290==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==815290==    by 0x517D2E5: VSIMalloc (cpl_vsisimple.cpp:561)
==815290==    by 0x516241E: CPLMalloc (cpl_conv.cpp:193)
==815290==    by 0x5162393: CPLCalloc (cpl_conv.cpp:152)
==815290==    by 0x52B4E87: CPLRecodeFromWCharIconv(wchar_t const*, char const*, char const*) (cpl_recode_iconv.cpp:219)
==815290==    by 0x51A202C: CPLRecodeFromWChar (cpl_recode.cpp:204)
==815290==    by 0x111CAF7D: OGRHANA::OGRHanaFeatureWriter::SetFieldValue(int, char16_t const*) (ogrhanafeaturewriter.cpp:158)
==815290==    by 0x111CDC35: OGRHANA::OGRHanaLayer::ReadFeature() (ogrhanalayer.cpp:597)
==815290==    by 0x111CCD54: OGRHANA::OGRHanaLayer::GetNextFeatureInternal() (ogrhanalayer.cpp:367)
==815290==    by 0x111CF347: OGRHANA::OGRHanaLayer::GetNextFeature() (ogrhanalayer.cpp:869)
==815290==    by 0x647F998: OGRLayer::FeatureIterator::FeatureIterator(OGRLayer*, bool) (ogrlayer.cpp:4304)
==815290==    by 0x647FBAB: OGRLayer::begin() (ogrlayer.cpp:4356)
==815290== 

[mrylov]

@rouault, I've taken all your points above into account.

• HANA is added to drivers.ini in b016de8
• Fixed memory leak and invalid read in f6e6cd9
• Fixed a conditional jump in bb651b4bd2050c7456b6d6009b43883e6a10c29e

The only issue that was not resolved is:
==815290== Conditional jump or move depends on uninitialised value(s)
==815290== at 0x1181E89A: checkODBCNumericIndicator (GenericOutputConverter.hpp:66)
==815290== by 0x1181E89A: lttc::enable_if<((((81u)==((Communication::Protocol::DataTypeCodeEnum)81))||((81u)==((Communication::Protocol::DataTypeCodeEnum)82)))||((81u)==((Communication::Protocol::DataTypeCodeEnum)76)))&&((18)==((SQLDBC_HostType)18)), SQLDBC_Retcode>::type SQLDBC::Conversion::convertDatabaseToHostValue<81u, 18>(SQLDBC::Conversion::DatabaseValue const&, SQLDBC::Conversion::HostValue&, SQLDBC::Conversion::ConversionOptions const&) (FixedOutputConverter.cpp:62)

To fix that I need to contact another team responsible for the HANA driver.

[mrylov]

@rouault, I have taken all your points into account. Could you please review this pull request again?

[rouault]

ok, seems ready for me to merge.
Can you edit https://github.com/OSGeo/gdal/wiki/Maintainers-per-sub-system to add a HANA line with maintainer(s) name/github handle, or some more generic contact (not sure if everybody has edit rights, otherwise just add it here)

[mrylov]

@rouault, thank you so much. I've added my name to the maintainers list.

[rouault]

actually I see there's no build testing for CMake builds. Would you mind adding at least build-time testing of the driver to the build-linux-ubuntu-focal job of .github/workflows/cmake_builds.yml (and a note as a commented line that there's no runtime checking of it for now if not adding it) ? There will be some more work in a couple month to rationalize a bit all those CI setups when we'll drop nmake & autoconf support.

[mrylov]

@rouault, I did some experiments with cmake_builds.yml, see here 11b213d0fabea0a5c3a2d3ed28744a1e3ec9afe5. Unfortunately, there is a failure in https://github.com/mrylov/gdal/runs/5767841355?check_suite_focus=true#step:14:1092. Is it an infrastructure issue or my GA profile simply does not have required credentials?

[rouault]

Is it an infrastructure issue or my GA profile simply does not have required credentials?

I assume this is just a temporary unrelated failure while retrieving https://gitlab.eumetsat.int/open-source/PublicDecompWT.git for the frmts/msg driver.
Could you revert the download of CMake 3.12 ? We want to check that GDAL can build with older CMake 3.9. Given that you don't put the old CMake in the path in the step where you build odbc-cpp , you should have the CMake from the Ubuntu 20.04 distribution (3.16).

[rouault]

merged!

[mrylov]

@rouault, excellent! Many thanks for your help!