Recent builds cause hanging on Windows #182
Labels
Comments
|
I'm still trying to make a smaller reproducible example beyond running all of Satpy's tests. However, Satpy's tests are very complicated as far as dependencies go. They include GDAL, netcdf4-python, xarray, dask, cython extensions, etc. So there could be a lot of things conflicting with libnetcdf beyond the changes to the builds. |
|
Ah @simonrp84 just noted that these errors show up when his pytest command hangs: |
|
Ok I've got a reproducible example without Satpy: Run with |
|
This also affects GDAL CI on Windows whose tests involving netcdf hang: https://github.com/OSGeo/gdal/actions/runs/5589991097/jobs/10222968884 And I also traced it to the nompi_h5902ca5_107 --> nompi_h624ddae_109 upgrade |
rouault
added a commit
to rouault/gdal
that referenced
this issue
Jul 19, 2023
|
A quick test on my CI shows 108 as the problem. Might have to revert the S3 support. CC @zklaus @DennisHeimbigner @dopplershift |
|
Just to confirm, you are saying 107 works as expected? |
rouault
added a commit
to OSGeo/gdal
that referenced
this issue
Jul 20, 2023
CI: avoid hang on Windows Conda testing related to conda-forge/libnetcdf-feedstock#182
djhoese
changed the title
|
It is a bit strange because the test you propose doesn't seem to have anything to do with S3, so I am wondering if it's a random upgrade in the environment that is causing this. To test, I'll add the test as a test to the recipe and see if de-/activating makes a difference. If the trace given above is any indication, h5netcdf or h5py might be involved. |
zklaus
pushed a commit
to zklaus/libnetcdf-feedstock
that referenced
this issue
Jul 20, 2023
5 tasks
|
I also thought h5netcdf or h5py could be the issue, but when I narrowed it down to my simple Python example it seems less likely. I can try doing one with only importing the lower-level netcdf4-python library. If @rouault knows the exact part of GDAL that does the NetCDF writing, or if there is a test function that writes a NetCDF file, maybe that would also trigger it and remove Python from the equation entirely. The exception I posted earlier is very strange, but may be a conflict between h5py and netcdf-python talking to HDF5/NetCDF-C internals. As far as runtime environment, it is exactly the same except for that libnetcdf version. Build environment...might be a different story. I suppose the logs for those are long gone though. |
From my manual testing, it is clearly the nompi_h5902ca5_107 --> nompi_h624ddae_108 upgrade. Manual installation of 107 results in OK behaviour. Manual installation in 108 results in hangs, both when reading or writing with GDAL utilities. Interstingly the ncdump and ncgen utilities don't seem to be affected |
2 tasks
|
@zklaus Got my python script down to this: I threw the |
|
Thanks, @djhoese, that's a nice and compact test! I have plugged your earlier test into the recipe in #183, but curiously, the tests seem not to be run on Windows?! Does anyone know what's up with that? I note that there is a |
|
@zklaus should we mark the Windows builds >=107 as broken until we figure this out? I can do that if you all agree. |
should be: >= 108 . 107 is fine |
|
Heads up @WardF |
|
Is there any chance someone can do the following:
Warning, the trace output may be voluminous. |
|
If someone can tell me how to compile my little C program above with conda-forge's preferred compilers and just the command line (something I can put into CI) we may be able to reduce the amount of output by a lot assuming the test C program triggers the same hanging issue. |
zklaus
pushed a commit
to zklaus/libnetcdf-feedstock
that referenced
this issue
Jul 21, 2023
Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com> Co-authored-by: Filipe <ocefpaf@gmail.com>
I tried to do this now in #185. Let me know what you think! |
|
Just to let everyone know, I will be on vacation from tomorrow until 23. August. Feel free to take over my branches and to play around. |
jeffypooo
pushed a commit
to IgnisTechnologies/gdal
that referenced
this issue
Aug 9, 2023
It does, in fact, still hang.
Your supposition sounds right to me; I've just started digging into the AWS SDK documentation looking to see if there is something subtle happening that I can suss out, but if it's Python garbage collection, is that something we are able to do anything about? I'll keep digging, and will see if I can replicate this in C, but the fact it's triggered by |
|
I guess that, at this point, we should probably open an issue in netcdf4-python. Or at least loop them in here. I don't think the tests upstream covers those options we just enabled. |
It has likely nothing to do with python. It's definitely reproducible in C/C++ since GDAL C++ command line utilities stall on exit when opening a netcdf file |
That's what makes it an interesting wrinkle; not sure what's happening in terms of the |
|
Ward- There is an experiment you can try to see if it is aws-sdk-cpp.
|
1 task
Thanks for the tip; I'll give it a try when I replicate the python script in C. Or, if I get to the point where I need to figure out how to get Python to invoke the netcdf.dll I'm compiling :). Thanks! |
|
I'm able to recreate a hang now in the netCDF-C tests. Interesting; it gives me a good place to start. |
Thanks @WardF for digging into this! It is not an easy debugging for sure! (Hoping for an easy fix though ;-p) |
|
@DennisHeimbigner I'm going to open an issue over on the Unidata/netcdf-c page. I'm seeing a hang in the first test run ( This obviates the
|
Hoping it's one of those cases like the other 99%, where the difficulty is in diagnosing, but the issue is straightforward. I have to shift context for a little bit, but immediately I see that a NULL pointer is being passed as part of |
This was referenced Aug 24, 2023
|
Hi folks, I'm wondering at what point we decide that an easy fix is not in the cards and roll back the changes so conda-forge can move on with various migrations that are stuck because of this issue. Is there any sign that a fix is imminent? |
|
I certainly don't mean to imply that I don't appreciate the work being done to debug this issue. Far from it! But I appreciate that it can be hard to find these issues and that we all have other demands on our time as well. |
|
|
|
@xylar IMO, it's perfectly reasonable to revert and move on until this is properly fixed upstream. But I'll leave it to those who are willing to do the work of reverting (and possibly un-reverting...). |
zklaus
pushed a commit
to zklaus/libnetcdf-feedstock
that referenced
this issue
Sep 5, 2023
Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com> Co-authored-by: Filipe <ocefpaf@gmail.com>
|
Hello all. Having popped back up the stack, I believe we may have a fix for this at the underlying C library. This was an interesting issue to try to diagnose and fix. As usual, the fix on our end (for the short term) was far easier than the effort put into figuring out what was going on in the first place. I'm not sure immediately how to test our fix against the Other PR's and issues have cropped up while I was focused on this, but I will be triaging and patching ASAP so that we might get a |
|
Hm. Unfortunately, it does not seem to work. |
|
-DENABLE_S3_SDK=FALSE is not a flag for netCDF-C; you should instead set |
|
I note that we did have an ENABLE_S3_SDK option in 4.9.2. |
|
Ok, things are more or less clear than now: An earlier version of @WardF's comment seemed to suggest that this fix could be applied to version 4.9.2. This was a misinterpretation on my part and we have to wait for 4.9.3. Thus, my attempt in #185 to use it with 4.9.2 was doomed. On the upside, I learned to replace |
|
My apologies for the confusion; I am working to get a v4.9.3 out ASAP. A number of other PR's that need to be addressed made their way into the queue during the time we spent hashing this out, but I'm hoping we can have something in the next week or so. The release of HDF5 1.14.2 is going to require some testing as well. Thanks! |
clrpackages
pushed a commit
to clearlinux-pkgs/gdal
that referenced
this issue
Nov 14, 2023
Aaron Boxer (1):
JP2OpenJPEG: refactor into re-usable template
Alan Thomas (2):
DXF: Preserve attributes in nested block insertions
DXF: Show constant ATTDEFs as text
Alessandro Pasotti (13):
MSSQL: fix crash in ogr2ogr when BCP is enabled (#7792)
MSSQL: fix GEOGRAPHY clockwise vertex order (#7826)
MSSQL: Fix cannot write binary field in BCP mode (#7841)
MSSQL: add test for geometry column identification (#7871)
OGCAPI: Fix TILES raster API not working
Add OGCAPI intial tests (#7926)
OGCAPI: more tests for errors (#8110)
Fix build with DEBUG_VERBOSE on
OPENCL: scan multiple devices for each platform (#8537)
OGR2OGR: fix preserve_fid and explodecollections incompatible options (#8539)
API: add GDALGetSubdatasetInfo() and related functions (#8155)
OAPIF: bump default limit to 1000 and honor schema from API (#8566)
OGR2OGR: test JSON types roundtrip (#8609)
Andrea Giudiceandrea (4):
[Shapefile] SHPRestoreSHX: update SHX content length even if error occurred
Docs: fix gdal_rasterize.rst
Doc: substitute remotesensing.org/geotiff/proj_list with gdal.org/proj_list/
Fix link in javadoc.java
Barry DeZonia (6):
SWIG JAVA: add Read/WriteRaster abilities for GDT_UInt64 & GDT_Int64 (#7893)
Java binding: eliminate some deprecations / compiler warnings (#8055)
Add a compatiblity method to swig bindings that is Java Only
Make sure to remove the %rename for GetFieldType from ogr_java.i
Java binding: make multidimensional API usable (#8048)
Make the 2d multidim test exercise itself harder
Bas Couwenberg (1):
Include <cstdint> for uint64_t.
Brad Hards (1):
Update gmlas.rst
Bryce Mecum (1):
Rename arrow repo links from master to main
Chris Toney (2):
updates to the documentation for LCP format
Doc: update url for R bindings
Craig de Stigter (1):
WFS: Don't issue STARTINDEX if feature count is small (#8146)
Dan Baston (11):
Update doc/source/drivers/raster/bag.rst
Update doc/source/drivers/raster/bag.rst
Update doc/source/drivers/vector/eeda.rst--
Update doc/source/drivers/raster/blx.rst
Update doc/source/drivers/vector/geoconcept.rst
Update doc/source/drivers/vector/elasticsearch.rst
Apply suggestions from code review
Update autotest/pyscripts/test_gdal_ls_py.py
autotest: skip test_vsicurl_streaming_1 on Travis
validate_geoparquet: avoid GitHub login via .netrc (#8096)
Doc: Update Python gotchas (#8489)
Daniel Baston (262):
Doc: Remove outdated information on building Python bindings
autotest: fix incomplete download when size is unknown
autotest: simplify gdaltest.download_file
ogr_sxf.py: Quiet debugging output
autotest: skip tests requiring unavailable downloads
hdf5.py: Update test data URL
autotest: Consolidate JSON validation logic, catch timeout errors
CI: Strip invalid characters from container tag
Doc: Add doxygen target to doc Makefile
Python bindings: Return context manager from Create, CreateDataSource
Doc: Add directives/roles for declaring and referencing options
configoptions.rst: Use source file links
configoptions.rst: Move CPL_VSIL_CURL* to Networking section
configoptions.rst: Remove GDAL_CACHEMAX history
configoptions.rst: Remove GDAL_VALIDATE_CREATION_OPTIONS
Doc: Remove old config options index
Doc: Use config directives in driver pages (raster)
Doc: Use config directives in driver pages (vector)
Doc: Use config directives in program pages
Doc: Use config directives in user docs
Doc: Remove :decl_configoption:
georaster.rst: format code blocks
configoptions.rst: reorganize global options
Doc: fix typos, formatting
configoptions.rst: Reformat options as list
bag.rst: fix typo
idb.rst: Distinguish between config options and env vars
Doc: Fix typos in libkml config option names
Doc: correct name of GDAL_VRT_PYTHON_TRUSTED_MODULES
Doc: correct name of PDF_USER_PWD config option
Doc: Ignore config option usages not in PDF output
GDALRasterBand: Document and validate GDAL_BAND_BLOCK_CACHE
Doc: Document OGR_PROMOTE_TO_INTEGER64
Doc: Document GDAL_MAX_BAND_COUNT
Doc: Add documentation for HTTP config options
Doc: Add entry for GDAL_CONFIG_FILE
Doc: Document GDAL_PYTHON_DRIVER_PATH
Doc: Migrate gdal_merge.py examples from wiki
Doc: Migrate gdalwarp information from wiki
Doc: Migrate gdal_translate information from wiki
Doc: Migrate ogrinfo information from wiki
configoptions.rst: Remove default for GDAL_NUM_THREADS
Doc: Improve documentation for OGR_ORGANIZE_POLYGONS
autotest: Add some tests for OGRGeometryFactory::organizePolygons
CI: Run ctest in parallel for most Linux builds
autotest: Add attribute_filter and spatial_filter context managers
autotest: Use context managers to remove "assert tr" pattern
autotest gdrivers/gpkg.py: Update code conventions
autotest: use asserts in ogrtest helpers
autotest: remove asserts of ogrtest helper function return values
autotest: call pytest.skip from skip_on_travis()
autotest: avoid calling post_reason in gdaltest.equal_srs_from_wkt
autotest: use assert in gdaltest.rpcs_equal, avoid post_reason
autotest: Move assert inside gdaltest.geotransform_equals
autotest: remove gdaltest.post_reason
autotest: skip test_http_1 on Travis
autotest: Enhancements to ogrtest.check_feature_geometry
autotest: check for mismatched dims in ogrtest.check_feature_geometry
Doc: Update PAM config options
organizePolygons: Remove handling of nonpolygonal geometries
autotest netcdf.py: make tests independent
autotest ogr_sql_sqlite.py: make tests independent
Python bindings: Invalidate band and layer refs when dataset closes
SWIG: Increment FeatureDefn ref count on ogr.Layer.GetLayerDefn()
autotest: remove global setting of ECW_DO_NOT_RESOLVE_DATUM_PROJECTION
Python bindings: Invalidate mask, overview references after dataset close
Python bindings: Invalidate refs from CreateLayer on datasource close
Python bindings: Invalidate refs from CopyLayer on datasource close
autotest ogr_sql_test.py: make tests independent
autotest vsizip.py: disable exceptions
autotest tiff_read.py: make tests independent
autotest pam.py: make tests independent
autotest rat.py: make tests independent
autotest ogr_sql_rfc28.py: make tests independent
autotest colortable.py: make tests independent
autotest gdal_stats.py: make tests independent
autotest hfa_write.py: make tests independent
GDAL_RASTERIO_RESAMPLING: add doc and test
Doc: Add entry for GDAL_READDIR_LIMIT_ON_OPEN
Doc: Document COMPRESS_GEOM configuration option
Doc: Document CHECK_DISK_FREE_SPACE
Doc: Add docs for VRT_SHARED_SOURCE
Doc: formatting fix in jp2openjpeg.rst
Doc: Add USE_TILE_AS_BLOCK to JP2KAK
Doc: Add USE_SPILL configuration option to HFA
OSR_WKT_FORMAT: Add doc and test
Doc: Add docs for some OSR options
OSR_DEFAULT_AXIS_MAPPING_STRATEGY: Add doc and test
autotest: add test to clarify configuration option/env var precedence
Python bindings: Fix gdal.config_options to prevent migration of config
autotest: remove unused globals, functions from gdaltest
autotest: remove usage of GDALTest return values
autotest: fix test_misc_6 incorrect early return before test completion
autotest: remove some always-true checks
autotest: remove assignment of non-existing return values
rasterlite: restore original state after temporarily setting OGR_SQLITE_EXACT_EXTENT
SWIG: Expose GetConfigOptions()
Python bindings: Allow passing options as dict
autotest: fix execution of commands on Windows with backslashes
autotest test_gdal_contour.py: make tests independent
autotest test_gdal_grid.py: make tests independent
autotest test_gdal_grid.py: split and parametrize tests, use tmp_path
autotest test_gdaladdo.py: make tests independent
autotest test_gdaldem.py: make tests independent
autotest test_nearblack.py: make tests independent
autotest test_ogrlineref.py: make tests independent
autotest test_gdal_translate.py: make tests independent
autotest test_gdalwarp.py: make tests independent
autotest test_gdalwarp_lib.py: make tests independent
autotest test_ogrtindex.py: move cleanup into fixture
autotest test_gdalbuildvrt.py: make tests independent, use tmp_path
autotest: disable random order for test_gdaltindex and test_gdalgnmutils
autotest test_gdal_merge.py: make tests independent, use tmp_path
gdal_pansharpen.py: avoid error trying to generate relative path
autotest test_gdal_pansharpen.py: make tests independent, use tmp_path
autotest test_gdal_retile.py: make tests independent, use tmp_path
autotest: Remove test for deprecated raster GMT driver
Doc: Document GDAL_OVR_PROPAGATE_NODATA
autotest test_pct.py: make tests independent, use tmp_path
autotest mask.py: make tests independent
autotest histogram.py: make tests independent
autotest numpy_rw_multidim.py: move setup to fixture
autotest vsicrypt.py: move setup to fixture
autotest vsihdfs.py: move setup into fixture
autotest gcore/tiff: make tests independent
autotest hfa_rfc40.py: mark to disabled random order
autotest gif.py: make tests independent
autotest bsb.py: use pytest.mark.require_driver
autotest bt.py: use pytest.mark.parameterize, move cleanup to fixture
autotest vrtrawlink.py: make tests independent, use tmp_path
autotest vrtwarp.py: combine two interdependent tests
autotest rasterlite.py: combine two interdependent tests
autotest postgisraster.py: move cleanup into fixture, add skip messages
autotest mbtiles.py: move webserver into fixture
autotest test_validate_jp2.py: move setup, cleanup into fixture
autotest kmlsuperoverlay.py: move cleanup into fixture
autotest jpeg.py: make tests independent
autotest hfa.py: combine interdependent tests
autotest srtmhgt.py: make tests independent
autotest srp.py: parametrize tests, use fixture for cleanup
autotest ehdr.py: merge two interdependent tests
autotest esric.py: use fixture for sample dataset
autotest fast.py: use fixture for test dataset
autotest: restore netcdf_cf checks
autotest wms.py: update SEDAC url, re-enable tests
autotest wms.py: use new ESRI test server
autotest wms.py: pull TileCache test server into fixture
autotest wms.py: make GIBS TWMS tests independent
autotest wms.py: fix always-skipped test, use tmp_path for wms cache
autotest wcs.py: parametrize test_wcs_6, remove global variables
autotest wcs.py: use fixture for geoserver.wcs
autotest plmosaic.py: make tests independent
autotest nitf.py: move cleanup to fixture, merge interdependent tests
autotest: disable random order for gdrivers/ngw.py
autotest pcidsk.py: combine interdependent tests
autotest: test_ogrinfo.py: use tmp_path fixture
autotest: test_gdal_create.py: use tmp_path fixture
autotest: test_gdal_footprint.py: use tmp_path fixture
autotest: test_ogr2ogr.py: use tmp_path fixture
autotest: test_gdal_rasterize.py: use tmp_path fixture
autotest: test_gdaladdo.py: use tmp_path fixture
autotest: test_gdalinfo.py: use tmp_path fixture
autotest: test_gdalsrsinfo.py: use tmp_path fixture
autotest: test_gdallocationinfo.py: use tmp_path fixture
autotest: test_gdaldem.py: use tmp_path fixture
autotest: test_ogrtindex.py: use tmp_path fixture
autotest: test_gdalwarp_lib.py: use tmp_path fixture
autotest: test_gdal_translate.py: use tmp_path fixture
autotest: test_gdalmdimtranslate.py: use tmp_path fixture
autotest: test_ogr2ogr_lib.py: use tmp_path fixture
autotest: test_gdal_rasterize_lib.py: use tmp_path fixture
autotest: test_gdalwarp.py: use tmp_path fixture
autotest: test_ogrlineref.py: use tmp_path fixture
autotest: test_nearblack.py: use tmp_path fixture
autotest: test_gdal_translate_lib.py: use tmp_path fixture
autotest: test_gdaltindex.py: use tmp_path fixture
autotest: test_sozip.py: use tmp_path fixture
autotest: test_gdal_viewshed.py: use tmp_path fixture
autotest: test_gnmutils.py: use tmp_path fixture
autotest: add tmp_vsimem fixture
autotest: ogr_rfc35_shape.py: make tests independent
autotest: combine ogr_rfc_35 tests
OGR: Add OGR_F_DumpReadableAsString
Python bindings: Use DumpReadable for Feature __repr__
Python bindings: Accept os.PathLike arguments where applicable
autotest: test that Python bool can be used in options dict
autotest: use gdal.quiet_errors()
Python bindings: Support os.PathLike inputs in utilities
Python bindings: accept options as dicts in CLI wrapper functions
autotest: contour.py: make tests independent
autotest: ogr_basic_test.py: make tests independent, use tmp_path
autotest: ogr_kml.py: combine two interdependent tests, use tmp_ fixtures
autotest: ogr_mem.py: make tests independent
autotest: ogr_shape.py: make tests independent, use tmp_path and tmp_vsimem
autotest: ogr_svg.py: make tests independent
autotest: ogr_tiger.py: make tests independent
autotest: ogr_pdf.py: make tests independent
autotest: ogr_vrt.py: make tests independent, use tmp_ fixtures
autotest: ogr_csv.py: make tests independent, use tmp_ fixtures
autotest: ogr_dgn.py: merge two interdependent tests
autotest: ogr_sxf.py: avoid assigning global variables
autotest: ogr_pcidsk.py: make tests independent
autotest: ogr_jml.py: use tmp_vsimem, split tests
autotest: ogr_mitab.py: make tests independent, parametrize and use tmp_ fixtures
autotest: ogr_libkml.py: combine interdependent tests, parametrize and use tmp_ fixtures
autotest: ogr_layer_algebra.py: split tests, use fixtures for inputs
autotest: ogr_gml.py: make tests independent, parametrize, use tmp_ fixtures
autotest: ogr_georss.py: make tests independent
autotest: ogr_pythondrivers.py: use fixture for setup and cleanup
autotest: ogr_ili: use tmp_path, mark as order-dependent
autotest: ogr_selafin.py: make tests independent
autotest: ogr_vfk.py: make tests independent
autotest: ogr_vdv.py: make tests independent, parametrize
autotest: ogr_s57.py: make tests independent
autotest: ogr_csw.py: make tests independent, use tmp_vsimem
autotest: ogr_plscenes.py: use pytest.mark.require_driver
autotest: ogr_gmlas.py: make tests independent
autotest: ogr_mitab.py: fix format parametrization of mapinfo_ds fixture
autotest: ogr_gml_read.py: copy xsd files with gml into tmp_path
autotest: ogr_gml_read.py: test that out-of-date gfs files are ignored
Doc: Add NASA logo to sponsorship page
Python bindings: Allow ExecuteSQL context manager to work on empty datasources
autotest: ogr_sqlite.py: make tests independent, use tmp_ fixtures
autotest ogr_mysql.py: make tests independent
autotest: ogr_pg.py: make tests independent
SWIG: Expose GDALClose
Python bindings: Avoid crash when using dataset after Destroy or Release
autotest: add gdaltest.reopen helper
autotest: ogr_gpkg.py: make tests independent, parallel-safe
Python bindings: Avoid crashes when using orphaned Geometry refs
Python bindings: Accept more os.PathLike inputs
autotest: utilities: use tmp_vsimem fixture
autotest: zarr_driver.py: use tmp_vsimem in test_zarr_create_array_bad_compressor
vsimem: implicitly create parent dirs when creating file
MIGRATION_GUIDE: Document behavior changes in /vsimem
ElasticSearch: modify paths to avoid having both /vsimem/foo file and /vsimem/foo/ directory
PLMosaic testing: Use webserver to avoid file and dir with same name in vsimem
autotest: daas.py: use tmp_vsimem fixture
autotest: ogr_oapif.py: avoid modifying FileHandler internal dict
autotest: ogr_elasticsearch.py: use webserver, make tests independent
autotest: ogr_plscenes.py: use webserver, split some tests
autotest: webserver.py: uninstall handler if final_check throws
autotest: ogr_elasticsearch.py: skip test failing on Windows
autotest: ogr_refcount.py: make tests independent
autotest: ogr_ogdi.py: make tests independent
autotest: ecrgtoc.py: make tests independent
autotest: kea.py: use setup fixture, tmp_path, parametrize
autotest: ecw.py: combine interdependent tests, use tmp_path
autotest: ogr_fgdb.py: make tests independent, use tmp_path
autotest: vsioss.py: make tests independent
autotest: vsiswift.py: make tests independent
autotest: vsiwebhdfs.py: make tests independent
autotest: vsicurl.py: use webserver fixture, skip tests of unavailable urls
autotest: ogr_mongodbv3.py: mark with require_driver
autotest: ogr_mvt.py: use webserver fixture
autotest: ogr_carto.py: move cleanup into fixture
autotest: ogr_mssqlspatial.py: make tests independent
autotest: arg.py: mark as order-dependent
autotest: overviewds.py: make tests independent, use tmp_path
autotest: idrisi.py: avoid creating new files in data dir
autotest: eedai.py: disable random order
autotest: ecw.py: avoid writing file to data dir
Erica Fischer (1):
GeoJSONand GeoJSONSeq: add AUTODETECT_JSON_STRINGS layer creation option (#8391)
Even Rouault (893):
Tag master as 3.8.0dev
mkgdaldist.sh: update to take account python bindings generated files are no longer included [ci skip]
HOWTO-RELEASE: update [ci skip]
Fix building without fuzzers/ and perftests/ directory
gdal.cmake: install data/gfs.xsd and data/gml_registry.xsd
Doc / NEWS: s/GetCompressedData/ReadCompressedData/
pytest.ini.in: fix issue when running autotest from git repo when GDAL_RELEASE_NAME in gdal_version.h.in has no dev suffix
C/C++ multidimensional API: add Rename() methods
SWIG: map multidimensional API Rename() methods
MEM multidim: implement Rename() for group, dimensions, arrays and attributes
netCDF multidim: implement Rename() for group, dimensions, arrays and attributes
ZARR: fix full name of attributes
ZARR multidim: implement Rename() for group
Zarr: avoid GetMDArrayNames() to report several times the same name in some circumstances
ZARR multidim: implement Rename() for array
ZARR multidim: implement Rename() for attribute
ZARR multiidm: implement Rename() for dimension
cppcheck-related tunings
Lint: remove useless variable
geopackage_aspatial.rst: mention it is superseded
Add third_party/fast_float header library for fast string->double conversion
scripts/clang-format.sh: exclude third_party/
CPLStrtodDelim(): use fast_float
bench_ogr_batch: add a --stream-opt switch
Parquet: add a COORDINATE_PRECISION layer creation option
Add a OGRWKTToWKBTranslator utility class
Parquet: add fast implementation of Arrow Array interface when requesting WKT as WKB
SDTS: fix reading of polygon geometries (fixes #7680)
WMTS: workaround buggy TileMatrix.TopLeftCorner of one server (fixes #5729)
Arrow/Parquet: emit ARROW:extension:name=ogc.wkb in Feature field metadata, and return it also through GetArrowStream() for Parquet; make Parquet driver recognize a geometry column if it has ARROW:extension:name=ogc.wkb/ogc.wkt field metadata
Add perftests/bench_ogr_to_geopandas.py (from RFC 90 benchmark programs)
Parquet: add a GEOM_POSSIBLE_NAMES and CRS open options for wider compatibility with datasets not following GeoParquet dataset-level metadata
Zarr: move code belonging to classes to dedicated files; rename classes ZarrGroupVx to ZarrVXGroup (no functional change)
NEWS.md: fix link formatting [ci skip]
Avoids warning with gcc 13.1 of mingw64
Fix issues on big endian hosts: recode from UTF-16, and broken tests
OGRSpatialReference::importFromPCI(): lint / avoid Coverity scan warning about leaked memory (CID 1509441)
Zarr: split ZarrArray in 2 classes (no functional change)
Zarr V3: update to current specification (breaks backward compatibility), but without codec/compression support for now
Zarr: robustify detection of existing array/group on creation
cpl_compressor.h: add missing inclusion guard
Zarr: fix SRS DataAxisToSRSAxisMapping
GDALDatasetFromArray: fix write/update support
Zarr: allow update support in classic mode
CPLBloscDecompressor(): fix logic error when providing an output buffer larger than needed
Add CPLJSONObject::Clone()
Python bindings: take into account Int64/UInt64 in gdal.Array.Write(array(...))
Zarr V3: add support for endian, transpose, gzip and blosc codecs
Zarr V3: implement group, array, attribute and dimension renaming
Multidim rename: factor code related to changing m_osFullName and add a NotifyChildrenOfRenaming() virtual method
Zarr rename: detect conflict between group and array within the same group
docker/ubuntu-full/bh-gdal.sh: fix Docker buildx arm64 build [ci skip]
docker/util.sh: optimize docker buildx --release --push mode [ci skip]
JP2OpenJPEG: avoid hard crash in multi-threaded writing mode when opj_write_tile() fails (fixes #7713)
MITAB: fix reading CRS with LCC_2SP and non-metre unit (fixes #7715)
[Refactor] GTiff: split geotiff.cpp content per class ...
Internal libjson: use locale insensitive CPLStrtod() to parse floating point numbers (fixes qgis/qgis#52731)
Add testing that internal libjson is using locale insensitive functions
Multidim GDALGroup::CopyFrom(): modify logic to copy first indexing variables. Fixes issue with gdalmdimtranslate from Zarr to netCDF
netCDF multidim: fix issues with data vs definition mode
Doc: option GDAL_USE_QB3 is actually called GDAL_USE_LIBQB3 (fixes #7727)
ISG: relax tolerence check to accept GEOIDEAR16_20160419.isg
Python bindings: make gdal.Group.GetGroupNames() and GetMDArrayNames() return an empty list instead of None
Multidim API: add DeleteGroup(), DeleteMDArray(), DeleteAttribute() methods in C/C++/SWIG API
MEM multidim: implement DeleteGroup(), DeleteMDArray(), DeleteAttribute()
netCDF multidim: implement DeleteAttribute()
Zarr multidim: implement DeleteGroup(), DeleteMDArray(), DeleteAttribute()
COG: only update mode if the IGNORE_COG_LAYOUT_BREAK=YES open option is specified (fixes #7735)
netcdf.py: add missing require_driver('CSV')
Doc: advertize 3.7.0
docker/README.md: advertize 3.7.0 [ci skip]
netCDF: add new member variables to make it easier for follow-up change (no functional change)
netCDF: on reading, set NETCDF_DIM_xxx band metadata items in on-demand way (helps with network accesses)
netCDF: do not set NETCDF_DIM_xxxx_VALUES dataset metadata items for variables of unlimited dimensions on network access for performance reasons
Update doc/source/drivers/raster/cog.rst
isg.rst: fix broken link (fixes #7738)
Warp (average,mode,min,max,med,Q1,Q3): fix issue on edge of target valid area when oversampling (fixes #7733)
minified_zutil.h: fix build error on Gentoo and derivatives (fixes #7739)
Warp average resampling: using Weighted incremental algorithm mean for numeric stability
CI: linux_build.yml: set travis_branch: ubuntu_1804
Zarr V3: add support for fill_value for complex data types
Zarr: implement GDALDriver::Rename(), Delete() and CopyFiles()
/vsimem/: make it safe to use in multi-threaded scenarios
Add perftests/gtiff_multi_ds_parallel_write.py to measure TIFF write performance
[Perf improvement] GTiff: avoid using block cache when writing whole blocks
Doc: avoid incorrect display of "--" as "—" (fixes #7752)
Revert "doc_build.yml: pin sphinx==6.0.1 to workround https://github.com/sphinx-doc/sphinx/issues/11163 (fixes #7251)"
gdallocationinfo: set exit code to 1 as soon as one coordinate is off the file (fixes #7759)
gdal_rasterize: issue explicit error message when specifying invalid layer name, and catch absence of bounds (refs #7763)
OGRProjCT::Transform(): do not emit generic error message if a specific one has already been emitted
Add OGRCoordinateTransformationOptions::SetOnlyBest() / OCTCoordinateTransformationOptionsSetOnlyBest()
autotest/osr: make it work with latest PROJ master
Add OGR_G_IsClockwise() and map it to SWIG
Parquet: add a way to write custom geoparquet metadata for unit testing purposes
CSV: do not allow CREATE_CSVT=YES with /vsistdout/ (fixes #7699), and make it possible to use GEOMETRY_NAME with /vsistdout/ (fixes #7700)
Revert "Conda: add libarchive.patch"
Addresses various Coverity Scan warnings about performance & bad use of iterators
CI: install jsonschema for build-windows-conda
PDFDataset::ExploreTree(): fix detection of loops
WCS: replace most uses of CPLString with std::string; addresses various Coverity Scan warnings about performance & bad use of iterators
scripts/clang-format.sh: skip pcidsk/sdk and third_party
Addresses various Coverity Scan warnings about performance & bad use of iterators
Add validate_geoparquet.py sample Python script
Warper: auto-enable OPTIMIZE_SIZE warping option when reasonable (fixes #7761)
SHPRestoreSHX(): add extra sanity checks
CPLParseXML(): do not call CPLErrorReset()
GDALIdentifyDriverEx() and GDALDriver::QuietDelete(): do not call CPLErrorReset()
Geoloc transformer: warn if the input dataset is larger in width or height compared to the geoloc array (fixes #7751)
Python bindings: detect invalid use of ReleaseResultSet() (fixes #7782)
Doc: fix case of SQLite3_INCLUDE_DIR/SQLite3_LIBRARY in example (fixes #7785)
netCDF multidim: deal with _FillValue attribute of different type as variable (e.g. for NASA GEDI L2B products)
HDF5 multidim: deal with _FillValue attribute of different type as variable (e.g. for NASA GEDI L2B products)
OpenFileGDB: allows to modify a record with a GlobalID field without regenerating it
OpenFileGDB: reduce debug traces on raster datasets
OpenFileGDB raster: do not emit error if SRS is unknown (refs #7794)
OpenFileGDB raster: correctly deal with rasters whose block_origin_x/y != (eminx, emaxy) (fixes #7794)
RequiresUnixPathSeparator(): takes into account /vsihdfs/ and /vsiwebhdfs/ (refs #7801)
Shape driver: in creation, uses w+b file opening mode instead of wb followed by r+b, to support network file systems having sequential write only and when using CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES (fixes #7801)
Fix building with -DDEBUG_CONFIG_OPTIONS (fixes #7789)
Python bindings: do not pass deprecated -py3 SWIG switch for SWIG >= 4.1
CMake: emit better error message when a raster driver cannot be enabled because of OGR_BUILD_OPTIONAL_DRIVERS=OFF
Doc: CSharp: add a paragraph about issues related to signing of build artifacts (fixes #7804)
CI: windows_build.yml: disable downloading of libdeflate.zip which now fails quasi systematically for unknown reason
Internal libtiff: resync with upstream
MySQL: add support for inserting new SRS in MySQL >= 8 INFORMATION_SCHEMA.ST_SPATIAL_REFERENCE_SYSTEMS table (refs #7781)
MySQL: fix compliance issues with test_ogrsf
netCDF: parse coordinates attribute from Sentinel-3 Synergy product
Move OGRWKBGetBoundingBox() to ogr_wkb.h/cpp so it can be reused by other drivers
OGR TileDB: fully apply spatial filter when using ArrowArray interface
gdalmdiminfo: output details of indexing variables that can be accessed only from the array (typically for TileDB dimension labels)
.github/workflows/ubuntu_20.04/Dockerfile.ci: update to TileDB 2.15.3
docker/ubuntu-full/Dockerfile: update to TileDB 2.15.3
TileDB: add read/write multidimensional support
GRIB: allow opening files with invalid Earth of shape (fixes #7811)
Avoid hanging on test_wms_19
GDALDataset::Open(): take into account open options for OF_SHARED mode (fixes #7824)
GeoTIFF SRS writer: make sure to write EPSG codes only in VerticalDatumGeoKey and VerticalUnitsGeoKey, and write VerticalCSTypeGeoKey = KvUserDefined, VerticalDatumGeoKey = KvUserDefined and VerticalCitationGeoKey=CRS name when the vertical CRS code is unknown (refs #7833)
GeoTIFF SRS reader: try to recover the vertical datum from the name of the vertical CRS if we only know it (refs #7833)
gdalsrsinfo: when -e flag is passed, report properly non EPSG authorities (refs #7833)
Fix gcc 13.1.1 -Werror=array-bounds false positive
Python bindings: GetArrowStreamAsNumPy(): fix wrong retrieval of content when ArrowArray::offset != 0 on a field of primitive type (integer, float)
OGRArrowLayer::CreateWKTArrayFromWKBArray(): fix when ArrowArray::offset != 0 (master only)
GPKG: sanitize type use of GPkgFieldToOGR()
GPKG: ArrowArray interface: make it tolerant to Spatialite geometries
GPKG: fix handling of geometry column from a view, ...
/vsiadls/: make it take into account Azure SAS token
Move parts of OGRLayer dealing with Arrow C interface in a dedicated ogrlayerarrow.cpp file
Parquet/Arrow: implement faster spatial filtering with ArrowArray interface
GetArrowStreamAsNumPy(): add a few sanity checks
GetArrowStreamAsNumPy(): remove likely erroneous offseting of data buffer for array of type 'list of strings'
GPKG: define a 'gdal_spatialite_computed_geom_column' extension when a view has its geometry column being the result of Spatialite SQL function
CMake: only try to detect HDF5 CXX component if KEA is detected
TileDB: FillPrimitiveListArray(): fix data corruption with spatial filter (Coverity CID 1512587, 1512576)
gdal_rasterize: fix logic error of fix of #7663 (CID 1512568)
CMake: capture dependency of MBTiles driver to MVT
GDALMDArrayMask: take into account attributes at construction time
GDALMDArray::GetMask(): add a UNMASK_FLAGS option
DGN: CreateFeature(): fix crash on empty geometries (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=56771)
RawRasterBand::IRasterIO(): avoid harmless unsigned integer overflow (https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57371)
ERS: avoid integer overflow (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57472)
DIPEx: fix memleak in error code path (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57478)
netCDF vector: fix crash on invalid layer (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=58469)
PAM .aux.xml: read geotransform from Esri GeodataXform.CoeffX/CoeffY elements (fixes qgis/qgis#53125)
GPKG: implement GeoPackage 1.4 RTree triggers, and add a VERSION=1.4 dsco (fixes #7823)
Doc: gpkg.rst: mention VERSION=1.4 (refs #7823)
gdal_translate: emit warning when -a_scale/-a_offset + -unscale is specified (fixes #7863)
Update rfc4_geolocate.rst: update status
GDALWarpResolveWorkingDataType(): take into account both source and target dataset
/vsiaz/: fix truncated Authorization header with very large SAS tokens
CI: try to avoid libjpeg-turbo 2.0 from non-conda-forge channel to be installed
[Lint] nearblack: isolate current algorithm in a dedicated function
[Lint] nearblack: better C++ification
nearblack: add "-alg floodfill" to select a flood fill algorithm
nearblack: make floodfill call twopasses when number of non-black pixels > 0
Parquet: fix crash when calling SetIgnoredFields() after SetAttributeFilter() (fixes qgis/QGIS#53301)
TileDB multidim: create TileDB attribute with the GDAL MDArray name (instead of 'TDB_VALUES')
OpenFileGDB: add support for FileGDB v9 raster
CI: fedora:rawhide: workarounds bug in Docker image
OGRArrowLayer::CreateWKTArrayFromWKBArray(): avoid minimal 2 GB allocation... (master only fix)
Doc: multidimensional_api_tut.rst: improve links
MVT generation: improve dealing with polygon inner rings (fixes #7890)
FlatGeobuf: fix build error when new Abseil headers are present (fixes #7894)
ZarrV3: fix crash on invalid shape/chunk_shape values (master only) (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=59607)
gdal_fuzzer: avoid false positive warning about div by zero (CID 1512809)
CPLStrtod(): avoid false positive warning about untrusted loop bound (CID 1512808)
MySQL: fix unused value warning (CID 1512807)
MAP: fix memleak in error code path (CID 1512806)
GDALDriver::DefaultRename(): avoid warning about unused return value (CID 1512805)
VSIMemHandle::Write(): disable coverity false positive (CID 1512585)
feat2kml(): use std::move() (CID 1509476)
Zarr: fix warning about useless copy (CID 1509443)
CI: fix build-mac
Fix typos
Fix build error with MSYS64 UCRT64 with gcc 13 (fixes #7909)
gdalwarp: in -tr mode (without -ts / -te), detect blank edge lines/columns before warping and remove them (fixes #7905)
tiff_ovr.py: add expected checksum for GISInternals build
/vsiaz/: add options to pass object_id/client_id/msi_res_id in IMDS authentication requests
Internal libtiff: resync with v4.5.1rc1
gdal_polygonize: add a -overwrite switch (fixes #7913)
ESRIC: directly open a .tpkx file
GTiff multi-threaded reader: catch errors emitted in worker threads and re-emit them in main thread
GTiff multi-threaded reader: avoid error when reading PlanarConfiguration=Separate and ExtraSamples > 0 (fixes #7921)
Internal libtiff: resync with v4.5.1rc3
CSV: change to use separator with the most occurences, (#7831)
TileDB multidim: fix creating an array with a dimension of same name as the array
GML: fix 3.7.0 regression when reading files with several layers, whose features have gml:boundedBy elements and the geometry element name is different between the different layers (fixes #7925)
OpenFileGDB write: fix memory leak in error code path (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=57365)
ossfuzz builds: use internal libpng
SAFE: fix non-contiguous number in subdataset names (fixes #7939)
ARG: mark it deprecated, removal planned for GDAL 3.9 (refs #7920)
MVT generation: fix wrong winding order for polygons after MakeValid()
Selafin: remove problematic trace in DEBUG_VERBOSE mode
Python bindings: works around issue with SWIG 4.1 related to module unloading
GML: add a USE_BBOX=YES open option to allow using gml:boundedBy as feature geometry
gml.rst: document USE_BBOX
VRT: fix crash in DEBUG_VERBOSE mode (fixes #7958)
CI/Docker: build libjxl with -DJPEGXL_ENABLE_JPEGLI_LIBJPEG=OFF to avoid install a libjpeg.so
Multidim API: add option to set actual_range in netCDF and Zarr when computing statistics
gdal_rasterize_lib: fix error message
gdal_rasterize_lib: other error messages fixes
GTiff multi-threaded reader: fix crash/read errors when reading a PlanarConfiguration=Separate file band per band (fixes rasterio/rasterio#2847)
fuzzers/build.sh: add back -lpng since it is needed by freetype
Internal Shapelib: update list of symbols to rename to avoid conflicts with external shapelib (fixes microsoft/vcpkg#32070)
MVT/MBTiles: take into account tileStats metadata item to decide if a field of type 'number' might be Integer or Integer64
CPLJSon: add setters for uint64_t
Add PMTiles read-only vector driver
VSIUnixStdioHandle / VSIWin32Handle: make Close() be callable multiple times to be friendly with VSIVirtualHandleUniquePtr
core and ogr2ogr: add logic so that ogr2ogr can try a driver specific implementation of GDALVectorTranslate()
MVT writer: clamp generated tile x, y coordinates to [0,(1<<z)-1]
PMTiles: add a MBTiles -> PMTiles converter mode through ogr2ogr
PMTiles: implement regular Create() mode, using MBTiles as an intermediate
mvtutils.h: change description of COMPRESS option to use GZip
GPKG: avoid Coverity Scan warning about nullptr deref (CID 1513124)
gdalmultidim: fix Coverity Scan warning about nullptr deref (CID 1513122) (master only)
vsis3.py: fix to make tests pass when port = 8081 (fixes #7952)
CMake: add a EXPAT_USE_STATIC_LIBS hint
CMake: add a CURL_USE_STATIC_LIBS hint
Doc: add hint for OpenSSL static linking (fixes #7955)
GeoTIFF SRS reader: try to retrieve the EPSG code of a CompoundCRS if the one of its horizontal and vertical part is known (fixes #7982)
typo fixes [ci skip]
FileGDB / OGRCreateFromShapeBin(): accept empty polygon (refs #7986)
OGRGeometryFactory::createGeometry(): take into account Z/M flags
OpenFileGDB: correctly read POINT EMPTY (fixes #7986)
Fix build against latest libjxl master
Add gdal_footprint utility
OGCAPI: make sure OGR_ENABLE_DRIVER_GML is set when the driver is built
OGCAPI: reduce the scope of code #ifdef'ed by OGR_ENABLE_DRIVER_GML
OGCAPI: make it work when the media type of links (expected to be application/json) is missing, using Accept content negotiation (fixes #7970)
OGCAPI: vector tiles: avoid potential infinite time to establish layer definition
PAM rasterband: on .aux.xml reading do not systematically set (offset,scale) to (0,1)
JPEG: allow QUALITY down to 1 (refs #7996) and redirect JPEG 'output message' to GDAL debug messages
gdal_calc.py: make --hideNoData imply --NoDataValue=none as documented (fixes #8009)
Internal libtiff: resync with upstream
ogrinfo: speed-up string concatenation
GPKG: speed-up HasMetadataTables()() on dataset with many layers
GPKG: speed-up unique constraint discovery on dataset with many layers
Changes to make ogr_p.h includable with CPL_SUPRESS_CPLUSPLUS (from SWIG bindings)
test_ogrinfo_lib.py: test reading field domains as JSON output
SWIG bindings: add ogr.CreateRangeFieldDomainDateTime() and Domain.GetMinAsString()/GetMaxAsString()
ogrinfo: add support for DateTime field domains
OpenFileGDB: add support for DateTime field domains
SQL SQLite: fix error on field of type String with GetWidth() > 0 and a field domain
Remove unused cmake/template/Makefile.PL.in
gdal.pc generation: use CMAKE_INSTALL_INCLUDEDIR/CMAKE_INSTALL_LIBDIR for includedir/libdir (refs #8012)
gdal-config: add a --plugindir switch (fixes #8012)
gdalwarp: make -cutline to work again with PostGIS datasources (fixes 3.7.0 regression, fixes #8023)
GDALBufferHasOnlyNoData() / GTiff SPARSE_OK=YES: recognize negative floating point 0 as 0 (fixes #8025)
Internal libtiff: WebP decoder: fix error when reading a 3-band blob in a RGBA image
XLSX: add FIELD_TYPES and HEADERS open options (fixes #8028)
ODS: add FIELD_TYPES and HEADERS open options (fixes #8028)
Python bindings: accept numpy.int64/float64 arguments for xoff, yoff, win_xsize, win_ysize, buf_xsize, buf_ysize arguments of ReadAsArray() (fixes #8026)
GTiff creation: do not emit 'WEBP_LEVEL is specified, but WEBP_LOSSLESS=YES' warning when only WEBP_LOSSLESS=YES is specified
Arrow/Parquet: do not hang on empty RecordBatch (fixes #8042)
GTiff: fix ff769d91f0d906b7ff0ae9cf6c77d604a7e5feda
/vsicurl/: avoid the same region to be downloaded at the same time from concurrent threads (fixes #8041)
fuzzers/build.sh: try to fix build [ci skip]
test_osr_set_proj_search_paths.cpp: override PROJ_DATA as well
CI: macos_build: update dependencies
CI: build-windows-msys2-mingw: workaround jsonschema 4.18 installation issue
CI: macos: install lxml and jsonschema from conda
SWIG: add gdal.SuggestedWarpOutput()
CI fedora: workaround jsonschema 4.18 installation issue
CI: add a coverage job
Use coverallsapp/github-action@v2
CI fedora: workaround jsonschema 4.18 installation issue (again)
netcdf_cf.py: remove use of 'imp' module removed in python 3.12
README.md: add coveralls badge [ci skip]
typo fixes [ci skip]
OpenFileGDB: make Open() to fail if requested to open in update mode and that files are in read-only (fixes qgis/QGIS#53715)
gdaladdo: add options to partially refresh existing overviews
CI: linux_build.yml: build and run tests as non-root
Doc: testing.rst: add a paragraph about test coverage
validate_geoparquet.py: make it use non-deprecated remote resource resolution with jsonschema >= 4.18
autotest: avoid deprecation warning with jsonschema 4.18
PMTiles: fix 2 Coverity scan warnings
Python samples: remove gdal_zip.py now we have the sozip utility (fixes #8062)
GeoPackage: deal with DateTime fields without milliseconds or seconds, as allowed by GeoPackage 1.4 (fixes #8037)
GeoPackage: add a DATETIME_PRECISION layer creation option (fixes #8037)
Add testing of OGRParseDateTimeYYYYMMDDTHHMMZ()
Internal libtiff: WebP codec: turn exact mode when creating lossless files to avoid altering R,G,B values in areas where alpha=0 (fixes #8038)
WFS: do not surround selected fields names in PROPERTYNAME with open and close parenthesis (fixes #8089)
autotest: ogr_wfs.py: make it work with pytest --random-order
Doc: advertize 3.7.1 release
docker/README.md: advertize v3.7.1 [ci skip]
Make gdal-config executable in build directory (refs #8029)
Python bindings: make setup.py check that libgdal version >= python bindings version (fixes #8029)
Add COPY_SRC_MDD=AUTO/YES/NO and SRC_MDD=domain_name creation options
vsioss.py: fix letter inversion in test names
autotest: skip tests that randomly fail on mac
gdal2tiles: fix exception with dataset in EPSG:4326 with longitudes > 180 in WebMercator profile (fixes #8100)
Ccache.cmake: fix warning
ogr2ogr: support -nlt GEOMETRY -nlt CONVERT_TO_LINEAR (fixes #8101)
vsis3.py: skip another randomly failing test on mac
Java bindings: implement ogr.CreateCodedFieldDomain() and FieldDomain.GetEnumeration() (fixes #8085)
WEBP: fix build against libwebp < 0.4.0 (fixes #8111)
autotest: replace deprecated numpy.alltrue() (will be removed in numpy 2.0) with numpy.all()
GTiff: fix reading .tif + .tif.aux.xml file with xml:ESRI SourceGCPs without TIFFTAG_RESOLUTIONUNIT (fixes #8083)
GTiff: fix reading .tif + .tif.aux.xml file with xml:ESRI SourceGCPs without TIFFTAG_RESOLUTIONUNIT - remove useless code (fixes #8083)
SWIG bindings: fix warning related to redefinition of CPLErr
Python bindings: fix CoordinateTransform.TransformPoint(sequence of 3 or 4 values)
Add driver for JSONFG (OGC Features and Geometries JSON)
tiledbsparse.cpp: fix compiler warning with MSVC
CI: avoid hang on Windows Conda testing related to https://github.com/conda-forge/libnetcdf-feedstock/issues/182
Java bindings: change default installation directory of JNI shared library and control it with GDAL_JAVA_JNI_INSTALL_DIR
TileMatrixSet::parse(): add support for OGC 2D Tile Matrix Set v2 (fixes #6882)
OGCAPI: do not try to use the 'uri' member of a tilematrixset definition document, as it might not be resolvable, and doesn't bring much
OGC API: resolve different issues with the raster tile API
STACIT: correctly process asset 'href' starting with 'file://' (fixes #8135)
GRIB: implement CCSDS Adaptive Entropy Coding decompression (template 5.42).
Add a DCAP_FLUSHCACHE_CONSISTENT_STATE capability
SWIG bindings: fix warning related to redefinition of CPLErr
OGRGeoJSONReader::FirstPassReadLayer(): fix potential memleak (but likely impossible to occur in practice) (CID 1518087)
JSONFG: fix wrong variable name in nullptr check (CID 1518086)
GDALDriver::DefaultCopyMetadata(): fix memleak (master only) (CID 1518085)
GTiff: fix memleak (master only) (CID 1518084)
JSONFG: silence Coverity warning (CID 1518083)
VRT: fix memleak (master only) (CID 1518082)
GDALFootprintOptionsNew(): remove dead code (CID 1518081)
Silence Coverity false positive (CID 1518080)
JSONFG: hopefully avoid Coverity false positive about memleak (CID 1518079)
Silence Coverity false positive (CID 1518078)
gdalinfo -json output: emit a stac['proj:epsg'] = null object when emitting proj:wkt2 or proj:projjson (fixes #8137)
STACIT: make it tolerant to missing proj:epsg if proj:wkt2 or proj:projjson are provided (fixes #8137)
COG: fix memleak (master only) (CID 1518077)
Silence Coverity false positive (CID 1518076)
PMTiles: fix Coverity scan warning (CID 1517749)
GeoJSON: modify to implement DCAP_FLUSHCACHE_CONSISTENT_STATE
ogr2ogr: calls FlushCache() (fixes #8033)
JSONFG: implement DCAP_FLUSHCACHE_CONSISTENT_STATE
GPKG: implement DCAP_FLUSHCACHE_CONSISTENT_STATE
Doc: sponsors.rst: remove sponsors which haven't renewed
XLS: avoid compiler warning when building against freexl 2.0
NITF: fix MIN/MAX_LONG/LAT when reading RPC00B
CI Alpine: fix pip install
CI Fedora: remove mongo-cxx-driver-devel. It has been removed per https://src.fedoraproject.org/rpms/mongo-cxx-driver/c/3cfdc2f6bee3fda0808a5eb91b76640074de176b?branch=rawhide
Revert "CI: fedora:rawhide: workarounds bug in Docker image"
CI: fedora:rawhide: use dnf5
/vsicurl/ / /vsicurl_streaming/: recognize IGNORE_FILENAME_RESTRICTIONS=YES open option to skip any extension based filtering (refs #8162)
Zarr: ignore filename restrictions when reading tile data files (fixes #8162)
/vsiaz/: fix cached URL names when listing /vsiaz/
Python bindings: handle 'tsn:' (DateTime in nanoseconds) data type for GetArrowStreamAsNumPy()
Arrow/Parquet: avoid data corruption through GetNextArrowArray() API when using spatial filters (master only)
ArrowArray interface: make PostFilterArrowArray() deal with attribute filter, and enable that in Parquet&Arrow drivers
Parquet: fix ExecuteSQL() MIN/MAX optimization on a UINT32 field on a Parquet 2 file
FindSPATIALITE.cmake: quote variable
CMakeLists.txt: update maximum range to 3.27
Add delete_untagged_containers.yml [ci skip]
CI: linux_build.yml: sanitize CACHE_CONTAINER_TAG in the same way as CONTAINER_TAG, to be able to actually use an up-to-date image
gdalwarp_lib.cpp: C++ify CropToCutline() (no functional change)
gdalwarp_lib.cpp: C++ify TransformCutlineToSource() (no functional change)
gdalwarp: fix cutline processing when warping with a cutline geometry in UTM 1/60 crossing the antimeridian, on a raster in long/lat SRS (fixes #8163)
gdalinfo.py: use math.isnan() (fixes #8196)
Doc: ogrinfo.rst: fix options fields and geom (fixes #8195)
gdalinfo.py: fix wrong order of long,lat corner coordinates (fixes #8199)
test_cpl.cpp: robustify test_cpl.CPLCopyTree
Add VSIGetCanonicalFilename()
Shapefile driver: use VSIGetCanonicalFilename() in GetFileList() (fixes #8164)
linux_build.yml: try to fix issue with Fedora CI
[Lint] CSV: replace GDAL_OVERRIDE with plain override
[Lint] CSV: have OGRCSVLayer and OGRCSVEditableLayer inherit from a same base class
CSV: implement GetFileList() and return .csvt if used (fixes #8165)
Revert "linux_build.yml: try to fix issue with Fedora CI"
Shapefile driver: make CreateLayer() + GetFileList() list the .prj file when it exists (fixes #8167)
VRT: ignore <OverviewList> when external .vrt.ovr is present, as documented and intended
CI: workaround weird caching issue with fedora_rawhide builds
/vsis3_streaming/ and the like: implement ReadDir() by forwarding to non-streaming filesystem (fixes #8191)
ZarrV2: fix duplicate array listing when both a 'foo' file and 'foo/' directory exist on the object storage (fixes #8192)
More accurate / fixes for M_PI/180 or 180/M_PI constants (fixes #8178)
CITATION.cff: update with contributors (#8201)
openjpeg: fix build with MSVC2017 (master only)
Python bindings: fix typo on slopeFormat parameter (fixes #8154)
OGRParseDate(): restrict valid times to HH:MM:SS(.sss) with at least 2 figures (fixes #8150)
Warper: do not modify bounds when doing geographic->geographic on a dataset with world extent but not in [-180,180] (fixes #8194)
ogrinfo: return non-zero ret code if -sql failed (fixes #8215)
Doc: removing Twitter, or X or whatever it will be named when I'll have commit this
Adjust test for libjson-c 0.17
PDF: update to support (and require) PDFium/5952
Doc: gdalwarp: clarify that -t_srs might still be needed when using -ct (fixes #8242)
Overview/RasterIO resampling: use Float64 as working data type for Float64 input data (fixes #8187)
RMS resampling: avoid potential integer overflow with UInt16 values
RasterIOResampled(): make mingw gcc happy
GDALResampleChunk_Mode(): use template for the different working data types
Fix ExecuteSQL(dialect='SQLITE', spatialFilter=...) on a Memory datasource
MEM: allocate a single buffer for band-interleaved data
HDF5: optimize code-paths for RasterIO() without resampling (depends on commit f9cc44abef)
/vsiaz/: implement Azure Active Directoy Workload Identity authentication, typically for Azure Kubernetes
NITF: add support for CSCSDB (Common Sensor Covariance Support Data) DES from GLAS/GFM SDEs
nitf_spec.xml: corrections to CSEXRB TRE
GDALChecksumImage(): fix 3.6.0 regression regarding integer overflow on images with more than 2 billion pixels (fixes #8254)
Doc: import https://github.com/OSGeo/libgeotiff / geotiff/html/proj_list under gdal.org/ (fixes #8221)
HDF5: more efficient metadata collection (no fonctional change)
HDF5: deal with int64/uint64 attributes
HDF5: remove trailing space in multi-valued metadata items, remove dataset name prefix in band level metadata, and address Planet's datacube band-specific metadata
Parquet: add OGRParquetLayer::GetMinMaxForField() and use in ExecuteSQL() (no fonctional change)
Parquet: optimize SELECT MIN(FID), MAX(FID)
Parquet: restrict FID column detection to Int32/Int64 data types
Arrow/Parquet: optimize attribute filter on FID column
OGRArrowLayer::GetNextArrowArray(): make sure to invalidate potentially modified batches at end of iteration
Parquet: use statistics to skip row groups that don't match attribute filter (fixes #8225)
Regenerate autotest/ogr/data/arrow/test.feather
cpl_json: add CPLJSONArray::AddNull() and CPLJSONObject constructor with primitive types
Arrow/Parquet: support/reading nested list/map datatypes as JSON (fixes #8227)
autotest: make tmp_vsimem fixture create the directory (similarly to pytest's tmp_path)
Arrow/Parquet: make TestCapability(OLCFastGetExtent) call FastGetExtent
Parquet: use statistics of bbox.minx/miny/max/maxy fields (as found in Ouverture Maps datasets) to implement fast GetExtent()
OGRArrowLayer::GetNextRawFeature(): save some use of shared_ptr refcount
Doc: Remove AWS from GDAL Sponsorship program (#8238)
OGRArrowLayer::GetNextRawFeature(): make it use bbox.minx/miny/max/maxy for faster spatial filter testing
Parquet: use statistics of bbox.minx/miny/max/maxy fields to skip row groups for faster spatial filter testing
[code cleanup] ogrlayerarrow.cpp: use helper functions
[code cleanup] ogrlayerarrow.cpp: rename pabyNull to pabyValidity
Doc: document performance issues with JPEG2000 through network access (fixes #8275)
PDF: prefix member variables of PDFDataset class with m_, and use inline initialization
PDF: avoid CSA false positive
PDF: use VSIVirtualHandleUniquePtr for m_fp
PDF: use unique_ptr for m_apoOvrDS and m_apoOvrDSBackup
pdf.py: add missing require_driver()
PDF: implement overviews for Poppler backend by adjusting the DPI value (fixes #8233)
PDF: improve performance of overviews
Python bindings: add a outputGeotransform option to gdal.Translate()
PDF: increase threshold to detect tile size and band count (fixes #8236, fixes #8240)
PDF: avoid int overflow on compressed stream whose decompressed size > 2 GB
PDF: limit decompression of compressed streams with Poppler backend
PDF: avoid potential OOM crash in PDFium backend
PDF: catch C++ exceptions from Poppler
PDF: limit memory allocations on huge PDF name objects
CPLvsnprintf(): deal with '%.*f' formatting
Python bindings: make Feature.SetField(field_idx_or_name, binary_values); add feature.SetFieldBinary(field_idx_or_name, binary_values)
autotest: replace use of SetFieldBinaryFromHexString() where possible
gdalmdimtranslate: fix wrong output dimension size when using syntax like '-array name=XXX,view=[::factor_gt_1]'
Zarr: add MULTIBAND=YES/NO, DIM_X and DIM_Y open options (fixes #8237)
Zarr: change MULTIBAND open option to default to YES (fixes #8237)
GDALMDArray::AsClassicDataset(): add a LOAD_EXTRA_DIM_METADATA_DELAY option, and expose it in Zarr driver
Zarr: fix writing partial tiles
ZarrArray::IWrite(): optimize inner loop
Zarr: classic raster API: write multi-band datasets as Zarr 3D arrays (writing them as several 2D arrays as in GDAL 3.7 can be asked with the SINGLE_ARRAY=NO creation option)
OGRCurve::isClockwise(): fix wrong result when lowest rightmost vertex is the first one (fixes #8296)
Test OGRCurve::isClockwise()
ogrinfo_output.schema.json: make schema reflect actual output of ogrinfo (fixes #8300)
autotest: display loaded shared objects (if psutil installed)
CI: skip test_driver_metadata on mingw64 since it now crashes for unknown reason
VRT: add 'min' and 'max' pixel functions (fixes #8292)
Docker: update dependencies, and bump Alpine to 3.18 [ci skip]
Python bindings: throw exceptions (when enabled) on gdal_array.OpenArray()/OpenNumPyArray()/OpenMultiDimensionalNumPyArray()
OGR MEM: add a FID layer creation option to specify the FID column name
SQLite dialect: when the underlying layer has a FID column name, enable the user to use it as an alias of ROWID
GPKG: remove duplicate call to CancelAsyncNextArrowArray() in DeleteFeature()
OGRGeoPackageLayer: prefix all member variables with m_, and use inline initialization
GPKG: implement SetNextByIndex() on table layers (by appending 'OFFSET -1 LIMIT index')
osgeo_utils: allign base.get_extension() and util.DoesDriverHandleExtension() on C++ versions, and add tests (fixes #8277)
TileDB ArrowArray: make sure ResetReading() does not invalidate array
OGRLayer::GetArrowStream(): do not issue ResetReading() at beginning of iteration, but at end instead, so SetNextByIndex() can be honoured
GPKG: make GetArrowStream() honour SetNextByIndex()
Doc: resync OGR_L_SetAttributeFilter() with OGRLayer::SetAttributeFilter() (fixes #8308)
Python: robustify typemaps, use more fragments
OGRGeometry classes: add addGeometry()/addRing()/addCurve() methods accepting a std::unique_ptr
gml2ogrgeometry: use std::unique_ptr
autotest: ogr_gml_geom.py: enhance tests for gml:Surface
gml2ogrgeometry: reject empty <gml:Triangle/>
Simplify GML2FaceExtRing()
Document and make --help and --help-general available in all utilities (fixes #3816)
Address warnings of cppcheck master
ISG: make it able to read headers > 1024 bytes
ISG: take into ISG format 2.0
Internal libtiff: resync
Doc: better document --formats (fixes #8315)
resync_from_upstream.sh: fix shellcheck issue
Build: fix building test_ofgdb_write (fixes #8321)
TileDBRasterBand::IRasterIO(): use correct band indexing
TileDB: read/write Int8, Int64 and UInt64
GPKG: add a CRS_WKT_EXTENSION=YES/NO dataset creation option to force addition of definition_12_063 column
GDALGeoPackageRasterBand::GetMetadata(): fix use after free
GDALGeoPackageRasterBand: fix missing GRID_CELL_ENCODING metadata item when there is other metadata
GPKG: remove .aux.xml file in Delete()
GPKG: add a METADATA_TABLES creation option to control creation of system metadata tables
/vsis3/ and /vsioss/: less error prone management of redirects / region discovery
/vsiaz/: do not append trailing slash for directories deduced from .gdal_marker_for_dir special file
typo fixes [ci skip]
/vsiaz/: implement server-side copy from /vsis3/, /vsigs/, /vsiadls/, /vsicurl/ to /vsiaz
VSISync() network to network: implement SyncStrategy::TIMESTAMP
gdal2tiles: raise explicit exception if bad value for --s_srs (fixes #8331)
frmts/mrf/CMakeLists.txt: ignore HAVE_JPEGTURBO_DUAL_MODE_8_12 if GDAL_USE_JPEG is OFF (tune #8336)
Python bindings: improve performance of CSLFromPySequence() on large sequences (like > 100,000 strings)
GDALGeneralCmdLineProcessor(): improve perf with lots of strings in --optfile
MEM driver: fix memleak in Create(nBands=0) (master only)
VSIFilesystemHandler: remove 'virtual' qualifier from ReadDir(), which now forwards to ReadDirEx(); modify all implementations to implement ReadDirEx()
Add /vsicached? virtual file system, as another way of doing the same as VSICreateCachedFile() / VSI_CACHE=YES
bench_ogr_batch: add a -v flag
Arrow/Parquet: fix crash on ArrowArray interface when there is a fid column
OGR2SQLITE_FeatureFromArgs(): fix memleak (master only)
GPKG SQL layer GetNextArrowArray(): fix wrong array size when spatial filtering occurs
OGRWKBGetBoundingBox(): properly initialize envelope
Add OGRWKBIsWithinPessimistic()
Add OGRLayer::FilterWKBGeometry()
JPEG: only take into account first Exif directory found
GPKG table layer GetNextArrowArray(): implement full spatial filtering (not just bbox intersection)
Arrow/Parquet GetNextArrowArray(): implement full spatial filtering (not just bbox intersection) (fixes #8347)
TileDB: GetNextArrowArray(): implement full spatial filtering (not just bbox intersection)
FlatGeoBuf: GetNextArrowArray(): implement full spatial filtering (not just bbox intersection
OpenFileGDB: fix opening of .gdb directories whose last component starts with 'a' and is 18 character long... (fixes #8357)
Add support for PoDoFo >= 0.10.0 (fixes #8356)
alpine/Dockerfile.ci: add libpodofo-dev (0.10.1 at time of writing)
autotest/gcore/vrt_read.py: fix random failure
AAIGRID output: remove leading space on each line (fixes #8344)
Doc: adjust & resync content of doc/source/api/python_bindings.rst and swig/python/README.rst
gdal_translate -scale: change dstMax value from 255.999 to 255
VRTComplexSource (scaling typically): make sure to take into account constraints from VRTRasterBand data type in RasterIO() (rather than just taking into account output buffer data type)
MITAB: avoid memory leak in edge case (fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=61127)
COG: add a STATISTICS=AUTO/YES/NO creation option and forward gdal_translate -stats to it (fixes #8169)
Doc: advertize 3.7.2
docker/README.md: update with 3.7.2 [ci skip]
vrt_read.py: fix breakage due to another concurrent change, and turn on exceptions
PDF: unique_ptr'ification of GDALPDFArray implementations
PDF: CreateCopy(): add #ifdef magick to produce a PDF to reproduce #8372
PDF: pdfreadvectors.cpp: improve debugging
PDF: pdfreadvectors.cpp: (minimal) take into account BMC operator to correctly handle BMC/EMC pairs w.r.t BDC/EMC ones (fixes #8372)
PDF read vector unstructured: take into account OCMD constructs as found in recent USGS GeoPDFs (fixes #8372)
PDF: ignore non-relevant StructTreeRoot in most recent USGS GeoPDFs (fixes #8372)
Internal giflib: fix memleak on animations (fixes #8380)
Internal giflib: fix memleak on truncate files (fixes #8383)
scripts/clang-format.sh: skip giflib
gdal_translate -ovr: properly rescale RPC (fixes #8386)
GTiff and COG: add a MAX_Z_ERROR_OVERVIEW creation option to separately control the error threshold of overviews w.r.t the one of the full resolution image
.github/workflows/fedora_rawhide/Dockerfile.ci: revert to dnf
Add support for TileDB 2.17
OSR SetFromUserInput(): add support for urn:ogc:def:coordinateMetadata (if PROJ >= 9.4)
OSR SetFromUserInput(): recognize 'EPSG:XXXX@YYYY' (PROJ >= 9.4)
Add OSRHasPointMotionOperation() (PROJ >= 9.4)
OGR_CT: handle point motion operations (PROJ >= 9.4)
gdalwarp: tune usage to allow both -s_coord_epoch and -t_coord_epoch
ogr2ogr: tune usage to allow both -s_coord_epoch and -t_coord_epoch
GDALCreateGenImgProjTransformer2(): correctly deal with same CRS but SRC_COORDINATE_EPOCH != DST_COORDINATE_EPOCH
gdaltransform: add -s_coord_epoch and -t_coord_epoch
OGR_CT: display epochs in debug messages
netCDF: better doc and error message when reading from /vsi is not possible (fixes #8398, #8378)
GDALCreateGenImgProjTransformer2(): deal with discontinuity of GCPs around antimeridian (fixes #8371)
docker/alpine-normal/Dockerfile: fix build of freexl 2.0 on arm64 architecture [ci skip]
docker/ubuntu-full/Dockerfile: workaround issue with GDAL not finding Arrow 13.0 on arm64 [ci skip]
netCDF: renormalize CRS and geotransform to metric, for EUMETSAT OSI SAF products with a proj4_string, units=km and geospatial_bounds_crs
CSV: avoid extra comma at end of header line with GEOMETRY=AS_XYZ and a single attribute (fixes #8410)
SQLite: ignore 'SRID=' layer creation option (fixes https://github.com/qgis/QGIS/issues/54560)
test_driver_metadata.py: skip on build-windows-conda due to crashes
/vsi network file systems: make Open() method common for all FS (no functional change)
/vsi network file system: support r+ access under CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE=YES by copying initial content of remove file locally
Parquet: emit GeoParquet 1.0.0 version number now it has been released
OGRGF_DetectArc(): harden tolerance when detecting consecutive arcs to avoid incorrect arc center computation (refs #8332)
ogr2ogr: better deal when reprojecting curve geometries to a non-curve geometry type (fixes #8332)
netCDF multidim: add write support for Int8 data type (fixes #8421)
GeoJSON writer: when writing with limited coordinate precision, run MakeValid() to avoid creating invalid geometries
GTiff: SRS writer: write Projected 3D built as a pseudo-compound in .aux.xml
netCDF: amend previous commit to make it less specific to EUMETSAT OSI SAF and add a PRESERVE_AXIS_UNIT_IN_CRS=YES/NO open option
netCDF (traditional): add support for EMIT band data ordering and geolocation array (not using glt_x/glt_y)
GDALMDArrayResampled::Create(): fix assertion if providing a input non-empty dimension for the non-X/Y dimensions
netCDF multidimensional: add support for EMIT band data ordering and geolocation array (not using glt_x/glt_y)
netCDF multidimensional: use glt_x/glt_y for EMIT data
gdalmdimtranslate: add support for resample=yes array spec option
netCDFVariable::IReadWrite(): emit error message from NCGetPutVaraFunc()
gdalwarp: fix error when using -ct and -cutline (master only), and actually use the -ct when possible if sourceCRS != targetCRS and targetCRS == cutlineCRS
Add RFC95 text: Use standard C/C++ integer types
RFC95: updates with GDAL 4.0, removing questions
ogrinfo: emit distinct error message if the file doesn't exist or can't be opened (fixes #8432)
OGR SQL: allow MIN() and MAX() on string fields
OpenFileGDB raster: add a RASTER_DATASE…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Solution to issue cannot be found in the documentation.
Issue
The CI in GitHub Actions for the Satpy library have started hanging on Windows. It seems to be related to the specific build of libnetcdf. With:
it works fine. With the same code and same dependencies except for the newer build of libnetcdf:
Pytest finishes running Satpy's tests (successfully) but then never exits. There is no other difference from what I can tell except the libnetcdf build. A fellow contributor @simonrp84 was able to reproduce this on his local Windows machine. His environment is what's providing the output for the below conda commands.
Otherwise, here is a passing Satpy CI job:
https://github.com/pytroll/satpy/actions/runs/5577565489/jobs/10190534313
And a hanging one:
https://github.com/pytroll/satpy/actions/runs/5594450997/jobs/10229326506
Installed packages
Environment info
The text was updated successfully, but these errors were encountered: