Incorrect Internal Mask Overviews when creating COGs

[tbonfort]

Using master, when creating a COG from a TIF with internal mask, the mask overviews are incorrectly filled with 255. Using an external mask as detailed in #754 works correctly

Steps to reproduce:

cat << EOF > test.geojson
{
"type": "FeatureCollection",
"name": "test",
"crs": { "type": "name", "properties": { "name": "urn:ogc:def:crs:OGC:1.3:CRS84" } },
"features": [
{ "type": "Feature", "properties": { }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -117.13404487689661, 47.744359501095211 ], [ -117.13404487689661, 47.743977636542496 ], [ -117.133624825888631, 47.743973941079084 ], [ -117.13404487689661, 47.744359501095211 ] ] ] } }
]
}
EOF
gdal_rasterize -burn 255 -burn 0 -burn 0 -ts 8000 8000 -ot Byte -of GTIFF -co TILED=YES -co COMPRESS=LZW test.geojson test.tif
nearblack -setmask -near 3 --config GDAL_TIFF_INTERNAL_MASK YES test.tif
gdaladdo -r average -minsize 512 -ro test.tif
gdal_translate test.tif cog.tif -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co COPY_SRC_OVERVIEWS=YES -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 --config GDAL_TIFF_OVR_BLOCKSIZE 512 -co JPEG_QUALITY=95 --config  GDAL_TIFF_INTERNAL_MASK YES --config JPEG_QUALITY_OVERVIEW 95

When opening cog.tif in qgis, or when using warp/translate, cog.tif shows up with a transparent "background" when using full resolution, but with a black background when using an overview level

[rouault]

@tbonfort I'm not sure we ever want to support that combination of internal mask + external overviews, becase I'm not sure in which file we should write the overviews of the mask : the .tif (since the mask is there, but as external overview is asked for, that would be odd), or a .msk.ovr one (but that would be odd since there is no .msk one) ?
0e555ef adds warnings and extra checks, and also tries to improve the documentation about the various configurations and their level of support

[rouault]

CC @palmerj who played recently in that area

[tbonfort]

Thanks for the explanation @rouault , OK to close for me