String attributes not supported by netCDF-Fortran

[ethanrd]

Topic for discussion

There is some discussion on the current lack of support for string attributes in a netcdf-fortran issue (#181).

While CF mentions differences in capabilities between netCDF-3 and-4, I don't think CF currently mentions any differences between what CF features different tools support. I think not discuss specific tools is probably appropriate. However, this is a widely used library (wrapper around nc-C) and I wonder if CF should consider/discuss its support for features as new capabilities are added to CF.

[ChrisBarker-NOAA]

Hmm -- and here we are talking about utf-8 encoding as well. What will Fortran users do with that?

I think we should keep major libraries in mind when expanding features.

But we also shouldn't keep stagnant. And this particular change is a netCDF feature -- I'd think it was up to the maintainers of the Fortran library to keep up to date.

[ChrisBarker-NOAA]

Hmm -- and here we are talking about utf-8 encoding as well. What will Fortran users do with that?

I think we should keep major libraries in mind when expanding features.

But we also shouldn't keep stagnant. And this particular change is a netCDF feature -- I'd think it was up to the maintainers of the Fortran library to keep up to date.

[taylor13]

I think if features are relied on by CF that can't be interpreted correctly by FORTRAN, they should be avoided unless they seem exceptionally valuable. In that case, I think we should prominently indicate in the conventions document that FORTRAN cannot interpret them. I think avoiding them might be o.k. for certain attributes, but not for the data and not the most important attributes.

[JonathanGregory]

Dear all

When designing the CF convention, we've often consulted the NUG, which we cite many times as the reference for netCDF. I've always assumed that Unidata's C and Fortran netCDF libraries implemented the functionality described by the NUG. Therefore I am surprised by the discussion in the netcdf-fortran issue. If I understand the issue correctly, the netcdf-fortran interface does not support the string type for attributes at all. This is not mentioned by the NUG, is it?

Since CF 1.8 (February 2020), section 2.2 has explicitly allowed variable-length strings as equivalent to char arrays for both attributes and variables. (We're currently discussing when arrays of strings should be allowed in CF attributes.) Consequently there is probably a lot of data in existence which is CF-compliant but can't be read by netcdf-fortran, including ERA5 data from Copernicus, apparently.

It would be against our backwards-compatibility principle (9 in sect 1.2) to introduce new restrictions or recommendations against use of strings now, despite our being blamed for having made strings acceptable in CF. I think we should add a note (not a warning or an error) in sect 2.2 that string attributes can't be read by the netcdf-fortran interface (currently 4.6.1, May 2023).

Cheers

Jonathan

[Dave-Allured]

Fortran can read and write netcdf string attributes with the use of C helper routines. Fortran has always supported strings in the form of character arrays, ever since character data type was added to fortran. Fortran supports variable length strings and variable length character sets by simple conventions. All this was long before modern fortran, netcdf,. or UTF-8.

The lack of support in the current netcdf-fortran wrapper is a complicating factor which IMO should be ignored for the purpose of developing netcdf conventions. If there is agreement, I suggest this topic be closed.

[JonathanGregory]

I don't think we should change our requirements or recommendations, but I do think it's worthwhile noting in the CF text that the netcdf-fortran interface does not support vlen strings.

[Dave-Allured]

it's worthwhile noting in the CF text that the netcdf-fortran interface does not support vlen strings.

Respectfully disagree. Such notation would perpetuate what I see as an unreasonable anti-string bias. Vlen strings are a fundamental and long standing data type. I already said that there are straightforward methods for dealing with them from fortran. Netcdf string data type was introduced 16 years ago, and netcdf-fortran is severely behind the times in failing to provide direct support. I do not like to see CF cluttered with unwarranted shortcomings of secondary software, such as this.

[JonathanGregory]

@Dave-Allured, I agree that the difficulty is that netcdf-fortran does not fully support netCDF-4. But is netcdf-fortran only "secondary software"? Since it appears at https://docs.unidata.ucar.edu/netcdf-fortran/current, I assume it's part of Unidata's netCDF software suite.

[Dave-Allured]

Absolutely, netcdf-fortran is part of Unidata's netCDF software suite. However Unidata's primary focus has been on the critical C library, for various reasons. Thus I am forced to label netcdf-fortran as "secondary".

[ChrisBarker-NOAA]

Hmm -- I'm postulating that Fortran is far more widely used to write netcdf than read it :-) -- At least in my world, a lot of models are written in Fortran, and thus are writing netcdf output, but even the developers of those models are using other tools (Python, Matlab, ...) to post-process / analyze the results.

In any case, the Fortran folks should be annoyed at / blame Unidata, not CF, if the Fortran lib doesn't fully support netcdf.

[larsbarring]

Before we continue this line of discussion I suggest that it would be useful to get some reaction from the Unidata folks:

The issue @ethanrd refers to was initiated in 2019, and in turn it refers to an issue from 2017. Early comments from back then (5 years ago) indicate that there were some obstacles. But things may have developed since then, and just very recently a potential solution was suggested. I am not a software developer so I cannot judge the potential of the suggestion.

From the perspective of Unidata/netCDF-Fortran does this suggestion solve -- or have the potential to solve --- the problem, at least with respect to single string variables (if not arrays of strings)?

[ChrisBarker-NOAA]

(if not arrays of strings)?

Good point -- I think we are still in process on deciding on the arrays of strings issue -- maybe lack of Fortran support is a good reason to hold off on that.