Error when dimension shares name with variable in netcdf-4 classic file #1772
Comments
|
It looks like this is driven by that bane of our existence: HDF5 dim scales. |
|
Short answer: you probably can't do this using the current HDF5+netcdf-c. |
|
Thanks Dennis. So to clarify: the NC_EHDFERR from nc_close() is expected netcdf behavior (i.e. limitation)? Or unexpected (i.e., bug) but there won't be a fix anytime soon? We'll fix our MATLAB code so we don't crash upon this error, but it would be great to know if this is a netcdf limitation (then we can document the lijmitation to our customers), or a netcdf bug. In our case, the customer has an easy workaround -- don't name the dimension the same name as the previously defined variable. I'm not sure why the were doing that in the first place, unless it has something to do with netcdf coordinate variables. I know a variable should only be given the same name as a dimension if it's to be used as a coordinate variable -- but does that apply in reverse? Meaning are there restrictions (other than the error) on naming a dimension the same name as a predefined variable? And does that have anything to do with coordinate variables? |
|
Technically, it is an error since the spec says nothing AFAIK about this being illegal. As for this question:
|
|
The best way to get a fix would be to submit a C test which demonstrates the problem. There are netCDF-4 limitations with respect to the naming of dimensions and variables. This is due to the lack of support for shared dimensions in HDF5. So creating a netCDF dimension actually creates a HDF5 dataset (i.e. variable) with some special metadata. If you then create a var with the same name, it needs to delete the dataset created for the dim, and create a new one, which is a coordinate variable, representing both the variable and the dimension in one HDF5 dataset. However, apparently the code does not handle the case where you want a var of the same name, but not a coordinate var, when you do things in a certain order. This will be order dependent, so try doing things in a different order and that could well work. (For example, I think if you create the var first, and then a dim with the same name, that will work correctly.) |
|
I stand corrected. Ed's answer is the one you should accept. |
|
Thanks Ed and Dennis. I mimicked what our nccreate function is doing by calling it twice in a row like the customer did -- you'll see it in the repro code. Can you please take a look at the C repro and let me know if this helps explain the problem? Let me know if you'd rather have me copy and paste the C repro directly i the comment. |
|
@edwardhartnett and @DennisHeimbigner - I was working on the crash that Ellen mentioned happened in MATLAB (after we got NC_EHDFERR error). I found this crash got triggered when nc_close(ncid) was called again after the first nc_close reported the error. I have attached the modified repro code in this post. Here is the sample output of running the code with netcdf 4.7.3: status after close = 0 Interestingly, I could only reproduce the crash using netcdf 4.6.1 and later versions. Calling nc_close(ncid) twice did not crash but gave the following output with 4.3.3.1: status after close = 0 which is expected as as you can see in the case of "no error" calling nc_close twice returned -33. I would expect the same in the case of any error as well. Can you please confirm if this is a bug in later versions of netcdf? Probably something changed between 4.3.3.1 and 4.6.1 which caused this crash to happen. Thanks, netcdfRepro_VarDimErrorAndCrash.txt . |
|
Hi @edwardhartnett and @DennisHeimbigner, Any comments on the crash I saw? Thanks, |
|
Firstly, thanks for all the hard work tracking this down! I'm sorry that this issue, like so many, does not get the attention it deserves. ;-) I wish that there was a giant team of programmers working on netCDF, but unfortunately there is only a small team of (in one case) giant programmers. The question of what happens when you call nc_close() again, after calling it once and getting an error, is undefined. Basically, don't do that. I agree it should not crash however. So this is a second bug. In terms of the first bug, the dimension scales issue is a difficult one. I will take a look when I get a chance. |
|
Hi @edwardhartnett and @DennisHeimbigner , I work closely with Nalini and Ellen for providing NetCDF functionality in MATLAB. I wanted to check if there is any update regarding the crash issue reported by Nalini. Please let me know if you have any questions for me to clarify our workflow. Thanks, |
|
Sorry, no update nor likely to be any progress for some time. |
|
Just noting that this is still an issue. Minimal: Resulting.. Alternatives:
We here are writing a generic library which just uses this as a standard storage format. |
|
Sorry this is a long-known issue, deeply embedded in the core netCDF-4 code and HDF5's way of dealing with dimensions. The work-around is to define the dimension first. |
|
Just to expand on the like : |
|
I think we hitting this bug also in julia's NCDatasets.jl. In addition to the For what is is worth, here are the steps in julia leading to an assertion error when the dimension name is used during a variable definition: using NCDatasets
ou = NCDataset("test.nc", "c") # nc_create
defDim(ou, "TIME1_10", 10) # nc_def_dim
nctax = defVar(ou, "tax", Int32,("TIME1_10",)) # nc_def_var
nctax[:] = 1:10 # nc_put_var of the data 1,2,3...10
defDim(ou, "tax", 10) # should netCDF-c rather raise an error already here?
ncmyvar = defVar(ou, "myvar", Int32, ("tax",))
ncmyvar[:] = 1:10 # HDF error
close(ou) # assertion errorGiven that this is a long-standing issue, I am wondering if it would be possible:
full stack trace on nc_close |
|
Is PR #2161 relevant to this? |
Hello everyone!
Short question: In a netcdf4-classic file, can a fixed-size dimension share the same name with a previously-defined variable?
A customer reported a MATLAB crash with following steps using our nccreate function. To me, this wouldn’t make sense to name the second variable’s dimension the same name as the first variable’s name. Wondering if this is related to issue #1528, but in our case they are fixed-size dimensions. We're using netcdf 4.7.3 and this is not OS-specific.
nccreate('test.nc','var1','Dimensions',{'dim1',3}); % one-dimension, vector of length 3
nccreate('test.nc','var2','Dimensions',{'var1',2}); % one-dimension, vector of length 2
This errors during nc_close() with error -101 NC_EHDFERR, and then we crash (oops). Note: nccreate defaults to netcdf4-classic. Calling it twice in succession like that means we first create new file test.nc and write var1 with dim1 to it, then close. Then reopen for append and create var2 with dimension name var1, then close.
I wrote a netcdf-c program (attached, netcdfRepro_VarDimError) to reproduce it (using netcdf-c v4.7.3), and still get the same NC_EHDFERR error, but no segfault or similar crash. (My repro is .c, but looks like github won't let me attach a .c file, so renamed it .txt).
netcdfRepro_VarDimError.txt
The text was updated successfully, but these errors were encountered: