-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cdscan fails on grib control files #75
Comments
@tangq Hi. I think it would be helpful to isolate if this is a cdscan or e3sm_unified problem. I will try test with standalone cdat environment to test if cdscan applies to the particular data. |
I tested cdscan within different cdms2 versions (3.0.0 and 3.1.4) was able to reproduce @tangq 's error in v3.1.4. Cdscan in v3.0.0 works. I'm directing this issue to CDAT development team. |
Thanks @chengzhuzhang for the help. It would be nice if the cdscan fix can be added to the newly released e3sm_unified. Since the older cdscan works, I assume the fix should be relatively minor and quick. |
@chengzhuzhang , did you run the old cdscan on cori? If so, is that possible for me to use the older version? I need to run it for multiple years of data. |
I tried in one of my old environment for e3sm_diags. I can create the .xmls for you. Just let me know how and where I should create those files. |
@tangq, this is the first I'm hearing about |
I see, after some investigation, Even if there's a bug fix, I'm hesitant to include it in this version of E3SM-Unified unless we can also be sure that E3SM_Diags works fine with the new version. And we'd need a new release really soon to not hold up the roll-out. Talking with @muryanto1, it sounded like they're testing a release candidate and a bug fix might be able to go in, but that we couldn't expect a new |
@xylar , thanks for the clarifications. I will use some older cdms2 as a temporary solution. |
Thanks, @chengzhuzhang . Some of the grib files are still downloading. I will let you know when it's complete. |
This seems to be getting addressed here: Once it is handled, I think I can patch the existing version of cdms2 to include this fix as well. If that happens very soon, I might be able to include it in E3SM-Unified 1.3.1 after all. |
@tangq, this has been fixed in the latest Once that is in, I will rebuild E3SM-Unified on compy and anvil, and ask you to try again. Hopefully, by Monday. |
@xylar , the issue reported there is for netcdf files. I am not sure if the fix can also solve the issue here for the grib control files. I will need to test it once you rebuild the ESM-Unified. Thanks. |
@tangq, E3SM-Unified really isn't a good place for testing this. I'll do it this time but we need to figure out a better workflow for noticing these issues earlier and testing with the new packages as they are released. That's way outside of my job description. |
@rljacob and @chengzhuzhang, can we add a discussion of how to better stress test E3SM-Unified to an IG agenda? |
@xylar , I agree that this is not an E3SM-Unified issue. It should be addressed by individual packages before it is integrated into E3SM-Unified. |
On the E3SM-Unified confluence page, I noticed that packages are assigned to different people. Perhaps, they know more about these packages and thus have better testing ideas? |
Right, but if E3SM-Unified isn't very useful for the next 6 months because it includes this issue, that's too bad. So if it's easy to address, that would be great!
Yes, that's true. Especially for the "main" packages, we do expect that these people have tested them before they go into E3SM-Unified. But at least this time, a lot of issues seem to have slipped through the cracks unnoticed. This suggests we may need a trial period for E3SM-Unified that's longer than a couple of days to catch these issues. This didn't seem to be needed previously so I hadn't thought about it too much. But I also thought that update every 2-3 months would be better than every 6-9. Project management disagrees. |
Yes. It will be a very useful discussion. Also we may want to roll out a reasonable schedule for deploying E3SM-Unified, which should balance the project need and development cycle of individual packages that are in active development. A fixed schedule with testing period might be helpful for each tools to be better prepared before adding to a E3SM-Unified release. |
@tangq, I have updated e3sm-unified on Anvil and Compy to include the patches in CDAT/cdms#399 and conda-forge/cdms2-feedstock#57. Could you please re-test cdscan to see if you still see problems? If so, could you make an issue on https://github.com/CDAT/cdms/issues? |
@xylar, I tested it on compy at /compyfs/tang338/tst. It got the same errors. I am not sure the env loaded by If there is another way to load the E3SM-Unified with the patches, please let me know and I can test again. |
@tangq oh, no! I'm sorry about that! Thing have moved to:
But I didn't update the path on Confluence. I'm doing my best to fix that now. |
Using 1.3.1 I got the same errors:
It looks like that the "list" command is not added on these lines...This is why I am not a fan of the non-backward compatible python updates. |
@tangq, That's too bad! As I said, this is an issue for CDAT folks. But it looks like a fix is unlikely to make it into this E3SM-Unified.
I don't understand why this is relevant. The update was incomplete but I don't think this is related to backwards compatibility... |
The cdscan issue is caused by that py3 is not backward compatible with py2. The fix CDAT put in is adding list() around some statements. I should have made it clearer that the update I meant is the py3 updates, not the E3SM-Unified updates. |
Ah, I see. Yes, python major versions are not meant to be backwards compatible. That may seem inconvenient but it comes with a lot more freedom to improve the performance and API over time. And there were 8 years of warning that python 2 would go away... |
I'm going to close this again. I am almost done deploying E3SM-Unified 1.3.1. All other issues that we identified have been addressed and it doesn't seem like there will be a quick enough fix for this one to make the cut. @tangq, please do open an issue with the CDAT folks if you haven't already. It would be nice to have this fixed and ready for the next E3SM-Unified release in 6 months or so. |
@xylar , sounds good. @chengzhuzhang opened an issue for this already. |
cdscan has been used to merge multiple grib control files into a single file. However, it fails with the newer versions in the e3sm_unified. The follow errors appears with e3sm_unified_1.2.6 and 1.3.0.
This is likely an issue with newer cdscan version. An older version from uvcdat 2.4.1 on the other machine works fine.
The files to reproduce this issue is at cori:/global/cscratch1/sd/tang30/ECMWF_Interim/download/2014
[tang30@cori06 2014]$ cdscan -x tst.xml *.ctl
Finding common directory ...
Common directory:
Scanning files ...
UVTQPS-20140101_T255.grib.ctl
Setting reference time units to
Traceback (most recent call last):
File "/global/cfs/cdirs/e3sm/software/anaconda_envs/base/envs/e3sm_unified_1.2.6/bin/cdscan", line 1840, in
main(sys.argv)
File "/global/cfs/cdirs/e3sm/software/anaconda_envs/base/envs/e3sm_unified_1.2.6/bin/cdscan", line 1282, in main
timeIsLinear = (referenceTime[0].lower().split() in
IndexError: string index out of range
The text was updated successfully, but these errors were encountered: