Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zero sst issue in low resolution s2s test run #149

Closed
junwang-noaa opened this issue Jul 30, 2020 · 44 comments
Closed

zero sst issue in low resolution s2s test run #149

junwang-noaa opened this issue Jul 30, 2020 · 44 comments
Assignees

Comments

@junwang-noaa
Copy link
Collaborator

Phil Pegion reported that when running the 1 degree low resolution test, he found an issue that there are some lakes in Northern Canada. Based on the coupler file (ufs.s2s.cold.cpl.hi.2012-01-01-03600.nc), it looks like the atmosphere think's these points are ocean, but the ocean model does not have anything at those points. And the result is a tmpsfc=0.0 at these points.

Phil provided test case on hera at:

/scratch2/BMC/gsienkf/Philip.Pegion/UFS-coupled

test1_1deg_cold
test1_1deg_cold.nems
test1_1deg_warm
test1_1deg_warm.nems

The CMEPS build code is in /scratch2/BMC/gsienkf/Philip.Pegion/ufs-s2s-model_1deg/
and NEMS build code is in /scratch2/BMC/gsienkf/Philip.Pegion/ufs-s2s-model_1deg.nems/NEMS/

@junwang-noaa
Copy link
Collaborator Author

From Mariana:
Thanks for looking into this. One difference between NEMS and CMEPS is that the mapping of SST from the ocean to the atm is scaled by the ocean fraction in CMEPS whereas it is not in NEMS. We had talked about this on a call with Tony at one point and he agreed that mapping between ocn->atm and ice->atm should be scaled by ofrac and ifrac respectively. At any rate, this should work in the fractional branch - but I am wondering if there is a problem in the non-fractional branch when we do this. It is a one line change to remove the scaling by the ocean fraction. I'll work through this with Denise tomorrow. We can also talk more in our Friday call.

It would be good for us to duplicate Phil's problem on cheyenne and then see if modifying the mapping call fixes it.

From Jun:
The zero sst happened in non-fractional grid runs.
Also, I am curious why temperature needs to be scaled with ocean frac or ice frac. I thought we only need ocean frac to determine if it is open water, if it is mediator provides sst from ocean to atm, or if it is ice, providing ice temperature on ice. Anyway, since this is a non-fractional grid run, the issue I see here is that atmosphere thinks it is an ocean point, but mediator thinks it is a land point. In NEMS it can output the atmosphere land sea mask in mediator, but now in CMEPS mediator file, the shade MED_FRAC_ATM_LFRIN and shade MED_FRAC_ATM_LFRAC are all zeros, may I ask how those fields are set up?

From Mariana:
Hi Jun,

The reason you want to scale with either the ocean fraction or the ice fraction is summarized in the CEMPS documentation https://escomp.github.io/CMEPS/versions/master/html/esmflds.html#exchange-of-fields
and I will also copy that section here.

Fractional normalization is needed to improve the accuracy of field exchanges between ice and ocean and atmosphere. Consider the case where one cell has an ice fraction of 0.3 and the other has a fraction of 0.5. Mapping the ice fraction to the atmospheric cell results in a value of 0.4. If the same temperatures are mapped in the same way, a temperature of -1.5 results which is reasonable, but not entirely accurate. Because of the relative ice fractions, the weight of the second cell should be greater than the weight of the first cell. Taking this into account properly results in a fraction weighted ice temperature of -1.625 in this example. This is the fraction correction that is carried out whenever ocean and ice fields are mapped to the atmosphere grid. Note that time varying fraction corrections are not required in other mappings to improve accuracy because their relative fractions remain static.

If you look at the NEMS code you will see that all ice->atm mapping is done with ice fraction normalization. The same "should" hold for ocn->atm mapping. I believe that in the fractional configuration this should be no problem - but it seems to sometimes be the case in the non-fractional configuration. We can confirm this. The reason that lfrac and lfrin are zero is that you have no land model communicating with the mediator. The land model sends lfrac and lfrin. In addition, you are doing all of the merging from the ocean, ice and internal FV3 land inside FV3 - not in the mediator. The merging done in the mediator for nems is in the prep_ocn and prep_ice only. So these fractions are set to zero. Please let me know if that makes sense.

@junwang-noaa
Copy link
Collaborator Author

From Mariana:
Hi Phil,

I'd like to keep Denise in the loop on our exchanges - to avoid a lot of pair-wise communication. I am happy to give you the one line change for you to try on hera or whatever machine you are using. It might be easier than working on cheyenne (although you could do that also).

in esmFldsExchange_nems_mod.F90 - change
call addmap(fldListFr(compocn)%flds, 'So_t', compatm, maptype, 'ofrac', 'unset')
to
call addmap(fldListFr(compocn)%flds, 'So_t', compatm, maptype, 'none', 'unset')

This is my hypothesis as to the problem - but of course I might be wrong. But that said, in my opinion moving forward we really should be working with the fractional configuration - since that one is the correct way to proceed.

@junwang-noaa
Copy link
Collaborator Author

@mvertens May I ask how the ofrac is created in mediator?

@mvertens
Copy link

@junwang-noaa - this is documented in https://escomp.github.io/CMEPS/versions/master/html/fractions.html.
So when you say ofrac- do you mean
ofrac on the ocean mesh
ofrac on the ice mesh
ofrac on the atm mesh
All of this is handled in the fractions modules.
The logic in the fractions module is that during the run:

ifrac on the ice mesh is received from the ice component.
ofrac on the ice mesh is then computed as ice_mask-ifrac on the ice mesh.

ifrac on the ocean mesh is just a redistribution of ifrac on the ice mesh
ofrac on the ocean mesh is just a redistribution of ofrac on the ice mesh

ifrac on the atm mesh is computed as either a nearest neighbor conservative or conservative mapping of ifrac on the ice mesh to ifrac on the atm mesh
ofrac on the atm mesh is computed as either a nearest neighbor conservative or conservative mapping of ofrac on the ice mesh to ofrac on the atm mesh

@pjpegion
Copy link

@mvertens that fixes the problem, but is also changes the answer around the ice-edge, which I suppose is why we can the ofrac in there.

@mvertens
Copy link

@pjpegion - great to hear that this works. I'm not sure what you mean by "can the ofrac there".

@pjpegion
Copy link

oops, I suppose I should read what I write. What I meant to say is I suppose this is why we should have the ofrac in there.

@mvertens
Copy link

@pjpegion - thanks for the clarification. This now duplicates what NEMS is doing. This should not be a problem in the fractional configuration since the masks should be consistent between the ocean and fv3 and at that point you should scale by ofrac. @DeniseWorthen and I will try this out on cheyenne.

@pjpegion
Copy link

@mvertens sounds good. Hopefully we can get the fractional grid working soon.

@mvertens
Copy link

@pjpegion - the fractional grid configuration should be working in other resolutions. I believe that @DeniseWorthen and I are waiting for Shan (not sure what her github name is) to give us initial data that we can use for the 1 degree ocean.

@pjpegion
Copy link

it is @shansun6

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jul 30, 2020 via email

@mvertens
Copy link

@junwang-noaa - the fix I provided @pjpegion means that CMEPS is doing exactly what NEMS was doing in the non-fractional configuration. The ocean SST is simply mapped from the ocean to the atm with conservative nearest neighbor. In the fractional grid implementation, the ocean always sets the mask - and FV3 is responsible for ensuring that internally its land fraction is consistent with that ocean mask. The only difference in the CMEPS between the fractional and non-fractional configuration is whether that ocn->atm and ice->atm mapping is conservative or conservative nearest neighbor.

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jul 30, 2020 via email

@JessicaMeixner-NOAA
Copy link
Collaborator

I thought with fractional grid, the ocean model dictates what the land/sea mask is and then the atmospheric grid is made to match. That was my understanding. @shansun6 is that correct?

@mvertens
Copy link

@JessicaMeixner-NOAA - that is exactly what I thought as well.

@junwang-noaa
Copy link
Collaborator Author

I don't see how ocean grid is used to create atmosphere land sea mask grid in fractional grid.

@JessicaMeixner-NOAA
Copy link
Collaborator

@junwang-noaa I think that's why @shansun6 was asking for the ocean grid land/sea mask (of whatever resolution) mapped onto the atmosphere grid, to be used as input when creating the oro data.

@mvertens
Copy link

@JessicaMeixner-NOAA - I agree. When the ocean model dictates the mask - the ocean fraction on the atm mesh is obtained by mapping the ocean mask to the atm mesh.

@junwang-noaa
Copy link
Collaborator Author

Then the issue should be resolved in non-fractional grid as well, as in the current s2s RT fv3 c96 grid is matching mediator ofrac grid, so there is no zero sst issue. But in Phil's run they do not match, therefore we see 0 sst. @pjpegion how do you create mom6 one degree grid? The issue we see in your test case may not show up if you recreate oro data then, atmosphere grid after you created 1 degree ocean grid, this way you don't need the fix in CMEPS.

@mvertens
Copy link

@Junwang - the problem is that we were normalizing by ofrac in the non-fractional grid and that was causing the problem. Once @pjpegion put in my fix not to do that - everything worked. And its consistent with the NEMS approach for the non-fractional configuration. I believe that we do need to normalize by ofrac for the fraction configuration and I will validate this with @DeniseWorthen.

@jiandewang
Copy link
Collaborator

jiandewang commented Jul 30, 2020 via email

@DeniseWorthen
Copy link
Collaborator

I think it would be helpful if Mariana and I could take some time to a) verify that the non-fractional grid is now working w/ the fix and then b) turn our attention to the fractional grid for low res once we have the required input. I also think it might be helpful maybe for us to have time set aside at one of the Friday technical calls to get everyone on the same page as to how fractional grid should work etc.

@junwang-noaa
Copy link
Collaborator Author

@mvertens I feel the issue is not about ofrac, it's the mismatching of grid between atmos/ocean. In fractional grid test, if there is mismatching points, I believe the ofrac normalizing would still be a problem because the ocean thinks it's complete land (ofrac=0) while fv3 is waiting for sst. I think we need to have consistent grid (fractional or non-fractional) . Basically if ocean grid is changed (in Phil's case), we need to regenerate atm grid to match that.

@pjpegion
Copy link

@junwang-noaa Jiande gave me the grid file, and I am using the oro_data file that you have me, which you said is from @shansun6. One potential issue is that the oro data file has a slmsk in it, but that one is ignored, and the slmsk in the sfc_data.tile*nc files is used.

@jiandewang
Copy link
Collaborator

jiandewang commented Jul 30, 2020 via email

@pjpegion
Copy link

you can ignore my previous comment about the slmsk. I tried initial conditions that pointed to the new oro file, and had the same problem.

@junwang-noaa
Copy link
Collaborator Author

I think both oro files are not matching with the new 1 degree ocean mask. The oro files I got from Shan (and the oro files in current RT) are matching with current s2s C96 RT test 1/4 degree ocean grid mask. @phil Pegion @jiande Wang - NOAA Affiliate Maybe you can provide the new 1 degree ocean file to Shan (@shansun6),  She can help to create new oro files and sfc ICs from that.

@jiandewang
Copy link
Collaborator

jiandewang commented Jul 30, 2020 via email

@DeniseWorthen
Copy link
Collaborator

The "ocean grid" file downloaded from gfdl at any given destination resolution is actually the supergrid---ie, 2x desired resolution. It is the grid that MOM reads in at run time (ocean_hgrid.nc). It cannot be used directly. You can however use the land_mask.nc in that directory.

@pjpegion
Copy link

@shansun6 have you tried to generate a C96 orography based on the 1-degree ocean grid file? We would like to test this file to see if it fixes our issue. If you don't have time to do it, can you please let me and @HenryWinterbottom-NOAA know where the code is to generate the new oro file? Thanks

@ShanSunNOAA
Copy link
Collaborator

ShanSunNOAA commented Aug 13, 2020 via email

@pjpegion
Copy link

@shansun6 we will give it a try.

@pjpegion
Copy link

@shansun6 @junwang-noaa using the correct orography file from Shan works. I think this issue can be marked as resolved.
Thanks!

@DeniseWorthen
Copy link
Collaborator

This fixes it without the one-line change Mariana provided?

@pjpegion
Copy link

yes

@ShanSunNOAA
Copy link
Collaborator

ShanSunNOAA commented Aug 13, 2020 via email

@HenryRWinterbottom
Copy link

@shansun6 Can you please provide the code that was used to make these changes to the FV3 oro_data files?

@rsdunlapiv
Copy link
Collaborator

We should have a plan to have several people test and vet the procedure and then move the instructions to the ufs-s2s-model wiki in a public place. This will be needed for the s2s public release. @arunchawla-NOAA what do you think?

@ShanSunNOAA
Copy link
Collaborator

ShanSunNOAA commented Aug 15, 2020 via email

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Aug 15, 2020

We have started a document for how to set up the ICs for the S2S model for posting to the UFS-S2S-model wiki.

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Aug 17, 2020 via email

@DeniseWorthen
Copy link
Collaborator

I will close this issue. The transfer of the required tools is in ufs-utils #143

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants