Excessively low T and high S in 0.25 degree

[aekiss]

@ezhilsabareesh8 has found T<3.1 and S>50 near the Siberian coast in December in the 0.25° configuration.

We suspect the excessive salinity is caused by excessive sea ice production, allowing excessively low temperatures without compensating frazil heating.

There may be some problem with the bathymetry. If so this tool may be useful for edits: https://github.com/COSIMA/topogtools/blob/master/editTopo.py
It records the changes made, allowing them to be aded to a reproducible topography generation workflow, as was done here: https://github.com/COSIMA/make_025deg_topo/blob/master/make_topog.sh#L32

[aekiss]

unfortunately editTopo is no longer working properly COSIMA/topogtools#9

now fixed

[aekiss]

Steps:

• identify the location or repeated crashes
• closeup plot (so we can see individual grid cells) of topography in crash location to see if anything looks weird
• if topography looks ok, try closeup plots of T, S, aice, etc from daily (or higher-frequency) data leading up to the crash

[ezhilsabareesh8]

I'm getting crashes while running the quarter degree configuration near the Laptev Sea, with excessive salinity, the following are the figures a day before the crash

WARNING from PE 209: Extreme surface sfc_state detected: i= 178 j= 950 lon=-240.586 lat= 73.068 x=-235.625 y= 76.219 D= 1.1806E+01 SSH= 8.8261E+00 SST=-3.2001E+00 SSS= 5.9101E+01 U-= 0.0000E+00 U+=-5.8223E-04 V-= 0.0000E+00 V+= 1.4975E-02

The location is marked by the red circle in the bathymetry figures below.

[ofa001]

Hi @ezhilsabareesh8 It looks like it is excessively shallow < 10m, we had a similar case in one of the latest access-cm2-025 cases under the ice off the Russian coast but outside the Arctic basin I was wondering if was the same point but its not, also over 50ppt salinity. Dave Bi used one of us old approaches to damp it down, but if we are re-editing the topography file we should include it in consideration too. as it may be an issue again.

[ezhilsabareesh8]

Thanks @ofa001! The depth at the crash location is approximately 11.805 meters, which is the smallest non-zero depth in the quarter-degree topog.nc file. This was fixed as part of issue 158, where the terracing at the Laptev Sea was removed. it seems like we may need to re-edit the topography file, any thoughts @aekiss?

[aekiss]

Thanks for the plots @ezhilsabareesh8. Nothing looks particularly weird in that location, and there's an extensive region with the same depth, so it's unclear to me what topographic change would be likely to help. That point is already at the minimum depth and deepening it would not make sense, given that the surroundings are equally shallow.

Presumably there's a coastal polynya in that region. To check, could you make some plots of the daily frazil_3d_int_z , sea ice concentration (aice) and thickness ('hi') just before the crash?

[ezhilsabareesh8]

Thanks @aekiss, I've looked into the Coastal Polynya and I didn't observe any abnormalities in the sea ice concentration, ice thickness, or frazil. However, I did notice a significant congelation ice growth at the crash location.

[aekiss]

Thanks @ezhilsabareesh8, that looks consistent with the hypothesis of very active sea ice growth.

[anton-seaice]

Is it possible this would stabilise later in the run? I know its ~11 months in, but could it be related to the initial conditions?

[ezhilsabareesh8]

The crash occurs after 2 years of the run, and the negative temperature continues to increase at the crash location, not sure if this relates to initial conditions.

[ofa001]

In Dave Bi's Run which was also at 55 ppt due to excess ice growth It was on the Russian coast but outside the Arctic basin, at the north-east corner of Shelekhov Gulf, it also led to a crash, he said today he hadn't checked its salinity since he put a fix in.

[aekiss]

What fix did Dave put in?

[aekiss]

What are the surface salinity restoring parameters? Maybe we should increase restoring when it's a long way from obs?

[ezhilsabareesh8]

These are the surface salinity parameters, I tried increasing the MAX_DELTA_SRESTORE to 5 but it didn't work

SALT_RESTORE_FILE = "salt_sfc_restore.nc" ! default = "salt_restore.nc"
                                ! A file in which to find the surface salinity to use for restoring.
SRESTORE_AS_SFLUX = True        !   [Boolean] default = False
                                ! If true, the restoring of salinity is applied as a salt flux instead of as a
                                ! freshwater flux.
MAX_DELTA_SRESTORE = 0.5        !   [PSU or g kg-1] default = 999.0
                                ! The maximum salinity difference used in restoring terms.

[aekiss]

Thanks. The salinity difference is about 20 so MAX_DELTA_SRESTORE=5 is still limiting the restoring rate. You could try 1000 (for debugging this issue, not for a production run).

Do you have any maps of salt restoring flux?
Maps of river runoff flux could also be relevant, as it's a coastal location.

I might take a look - where is the output directory?

[AndyHoggANU]

I agree it would be worthwhile increasing the MAX_DELTA_SRESTORE just to see what happens.

[aekiss]

Relevant code is here https://github.com/mom-ocean/MOM6/blob/87913b5e21d3c08b2634f68ed9f8b9faf51d8c9e/config_src/drivers/nuopc_cap/mom_surface_forcing_nuopc.F90#L373-L388

What is MASK_SRESTORE_UNDER_ICE set to? It defaults to false, meaning restore under sea ice (which is what we want).
https://github.com/mom-ocean/MOM6/blob/87913b5e21d3c08b2634f68ed9f8b9faf51d8c9e/config_src/drivers/nuopc_cap/mom_surface_forcing_nuopc.F90#L1234-L1238

[ezhilsabareesh8]

Thanks @aekiss and @AndyHoggANU. I will have a look at the salt restoring flux, the output files are in the following directory.

/g/data/tm70/ek4684/MOM6-CICE6/work

MASK_SRESTORE_UNDER_ICE is not set in the current configuration, which is FALSE by default.

MASK_SRESTORE_UNDER_ICE = FALSE ! [Boolean] default = False ! If true, disables SSS restoring under sea-ice based on a frazil criteria ! (SST<=Tf). Only used when RESTORE_SALINITY is True.

Since the congelation ice growth is significant at the crash location, I also suspect that we should turn on MASK_SRESTORE_UNDER_ICE. Currently, I am running two experiments one with MAX_DELTA_SRESTORE = 999.0 and the other with MASK_SRESTORE_UNDER_ICE = TRUE.

[aekiss]

Thanks Ezhil. I don't think we want MASK_SRESTORE_UNDER_ICE = TRUE - according to the comments this will disable restoring under ice.

[AndyHoggANU]

We may not want it long-term, but I still think it's worth testing to see what we find. Sometimes the comments describing parameter choices have been inaccurate ...

[aekiss]

The comments seem to agree with the code.

In any case, it won't detect sea ice correctly using the frazil criterion, because SSS is so high that frazil isn't forming (which is why the growth is mostly congelation, and also why it's getting so cold, as there's no latent heat from frazil formation).

[aekiss]

This formula sets a notional frazil formation temperature of -0.0539*SSS, which is -3.19C when SSS= 5.9101E+01, consistent with SST=-3.2001E+00 at the crash location. So the water is at its freezing point (assuming the ocean model uses the same freezing point depression formula as the cap).

[ezhilsabareesh8]

Salt restoring flux

River runoff flux

This formula sets a notional frazil formation temperature of -0.0539*SSS,

Is this formula for not restoring salt flux under ice (as mentioned here) ?

[ofa001]

Dave Bi's fix is a diffusive term, similar to that he used in the Red Sea, just seeing this issue about the lack of salt restoring under ice, Its a fix thats been used in previous generation models ( CSIRO MK2, Mk3) at tricky coastal points

[aekiss]

Thanks Ezhil.

Looks like you're plotting on xt_ocean, yt_ocean, which is why your coastlines and red dot are misaligned with the colour map.

If so, there's a strong negative salt flux in the problem area - does negative mean it's a salt flux out of the ocean? That would make sense.

It's unclear if the runoff is exactly zero at the coast, as it might not be visible with this colormap.

Can you try replotting both of these with different colour scales, to highlight what's happening in the crash location?

Is this formula for not restoring salt flux under ice (as mentioned here) ?

Yep

[ezhilsabareesh8]

Thanks Andrew, I fixed the plots.

If so, there's a strong negative salt flux in the problem area - does negative mean it's a salt flux out of the ocean? That would make sense.

Yes a negative restoration.

Can you try replotting both of these with different colour scales, to highlight what's happening in the crash location?

I tried changing the scales still it's the same, I am plotting here, Friver (Water Flux into Sea Water From Rivers)

Salinity

[minghangli-uni]

I tried changing the scales still it's the same, I am plotting here, Friver (Water Flux into Sea Water From Rivers)

From the plot, it is very small. For a more definitive check, can you provide the values of Friver in that area to see if it is exactly zero at the coast?

[ezhilsabareesh8]

From the plot, it is very small. For a more definitive check, can you provide the values of Friver in that area to see if it is exactly zero at the coast?

It is exactly zero

[aekiss]

Thanks Ezhil. I guess the real question is, should Friver be zero? Might need to look at the runoff file being fed into DROF to see if runoff is any supposed to be happening there (possibly not, as runoff is weak in Arctic winter).

[aekiss]

Could you try plotting this with a broader scale, so the negative values aren't off-scale? I'm wondering what the structure looks like at the crash location but it's all the same colour at the moment.

[ezhilsabareesh8]

Could you try plotting this with a broader scale, so the negative values aren't off-scale? I'm wondering what the structure looks like at the crash location but it's all the same colour at the moment.

I zoomed the crash location and included the negative values in the scale, it seems the salinity restoration is almost constant at the crash location, may be limited by MAX_DELTA_SRESTORE?

[ezhilsabareesh8]

Might need to look at the runoff file being fed into DROF to see if runoff is any supposed to be happening there (possibly not, as runoff is weak in Arctic winter).

I have turned on the history outputs and checking, will update shortly on this

[aekiss]

may be limited by MAX_DELTA_SRESTORE

thanks, yes, that seems to be what's happening. I thought the triangular ends of your colourbar indicated it was saturated, but I guess not.

[ezhilsabareesh8]

thanks, yes, that seems to be what's happening

When I increased the MAX_DELTA_SRESTORE to 999.0 (default value), salinity restoring is not limited at the crash location and the simulation didn't crash yet. The minimum temperature at the crash location is now -2.562991 celsius and the salinity is 47.3

Salinity restoring flux (restoring is high at the crash location when compared to previous run), salinity and SST plots when MAX_DELTA_SRESTORE is set to 999.0

[DaveBi]

extremely high salinty at the north-east corner of Shelekhov Gulf due to ice formation and no efficient communication with outside through the narrow "passage" -- better to make the outlet passage at least 4-gridpoints wide.

Weddel Sea (coastline) thick sea ice due to, again, communication constraint by topograpghy (2-point, too narrow, should be "smoothed" off...)

[ezhilsabareesh8]

When I increased the MAX_DELTA_SRESTORE to 999.0 (default value), salinity restoring is not limited at the crash location and the simulation didn't crash yet

Increasing MAX_DELTA_SRESTORE to 999.0 has temporarily prevented higher salinity issues and allowed the simulation to run for four years, I recognize that this may not be a permanent solution. Any suggestions @aekiss to address this issue without relying on setting high MAX_DELTA_SRESTORE value?

[ezhilsabareesh8]

Might need to look at the runoff file being fed into DROF to see if runoff is any supposed to be happening there (possibly not, as runoff is weak in Arctic winter).

There is no runoff is happening at or near the crash location, both the rofImp_Forr_rofl and rofImp_Forr_rofi from the mediator outputs are exactly zero at the crash location

[aekiss]

I think it's OK for now to use MAX_DELTA_SRESTORE=999.0 (it's the default value, after all).

We don't particularly care about this bit of ocean, but we should keep an eye on the salinity restoring to see if it becomes excessive anywhere else.

[ezhilsabareesh8]

Time-mean sfc_salt_flux_restore over the seven years (1900-1907) of the 0.25 deg run.

we should keep an eye on the salinity restoring to see if it becomes excessive anywhere else.

It looks similar to the SSS restoring here and the mean salinity looks normal. However, it appears that restoring is excessively negative in the Laptev Sea, is that normal?

[aekiss]

Thanks @ezhilsabareesh8.

For comparison, there are some maps of restoring with various iterations of the ACCESS-OM2 1° and 0.25° topography here, here and here.

For example 025deg_jra55_iaf_omip2_cycle[1-6] (from here) doesn't have strong negative restoring in the Laptev Sea, north of Greenland, or northwest Canada. We should probably look at the runoff fluxes to see if they're comparable, and also the salt and freshwater fluxes with CICE.

An update to the topography in /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo2_cycle1 achieves biases in the Mediterranean Sea, White Sea, Black Sea, Red Sea and Persian Gulf that are lower than OM3 (fig from here) - we'll need to address this by tweaking topography at the mouths of these regional seas when we redo the topography for the C-grid.

We haven't made a global map of SSS restoring with the 5th iteration /g/data/ik11/outputs/access-om2-025/025deg_jra55_iaf_omip_straits_topo5_cycle1 - that would be an even better point of comparison (though the differences would mainly be in marginal seas, especially the Baltic).

[ezhilsabareesh8]

Thanks @aekiss. Here are the time series plots of salinity restoring in marginal seas.

1. The Baltic salinity restoring is almost zero. Here is the baltic sea salinity restoring time series from 0M2 run.

2. Persian Gulf: The salinity restoring shows large seasonal drops, particularly negative fluxes, indicating significant removal of salt during certain periods which appears suspicious. Such a seasonal drops are not observed in OM2 runs.

3. Med Sea: Shows a consistent positive trend over time, indicating a continuous addition of salt (It may stabilise after a long run). In OM2 the salinity restoring is negative in this region and a consistent positive trend over time

4: Red Sea: Shows a negative restoring within a moderate range. OM2 runs doesn't show a consistent negative restoring in this region.

[aekiss]

Great, thanks @ezhilsabareesh8 - looks like we'll need to adjust the Red Sea, Persian Gulf and maybe Mediterranean when we make the new topography. Great that the Baltic looks ok - that was a pain in OM2.

[ezhilsabareesh8]

3. Med Sea: Shows a consistent positive trend over time, indicating a continuous addition of salt (It may stabilise after a long run). In OM2 the salinity restoring is negative in this region and a consistent positive trend over time

In this plot I summed up the salinity restoring over the entire region (similar to Andy's) instead of taking mean. Here the Med Sea looks suspicious, for a longer run it shows a big upward trend in added salt flux over time and also the salinity of Med sea is dropping steadily.

[aekiss]

Related discussion: #167