Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Complete issue #607 (Optimization of multiscale regional runs) #670

Merged
merged 62 commits into from
Jan 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
b846b29
GitHub Issue NOAA-EMC/GSI#175. Use the global 127L B-Matrix in regio…
jderber-NOAA Aug 13, 2021
05f1c1f
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Sep 30, 2021
059e402
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Oct 15, 2021
f938842
GitHub Issue NOAA-EMC/GSI#219 Improve Minimization and fix bug in vqc
jderber-NOAA Oct 22, 2021
fafadac
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Nov 3, 2021
52c5ae6
fix setupw
jderber-NOAA Nov 5, 2021
f00e377
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Nov 16, 2021
7703367
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Feb 17, 2022
9eb9606
Merge remote-tracking branch 'upstream/master'
jderber-NOAA Feb 23, 2022
3eb0e13
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jun 17, 2022
ddced98
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jun 29, 2022
f60343b
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jul 11, 2022
8dbfbd1
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jul 14, 2022
1554f65
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jul 21, 2022
bf060fd
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Aug 29, 2022
3f073fa
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Nov 4, 2022
7f62d1c
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jan 25, 2023
85cbdb1
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Feb 23, 2023
51a444b
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Mar 3, 2023
0be4126
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Mar 22, 2023
57fda95
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Apr 13, 2023
6336b79
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Apr 17, 2023
a10841d
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA May 10, 2023
7261674
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA May 30, 2023
9bf983b
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jun 27, 2023
6ba6024
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Jul 26, 2023
5c0d74d
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Aug 30, 2023
00dc30d
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Sep 3, 2023
7e4f656
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Sep 21, 2023
f8f1bbf
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Sep 22, 2023
e1ebee6
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Sep 24, 2023
fb7f6ab
A few minor changes to stpcalc.f90
jderber-NOAA Oct 2, 2023
5ccfb02
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Oct 2, 2023
526d8ad
Merge branch 'develop' into optimize6
jderber-NOAA Oct 2, 2023
bd78ee1
optimizations
jderber-NOAA Oct 11, 2023
489d2f9
optimization changes again.
jderber-NOAA Oct 16, 2023
64b91b6
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Oct 16, 2023
2b0d8e9
Merge branch 'develop' into optimize6
jderber-NOAA Oct 16, 2023
7143186
Fix bug in convthin.f90
jderber-NOAA Oct 20, 2023
1ae500e
Improvement to thinning
jderber-NOAA Oct 21, 2023
67b867c
Improve satwnd.f90
jderber-NOAA Oct 30, 2023
976da68
Improvements to thinning and read_satwnd.
jderber-NOAA Nov 3, 2023
9430a23
Add thinning and data retention options.
jderber-NOAA Nov 20, 2023
9358952
Read options 2-3 fixes
jderber-NOAA Nov 28, 2023
eae087e
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Nov 28, 2023
5e82291
Merge branch 'develop' into optimize6
jderber-NOAA Nov 28, 2023
603b5c8
Final set of changes so that it will pass regression tests.
jderber-NOAA Dec 10, 2023
e7b3764
Merge remote-tracking branch 'upstream/develop' into develop
jderber-NOAA Dec 10, 2023
f544ed4
Merge branch 'develop' into optimize6
jderber-NOAA Dec 10, 2023
2a4ff98
Clean up
jderber-NOAA Dec 13, 2023
33d60d8
Remove code to reproduce order of data to optimize execution time.
jderber-NOAA Dec 13, 2023
9f6103b
Complete use_pbl usage change
jderber-NOAA Dec 16, 2023
19d43e6
Update code based on second set of Ting's comments.
jderber-NOAA Dec 19, 2023
7cf1df5
Fix bugs found in review.
jderber-NOAA Dec 20, 2023
593ed20
Fixing some of the issues pointed out by reviewer shoyokota.
jderber-NOAA Jan 5, 2024
7a92c3b
Changes based on shoyokota review #2.
jderber-NOAA Jan 5, 2024
e123548
Changes to code based on shoyokota changes #3
jderber-NOAA Jan 7, 2024
91d1f79
Modify code based shoyokota comments #4 (and last for first set of co…
jderber-NOAA Jan 10, 2024
4c4008a
Update to reproduce initial results (except hafs_3d)
jderber-NOAA Jan 18, 2024
b7bdbc4
Changes to make all regression tests to duplicate earlier results.
jderber-NOAA Jan 18, 2024
1fdc927
Merge remote-tracking branch 'upstream/develop' into optimize6
jderber-NOAA Jan 20, 2024
79cdb68
A few more minor changes.
jderber-NOAA Jan 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/enkf/controlvec.f90
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ subroutine read_control()
! read ensemble members on IO tasks
implicit none
real(r_double) :: t1,t2
integer(i_kind) :: nb,nlev,ne
integer(i_kind) :: nb,ne
integer(i_kind) :: q_ind
integer(i_kind) :: ierr

Expand Down
19 changes: 11 additions & 8 deletions src/enkf/letkf.f90
Original file line number Diff line number Diff line change
Expand Up @@ -172,8 +172,8 @@ subroutine letkf_update()
if (nproc == 0) print *,'using',nthreads,' openmp threads'

! define a few frequently used parameters
r_nanals=one/float(nanals)
r_nanalsm1=one/float(nanals-1)
r_nanals=one/real(nanals,r_kind)
r_nanalsm1=one/real(nanals-1,r_kind)
mincorrlength_factsq = mincorrlength_fact**2

kdobs=associated(kdtree_obs2)
Expand Down Expand Up @@ -541,31 +541,34 @@ subroutine letkf_update()
enddo
!$omp end parallel do

tmean=zero
tmin=zero
tmax=zero
tend = mpi_wtime()
call mpi_reduce(tend-tbegin,tmean,1,mpi_real8,mpi_sum,0,mpi_comm_world,ierr)
tmean = tmean/numproc
tmean = tmean/real(numproc,r_kind)
call mpi_reduce(tend-tbegin,tmin,1,mpi_real8,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(tend-tbegin,tmax,1,mpi_real8,mpi_max,0,mpi_comm_world,ierr)
if (nproc .eq. 0) print *,'min/max/mean time to do letkf update ',tmin,tmax,tmean
t2 = t2/nthreads; t3 = t3/nthreads; t4 = t4/nthreads; t5 = t5/nthreads
if (nproc == 0) print *,'time to process analysis on gridpoint = ',t2,t3,t4,t5,' secs on task',nproc
call mpi_reduce(t2,tmean,1,mpi_real8,mpi_sum,0,mpi_comm_world,ierr)
tmean = tmean/numproc
tmean = tmean/real(numproc,r_kind)
call mpi_reduce(t2,tmin,1,mpi_real8,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(t2,tmax,1,mpi_real8,mpi_max,0,mpi_comm_world,ierr)
if (nproc .eq. 0) print *,',min/max/mean t2 = ',tmin,tmax,tmean
call mpi_reduce(t3,tmean,1,mpi_real8,mpi_sum,0,mpi_comm_world,ierr)
tmean = tmean/numproc
tmean = tmean/real(numproc,r_kind)
call mpi_reduce(t3,tmin,1,mpi_real8,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(t3,tmax,1,mpi_real8,mpi_max,0,mpi_comm_world,ierr)
if (nproc .eq. 0) print *,',min/max/mean t3 = ',tmin,tmax,tmean
call mpi_reduce(t4,tmean,1,mpi_real8,mpi_sum,0,mpi_comm_world,ierr)
tmean = tmean/numproc
tmean = tmean/real(numproc,r_kind)
call mpi_reduce(t4,tmin,1,mpi_real8,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(t4,tmax,1,mpi_real8,mpi_max,0,mpi_comm_world,ierr)
if (nproc .eq. 0) print *,',min/max/mean t4 = ',tmin,tmax,tmean
call mpi_reduce(t5,tmean,1,mpi_real8,mpi_sum,0,mpi_comm_world,ierr)
tmean = tmean/numproc
tmean = tmean/real(numproc,r_kind)
call mpi_reduce(t5,tmin,1,mpi_real8,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(t5,tmax,1,mpi_real8,mpi_max,0,mpi_comm_world,ierr)
if (nproc .eq. 0) print *,',min/max/mean t5 = ',tmin,tmax,tmean
Expand All @@ -590,7 +593,7 @@ subroutine letkf_update()
call mpi_reduce(nobslocal_max,nobslocal_maxall,1,mpi_integer,mpi_max,0,mpi_comm_world,ierr)
call mpi_reduce(nobslocal_min,nobslocal_minall,1,mpi_integer,mpi_min,0,mpi_comm_world,ierr)
call mpi_reduce(nobslocal_mean,nobslocal_meanall,1,mpi_integer,mpi_sum,0,mpi_comm_world,ierr)
if (nproc == 0) print *,'min/max/mean number of obs in local volume',nobslocal_minall,nobslocal_maxall,nint(nobslocal_meanall/float(numproc))
if (nproc == 0) print *,'min/max/mean number of obs in local volume',nobslocal_minall,nobslocal_maxall,nint(nobslocal_meanall/real(numproc,r_kind))
endif
call mpi_reduce(nobslocal_max,nobslocal_maxall,1,mpi_integer,mpi_max,0,mpi_comm_world,ierr)
call mpi_reduce(nobslocal_min,nobslocal_minall,1,mpi_integer,mpi_max,0,mpi_comm_world,ierr)
Expand Down
2 changes: 1 addition & 1 deletion src/gsi/adjtest.f90
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ module adjtest
use gsi_bundlemod, only: assignment(=)
use bias_predictors, only: predictors,allocate_preds,deallocate_preds, &
assignment(=)
use control2state_mod, only: control2state,c2sset,control2state_ad
use control2state_mod, only: control2state,control2state_ad

implicit none
private
Expand Down
1 change: 0 additions & 1 deletion src/gsi/apply_scaledepwgts.f90
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,6 @@ subroutine apply_scaledepwgts(m,grd_in,sp_in)
use general_specmod, only: spec_vars
use general_sub2grid_mod, only: sub2grid_info
use hybrid_ensemble_parameters, only: spc_multwgt,en_perts,nsclgrp,n_ens
use mpimod, only: mype
implicit none

! Declare passed variables
Expand Down
14 changes: 2 additions & 12 deletions src/gsi/balmod.f90
Original file line number Diff line number Diff line change
Expand Up @@ -689,13 +689,7 @@ subroutine balance(t,p,st,vp,fpsproj,fut2ps)

!! Strong balance constraint
!! Pass uvflag=.false.
if(lsqrtb) then
call strong_bk(st,vp,p,t,.false.)
else
if(tlnmc_option==1 .or. tlnmc_option==4) call strong_bk(st,vp,p,t,.false.)
endif


if(lsqrtb .or. tlnmc_option==1 .or. tlnmc_option==4) call strong_bk(st,vp,p,t,.false.)

return
end subroutine balance
Expand Down Expand Up @@ -777,11 +771,7 @@ subroutine tbalance(t,p,st,vp,fpsproj,fut2ps)

! Adjoint of strong balance constraint
! pass uvflag=.false.
if(lsqrtb) then
call strong_bk_ad(st,vp,p,t,.false.)
else
if(tlnmc_option==1 .or. tlnmc_option==4) call strong_bk_ad(st,vp,p,t,.false.)
endif
if(lsqrtb .or. tlnmc_option==1 .or. tlnmc_option==4) call strong_bk_ad(st,vp,p,t,.false.)

! REGIONAL BRANCH
if (regional) then
Expand Down
3 changes: 1 addition & 2 deletions src/gsi/berror.f90
Original file line number Diff line number Diff line change
Expand Up @@ -844,8 +844,7 @@ subroutine create_berror_vars_reg

! Grid constant for background error

allocate(be(ndeg), &
qvar3d(lat2,lon2,nsig))
allocate(be(ndeg),qvar3d(lat2,lon2,nsig))
if(nc3d>0)then
allocate(alv(llmin:llmax,ndeg,nsig,nc3d), &
dssv(lat2,lon2,nsig,nc3d))
Expand Down
3 changes: 2 additions & 1 deletion src/gsi/calctends.f90
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ subroutine calctends(mype,teta,pri,guess,xderivative,yderivative,tendency)
use gsi_bundlemod, only: gsi_bundlegetpointer

use mpeu_util, only: die
use turblmod, only: use_pbl
implicit none

! Declare passed variables
Expand Down Expand Up @@ -357,7 +358,7 @@ subroutine calctends(mype,teta,pri,guess,xderivative,yderivative,tendency)
end do
end do !end do k

call turbl(u,v,pri,t,teta,z,u_t,v_t,t_t,jtstart(kk),jtstop(kk))
if(use_pbl)call turbl(u,v,pri,t,teta,z,u_t,v_t,t_t,jtstart(kk),jtstop(kk))

if(.not.wrf_nmm_regional.and..not.nems_nmmb_regional)then
do k=1,nsig
Expand Down
3 changes: 2 additions & 1 deletion src/gsi/calctends_ad.f90
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ subroutine calctends_ad(fields,fields_dt,mype)
use mpeu_util, only: die
use derivsmod, only: gsi_xderivative_bundle
use derivsmod, only: gsi_yderivative_bundle
use turblmod, only: use_pbl
implicit none

! Declare passed variables
Expand Down Expand Up @@ -356,7 +357,7 @@ subroutine calctends_ad(fields,fields_dt,mype)
end do
end if

call turbl_ad(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
if(use_pbl)call turbl_ad(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
u,v,pri,t,u_t,v_t,t_t,jtstart(kk),jtstop(kk))

do k=nsig,1,-1
Expand Down
3 changes: 2 additions & 1 deletion src/gsi/calctends_no_ad.f90
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ subroutine calctends_no_ad(st,vp,t,p,mype,u_t,v_t,t_t,p_t,uvflag)
use gsi_bundlemod, only: gsi_bundlegetpointer
use derivsmod, only: gsi_xderivative_bundle
use derivsmod, only: gsi_yderivative_bundle
use turblmod, only: use_pbl
implicit none

! Declare passed variables
Expand Down Expand Up @@ -210,7 +211,7 @@ subroutine calctends_no_ad(st,vp,t,p,mype,u_t,v_t,t_t,p_t,uvflag)
end do
end if

call turbl_ad(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
if(use_pbl)call turbl_ad(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jderber-NOAA a question about why , now use_pbl condition is required to call turbl_ad in calctends_no_ad.f90 and to call turbl_tl in calctends_no.f90 but not used to call turbl in calctends.f90.? And, seems this change could cause differences in results, right? Will this be able to explain those tiny differences in your regression test?

Copy link
Contributor Author

@jderber-NOAA jderber-NOAA Dec 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is one of those things that I started to do, but just didn't have time to finish. The turbl_ad, turbl_tl and turbl routines all had the first executable line saying if(.not.use_pbl)return. So the use_pbl in the call for turbl_ad and turbl_tl was redundant and results should not change.

I have completed what I was doing and added a "if (use_pbl) call turbl..." for all the calls to the turbl routines (turbl_tl, turbl_ad, and turbl). This is added in all of the calctends routines. The use_pbl has been removed from all of the turbl routines.

Regression tests rerun. Results are identical to above as expected.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation and clarification!

u,v,pri,t,u_t,v_t,t_t,jtstart(kk),jtstop(kk))

do k=nsig,1,-1
Expand Down
5 changes: 3 additions & 2 deletions src/gsi/calctends_no_tl.f90
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ subroutine calctends_no_tl(st,vp,t,p,mype,u_t,v_t,t_t,p_t,uvflag)
! v - meridional wind on subdomain
! t - virtual temperature on subdomain
! mype - task id
! uvflag - logical, set to true for st,vp wind components, instead of stream/potential function
! uvflag - logical, set to true for u,v wind components, instead of stream/potential function
!
! output argument list:
! u_t - time tendency of u
Expand All @@ -64,6 +64,7 @@ subroutine calctends_no_tl(st,vp,t,p,mype,u_t,v_t,t_t,p_t,uvflag)
use gsi_bundlemod, only: gsi_bundlegetpointer
use derivsmod, only: gsi_xderivative_bundle
use derivsmod, only: gsi_yderivative_bundle
use turblmod, only: use_pbl
implicit none

! Declare passed variables
Expand Down Expand Up @@ -364,7 +365,7 @@ subroutine calctends_no_tl(st,vp,t,p,mype,u_t,v_t,t_t,p_t,uvflag)
end do !end do j
end do !end do k

call turbl_tl(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
if(use_pbl)call turbl_tl(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
u,v,pri,t,u_t,v_t,t_t,jtstart(kk),jtstop(kk))

if(.not.wrf_nmm_regional.and..not.nems_nmmb_regional)then
Expand Down
3 changes: 2 additions & 1 deletion src/gsi/calctends_tl.f90
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ subroutine calctends_tl(fields,fields_dt,mype)
use mpeu_util, only: die, getindex
use derivsmod, only: gsi_xderivative_bundle
use derivsmod, only: gsi_yderivative_bundle
use turblmod, only: use_pbl
implicit none

! Declare passed variables
Expand Down Expand Up @@ -474,7 +475,7 @@ subroutine calctends_tl(fields,fields_dt,mype)
end do !end do j
end do !end do k

call turbl_tl(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
if(use_pbl)call turbl_tl(ges_prsi(1,1,1,it),ges_tv,ges_teta(1,1,1,it),&
u,v,pri,t,u_t,v_t,t_t,jtstart(kk),jtstop(kk))

if(.not.wrf_nmm_regional.and..not.nems_nmmb_regional)then
Expand Down
Loading