Loop ordering in diagnostic_fluxes.fpp
Issue #79
new
I’m not familiar with this part of the code, but the loop at line 626 of diagnostic fluxes looks dubious:
! below we define gparity = J0 * h, where delta f = h - (q phi / T) F0
! because we're ultimately interested in int J0 h * phi (i.e. particle flux)
do isgn = 1, 2
do ig = -ntgrid, ntgrid
do iglo = g_lo%llim_world, g_lo%ulim_world
ik = ik_idx(g_lo,iglo)
ie = ie_idx(g_lo,iglo)
il = il_idx(g_lo,iglo)
is = is_idx(g_lo,iglo)
it = it_idx(g_lo,iglo)
! count total index for given (ik,il,ie,is) combo (dependent on layout)
iplo = idx(p_lo,ik,il,ie,is)
! check to see if value of g corresponding to iglo is on this processor
if (idx_local(g_lo,iglo)) then
if (idx_local(p_lo,iplo)) then
! if g value corresponding to iplo should be on this processor, then get it
gparity(ig,it,isgn,iplo) = gnew(ig,isgn,iglo)*aj0(ig,iglo)
else
! otherwise, send g for this iplo value to correct processor
call send (gnew(ig,isgn,iglo)*aj0(ig,iglo),proc_id(p_lo,iplo))
end if
else if (idx_local(p_lo,iplo)) then
! if g for this iplo not on processor, receive it
call receive (gparity(ig,it,isgn,iplo),proc_id(g_lo,iglo))
end if
end do
end do
end do
where p_lo
has ky
, l
, e
and s
distributed and kx
, theta
, sgn
local.
- It is a loop over
iglo_world
, but the first if statement skips all work ifiglo
is not on proc. - The
iglo
loop should be outside theig
andisgn
loops. The loop content doesn't seem to have any dependence onig
andisgn
, and both these dimensions are local ing_lo
andp_lo
. At the very least, the send/receives should be done as an array containing allig
,isgn
.
I think this should be rewritten as a redistribute – I think that’s what it’s trying to do here anyway. I think this shouldn’t be too much of an issue as this diagnostic should (hopefully) be off by default and not be called very often. The communication pattern is also likely sub-optimal – it’s sending individual elements at a time and done in a blocking manner (it’s not impossible that this could hang in some cases).