Loop ordering in diagnostic_fluxes.fpp

Issue #79 new
Joseph Parker created an issue

I’m not familiar with this part of the code, but the loop at line 626 of diagnostic fluxes looks dubious:

    ! below we define gparity = J0 * h, where delta f = h - (q phi / T) F0
    ! because we're ultimately interested in int J0 h * phi (i.e. particle flux)
    do isgn = 1, 2
       do ig = -ntgrid, ntgrid
          do iglo = g_lo%llim_world, g_lo%ulim_world
             ik = ik_idx(g_lo,iglo)
             ie = ie_idx(g_lo,iglo)
             il = il_idx(g_lo,iglo)
             is = is_idx(g_lo,iglo)
             it = it_idx(g_lo,iglo)
             ! count total index for given (ik,il,ie,is) combo (dependent on layout)
             iplo = idx(p_lo,ik,il,ie,is)
             ! check to see if value of g corresponding to iglo is on this processor
             if (idx_local(g_lo,iglo)) then
                if (idx_local(p_lo,iplo)) then
                   ! if g value corresponding to iplo should be on this processor, then get it     
                   gparity(ig,it,isgn,iplo) = gnew(ig,isgn,iglo)*aj0(ig,iglo)
                else
                   ! otherwise, send g for this iplo value to correct processor
                   call send (gnew(ig,isgn,iglo)*aj0(ig,iglo),proc_id(p_lo,iplo))
                end if
             else if (idx_local(p_lo,iplo)) then
                ! if g for this iplo not on processor, receive it
                call receive (gparity(ig,it,isgn,iplo),proc_id(g_lo,iglo))
             end if
          end do
       end do
    end do

where p_lo has ky, l, e and s distributed and kx, theta, sgn local.

  • It is a loop over iglo_world, but the first if statement skips all work if iglo is not on proc.
  • The iglo loop should be outside the ig and isgn loops. The loop content doesn't seem to have any dependence on ig and isgn, and both these dimensions are local in g_lo and p_lo. At the very least, the send/receives should be done as an array containing all ig, isgn.

Comments (1)

  1. David Dickinson

    I think this should be rewritten as a redistribute – I think that’s what it’s trying to do here anyway. I think this shouldn’t be too much of an issue as this diagnostic should (hopefully) be off by default and not be called very often. The communication pattern is also likely sub-optimal – it’s sending individual elements at a time and done in a blocking manner (it’s not impossible that this could hang in some cases).

  2. Log in to comment