title: Reference Argument Passing with Nested OpenACC Routines
body: <p>I'm attempting to parallelize some Fortran 90 code using OpenACC, where a parallelized loop calls a sequential routine. When I attempt to run the code using the PGI Fortran compiler (2020.4), I obtain an error message saying that reference argument passing prevents parallelization.</p>
  <p>My understanding is that this is likely because one routine exists on the Host while the other is on the Device, but I'm unclear on where I might be missing a pragma that would lead to this outcome.</p>
  <p>The basic structure of the calling routine is:</p>
  <pre><code>subroutine OuterRoutine(F,G,X,Y) 
  
        real(wp), dimension(:,:), intent(IN) :: X      
        real(wp), dimension(:,:), intent(IN) :: Y
        real(wp), dimension(1,PT), intent(OUT) :: F
        real(wp), dimension(N_p,PT), intent(OUT) :: G
          
        ! Local Variables
        integer :: t, i, j
  
        !$acc data copyin(X,Y), copyout(F,G)
  
        !$acc parallel loop
        do t = 1,PT,1
              
            !$acc loop collapse(2) reduction(+:intr)
            do i = 1,N_int-1,1
              do j = 1,N_int-1,1
                G(i,j) = intgrdJ2(X(i,j),X(j,i),Y(i,j),Y(j,i),t)
              end do
            end do
            !$acc end loop
  
        !$acc end parallel loop
  
        !$acc end data
  
  end subroutine OuterRoutine 
  </code></pre>
  <p>And the function being called is:</p>
  <pre><code>function intgrdJ2(z,mu,p,q,t)
      !$acc routine seq
      
      real(wp), intent(IN) :: z, mu, p, q
      integer, intent(IN) :: t
      real(wp) :: intgrdJ2
          
      ! Local Variables
      real(wp) :: mu2
      real(wp), dimension(N_p) :: nu_m2, psi_m2
      integer :: i
          
      mu2 = (mu*fh_pdf(z,mu,p))/f_pdf(z,mu,p)
          
      do i = 1,N_p,1
          nu_m2(i) = interpValue(mu2,mugrid,nu_knots(:,i,t))
          psi_m2(i) = interpValue(mu2,mugrid,psi_knots(:,i,t))
      end do
      
      intgrdJ2 = nu_m2(i)*psi_m2(i)
      
  end function intgrdJ2
  </code></pre>
  <p>The routines <em>interpValue</em>, <em>fh_pdf</em>, and <em>f_pdf</em> are all contained in a used module, and denoted as <em>!$acc routine seq</em>. The variables <em>mugrid</em>, <em>nu_knots</em>, and <em>psi_knots</em> are all module-level variables, which are copied-in to the Device prior to calling <em>OuterRoutine</em>.</p>
  <p>When I run the code, I get this sort of output from the compiler:</p>
  <pre><code>intgrdj2:
      576, Generating acc routine seq
           Generating Tesla code
      593, Reference argument passing prevents parallelization: mu2
  </code></pre>
  <p>Where 593 refers to the &quot;nu_m2(i) = ...&quot; line.</p>
  <p>My understanding is that since the variable <em>mu2</em> is a scalar declared inside of the sequential routine, each thread should have it's own copy of the variable, and I don't need to explicitly declare it to be private when I declare the data region. From reading <a href="https://stackoverflow.com/questions/64112821/routines-inside-acc-parallel-region">this post</a> it seems that the problem may be related to where the routines are located (Host vs Device). However, it seems as though all of the relevant pieces should be on the device because I'm specifying that routines are sequential.</p>
  <p>As a first-time OpenACC user, any explanations about what I might be overlooking would be greatly appreciated!</p>