<p dir="auto">I discovered this issue while using Flux/Zygote. I am training a big and complex model with Julia distributed pmap. It used to be fine under Flux/Tracker but after moving to Zygote, the memory consumption seems to grow overtime. GC.gc() does not work, if one observes memory usage using <code class="notranslate">top</code> in linux or activity monitor in osx, it seems to grow until the program terminates due to out-of-memory by the OS.</p>
<p dir="auto">It looks like pmap doesn't free the memory generated through the use of Julia's metaprogramming features.</p>
<p dir="auto">The machines I have experimented on:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="versioninfo()

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)"><pre class="notranslate"><code class="notranslate">versioninfo()

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

Julia Version 1.5.1
Commit 697e782ab8 (2020-08-25 20:08 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, broadwell)
</code></pre></div>
<p dir="auto"><strong>Minimum working example</strong></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="using Distributed
addprocs(4)

@everywhere mutable struct A
    a::Float32
end

@everywhere function genprog(n, p::A)
    map(1:n) do i
        y = rand()
        mdname = gensym()
        expr = :(module $mdname
            f(x) = 2*x + $y + $p.a
            end
        )
        m = eval(expr)
        Base.invokelatest(m.f, p.a)
    end
end

function main()
    i = 0
    x = A(rand())
    while true
        println(&quot;epoch $(i)&quot;)
        @everywhere GC.gc()
        
        tasks = rand(1:100, 100)
        _, timeTaken, _, _, _ = @timed let x=x
            pmap(tasks) do n
                genprog(n, x)
            end
        end
        @show timeTaken
        x.a = rand()
        i += 1
    end
end

main()"><pre class="notranslate"><code class="notranslate">using Distributed
addprocs(4)

@everywhere mutable struct A
    a::Float32
end

@everywhere function genprog(n, p::A)
    map(1:n) do i
        y = rand()
        mdname = gensym()
        expr = :(module $mdname
            f(x) = 2*x + $y + $p.a
            end
        )
        m = eval(expr)
        Base.invokelatest(m.f, p.a)
    end
end

function main()
    i = 0
    x = A(rand())
    while true
        println("epoch $(i)")
        @everywhere GC.gc()
        
        tasks = rand(1:100, 100)
        _, timeTaken, _, _, _ = @timed let x=x
            pmap(tasks) do n
                genprog(n, x)
            end
        end
        @show timeTaken
        x.a = rand()
        i += 1
    end
end

main()
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/MikeInnes/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/MikeInnes">@MikeInnes</a> , trying to get your attention since you are the author on Zygote/src/compiler/*.jl files</p>
<p dir="auto">References</p>
<ol dir="auto">
<li>probably related issue <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="124051061" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/14495" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/14495/hovercard" href="https://github.com/JuliaLang/julia/issues/14495">#14495</a></li>
<li><a href="https://discourse.julialang.org/t/dynamically-create-and-destoy-modules-to-avoid-memory-leaks-in-program-synthesis/39729" rel="nofollow">https://discourse.julialang.org/t/dynamically-create-and-destoy-modules-to-avoid-memory-leaks-in-program-synthesis/39729</a></li>
<li><a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="397052933" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/30653" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/30653/hovercard" href="https://github.com/JuliaLang/julia/issues/30653">#30653</a></li>
</ol>
<p dir="auto">Hello guys!</p>
<p dir="auto">I found a generated code related memory leak issue. I wrote a <a href="https://groups.google.com/forum/#!topic/julia-users/J62di4skPOU" rel="nofollow">post</a> about it on julia-lang users group, but it looks like everyone have no time for that. I emailed to <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/JeffBezanson/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/JeffBezanson">@JeffBezanson</a>  about this issue and he knows about it. I have no possibility to fix this by myself, because i have no such experience in C. So, i decided to create this issue to track it somehow.</p>
<p dir="auto">So, here is a problem:</p>
<div class="highlight highlight-source-julia notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="function leak()
    for i=1:100000
        t = Task(eval(:(function() produce() end)))
        consume(t)
        try
          Base.throwto(t, null)
        end
    end
    gc()
end"><pre class="notranslate"><span class="pl-k">function</span> <span class="pl-en">leak</span>()
    <span class="pl-k">for</span> i<span class="pl-k">=</span><span class="pl-c1">1</span><span class="pl-k">:</span><span class="pl-c1">100000</span>
        t <span class="pl-k">=</span> <span class="pl-c1">Task</span>(<span class="pl-c1">eval</span>(:(<span class="pl-k">function</span>() <span class="pl-c1">produce</span>() <span class="pl-k">end</span>)))
        <span class="pl-c1">consume</span>(t)
        <span class="pl-k">try</span>
          Base<span class="pl-k">.</span><span class="pl-c1">throwto</span>(t, null)
        <span class="pl-k">end</span>
    <span class="pl-k">end</span>
    <span class="pl-c1">gc</span>()
<span class="pl-k">end</span></pre></div>
<p dir="auto">Every call of <code class="notranslate">leak()</code> eats ~30mb of memory on my PC. This problem exists on both Linux and Windows platforms.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="julia&gt; versioninfo()
Julia Version 0.4.2
Commit bb73f34 (2015-12-06 21:47 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3"><pre class="notranslate"><code class="notranslate">julia&gt; versioninfo()
Julia Version 0.4.2
Commit bb73f34 (2015-12-06 21:47 UTC)
Platform Info:
  System: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz
  WORD_SIZE: 64
  BLAS: libopenblas (NO_LAPACK NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: liblapack.so.3
  LIBM: libopenlibm
  LLVM: libLLVM-3.3
</code></pre></div>