<p dir="auto">This is related to the discussion in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="476496324" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/Pkg.jl/issues/1285" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/Pkg.jl/issues/1285/hovercard" href="https://github.com/JuliaLang/Pkg.jl/issues/1285">JuliaLang/Pkg.jl#1285</a> but the plan I propose doesn't touch Pkg at all, so I'm writing it up here. <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/Wimmerer/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/Wimmerer">@Wimmerer</a> contacted me about wanting to work on this feature and we had a back and forth about what needs to be done. Here's what I propose as a concrete solution to the glue code problem, aka conditional dependencies.</p>
<p dir="auto">The problem we want to address occurs when there are packages, say <code class="notranslate">A</code> and <code class="notranslate">B</code>, which don't depend on each other, but when both are loaded at the same time, there is are some additional definitions—usually of methods—that are needed to make them work well together. For example, suppose <code class="notranslate">A</code> provides a function <code class="notranslate">A.f</code> and <code class="notranslate">B</code> defines a type <code class="notranslate">B.T</code> and there's a method <code class="notranslate">A.f(::B.T)</code> that would be useful to have. This method cannot be defined in <code class="notranslate">A</code> since <code class="notranslate">B.T</code> isn't available there and it can't be defined in <code class="notranslate">B</code> since <code class="notranslate">A.f</code> isn't available there. We need a way to provide that definition at the point when both <code class="notranslate">A</code> and <code class="notranslate">B</code> have been loaded.</p>
<p dir="auto">The solution comes in two parts.</p>
<h3 dir="auto">Part 1: post-load hooks for sets of packages</h3>
<p dir="auto">The first part is to implement and expose a mechanism whereby you can register a callback to be called when a set of packages have been loaded. The packages should be identified by UUID and it should be an arbitrary set of packages, not just one or two packages. This logic goes right after a new package has been loaded and should check for each registered hook, whether the set of loaded package UUIDs is a superset of the set of required package UUIDs, and if it is, then call the hook callback and then delete it.</p>
<h3 dir="auto">Part 2: <code class="notranslate">glue/*.jl</code> entry-points</h3>
<p dir="auto">The second part leverages the first mechanism and uses it to load some glue code that is included in packages that are loaded. The idea is that package <code class="notranslate">A</code> will have a top-level directory called <code class="notranslate">glue</code> and a file <code class="notranslate">glue/B.jl</code> which contains the code that implements the methods that are useful when both <code class="notranslate">A</code> and <code class="notranslate">B</code> are loaded. In more detail:</p>
<ul dir="auto">
<li>When Julia loads a new package, say <code class="notranslate">A</code>, it should look at all <code class="notranslate">glue/*.jl</code> files</li>
<li>For each one, split the name on commas—those are the names of the glue dependencies
<ul dir="auto">
<li>e.g. <code class="notranslate">glue/B.jl</code> in the package <code class="notranslate">A</code> would be the file providing glue definitions for <code class="notranslate">A</code> and <code class="notranslate">B</code></li>
<li>If any part of a name isn't a valid package name, ignore the file and continue</li>
</ul>
</li>
<li>To resolve each glue dependency name to a package UUID, look it up in a new section of <code class="notranslate">A</code>'s project file entitled <code class="notranslate">[glue]</code> with the same format as the <code class="notranslate">[deps]</code> and <code class="notranslate">[extras]</code> sections, i.e. mapping names to UUIDs
<ul dir="auto">
<li>If a name doesn't exist there, error or maybe ignore?</li>
</ul>
</li>
<li>Use the contents of the glue file to define a hook that will trigger when the set of glue dependencies have all been loaded, using the mechanism created in part 1.</li>
</ul>
<p dir="auto">So, <code class="notranslate">glue/B.jl</code> in the package <code class="notranslate">A</code> would be the glue for <code class="notranslate">A</code> and <code class="notranslate">B</code> and <code class="notranslate">glue/B,C.jl</code> would be glue for the packages <code class="notranslate">B</code> and <code class="notranslate">C</code> together, i.e. definitions that depend on all three of <code class="notranslate">A</code>, <code class="notranslate">B</code> and <code class="notranslate">C</code>.</p>
<p dir="auto">The next question is how to use the glue file to define a hook. We want the behavior of the glue files to be fairly constrained, so I propose that the hook generated look something like this:</p>
<div class="highlight highlight-source-julia notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="()-&gt;@eval Module() begin
    import A, B
    include($(abspath(glue_file)))
end"><pre class="notranslate">()<span class="pl-k">-&gt;</span><span class="pl-c1">@eval</span> <span class="pl-c1">Module</span>() <span class="pl-k">begin</span>
    <span class="pl-k">import</span> A, B
    <span class="pl-c1">include</span>(<span class="pl-k">$</span>(<span class="pl-c1">abspath</span>(glue_file)))
<span class="pl-k">end</span></pre></div>
<p dir="auto">This evaluates the glue code in an anonymous module that imports <code class="notranslate">A</code> and <code class="notranslate">B</code>. I'm not sure how easy it would be to rig it up so that this is done in a context where only <code class="notranslate">A</code> and <code class="notranslate">B</code> can be loaded and if that's really necessary or desirable, but otherwise the code can load anything that can be loaded in Main, which probably isn't ideal or sensible for glue hooks.</p>
<p dir="auto">This hook could probably be expressed generically and be written as <code class="notranslate">glue_hook(glue_path, A, B)</code> or something like that, which might avoid code some code generation. Or in the other direction, we could insert the contents of the glue file into the above template and generate that function. But I suspect that's not what we want. Probably better to be late binding and do less code generation.</p>
<p dir="auto">The actual contents of a glue file would be pretty simple. For example, the glue file for defining the method <code class="notranslate">A.f(::B.T)</code> would simply contain that method definition:</p>
<div class="highlight highlight-source-julia notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="A.f(::B.T) = definition"><pre class="notranslate">A<span class="pl-k">.</span><span class="pl-en">f</span>(<span class="pl-k">::</span><span class="pl-c1">B.T</span>) <span class="pl-k">=</span> definition</pre></div>
<p dir="auto">This would either be <code class="notranslate">glue/B.jl</code> in package A or <code class="notranslate">glue/A.jl</code> package B.</p>
<h3 dir="auto">Questions</h3>
<p dir="auto">What happens if <code class="notranslate">glue/B.jl</code> in package A and <code class="notranslate">glue/A.jl</code> in package B both exist? Load them both.</p>
<p dir="auto">What about the order? Eh, whatever order they happen in is probably fine, but we could maybe have a defined ordering based on UUIDs or something.</p>
<p dir="auto">What if they have conflicting definitions and it breaks? That's a normal package incompatibility between those versions of <code class="notranslate">A</code> and <code class="notranslate">B</code>, although it's a slightly weird one because neither <code class="notranslate">A</code> nor <code class="notranslate">B</code> depends on the other but their versions are incompatible. I'm not certain if we can express that in <code class="notranslate">[compat]</code>—it's possible that we can't. At the very least, <code class="notranslate">[compat]</code> probably needs to know about the <code class="notranslate">[glue]</code> section, which may end up being a reason to put glue dependencies in the <code class="notranslate">[extras]</code> section since I think <code class="notranslate">[compat]</code> may already know about that.</p>
<p dir="auto">Why not make glue packages real registered external packages that live outside of both <code class="notranslate">A</code> and <code class="notranslate">B</code>? Because that complicates things massively. In that design, Pkg needs to know about glue packages and needs to makes sure that whenever <code class="notranslate">A</code> and <code class="notranslate">B</code> are both in a manifest, then <code class="notranslate">Glue_A_B</code> is also included in the manifest. Also, registering glue packages and versioning them separately seems like a lot of overhead.</p>
<p dir="auto">What if the glue behavior depends on what features the particular version of <code class="notranslate">B</code> that gets loaded has? The glue code can do whatever metaprogramming it needs to, reflecting on <code class="notranslate">B</code>, it’s version, and features.</p>
<p dir="auto"><strong>Edit: couple of added questions</strong></p>
<p dir="auto">Why does the glue code go in <code class="notranslate">A/glue/</code> instead of somewhere in <code class="notranslate">A/src/</code>—isn't it part of the <code class="notranslate">A</code> package? No, not really. If you just load <code class="notranslate">A</code> then you load what's in <code class="notranslate">A/src</code> and you don't load anything from <code class="notranslate">A/glue</code> at all. The glue stuff is really external lightweight packages that depend on <code class="notranslate">A</code>, which is why it makes sense to put them outside of the main codebase of <code class="notranslate">A</code>.</p>
<p dir="auto">What's really so bad about making glue packages explicit separate packages? For one thing, we don't even have a way of expressing that to Julia's version resolver and it's unclear how we would even do this. The resolver currently understands two things: (1) that a version of a package depends on some other set of packages and (2) that some versions of packages are incompatible with each other. The conditional dependency pattern can't be expressed in terms of these two features—it's a new kind of thing entirely. You need to express that for some subset of pairs of versions of two packages, you need to load yet another package if you've chosen both those versions. What about for other pairs of version of those two packages? Are those incompatible or are they just pairs for which the glue isn't necessary or doesn't work? Both are valid concepts. So now you need to express several things:</p>
<ol dir="auto">
<li>That there are certain n-tuples of packages, which, if all present in a resolver solution, require that yet another package be present in the solution.</li>
<li>Which versions of those packages that reverse dependency applies to. Do you express this as a Cartesian n-product of version specs? There are sets of version tuples that can't be expressed that way. Do you allow a union of Cartesian n-products of version specs? Or allow cutting out n-products in a nested fashion. Note that this is just for specifying which version of the n-tuple of packages the reverse depedency applies to.</li>
<li>You still need to express compatibility. Which versions of each dependency is the glue package compatible with. The good news is that this would just be a normal compatibility constraint that we already know how to express and solve for. The bad news is that this is a different and separate concept from (2): you have to know when you need the glue package at all before you can decide which versions will work.</li>
</ol>
<p dir="auto">This is a lot of complicated new crap to cram into the registry and teach the resolver about. How does this proposal avoid this problem? It locks the glue code to the version of one or both reverse dependencies and then it's just the usual problem of picking those such that they're compatible.</p>
<p dir="auto">currently, some modules (incorrectly and unfortunately) use <code class="notranslate">Pkg.installed</code> to decide whether it should include some additional functionality. I would like to propose the following extension to module initialization that could help simplify this situation of optional dependencies:</p>
<hr>
<h3 dir="auto">Rules:</h3>
<ol dir="auto">
<li>During module parse time, a module can declare a dependency on another module, given by name</li>
<li>Following the closing of a module (without error), and after all <code class="notranslate">__init__</code> functions have run, any modules that declared <code class="notranslate">module B require Main.C</code> will be (re-)loaded.</li>
</ol>
<h3 dir="auto">Example:</h3>
<div class="highlight highlight-source-julia notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="module A
  module B requires Main.C
    # code in here is part of A iff Main.C is defined as a module
    # it's execution time is defined as after the call to A.__init__ and C.__init__
    using Main.C
    import ..A.something
    something(::C.CType) = 2
  end
  # code in here is always part of A
  something(::Any) = 1
end
# Accessing A.B here would throw an UndefRef-like error, with a message that B requires C
module C
  type CType end
end
# Now A.B exists, and A.something() has extra functionality"><pre class="notranslate"><span class="pl-k">module</span> A
  <span class="pl-k">module</span> B requires Main<span class="pl-k">.</span>C
    <span class="pl-c"><span class="pl-c">#</span> code in here is part of A iff Main.C is defined as a module</span>
    <span class="pl-c"><span class="pl-c">#</span> it's execution time is defined as after the call to A.__init__ and C.__init__</span>
    <span class="pl-k">using</span> Main<span class="pl-k">.</span>C
    <span class="pl-k">import</span> <span class="pl-k">..</span>A<span class="pl-k">.</span>something
    <span class="pl-en">something</span>(<span class="pl-k">::</span><span class="pl-c1">C.CType</span>) <span class="pl-k">=</span> <span class="pl-c1">2</span>
  <span class="pl-k">end</span>
  <span class="pl-c"><span class="pl-c">#</span> code in here is always part of A</span>
  <span class="pl-en">something</span>(<span class="pl-k">::</span><span class="pl-c1">Any</span>) <span class="pl-k">=</span> <span class="pl-c1">1</span>
<span class="pl-k">end</span>
<span class="pl-c"><span class="pl-c">#</span> Accessing A.B here would throw an UndefRef-like error, with a message that B requires C</span>
<span class="pl-k">module</span> C
  type CType <span class="pl-k">end</span>
<span class="pl-k">end</span>
<span class="pl-c"><span class="pl-c">#</span> Now A.B exists, and A.something() has extra functionality</span></pre></div>