<p>I'm not exactly an expert on optimization, but: it depends on what you mean by "nondifferentiable".</p>

<p>For many mathematical functions that are used, "nondifferentiable" will just mean "not everywhere differentiable" -- but that's still "differentiable almost everywhere, except on countably many points" (e.g., <code>abs</code>, <code>relu</code>).  These functions are not a problem at all -- you can just chose <a href="https://en.wikipedia.org/wiki/Subgradient_method" rel="nofollow noreferrer">any subgradient</a> and apply any normal gradient method.  That's what basically all AD systems for machine learning do.    The case for non-singular subgradients will happen with low probability anyway. An alternative for certain forms of convex objectives are <a href="https://en.wikipedia.org/wiki/Proximal_gradient_methods_for_learning" rel="nofollow noreferrer">proximal gradient methods</a>, which "smooth" the objective in an efficient way that preserves optima (cf. <a href="https://github.com/kul-forbes/ProximalOperators.jl" rel="nofollow noreferrer">ProximalOperators.jl</a>).</p>

<p>Then there's those functions that seem like they can't be differentiated at all, since they seem "combinatoric" or discrete, but are in fact piecewise differentiable (if seen from the correct point of view).  This includes <a href="https://arxiv.org/abs/2002.08871" rel="nofollow noreferrer">sorting and ranking</a>.  But you have to find them, and describing and implementing the derivative is rather complicated.  Whether such functions are supported by an AD system depends on how sophisticated its "standard library" is.  Some variants of this, like "permute", can just fall out AD over control structures, while move complex ones require the primitive adjoints to be manually defined.</p>

<p>For certain kinds of problems, though, we just work in an intrinsically discrete space -- like, integer parameters of some probability distributions.  In these case, differentiation makes no sense, and hence AD libraries define their primitives not to work on these parameters.  Possible alternatives are to use (mixed) integer programming, approximations, search, and model selection.  This case also occurs for problems where the optimized space itself depends on the parameter in question, like the second argument of <code>fill</code>.  We also have things like the <a href="https://en.wikipedia.org/wiki/Lp_space#When_p_=_0" rel="nofollow noreferrer">ℓ0 "norm"</a> or the rank of a matrix, for which there exist well-known continuous relaxations, but that's outside of the scope of AD).</p>

<p>(In the specific case of MCMC for discrete or dimensional parameters, there's other ways to deal with that, like combining HMC with other MC methods in a Gibbs sampler, or using a nonparametric model instead.  Other tricks <a href="http://arxiv.org/abs/2003.00704" rel="nofollow noreferrer">are possible for VI</a>.)</p>

<p>That being said, you will rarely encounter complicated <a href="https://en.wikipedia.org/wiki/Weierstrass_function" rel="nofollow noreferrer">nowhere differentiable continuous</a> functions in optimization.  They are already complicated to describe, are just unlikely to arise in the kind of math we use for modelling.  </p>
Can I use automatic differentiation for non-differentiable functions?
<p>I am testing performance of different solvers on minimizing an objective function derived from simulated method of moments. Given that my objective function is not differentiable, I wonder if automatic differentiation would work in this case? I tried my best to read some introduction on this method, but I couldn't figure it out.</p>

<p>I am actually trying to use Ipopt+JuMP in Julia for this test. Previously, I have tested it using BlackBoxoptim in Julia. I will also appreciate if you could provide some insights on optimization of non-differentiable functions in Julia.</p>

<hr>

<p>It seems that I am not clear on "non-differentiable". Let me give you an example. Consider the following <a href="https://i.stack.imgur.com/UzMDf.png" rel="nofollow noreferrer">objective function</a>. X is dataset, B is unobserved random errors which will be integrated out, \theta is parameters. However, A is discrete and therefore not differentiable.</p>