{"action":"created","issue":{"url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/15","repository_url":"https://api.github.com/repos/dfdx/XGrad.jl","labels_url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/15/comments","events_url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/15/events","html_url":"https://github.com/dfdx/XGrad.jl/issues/15","id":317942578,"number":15,"title":"Jacobians","user":{"login":"cortner","id":7688515,"avatar_url":"https://avatars0.githubusercontent.com/u/7688515?v=4","gravatar_id":"","url":"https://api.github.com/users/cortner","html_url":"https://github.com/cortner","followers_url":"https://api.github.com/users/cortner/followers","following_url":"https://api.github.com/users/cortner/following{/other_user}","gists_url":"https://api.github.com/users/cortner/gists{/gist_id}","starred_url":"https://api.github.com/users/cortner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cortner/subscriptions","organizations_url":"https://api.github.com/users/cortner/orgs","repos_url":"https://api.github.com/users/cortner/repos","events_url":"https://api.github.com/users/cortner/events{/privacy}","received_events_url":"https://api.github.com/users/cortner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-26T09:14:53Z","updated_at":"2018-04-29T00:36:52Z","closed_at":null,"author_association":"NONE","body":"Would you consider extending this package to compute jacobians?\r\n\r\nI suppose on some level this can be \"hacked\" by differentiating the individual components, but if there are a lot of intermediate expressions that go into several components of the final `f(x)` vector, then this could really improve performance.\r\n\r\n(My current use-case is differentiation of vector-valued multi-variate polynomials in up to 10, later maybe more, variables.)"},"comment":{"url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/comments/385215720","html_url":"https://github.com/dfdx/XGrad.jl/issues/15#issuecomment-385215720","issue_url":"https://api.github.com/repos/dfdx/XGrad.jl/issues/15","id":385215720,"user":{"login":"dfdx","id":428171,"avatar_url":"https://avatars2.githubusercontent.com/u/428171?v=4","gravatar_id":"","url":"https://api.github.com/users/dfdx","html_url":"https://github.com/dfdx","followers_url":"https://api.github.com/users/dfdx/followers","following_url":"https://api.github.com/users/dfdx/following{/other_user}","gists_url":"https://api.github.com/users/dfdx/gists{/gist_id}","starred_url":"https://api.github.com/users/dfdx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfdx/subscriptions","organizations_url":"https://api.github.com/users/dfdx/orgs","repos_url":"https://api.github.com/users/dfdx/repos","events_url":"https://api.github.com/users/dfdx/events{/privacy}","received_events_url":"https://api.github.com/users/dfdx/received_events","type":"User","site_admin":false},"created_at":"2018-04-29T00:36:52Z","updated_at":"2018-04-29T00:36:52Z","author_association":"OWNER","body":"Here's a manual code for calculating \"something like jacobian\":\r\n\r\n```\r\n# example vector-valued expression\r\nbase_ex = quote\r\n    x1 = exp.(W*x)\r\n    x2 = exp.(W*x)\r\n    x3 = exp.(W*x2)\r\nend\r\n\r\n# adding `x3[1]` and `x3[2]` to run normal differentiation pipeline\r\nex1 = copy(base_ex) |> sanitize; push!(ex1.args, :(z = x3[1]))\r\nex2 = copy(base_ex) |> sanitize; push!(ex2.args, :(z = x3[2]))\r\n\r\n# finding derivative expressions for these 2 components\r\nW = rand(3,3); x = rand(3)\r\ninputs = [:W => W, :x => x]\r\nctx = Dict(:codegen => VectorCodeGen())\r\ndex1 = xdiff(ex1; ctx=ctx, inputs...)\r\ndex2 = xdiff(ex2; ctx=ctx, inputs...)\r\n\r\n# joining diff expressions for the 2 components\r\ndex = copy(dex1)\r\nfor subex in dex2.args\r\n    push!(dex.args, subex)\r\nend\r\n# auxiliary variable to return all the results\r\npush!(dex.args, :(result = ($(dex1.args[end].args[1]), $(dex2.args[end].args[2]))))\r\n\r\n# create graph from the joint diff expression and eliminate common subexpressions\r\ng = ExGraph(dex; inputs...)\r\nevaluate!(g)\r\ng = eliminate_common(g)\r\nto_expr(g)\r\n```\r\nwhich generates: \r\n\r\n```\r\nquote  # /home/azhabinski/.julia/v0.6/Espresso/src/exgraph.jl, line 99:\r\n    tmp1342 = 1\r\n    tmp1357 = transpose(W)\r\n    dz!dz = 1.0\r\n    tmp1338 = W * x\r\n    tmp1352 = exp.(tmp1338)\r\n    tmp1348 = transpose(tmp1352)\r\n    tmp1340 = W * tmp1352\r\n    tmp1346 = exp.(tmp1340)\r\n    dz!dx3 = ungetindex(tmp1346, dz!dz, tmp1342)\r\n    dz!dtmp1340 = tmp1346 .* dz!dx3\r\n    dz!dx2 = tmp1357 * dz!dtmp1340\r\n    dz!dtmp1338 = tmp1352 .* dz!dx2\r\n    dz!dx = tmp1357 * dz!dtmp1338\r\n    dz!dW__1 = dz!dtmp1340 * tmp1348\r\n    z = getindex(tmp1346, tmp1342)\r\n    tmp1354 = transpose(x)\r\n    dz!dW__2 = dz!dtmp1338 * tmp1354\r\n    dz!dW = dz!dW__1 .+ dz!dW__2\r\n    tmp1359 = (z, dz!dW, dz!dx)\r\n    tmp1366 = 2\r\n    dz!dx4 = ungetindex(tmp1346, dz!dz, tmp1366)\r\n    dz!dtmp1364 = tmp1346 .* dz!dx4\r\n    dz!dx5 = tmp1357 * dz!dtmp1364\r\n    dz!dW__3 = dz!dtmp1364 * tmp1348\r\n    z2 = getindex(tmp1346, tmp1366)\r\n    dz!dtmp1362 = tmp1352 .* dz!dx5\r\n    dz!dx6 = tmp1357 * dz!dtmp1362\r\n    dz!dW__4 = dz!dtmp1362 * tmp1354\r\n    dz!dW2 = dz!dW__3 .+ dz!dW__4\r\n    tmp1428 = (z2, dz!dW2, dz!dx6)\r\n    result = (tmp1359, tmp1428)\r\nend\r\n```\r\nThe result is a tuple where each element is itself a tuple with usual diff results for each element (2 in this case) - we can concatenate these results to obtain jacobians. \r\n\r\nInteresting findings: \r\n\r\n1. Forward pass (calculating result of the original expression) is the same for all components, so it's successfully eliminated. \r\n2. Reverse pass is mostly unique for each component. \r\n3. Number of components should known in advance. \r\n\r\nSo if forward pass takes `m` seconds and reverse pass takes `n` seconds, constructing jacobians this way for 10 components will result in approximately `m + 10n` seconds (compared to `10m + 10n` in case of naive approach). \r\n\r\nWe may also be able to reuse buffers for variables in reverse pass (not shown here), but I haven't invested much time in memory management strategies yet. "}}
{"actor":{"display_login":"dfdx"}}