{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/14556","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/14556/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/14556/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/14556/events","html_url":"https://github.com/JuliaLang/julia/issues/14556","id":124687516,"number":14556,"title":"Compilation bottleneck","user":{"login":"timholy","id":1525481,"avatar_url":"https://avatars.githubusercontent.com/u/1525481?v=3","gravatar_id":"","url":"https://api.github.com/users/timholy","html_url":"https://github.com/timholy","followers_url":"https://api.github.com/users/timholy/followers","following_url":"https://api.github.com/users/timholy/following{/other_user}","gists_url":"https://api.github.com/users/timholy/gists{/gist_id}","starred_url":"https://api.github.com/users/timholy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timholy/subscriptions","organizations_url":"https://api.github.com/users/timholy/orgs","repos_url":"https://api.github.com/users/timholy/repos","events_url":"https://api.github.com/users/timholy/events{/privacy}","received_events_url":"https://api.github.com/users/timholy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-01-04T02:36:44Z","updated_at":"2016-01-04T15:08:02Z","closed_at":null,"body":"I don't understand the compile toolchain very well, and I haven't yet gotten good profile info (see https://groups.google.com/d/msg/julia-dev/RKAahOwppNs/Kg0TFx_SBwAJ), but at least for the SubArray tests I'm guessing that [this line](https://github.com/JuliaLang/julia/blob/c70ab26bb677c92f0d8e0ae41c3035217a4b111f/src/dump.c#L622), via `jl_compress_ast`, is an enormous bottleneck in compilation. I inserted a debugging line,\r\n```diff\r\ndiff --git a/src/dump.c b/src/dump.c\r\nindex e11bff5..9c3c96f 100644\r\n--- a/src/dump.c\r\n+++ b/src/dump.c\r\n@@ -618,6 +618,7 @@ static int is_ast_node(jl_value_t *v)\r\n \r\n static int literal_val_id(jl_value_t *v)\r\n {\r\n+    jl_printf(JL_STDOUT, \"len = %d\\n\", jl_array_len(tree_literal_values));\r\n     for(int i=0; i < jl_array_len(tree_literal_values); i++) {\r\n         if (jl_egal(jl_cellref(tree_literal_values,i), v))\r\n             return i;\r\n```\r\nand saw that most of the lengths were around 10k or so. That's a heck of a lot of calls to `jl_egal`. Is there anything that can be done to reduce this number?"},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/168701315","html_url":"https://github.com/JuliaLang/julia/issues/14556#issuecomment-168701315","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/14556","id":168701315,"user":{"login":"timholy","id":1525481,"avatar_url":"https://avatars.githubusercontent.com/u/1525481?v=3","gravatar_id":"","url":"https://api.github.com/users/timholy","html_url":"https://github.com/timholy","followers_url":"https://api.github.com/users/timholy/followers","following_url":"https://api.github.com/users/timholy/following{/other_user}","gists_url":"https://api.github.com/users/timholy/gists{/gist_id}","starred_url":"https://api.github.com/users/timholy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timholy/subscriptions","organizations_url":"https://api.github.com/users/timholy/orgs","repos_url":"https://api.github.com/users/timholy/repos","events_url":"https://api.github.com/users/timholy/events{/privacy}","received_events_url":"https://api.github.com/users/timholy/received_events","type":"User","site_admin":false},"created_at":"2016-01-04T15:08:02Z","updated_at":"2016-01-04T15:08:02Z","body":"I decided to profile the test suite. I had to exclude a few tests (`profile` because it calls `Profile.clear()`; also `\"cmdlineargs\", \"file\", \"libdl\", \"parallel\"` because they gave problems for reasons I didn't track down). I ran this on release-0.4 (`0.4.3-pre6`), because I don't trust profiling on 0.5 due to #14281. I collected one sample every 10 ms (100samples/sec). \r\n\r\nAnalysis script (whose purpose was to show \"flat\" results that don't suffer from the [duplicates-due-to-recursion problem as illustrated by `dumbsum`](http://docs.julialang.org/en/stable/manual/profile/#basic-usage)) and results are here: https://gist.github.com/timholy/a77b00400037bd9fb708. I only included lines that accounted for at least 100 samples, i.e., at least 1s of time. Scroll to the end to see the worst offenders.\r\n\r\nWhile it's a shame that so many important lines lack good backtrace info (here's hoping that @Keno's work will help), to me it looks as if `jl_egal` is the single worst \"low-level\" offender, accounting for more than 8% of the total number of samples collected. That, combined with the fact that the number of `jl_egal` calls is clearly not as small as seems to have been expected, suggests there's an opportunity for measurable improvement."}}