{"action":"created","issue":{"url":"https://api.github.com/repos/ceylon/ceylon/issues/5968","repository_url":"https://api.github.com/repos/ceylon/ceylon","labels_url":"https://api.github.com/repos/ceylon/ceylon/issues/5968/labels{/name}","comments_url":"https://api.github.com/repos/ceylon/ceylon/issues/5968/comments","events_url":"https://api.github.com/repos/ceylon/ceylon/issues/5968/events","html_url":"https://github.com/ceylon/ceylon/issues/5968","id":130554921,"number":5968,"title":"Dependency resolution does not work correctly for Maven dependencies","user":{"login":"renatoathaydes","id":1901277,"avatar_url":"https://avatars.githubusercontent.com/u/1901277?v=3","gravatar_id":"","url":"https://api.github.com/users/renatoathaydes","html_url":"https://github.com/renatoathaydes","followers_url":"https://api.github.com/users/renatoathaydes/followers","following_url":"https://api.github.com/users/renatoathaydes/following{/other_user}","gists_url":"https://api.github.com/users/renatoathaydes/gists{/gist_id}","starred_url":"https://api.github.com/users/renatoathaydes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/renatoathaydes/subscriptions","organizations_url":"https://api.github.com/users/renatoathaydes/orgs","repos_url":"https://api.github.com/users/renatoathaydes/repos","events_url":"https://api.github.com/users/renatoathaydes/events{/privacy}","received_events_url":"https://api.github.com/users/renatoathaydes/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ceylon/ceylon/labels/c-module%20system","name":"c-module system","color":"02d7e1"},{"url":"https://api.github.com/repos/ceylon/ceylon/labels/m-compiler-java","name":"m-compiler-java","color":"00aa00"},{"url":"https://api.github.com/repos/ceylon/ceylon/labels/t-bug","name":"t-bug","color":"e10c02"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/ceylon/ceylon/milestones/14","html_url":"https://github.com/ceylon/ceylon/milestones/1.2.2","labels_url":"https://api.github.com/repos/ceylon/ceylon/milestones/14/labels","id":1497228,"number":14,"title":"1.2.2","description":"","creator":{"login":"FroMage","id":179265,"avatar_url":"https://avatars.githubusercontent.com/u/179265?v=3","gravatar_id":"","url":"https://api.github.com/users/FroMage","html_url":"https://github.com/FroMage","followers_url":"https://api.github.com/users/FroMage/followers","following_url":"https://api.github.com/users/FroMage/following{/other_user}","gists_url":"https://api.github.com/users/FroMage/gists{/gist_id}","starred_url":"https://api.github.com/users/FroMage/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/FroMage/subscriptions","organizations_url":"https://api.github.com/users/FroMage/orgs","repos_url":"https://api.github.com/users/FroMage/repos","events_url":"https://api.github.com/users/FroMage/events{/privacy}","received_events_url":"https://api.github.com/users/FroMage/received_events","type":"User","site_admin":false},"open_issues":129,"closed_issues":5,"state":"open","created_at":"2016-01-11T13:38:44Z","updated_at":"2016-02-03T17:05:57Z","due_on":null,"closed_at":null},"comments":10,"created_at":"2016-02-02T02:52:28Z","updated_at":"2016-02-04T15:04:37Z","closed_at":null,"body":"Using Maven dependencies with Ceylon has been incredibly difficult. I wrote the Gradle plugin to help do that, and it works great on the surface... but there are several issues that make it not very reliable in the current state... which is what I am trying to fix.\r\n\r\nFirst of all, looks like Ceylon does a good job of downloading dependencies. However, it just doesn't seem to use the information about transitive dependencies when compiling at all.\r\n\r\nFor example, here's a very simple example of a module that tries to use log4j:\r\n\r\n```ceylon\r\nnative(\"jvm\")\r\nmodule com.athaydes.maven \"1.0.0\" {\r\n\timport java.base \"8\";\r\n\timport \"org.apache.logging.log4j:log4j-core\" \"2.4.1\";\r\n}\r\n```\r\n\r\nIt declares a direct dependency on `log4j-core` as it would do with Maven or Gradle... then, uses `log4j-core` in some simple code:\r\n\r\n```ceylon\r\nimport org.apache.logging.log4j {\r\n\tLogger,\r\n\tLogManager\r\n}\r\nimport java.lang {\r\n\tThread\r\n}\r\n\r\nshared class MyTest() {}\r\n\r\nLogger logger = LogManager.getLogger(`class MyTest`);\r\n\r\n\"Run the module `com.athaydes.maven`.\"\r\nshared void run() {\r\n\tlogger.info(\"Hello Maven!\");\r\n}\r\n```\r\n\r\nMaven and Gradle would download the `log4j-core`, put it into the classpath for you along with the transitive dependencies, in this case just `log4j-api`, and everything would work.\r\n\r\nWith Ceylon, it doesn't work out-of-the-box... you didn't say you have a dependency on `log4j-api`, so the Ceylon compiler doesn't seem to see classes in it... you get lots of errors that look like this:\r\n\r\n```\r\nsource/com/athaydes/maven/run.ceylon:1: error: package not found in imported modules: 'org.apache.logging.log4j' (add module import to module descriptor of 'com.athaydes.maven')\r\nimport org.apache.logging.log4j {\r\n       ^\r\nsource/com/athaydes/maven/run.ceylon:11: error: type declaration does not exist: 'Logger'\r\n```\r\n\r\n`Logger` is in the `log4j-api` jar... before we fix that, let's look at what the compiler says in verbose mode (just a small part of it, which is quite interesting):\r\n\r\n```\r\nDebug: Repository http://repo1.maven.org/maven2 have been made available for artifact resolution \r\nDebug: Resolving artifact org.apache.logging.log4j:log4j-core:pom:2.4.1 \r\nDebug: Resolved artifact org.apache.logging.log4j:log4j-core:pom:2.4.1 from central (https://nexus.2botech.com:10123/nexus/content/repositories/central/, releases+snapshots) \r\nDebug: Resolving artifact org.apache.logging.log4j:log4j:pom:2.4.1 \r\nDebug: Resolved artifact org.apache.logging.log4j:log4j:pom:2.4.1 from /Users/renato/.m2/repository (enhanced) \r\nDebug: Resolving artifact org.apache:apache:pom:9 \r\nDebug: Resolved artifact org.apache:apache:pom:9 from /Users/renato/.m2/repository (enhanced) \r\nDebug: Resolving artifact org.apache.logging.log4j:log4j-api:pom:2.4.1 \r\nDebug: Resolved artifact org.apache.logging.log4j:log4j-api:pom:2.4.1 from /Users/renato/.m2/repository (enhanced) \r\nDebug: Resolving artifact org.apache.logging.log4j:log4j:pom:2.4.1 \r\nDebug: Resolved artifact org.apache.logging.log4j:log4j:pom:2.4.1 from /Users/renato/.m2/repository (enhanced) \r\nDebug: Resolving artifact org.apache:apache:pom:9 \r\nDebug: Resolved artifact org.apache:apache:pom:9 from /Users/renato/.m2/repository (enhanced) \r\nDebug: Resolving artifact org.osgi:org.osgi.core:pom:4.3.1 \r\nDebug: Resolved artifact org.osgi:org.osgi.core:pom:4.3.1 from central (https://nexus.2botech.com:10123/nexus/content/repositories/central/, releases+snapshots) \r\nDebug: Dependency collection stats: {ConflictMarker.analyzeTime=0, ConflictMarker.markTime=1, ConflictMarker.nodeCount=5, ConflictIdSorter.graphTime=0, ConflictIdSorter.topsortTime=1, ConflictIdSorter.conflictIdCount=3, ConflictIdSorter.conflictIdCycleCount=0, ConflictResolver.totalTime=2, ConflictResolver.conflictItemCount=4, DefaultDependencyCollector.collectTime=80, DefaultDependencyCollector.transformTime=5} \r\n```\r\n\r\nSo, wait a minute... not only can it see that `log4j-core` depends on `log4j-api`, it can also see a dependency on [`osgi` that is marked as `provided`](https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-core/2.5/log4j-core-2.5.pom) in the log4j-core pom... which means that if I try to use an offline repository that does not contain `osgi` in it, it fails (yes, I tried).\r\n\r\nBut if the Ceylon compiler, or perhaps the JBoss module system, knows the transitive dependencies of `log4j-core` and indeed goes get them,  why does it not automatically make them available in the classpath for my module?\r\n\r\nFixing this issue, for this particular dependency on `log4j-core` is a matter of declaring the its reliance on `log4j-api` either in the module file or in an `overrides` file, which is what the Gradle plugin does automatically (generates the following file):\r\n\r\n```xml\r\n<overrides>\r\n  <artifact groupId='org.apache.logging.log4j' artifactId='log4j-core' version='2.4.1'>\r\n    <add groupId='org.apache.logging.log4j' artifactId='log4j-api' version='2.4.1' shared='true' />\r\n  </artifact>\r\n</overrides>\r\n```\r\n\r\nNow it works when we try to compile.\r\n\r\nHowever, even after doing this, when I try to use Ceylon in offline mode and give it an `eather` local repository where I have both `log4j-core` and `log4j-api` installed, Ceylon will fail to compile my module.\r\n\r\nHere's the full command:\r\n\r\n```\r\nceylon compile --offline \r\n--overrides auto-generated/overrides.xml\r\n--rep=aether:auto-generated/settings.xml \r\n--no-default-repositories com.athaydes.maven\r\n```\r\n\r\nHere's the error message:\r\n\r\n```\r\nsource/com/athaydes/maven/module.ceylon:4: error: cannot find module artifact org.apache.logging.log4j:log4j-core-2.4.1(.car|.jar)\r\n        import \"org.apache.logging.log4j:log4j-core\" \"2.4.1\";\r\n        ^\r\n        - dependency tree: com.athaydes.maven/1.0.0 -> org.apache.logging.log4j:log4j-core/2.4.1\r\nNote: Created module com.athaydes.maven/1.0.0\r\nceylon compile: There was 1 error\r\n```\r\n\r\nEven though the error message tells us that Ceylon can't find `log4j-core`, in fact, what it can't find is `osgi` and other things referenced from the poms that could be safely ignored...  it is a `provided` dependency (probably should be marked as optional also, but let's ignore that for the moment).\r\n\r\nHow do I know? Run the above without `--offline` (will succeed because Ceylon will take stuff from remote repos to build) and with `--verbose`:\r\n\r\n```\r\nDebug: Resolved artifact org.osgi:org.osgi.core:pom:4.3.1 from central (http://repo1.maven.org/maven2, releases+snapshots) \r\n```\r\n\r\nAnd lots more like this...\r\n\r\nLooking through my local repo which only had `log4j-core` and `log4j-api` in it before, I can see that Ceylon seems to have installed in it other things... like `osgi-core` and `apache-9` whatever that is!? It also installed the parent `log4j` pom...\r\n\r\nNow, running `compile` again with the `--offline` switch, it also succeeds!\r\n\r\nSo, it seems Ceylon tries to do the whole Maven pom resolution to find out every single dependency of my module, but when it comes to actually compiling, I need to manually add the mandatory (at least) transitive dependencies by hand... which is pretty crazy if you ask me....\r\n\r\nThe Gradle plugin was meant to do all the downloading and version resolution (where applicable) not only from Maven repositories, but from Ivy, local files etc. but because Ceylon tries to do it all itself, even when I try to ask it not to (using the switches mentioned above, as well as `no-default-repositories` etc)... notice that the only runtime dependencies are the 2 jars, in this case: `log4j-api` and `log4j-core`, both of which I had installed in my repository locally... so Ceylon didn't need to do any dependency resolution at all after that.\r\n\r\nI don't know what solution would be best... I suspect it would be to be able to completely turn off ceylon's dependency resolution... but I can see how that would be an issue to resolve modules packages properly (which, btw, is impossible when you have these terrible Maven optional dependencies, as the jars don't tell which package are optional)... anything that makes it working with transitive dependencies as easy as with Groovy/Java/Kotlin, at least when you use `--flat-classpath` and module isolation at runtime is a non-issue...\r\n\r\nAs you can see, it takes a huge amount of time just to explain what's wrong... I hope this explanation is enough for you guys to sort something out... for now, I can see no good solution to make Java interop with several dependencies work in Ceylon... will need your help to make that at least tolerable..."},"comment":{"url":"https://api.github.com/repos/ceylon/ceylon/issues/comments/179886728","html_url":"https://github.com/ceylon/ceylon/issues/5968#issuecomment-179886728","issue_url":"https://api.github.com/repos/ceylon/ceylon/issues/5968","id":179886728,"user":{"login":"renatoathaydes","id":1901277,"avatar_url":"https://avatars.githubusercontent.com/u/1901277?v=3","gravatar_id":"","url":"https://api.github.com/users/renatoathaydes","html_url":"https://github.com/renatoathaydes","followers_url":"https://api.github.com/users/renatoathaydes/followers","following_url":"https://api.github.com/users/renatoathaydes/following{/other_user}","gists_url":"https://api.github.com/users/renatoathaydes/gists{/gist_id}","starred_url":"https://api.github.com/users/renatoathaydes/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/renatoathaydes/subscriptions","organizations_url":"https://api.github.com/users/renatoathaydes/orgs","repos_url":"https://api.github.com/users/renatoathaydes/repos","events_url":"https://api.github.com/users/renatoathaydes/events{/privacy}","received_events_url":"https://api.github.com/users/renatoathaydes/received_events","type":"User","site_admin":false},"created_at":"2016-02-04T15:04:37Z","updated_at":"2016-02-04T15:04:37Z","body":"I think this flag would be useful... but I 've gotten around that by just using `--offline` and generating my own poms based on Gradle's resolutions, so no optional dependencies are visible to Ceylon's tools.\r\n\r\nThe option to load extra modules is probably what Maven/Gradle call `runtime` scoped dependency. Which is very useful when you have automatically found services in a jar... to provide security algorithms, for example, or just the implementation to be used by `slf4j` (Which you don't want to be visible at compile time, but which should be visible by the framework discovering the implementation from the classpath at runtime). Yes, this would be really really useful."}}