{"action":"created","issue":{"url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/89","repository_url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner","labels_url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/89/labels{/name}","comments_url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/89/comments","events_url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/89/events","html_url":"https://github.com/lukehutch/fast-classpath-scanner/issues/89","id":185371746,"number":89,"title":"Consider other alternatives to filename canonicalization","user":{"login":"Alexander--","id":1107390,"avatar_url":"https://avatars.githubusercontent.com/u/1107390?v=3","gravatar_id":"","url":"https://api.github.com/users/Alexander--","html_url":"https://github.com/Alexander--","followers_url":"https://api.github.com/users/Alexander--/followers","following_url":"https://api.github.com/users/Alexander--/following{/other_user}","gists_url":"https://api.github.com/users/Alexander--/gists{/gist_id}","starred_url":"https://api.github.com/users/Alexander--/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alexander--/subscriptions","organizations_url":"https://api.github.com/users/Alexander--/orgs","repos_url":"https://api.github.com/users/Alexander--/repos","events_url":"https://api.github.com/users/Alexander--/events{/privacy}","received_events_url":"https://api.github.com/users/Alexander--/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-26T11:51:52Z","updated_at":"2016-10-26T12:40:41Z","closed_at":null,"body":"Depending on implementation in underlying filesystem, creating canonicalized filenames may be (and usually is) exponentially expensive. Not sure about Windows, but in Linux it goes like this:\r\n\r\n1. A `stat` syscall for *every section* of file path\r\n2. If the path element appears to be a symlink, try to resolve it via `readlink` syscall\r\n\r\nAnd that is without considering extra resources spent on creating and storing Java Strings with canonical paths.\r\n\r\nModern Java versions (since Java 7) include much better alternative — https://docs.oracle.com/javase/7/docs/api/java/nio/file/attribute/BasicFileAttributes.html#fileKey(). It amounts to inode+devid on Unix and should be supported on NTFS as well. Obtaining the inode of filesystem path requires a single syscall per *entire path*.\r\n\r\nFor files on file systems without the stable key you can always fall back to canonicalization (which is basically no-op for FAT and the likes)."},"comment":{"url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/comments/256335056","html_url":"https://github.com/lukehutch/fast-classpath-scanner/issues/89#issuecomment-256335056","issue_url":"https://api.github.com/repos/lukehutch/fast-classpath-scanner/issues/89","id":256335056,"user":{"login":"lukehutch","id":811305,"avatar_url":"https://avatars.githubusercontent.com/u/811305?v=3","gravatar_id":"","url":"https://api.github.com/users/lukehutch","html_url":"https://github.com/lukehutch","followers_url":"https://api.github.com/users/lukehutch/followers","following_url":"https://api.github.com/users/lukehutch/following{/other_user}","gists_url":"https://api.github.com/users/lukehutch/gists{/gist_id}","starred_url":"https://api.github.com/users/lukehutch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lukehutch/subscriptions","organizations_url":"https://api.github.com/users/lukehutch/orgs","repos_url":"https://api.github.com/users/lukehutch/repos","events_url":"https://api.github.com/users/lukehutch/events{/privacy}","received_events_url":"https://api.github.com/users/lukehutch/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T12:40:41Z","updated_at":"2016-10-26T12:40:41Z","body":"These are all good points, and I certainly noticed that canonicalization was expensive when I profiled FCS extensively. I managed to dramatically reduce the use of canonicalization back when I spent a lot of time profiling and tuning FCS for the 2.0 release.\r\n\r\nThe reason for canonicalization is to prevent extra work scanning the same jar or directory twice. It's quite common to have the same jars included multiple times on the classpath with different paths (e.g. once via symlinks, once via absolute path). Really this is a tradeoff between the work performed in canonicalization and the work performed during jar or directory scanning -- and as far as I could tell, in the general case, it was cheaper to simply canonicalize everything to be safe. For specific cases, not canonicalizing at all would certainly be the fastest, but in the worst case scenario, the scanning time could double or worse. (The whole reason I'm canonicalizing the paths now is because a user reported that their code worked twice as fast in Eclipse as in Maven, due to Maven including the jars twice, once using symlinks.)\r\n\r\nI tried using `BasicFileAttributes#fileKey()` before, but based on my own testing, this was actually *slower* on Linux, because the Java 7 APIs related to `Path` actually incur a significant overhead of their own. The overhead of using `Path` (in order to get at the fileKey) was actually greater than that of canonicalization. I switched all of FCS to use `Path` at one point, but performance suffered so much that I had to switch it all back. I didn't profile on Windows or MacOS, but assumed (maybe incorrectly) that performance differences of these two methods would be in the same ballpark across all OSes.\r\n\r\nNote that only directories and jarfiles have their paths canonicalized; files within directories do not.\r\n\r\nBased on jvisualvm profiling, overall the total time spent in canonicalization is currently 0.1% for a large classpath (admittedly consisting mostly of jarfiles; it would be more if these jarfiles were decompressed, because multiple directories would need to be canonicalized per jarfile entry). However, the time is dominated by unzipping and classfile header parsing.\r\n\r\nSince the time spent in canonicalization is quite minor according to my own benchmarks, I don't think this is worth changing. However, if you find a usecase (or a platform) for which canonicalization is a major bottleneck, please let me know, I'm all ears :-)"}}
{"actor":{"display_login":"lukehutch"}}