{"action":"created","issue":{"url":"https://api.github.com/repos/ome/ome-common-java/issues/33","repository_url":"https://api.github.com/repos/ome/ome-common-java","labels_url":"https://api.github.com/repos/ome/ome-common-java/issues/33/labels{/name}","comments_url":"https://api.github.com/repos/ome/ome-common-java/issues/33/comments","events_url":"https://api.github.com/repos/ome/ome-common-java/issues/33/events","html_url":"https://github.com/ome/ome-common-java/pull/33","id":366867715,"node_id":"MDExOlB1bGxSZXF1ZXN0MjIwNDM3Njg1","number":33,"title":"Add S3Handle","user":{"login":"manics","id":1644105,"node_id":"MDQ6VXNlcjE2NDQxMDU=","avatar_url":"https://avatars2.githubusercontent.com/u/1644105?v=4","gravatar_id":"","url":"https://api.github.com/users/manics","html_url":"https://github.com/manics","followers_url":"https://api.github.com/users/manics/followers","following_url":"https://api.github.com/users/manics/following{/other_user}","gists_url":"https://api.github.com/users/manics/gists{/gist_id}","starred_url":"https://api.github.com/users/manics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manics/subscriptions","organizations_url":"https://api.github.com/users/manics/orgs","repos_url":"https://api.github.com/users/manics/repos","events_url":"https://api.github.com/users/manics/events{/privacy}","received_events_url":"https://api.github.com/users/manics/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/ome/ome-common-java/milestones/9","html_url":"https://github.com/ome/ome-common-java/milestone/9","labels_url":"https://api.github.com/repos/ome/ome-common-java/milestones/9/labels","id":3732275,"node_id":"MDk6TWlsZXN0b25lMzczMjI3NQ==","number":9,"title":"6.0.0","description":"","creator":{"login":"sbesson","id":1355463,"node_id":"MDQ6VXNlcjEzNTU0NjM=","avatar_url":"https://avatars1.githubusercontent.com/u/1355463?v=4","gravatar_id":"","url":"https://api.github.com/users/sbesson","html_url":"https://github.com/sbesson","followers_url":"https://api.github.com/users/sbesson/followers","following_url":"https://api.github.com/users/sbesson/following{/other_user}","gists_url":"https://api.github.com/users/sbesson/gists{/gist_id}","starred_url":"https://api.github.com/users/sbesson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbesson/subscriptions","organizations_url":"https://api.github.com/users/sbesson/orgs","repos_url":"https://api.github.com/users/sbesson/repos","events_url":"https://api.github.com/users/sbesson/events{/privacy}","received_events_url":"https://api.github.com/users/sbesson/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":3,"state":"open","created_at":"2018-10-12T05:50:06Z","updated_at":"2018-11-05T11:32:38Z","due_on":null,"closed_at":null},"comments":11,"created_at":"2018-10-04T16:17:29Z","updated_at":"2018-11-05T11:32:38Z","closed_at":"2018-11-05T11:32:38Z","author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/ome/ome-common-java/pulls/33","html_url":"https://github.com/ome/ome-common-java/pull/33","diff_url":"https://github.com/ome/ome-common-java/pull/33.diff","patch_url":"https://github.com/ome/ome-common-java/pull/33.patch"},"body":"These are the commits from #32 with additional commits on top.\r\n\r\nIt should be possible to read a file off AWS S3, e.g. `./showinf s3://s3.us-east-1.amazonaws.com/czi.starfish.data.public/browse/raw/20180820/merfish_u2os/fov_57.tif` if you don't mind waiting\r\n```\r\nInitialization took 2172.948s\r\n54.351s elapsed (3019.5ms per plane)\r\n```\r\n\r\nAlso includes primitive caching: `BF_REMOTE_CACHE_ROOTDIR=$HOME/tmp/ BF_DEVEL=true BF_FLAGS=\"-cp $PWD/../../ome-common-java/target/ome-common-5.3.7-SNAPSHOT.jar:$PWD/bioformats_package.jar:$PWD\" ./showinf s3://s3.us-east-1.amazonaws.com/czi.starfish.data.public/browse/raw/20180820/merfish_u2os/fov_57.tif` though it still attempts to check for the existence of many other remote files\r\n```\r\nInitialization took 12.244s\r\n2.402s elapsed (133.44444ms per plane)\r\n```\r\n\r\nExample full timings (including the time taken before the \"Initialization\" time is calculated) for `s3://s3.us-east-1.amazonaws.com/czi.starfish.data.public/browse/raw/20180820/merfish_u2os/fov_57.tif`:\r\n- Cache enabled, not cached: 1m19s\r\n- Cache enabled, cached: 32s\r\n- Cache disabled: 24m22s\r\n\r\nExamples using a local minio server with the same file `s3://INTERNAL.HOST/test.bucket/fov_57.tif`\r\n- Cache enabled, not cached: 7s\r\n- Cache enabled, cached: 3s\r\n- Cache disabled: 1m48s\r\n\r\nConclusions so far:\r\n- Streaming a file is not practical due to the reader, probably due to the reader seeking backwards and therefore causing the stream to reset to the beginning\r\n- The initialisation (or pre-initialisation) steps which involve checking the existence of many other related files is very time-consuming, in this case approximately the same time as it takes to download the the 144MB file"},"comment":{"url":"https://api.github.com/repos/ome/ome-common-java/issues/comments/435843452","html_url":"https://github.com/ome/ome-common-java/pull/33#issuecomment-435843452","issue_url":"https://api.github.com/repos/ome/ome-common-java/issues/33","id":435843452,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNTg0MzQ1Mg==","user":{"login":"manics","id":1644105,"node_id":"MDQ6VXNlcjE2NDQxMDU=","avatar_url":"https://avatars2.githubusercontent.com/u/1644105?v=4","gravatar_id":"","url":"https://api.github.com/users/manics","html_url":"https://github.com/manics","followers_url":"https://api.github.com/users/manics/followers","following_url":"https://api.github.com/users/manics/following{/other_user}","gists_url":"https://api.github.com/users/manics/gists{/gist_id}","starred_url":"https://api.github.com/users/manics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manics/subscriptions","organizations_url":"https://api.github.com/users/manics/orgs","repos_url":"https://api.github.com/users/manics/repos","events_url":"https://api.github.com/users/manics/events{/privacy}","received_events_url":"https://api.github.com/users/manics/received_events","type":"User","site_admin":false},"created_at":"2018-11-05T11:32:38Z","updated_at":"2018-11-05T11:32:38Z","author_association":"MEMBER","body":"Updated"}}
{"actor":{"display_login":"manics"}}