{"action":"created","issue":{"url":"https://api.github.com/repos/mapbox/rasterio/issues/538","labels_url":"https://api.github.com/repos/mapbox/rasterio/issues/538/labels{/name}","comments_url":"https://api.github.com/repos/mapbox/rasterio/issues/538/comments","events_url":"https://api.github.com/repos/mapbox/rasterio/issues/538/events","html_url":"https://github.com/mapbox/rasterio/issues/538","id":125833187,"number":538,"title":"Support or supplant vsi* networking virtual file system functionality in GDAL (e.g. vsicurl, vsis3)","user":{"login":"lossyrob","id":2320142,"avatar_url":"https://avatars.githubusercontent.com/u/2320142?v=3","gravatar_id":"","url":"https://api.github.com/users/lossyrob","html_url":"https://github.com/lossyrob","followers_url":"https://api.github.com/users/lossyrob/followers","following_url":"https://api.github.com/users/lossyrob/following{/other_user}","gists_url":"https://api.github.com/users/lossyrob/gists{/gist_id}","starred_url":"https://api.github.com/users/lossyrob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lossyrob/subscriptions","organizations_url":"https://api.github.com/users/lossyrob/orgs","repos_url":"https://api.github.com/users/lossyrob/repos","events_url":"https://api.github.com/users/lossyrob/events{/privacy}","received_events_url":"https://api.github.com/users/lossyrob/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-10T17:20:33Z","updated_at":"2016-01-10T17:26:33Z","closed_at":null,"body":"Between 0.29.0 and 0.30.0, functionality was introduced to specifically support the virtual file system functinality `vsigzip`, `vsizip`, and `vsitar`. [As can be seen in this code](https://github.com/mapbox/rasterio/blob/0.30.0/rasterio/vfs.py#L6-L9), along with explicit support for some virtual file systems, some functionality for other virtual file system support that was previously and unintentionally exposed was explicitly disallowed, and calls that had taken advantage of that GDAL functionality began throwing a \"VFS scheme {0} unknown\" exception upon upgrade of rasterio.\r\n\r\nPart of the GDAL functionality that became inaccessible through rasterio was accessing rasters over HTTP with `vsicurl`. That feature in particular was useful to myself and others, since we were relying on it to do windowed reads of large rasters over the network, without pulling down the whole file. In this way we can have multiple processes pulling byte ranges of a GeoTiff off of S3 (which supports the byte range HTTP request header), as a mechanism of doing tiling work using rasterio and PySpark (see the [OpenAerialMap tiler code here](https://github.com/hotosm/oam-server-tiler/blob/4ac1eec8c1974be6f6681730240123c1824a20e7/chunk/chunk.py)...this \"chunking\" step had some problems dealing with very large rasters so if you're interested in using these techniques make sure to look to the master branch's version and any unmerged PR's).\r\n\r\nI started a discussion on a merged pull request, which is the place I found the most definitive language about why `vsicurl` was no longer supported. It was smartly recommended that we start a separate ticket instead of talking on a merged PR, and this ticket can serve this purpose. [Please read the prior discussion](https://github.com/mapbox/rasterio/pull/472#issuecomment-169817208) as if they were the first comments on this issue.\r\n\r\n/cc @sgillies @perrygeo @mojodna\r\n\r\n "},"comment":{"url":"https://api.github.com/repos/mapbox/rasterio/issues/comments/170372759","html_url":"https://github.com/mapbox/rasterio/issues/538#issuecomment-170372759","issue_url":"https://api.github.com/repos/mapbox/rasterio/issues/538","id":170372759,"user":{"login":"lossyrob","id":2320142,"avatar_url":"https://avatars.githubusercontent.com/u/2320142?v=3","gravatar_id":"","url":"https://api.github.com/users/lossyrob","html_url":"https://github.com/lossyrob","followers_url":"https://api.github.com/users/lossyrob/followers","following_url":"https://api.github.com/users/lossyrob/following{/other_user}","gists_url":"https://api.github.com/users/lossyrob/gists{/gist_id}","starred_url":"https://api.github.com/users/lossyrob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lossyrob/subscriptions","organizations_url":"https://api.github.com/users/lossyrob/orgs","repos_url":"https://api.github.com/users/lossyrob/repos","events_url":"https://api.github.com/users/lossyrob/events{/privacy}","received_events_url":"https://api.github.com/users/lossyrob/received_events","type":"User","site_admin":false},"created_at":"2016-01-10T17:26:33Z","updated_at":"2016-01-10T17:26:33Z","body":"@sgillies to answer your question in the previous thread, \r\n> Does vsicurl raise HTTP errors adequately?\r\n\r\nThe answer as you might have expected is, in my experience, no. For example, I didn't set the ACL correctly when uploading tiles, and instead of a `403` being returned, it gave the generic\r\n\r\n```console\r\nERROR 4: `/vsicurl/http://foo-bucket/bar.tiff' does not exist in the file system,\r\nand is not recognised as a supported dataset name.\r\n```\r\n\r\nerror. I don't think relying on GDAL to give us nice information about what is happening over the network is not going to be a good path forward, without some PR's to GDAL.\r\n\r\nPerhaps @rouault or @warmerdam would have some more informed ideas about this?"}}