{"action":"created","comment":{"url":"https://api.github.com/repos/CartoDB/cartodb/pulls/comments/92785302","pull_request_review_id":13306099,"id":92785302,"diff_hunk":"@@ -0,0 +1,38 @@\n+# encoding: utf-8\n+\n+module Carto\n+  module StorageOptions\n+    class S3\n+      def self.conf\n+        Cartodb.get_config_if_present(:aws, 's3')\n+      end\n+\n+      def initialize(bucket_name)\n+        @bucket_name = bucket_name\n+\n+        AWS::config(self.class.conf)\n+      end\n+\n+      def upload(namespace, file)\n+        filename = Pathname.new(file.path).basename.to_s\n+        mime_type = MIME::Types.type_for(filename).first.to_s\n+        identifier = File.join(namespace, filename)\n+\n+        asset = bucket.objects[identifier]","path":"lib/carto/storage_options/s3.rb","position":21,"original_position":21,"commit_id":"09b634a81fc9f697d01d1e04b3811981c82c9fec","original_commit_id":"86bd076d7bcff8000ce5b2c2bd8e0b3cb884ab59","user":{"login":"gfiorav","id":4222826,"avatar_url":"https://avatars.githubusercontent.com/u/4222826?v=3","gravatar_id":"","url":"https://api.github.com/users/gfiorav","html_url":"https://github.com/gfiorav","followers_url":"https://api.github.com/users/gfiorav/followers","following_url":"https://api.github.com/users/gfiorav/following{/other_user}","gists_url":"https://api.github.com/users/gfiorav/gists{/gist_id}","starred_url":"https://api.github.com/users/gfiorav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gfiorav/subscriptions","organizations_url":"https://api.github.com/users/gfiorav/orgs","repos_url":"https://api.github.com/users/gfiorav/repos","events_url":"https://api.github.com/users/gfiorav/events{/privacy}","received_events_url":"https://api.github.com/users/gfiorav/received_events","type":"User","site_admin":false},"body":"It's an asset indeed.","created_at":"2016-12-16T10:16:33Z","updated_at":"2016-12-16T10:16:33Z","html_url":"https://github.com/CartoDB/cartodb/pull/11034#discussion_r92785302","pull_request_url":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034","_links":{"self":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/comments/92785302"},"html":{"href":"https://github.com/CartoDB/cartodb/pull/11034#discussion_r92785302"},"pull_request":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034"}}},"pull_request":{"url":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034","id":97780616,"html_url":"https://github.com/CartoDB/cartodb/pull/11034","diff_url":"https://github.com/CartoDB/cartodb/pull/11034.diff","patch_url":"https://github.com/CartoDB/cartodb/pull/11034.patch","issue_url":"https://api.github.com/repos/CartoDB/cartodb/issues/11034","number":11034,"state":"open","locked":false,"title":"Organization assets","user":{"login":"gfiorav","id":4222826,"avatar_url":"https://avatars.githubusercontent.com/u/4222826?v=3","gravatar_id":"","url":"https://api.github.com/users/gfiorav","html_url":"https://github.com/gfiorav","followers_url":"https://api.github.com/users/gfiorav/followers","following_url":"https://api.github.com/users/gfiorav/following{/other_user}","gists_url":"https://api.github.com/users/gfiorav/gists{/gist_id}","starred_url":"https://api.github.com/users/gfiorav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gfiorav/subscriptions","organizations_url":"https://api.github.com/users/gfiorav/orgs","repos_url":"https://api.github.com/users/gfiorav/repos","events_url":"https://api.github.com/users/gfiorav/events{/privacy}","received_events_url":"https://api.github.com/users/gfiorav/received_events","type":"User","site_admin":false},"body":"Fixes #10754\r\n\r\n## Synopsis\r\n\r\nThis PR present 3 distinct elements. I've decided to keep them together in 1 PR since each one helps explain the other's existence, and it's not that much code base. These elements are:\r\n\r\n1. A implementation-transparent storage for s3/local. Works for any file.\r\n2. An abstracted \"Organisation asset\" class that handles downloads and uploads (using 1.) with limits.\r\n3. An API for managing organisation assets, that uses 2. for handling an organisations assets. Standard REST without update.\r\n\r\nThere is also a small refactor of a previous controller which I couldn't help but include since it's only one method and the asset presenter has been renamed (which appears as 2 different files).\r\n\r\nThe Asset model has also been modified and 2 columns have been added:\r\n+ Organization id\r\n+ Storage info (contains type, location and identifier)\r\n\r\nLet me explain the terminology here. I've decided to think of if like this: A storage system (wether it's local or S3) need's 2 elements to define a file. It needs a location (organisation assets are in a separate place as a whole than user assets) and an identifier (that separates one asset in that location from another in the same location). Storage type is simply local, s3, or whatever comes next. In S3, a location is a bucket while on local it's a subfolder. In both cases, the identifier has a full path format (`organization_id/manolito_escobar.png`). So, basically, storage info is a validated JSON with enough info to locate an asset in a storage service.\r\n\r\nSo, for s3, you can get the following:\r\n```\r\nstorage_type: s3\r\nlocation: com.carto.organization-assets.production\r\nidentifier: /<some org uuid>/file.png\r\n```\r\n\r\nor, for local\r\n```\r\nstorage_type: local\r\nlocation: organisation_assets\r\nidentier: /<some org uuid>/file.png\r\n```\r\n\r\nIt might seem that everything apart from organisation id is redundant. After all, users assets work without them. They're not. Previous implementation relied on parsing the asset's public url to infer information about the full path. Storage type (s3 or local) was assumed to be the one that was configured at the that moment (although you might turn on or off s3 at some point in the future and there is no technological limitation for older assets to work). Location wasn't necessary because all user assets live in the same folder (in case of local) or bucket (in case of s3). I find this better, and it supports switching systems after some assets exist in other storages (we know where to find the asset). It also has the benefit that it can survive url changes (if we wan't to use ACL other than public, for instance).\r\n\r\n## Acceptance instructions\r\n\r\nFirst of all, you need to change (with `knife`) the carto config to include this in the `assets` section:\r\n\r\n```yaml\r\nassets:\r\n    s3_bucket_name: \"tests\"\r\n    max_file_size: 5242880 # 5.megabytes\r\n    organizations:\r\n      s3_bucket_name: 'com.cartodb.users-assets.staging'\r\n      max_file_size: <whatever you feel is right> # 1.megabytes by default if not specified\r\n```\r\n\r\nIn order to test this PR, 2 things should be checked:\r\n\r\n1. Old editor assets still work.\r\n2. New builder organisations assets work.\r\n\r\nFor 2., you need to `curl` your way into the endpoint (see `bundle exec rake routes`) and check both with s3 configured and without that you can create and delete assets (check the files disappear from the corresponding storage accordingly).","created_at":"2016-12-13T16:05:32Z","updated_at":"2016-12-16T10:16:33Z","closed_at":null,"merged_at":null,"merge_commit_sha":"268e023763d08f16ef91206a5c2689658de6bb41","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034/commits","review_comments_url":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034/comments","review_comment_url":"https://api.github.com/repos/CartoDB/cartodb/pulls/comments{/number}","comments_url":"https://api.github.com/repos/CartoDB/cartodb/issues/11034/comments","statuses_url":"https://api.github.com/repos/CartoDB/cartodb/statuses/09b634a81fc9f697d01d1e04b3811981c82c9fec","head":{"label":"CartoDB:10838-new-assests-endpoints","ref":"10838-new-assests-endpoints","sha":"09b634a81fc9f697d01d1e04b3811981c82c9fec","user":{"login":"CartoDB","id":1799254,"avatar_url":"https://avatars.githubusercontent.com/u/1799254?v=3","gravatar_id":"","url":"https://api.github.com/users/CartoDB","html_url":"https://github.com/CartoDB","followers_url":"https://api.github.com/users/CartoDB/followers","following_url":"https://api.github.com/users/CartoDB/following{/other_user}","gists_url":"https://api.github.com/users/CartoDB/gists{/gist_id}","starred_url":"https://api.github.com/users/CartoDB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CartoDB/subscriptions","organizations_url":"https://api.github.com/users/CartoDB/orgs","repos_url":"https://api.github.com/users/CartoDB/repos","events_url":"https://api.github.com/users/CartoDB/events{/privacy}","received_events_url":"https://api.github.com/users/CartoDB/received_events","type":"Organization","site_admin":false},"repo":{"id":4909355,"name":"cartodb","full_name":"CartoDB/cartodb","owner":{"login":"CartoDB","id":1799254,"avatar_url":"https://avatars.githubusercontent.com/u/1799254?v=3","gravatar_id":"","url":"https://api.github.com/users/CartoDB","html_url":"https://github.com/CartoDB","followers_url":"https://api.github.com/users/CartoDB/followers","following_url":"https://api.github.com/users/CartoDB/following{/other_user}","gists_url":"https://api.github.com/users/CartoDB/gists{/gist_id}","starred_url":"https://api.github.com/users/CartoDB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CartoDB/subscriptions","organizations_url":"https://api.github.com/users/CartoDB/orgs","repos_url":"https://api.github.com/users/CartoDB/repos","events_url":"https://api.github.com/users/CartoDB/events{/privacy}","received_events_url":"https://api.github.com/users/CartoDB/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/CartoDB/cartodb","description":"Location Intelligence & Data Visualization tool","fork":false,"url":"https://api.github.com/repos/CartoDB/cartodb","forks_url":"https://api.github.com/repos/CartoDB/cartodb/forks","keys_url":"https://api.github.com/repos/CartoDB/cartodb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CartoDB/cartodb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CartoDB/cartodb/teams","hooks_url":"https://api.github.com/repos/CartoDB/cartodb/hooks","issue_events_url":"https://api.github.com/repos/CartoDB/cartodb/issues/events{/number}","events_url":"https://api.github.com/repos/CartoDB/cartodb/events","assignees_url":"https://api.github.com/repos/CartoDB/cartodb/assignees{/user}","branches_url":"https://api.github.com/repos/CartoDB/cartodb/branches{/branch}","tags_url":"https://api.github.com/repos/CartoDB/cartodb/tags","blobs_url":"https://api.github.com/repos/CartoDB/cartodb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/CartoDB/cartodb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CartoDB/cartodb/git/refs{/sha}","trees_url":"https://api.github.com/repos/CartoDB/cartodb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CartoDB/cartodb/statuses/{sha}","languages_url":"https://api.github.com/repos/CartoDB/cartodb/languages","stargazers_url":"https://api.github.com/repos/CartoDB/cartodb/stargazers","contributors_url":"https://api.github.com/repos/CartoDB/cartodb/contributors","subscribers_url":"https://api.github.com/repos/CartoDB/cartodb/subscribers","subscription_url":"https://api.github.com/repos/CartoDB/cartodb/subscription","commits_url":"https://api.github.com/repos/CartoDB/cartodb/commits{/sha}","git_commits_url":"https://api.github.com/repos/CartoDB/cartodb/git/commits{/sha}","comments_url":"https://api.github.com/repos/CartoDB/cartodb/comments{/number}","issue_comment_url":"https://api.github.com/repos/CartoDB/cartodb/issues/comments{/number}","contents_url":"https://api.github.com/repos/CartoDB/cartodb/contents/{+path}","compare_url":"https://api.github.com/repos/CartoDB/cartodb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/CartoDB/cartodb/merges","archive_url":"https://api.github.com/repos/CartoDB/cartodb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CartoDB/cartodb/downloads","issues_url":"https://api.github.com/repos/CartoDB/cartodb/issues{/number}","pulls_url":"https://api.github.com/repos/CartoDB/cartodb/pulls{/number}","milestones_url":"https://api.github.com/repos/CartoDB/cartodb/milestones{/number}","notifications_url":"https://api.github.com/repos/CartoDB/cartodb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/CartoDB/cartodb/labels{/name}","releases_url":"https://api.github.com/repos/CartoDB/cartodb/releases{/id}","deployments_url":"https://api.github.com/repos/CartoDB/cartodb/deployments","created_at":"2012-07-05T15:23:45Z","updated_at":"2016-12-15T17:30:19Z","pushed_at":"2016-12-16T10:15:44Z","git_url":"git://github.com/CartoDB/cartodb.git","ssh_url":"git@github.com:CartoDB/cartodb.git","clone_url":"https://github.com/CartoDB/cartodb.git","svn_url":"https://github.com/CartoDB/cartodb","homepage":"http://carto.com","size":343583,"stargazers_count":1352,"watchers_count":1352,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":359,"mirror_url":null,"open_issues_count":578,"forks":359,"open_issues":578,"watchers":1352,"default_branch":"master"}},"base":{"label":"CartoDB:master","ref":"master","sha":"839ced471a23fd08712dc5c076e9597229fc6315","user":{"login":"CartoDB","id":1799254,"avatar_url":"https://avatars.githubusercontent.com/u/1799254?v=3","gravatar_id":"","url":"https://api.github.com/users/CartoDB","html_url":"https://github.com/CartoDB","followers_url":"https://api.github.com/users/CartoDB/followers","following_url":"https://api.github.com/users/CartoDB/following{/other_user}","gists_url":"https://api.github.com/users/CartoDB/gists{/gist_id}","starred_url":"https://api.github.com/users/CartoDB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CartoDB/subscriptions","organizations_url":"https://api.github.com/users/CartoDB/orgs","repos_url":"https://api.github.com/users/CartoDB/repos","events_url":"https://api.github.com/users/CartoDB/events{/privacy}","received_events_url":"https://api.github.com/users/CartoDB/received_events","type":"Organization","site_admin":false},"repo":{"id":4909355,"name":"cartodb","full_name":"CartoDB/cartodb","owner":{"login":"CartoDB","id":1799254,"avatar_url":"https://avatars.githubusercontent.com/u/1799254?v=3","gravatar_id":"","url":"https://api.github.com/users/CartoDB","html_url":"https://github.com/CartoDB","followers_url":"https://api.github.com/users/CartoDB/followers","following_url":"https://api.github.com/users/CartoDB/following{/other_user}","gists_url":"https://api.github.com/users/CartoDB/gists{/gist_id}","starred_url":"https://api.github.com/users/CartoDB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/CartoDB/subscriptions","organizations_url":"https://api.github.com/users/CartoDB/orgs","repos_url":"https://api.github.com/users/CartoDB/repos","events_url":"https://api.github.com/users/CartoDB/events{/privacy}","received_events_url":"https://api.github.com/users/CartoDB/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/CartoDB/cartodb","description":"Location Intelligence & Data Visualization tool","fork":false,"url":"https://api.github.com/repos/CartoDB/cartodb","forks_url":"https://api.github.com/repos/CartoDB/cartodb/forks","keys_url":"https://api.github.com/repos/CartoDB/cartodb/keys{/key_id}","collaborators_url":"https://api.github.com/repos/CartoDB/cartodb/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/CartoDB/cartodb/teams","hooks_url":"https://api.github.com/repos/CartoDB/cartodb/hooks","issue_events_url":"https://api.github.com/repos/CartoDB/cartodb/issues/events{/number}","events_url":"https://api.github.com/repos/CartoDB/cartodb/events","assignees_url":"https://api.github.com/repos/CartoDB/cartodb/assignees{/user}","branches_url":"https://api.github.com/repos/CartoDB/cartodb/branches{/branch}","tags_url":"https://api.github.com/repos/CartoDB/cartodb/tags","blobs_url":"https://api.github.com/repos/CartoDB/cartodb/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/CartoDB/cartodb/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/CartoDB/cartodb/git/refs{/sha}","trees_url":"https://api.github.com/repos/CartoDB/cartodb/git/trees{/sha}","statuses_url":"https://api.github.com/repos/CartoDB/cartodb/statuses/{sha}","languages_url":"https://api.github.com/repos/CartoDB/cartodb/languages","stargazers_url":"https://api.github.com/repos/CartoDB/cartodb/stargazers","contributors_url":"https://api.github.com/repos/CartoDB/cartodb/contributors","subscribers_url":"https://api.github.com/repos/CartoDB/cartodb/subscribers","subscription_url":"https://api.github.com/repos/CartoDB/cartodb/subscription","commits_url":"https://api.github.com/repos/CartoDB/cartodb/commits{/sha}","git_commits_url":"https://api.github.com/repos/CartoDB/cartodb/git/commits{/sha}","comments_url":"https://api.github.com/repos/CartoDB/cartodb/comments{/number}","issue_comment_url":"https://api.github.com/repos/CartoDB/cartodb/issues/comments{/number}","contents_url":"https://api.github.com/repos/CartoDB/cartodb/contents/{+path}","compare_url":"https://api.github.com/repos/CartoDB/cartodb/compare/{base}...{head}","merges_url":"https://api.github.com/repos/CartoDB/cartodb/merges","archive_url":"https://api.github.com/repos/CartoDB/cartodb/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/CartoDB/cartodb/downloads","issues_url":"https://api.github.com/repos/CartoDB/cartodb/issues{/number}","pulls_url":"https://api.github.com/repos/CartoDB/cartodb/pulls{/number}","milestones_url":"https://api.github.com/repos/CartoDB/cartodb/milestones{/number}","notifications_url":"https://api.github.com/repos/CartoDB/cartodb/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/CartoDB/cartodb/labels{/name}","releases_url":"https://api.github.com/repos/CartoDB/cartodb/releases{/id}","deployments_url":"https://api.github.com/repos/CartoDB/cartodb/deployments","created_at":"2012-07-05T15:23:45Z","updated_at":"2016-12-15T17:30:19Z","pushed_at":"2016-12-16T10:15:44Z","git_url":"git://github.com/CartoDB/cartodb.git","ssh_url":"git@github.com:CartoDB/cartodb.git","clone_url":"https://github.com/CartoDB/cartodb.git","svn_url":"https://github.com/CartoDB/cartodb","homepage":"http://carto.com","size":343583,"stargazers_count":1352,"watchers_count":1352,"language":"HTML","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":359,"mirror_url":null,"open_issues_count":578,"forks":359,"open_issues":578,"watchers":1352,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034"},"html":{"href":"https://github.com/CartoDB/cartodb/pull/11034"},"issue":{"href":"https://api.github.com/repos/CartoDB/cartodb/issues/11034"},"comments":{"href":"https://api.github.com/repos/CartoDB/cartodb/issues/11034/comments"},"review_comments":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034/comments"},"review_comment":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/CartoDB/cartodb/pulls/11034/commits"},"statuses":{"href":"https://api.github.com/repos/CartoDB/cartodb/statuses/09b634a81fc9f697d01d1e04b3811981c82c9fec"}}}}
{"actor":{"display_login":"gfiorav"}}