{"action":"created","comment":{"url":"https://api.github.com/repos/spotify/luigi/pulls/comments/234977081","pull_request_review_id":176496092,"id":234977081,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIzNDk3NzA4MQ==","diff_hunk":"@@ -0,0 +1,164 @@\n+# -*- coding: utf-8 -*-\n+#\n+# Copyright 2018 Microsoft Corporation\n+#\n+# Licensed under the Apache License, Version 2.0 (the \"License\");\n+# you may not use this file except in compliance with the License.\n+# You may obtain a copy of the License at\n+#\n+# http://www.apache.org/licenses/LICENSE-2.0\n+#\n+# Unless required by applicable law or agreed to in writing, software\n+# distributed under the License is distributed on an \"AS IS\" BASIS,\n+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n+# See the License for the specific language governing permissions and\n+# limitations under the License.\n+#\n+\"\"\"\n+Integration tests for azureblob module.\n+\"\"\"\n+\n+import os\n+import unittest\n+\n+import luigi\n+from luigi.contrib.azureblob import AzureBlobClient, AzureBlobTarget\n+\n+client = AzureBlobClient(os.environ.get(\"ACCOUNT_NAME\"), os.environ.get(\"ACCOUNT_KEY\"), os.environ.get(\"SAS_TOKEN\"))","path":"test/contrib/azureblob_test.py","position":27,"original_position":27,"commit_id":"40b69394e6093ff44261aec622dcab10000e7079","original_commit_id":"0651a440ea146901f2d1fbdb5ba0148fd8a12d26","user":{"login":"dlstadther","id":7432199,"node_id":"MDQ6VXNlcjc0MzIxOTk=","avatar_url":"https://avatars3.githubusercontent.com/u/7432199?v=4","gravatar_id":"","url":"https://api.github.com/users/dlstadther","html_url":"https://github.com/dlstadther","followers_url":"https://api.github.com/users/dlstadther/followers","following_url":"https://api.github.com/users/dlstadther/following{/other_user}","gists_url":"https://api.github.com/users/dlstadther/gists{/gist_id}","starred_url":"https://api.github.com/users/dlstadther/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dlstadther/subscriptions","organizations_url":"https://api.github.com/users/dlstadther/orgs","repos_url":"https://api.github.com/users/dlstadther/repos","events_url":"https://api.github.com/users/dlstadther/events{/privacy}","received_events_url":"https://api.github.com/users/dlstadther/received_events","type":"User","site_admin":false},"body":"There look to be some some way to test this stuff without real accounts: https://github.com/Azure/azure-storage-python/tree/master/tests\r\n\r\nAnd this is from the python SDK already being used by your additions.","created_at":"2018-11-20T12:10:16Z","updated_at":"2018-11-20T12:11:20Z","html_url":"https://github.com/spotify/luigi/pull/2585#discussion_r234977081","pull_request_url":"https://api.github.com/repos/spotify/luigi/pulls/2585","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/spotify/luigi/pulls/comments/234977081"},"html":{"href":"https://github.com/spotify/luigi/pull/2585#discussion_r234977081"},"pull_request":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2585"}},"in_reply_to_id":234739528},"pull_request":{"url":"https://api.github.com/repos/spotify/luigi/pulls/2585","id":231795978,"node_id":"MDExOlB1bGxSZXF1ZXN0MjMxNzk1OTc4","html_url":"https://github.com/spotify/luigi/pull/2585","diff_url":"https://github.com/spotify/luigi/pull/2585.diff","patch_url":"https://github.com/spotify/luigi/pull/2585.patch","issue_url":"https://api.github.com/repos/spotify/luigi/issues/2585","number":2585,"state":"open","locked":false,"title":"Support for Azure Blob Storage Target","user":{"login":"manish","id":16161,"node_id":"MDQ6VXNlcjE2MTYx","avatar_url":"https://avatars3.githubusercontent.com/u/16161?v=4","gravatar_id":"","url":"https://api.github.com/users/manish","html_url":"https://github.com/manish","followers_url":"https://api.github.com/users/manish/followers","following_url":"https://api.github.com/users/manish/following{/other_user}","gists_url":"https://api.github.com/users/manish/gists{/gist_id}","starred_url":"https://api.github.com/users/manish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manish/subscriptions","organizations_url":"https://api.github.com/users/manish/orgs","repos_url":"https://api.github.com/users/manish/repos","events_url":"https://api.github.com/users/manish/events{/privacy}","received_events_url":"https://api.github.com/users/manish/received_events","type":"User","site_admin":false},"body":"Azure Blob Storage backend for Target. Transparently uploads and downloads text and binary objects to Azure Blob Storage\r\n\r\n## Description\r\n\r\nProvides a `AzureBlobTarget` and `AzureBlobClient`. When reading, `ReadableAzureBlobFile` is used and when writing `AtomicAzureBlobFile` is used.\r\n\r\nUse account name + account key for authentication or use shared-access-signatures (like an OAuth token). For saving/loading file, provide the azure blob container and azure blob name. There is an option to not download the file and just get the specific bytes in memory including specific bytes range (advanced usage).\r\n\r\n## Motivation and Context\r\n\r\nWhile working on a project, I noticed that `S3Target` exists, but no such exists for Azure Blob Storage and I wanted to implement it since I wanted all my artifacts to remain under single cloud subscription (Azure)\r\n\r\n## Have you tested this? If so, how?\r\n\r\nI have written united tests which is `test/contrib/azureblob_test.py`. It requires azure account name + key or a SAS token to run. The values can be passed or can be exported as environment variables \r\n\r\n`ACCOUNT_NAME` - this is the [actual name of your azure blob storage account](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-dotnet?tabs=macos#copy-your-credentials-from-the-azure-portal).\r\n\r\n`ACCOUNT_KEY` - this is the key (password) for the storage account. It has `root` like privileges, so not always a good idea to use it everywhere. [(Documentation)](https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1?toc=%2fazure%2fstorage%2fblobs%2ftoc.json)\r\n\r\n`SAS_TOKEN` - A shared access signature (SAS) provides you with a way to grant limited access to objects in your storage account to other clients, without exposing your account key. [(Documentation)](https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1?toc=%2fazure%2fstorage%2fblobs%2ftoc.json)\r\n\r\n","created_at":"2018-11-18T15:47:25Z","updated_at":"2018-11-20T12:11:20Z","closed_at":null,"merged_at":null,"merge_commit_sha":"e7414e008a3524064e279894ca79300054230f90","assignee":null,"assignees":[],"requested_reviewers":[{"login":"ulzha","id":84229,"node_id":"MDQ6VXNlcjg0MjI5","avatar_url":"https://avatars3.githubusercontent.com/u/84229?v=4","gravatar_id":"","url":"https://api.github.com/users/ulzha","html_url":"https://github.com/ulzha","followers_url":"https://api.github.com/users/ulzha/followers","following_url":"https://api.github.com/users/ulzha/following{/other_user}","gists_url":"https://api.github.com/users/ulzha/gists{/gist_id}","starred_url":"https://api.github.com/users/ulzha/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ulzha/subscriptions","organizations_url":"https://api.github.com/users/ulzha/orgs","repos_url":"https://api.github.com/users/ulzha/repos","events_url":"https://api.github.com/users/ulzha/events{/privacy}","received_events_url":"https://api.github.com/users/ulzha/received_events","type":"User","site_admin":false},{"login":"Tarrasch","id":294349,"node_id":"MDQ6VXNlcjI5NDM0OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/294349?v=4","gravatar_id":"","url":"https://api.github.com/users/Tarrasch","html_url":"https://github.com/Tarrasch","followers_url":"https://api.github.com/users/Tarrasch/followers","following_url":"https://api.github.com/users/Tarrasch/following{/other_user}","gists_url":"https://api.github.com/users/Tarrasch/gists{/gist_id}","starred_url":"https://api.github.com/users/Tarrasch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Tarrasch/subscriptions","organizations_url":"https://api.github.com/users/Tarrasch/orgs","repos_url":"https://api.github.com/users/Tarrasch/repos","events_url":"https://api.github.com/users/Tarrasch/events{/privacy}","received_events_url":"https://api.github.com/users/Tarrasch/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/spotify/luigi/pulls/2585/commits","review_comments_url":"https://api.github.com/repos/spotify/luigi/pulls/2585/comments","review_comment_url":"https://api.github.com/repos/spotify/luigi/pulls/comments{/number}","comments_url":"https://api.github.com/repos/spotify/luigi/issues/2585/comments","statuses_url":"https://api.github.com/repos/spotify/luigi/statuses/40b69394e6093ff44261aec622dcab10000e7079","head":{"label":"manish:luigi-azure-blob","ref":"luigi-azure-blob","sha":"40b69394e6093ff44261aec622dcab10000e7079","user":{"login":"manish","id":16161,"node_id":"MDQ6VXNlcjE2MTYx","avatar_url":"https://avatars3.githubusercontent.com/u/16161?v=4","gravatar_id":"","url":"https://api.github.com/users/manish","html_url":"https://github.com/manish","followers_url":"https://api.github.com/users/manish/followers","following_url":"https://api.github.com/users/manish/following{/other_user}","gists_url":"https://api.github.com/users/manish/gists{/gist_id}","starred_url":"https://api.github.com/users/manish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manish/subscriptions","organizations_url":"https://api.github.com/users/manish/orgs","repos_url":"https://api.github.com/users/manish/repos","events_url":"https://api.github.com/users/manish/events{/privacy}","received_events_url":"https://api.github.com/users/manish/received_events","type":"User","site_admin":false},"repo":{"id":158091230,"node_id":"MDEwOlJlcG9zaXRvcnkxNTgwOTEyMzA=","name":"luigi","full_name":"manish/luigi","private":false,"owner":{"login":"manish","id":16161,"node_id":"MDQ6VXNlcjE2MTYx","avatar_url":"https://avatars3.githubusercontent.com/u/16161?v=4","gravatar_id":"","url":"https://api.github.com/users/manish","html_url":"https://github.com/manish","followers_url":"https://api.github.com/users/manish/followers","following_url":"https://api.github.com/users/manish/following{/other_user}","gists_url":"https://api.github.com/users/manish/gists{/gist_id}","starred_url":"https://api.github.com/users/manish/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/manish/subscriptions","organizations_url":"https://api.github.com/users/manish/orgs","repos_url":"https://api.github.com/users/manish/repos","events_url":"https://api.github.com/users/manish/events{/privacy}","received_events_url":"https://api.github.com/users/manish/received_events","type":"User","site_admin":false},"html_url":"https://github.com/manish/luigi","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ","fork":true,"url":"https://api.github.com/repos/manish/luigi","forks_url":"https://api.github.com/repos/manish/luigi/forks","keys_url":"https://api.github.com/repos/manish/luigi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/manish/luigi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/manish/luigi/teams","hooks_url":"https://api.github.com/repos/manish/luigi/hooks","issue_events_url":"https://api.github.com/repos/manish/luigi/issues/events{/number}","events_url":"https://api.github.com/repos/manish/luigi/events","assignees_url":"https://api.github.com/repos/manish/luigi/assignees{/user}","branches_url":"https://api.github.com/repos/manish/luigi/branches{/branch}","tags_url":"https://api.github.com/repos/manish/luigi/tags","blobs_url":"https://api.github.com/repos/manish/luigi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/manish/luigi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/manish/luigi/git/refs{/sha}","trees_url":"https://api.github.com/repos/manish/luigi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/manish/luigi/statuses/{sha}","languages_url":"https://api.github.com/repos/manish/luigi/languages","stargazers_url":"https://api.github.com/repos/manish/luigi/stargazers","contributors_url":"https://api.github.com/repos/manish/luigi/contributors","subscribers_url":"https://api.github.com/repos/manish/luigi/subscribers","subscription_url":"https://api.github.com/repos/manish/luigi/subscription","commits_url":"https://api.github.com/repos/manish/luigi/commits{/sha}","git_commits_url":"https://api.github.com/repos/manish/luigi/git/commits{/sha}","comments_url":"https://api.github.com/repos/manish/luigi/comments{/number}","issue_comment_url":"https://api.github.com/repos/manish/luigi/issues/comments{/number}","contents_url":"https://api.github.com/repos/manish/luigi/contents/{+path}","compare_url":"https://api.github.com/repos/manish/luigi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/manish/luigi/merges","archive_url":"https://api.github.com/repos/manish/luigi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/manish/luigi/downloads","issues_url":"https://api.github.com/repos/manish/luigi/issues{/number}","pulls_url":"https://api.github.com/repos/manish/luigi/pulls{/number}","milestones_url":"https://api.github.com/repos/manish/luigi/milestones{/number}","notifications_url":"https://api.github.com/repos/manish/luigi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/manish/luigi/labels{/name}","releases_url":"https://api.github.com/repos/manish/luigi/releases{/id}","deployments_url":"https://api.github.com/repos/manish/luigi/deployments","created_at":"2018-11-18T14:16:48Z","updated_at":"2018-11-18T14:16:51Z","pushed_at":"2018-11-20T00:26:51Z","git_url":"git://github.com/manish/luigi.git","ssh_url":"git@github.com:manish/luigi.git","clone_url":"https://github.com/manish/luigi.git","svn_url":"https://github.com/manish/luigi","homepage":null,"size":9736,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"spotify:master","ref":"master","sha":"7d3dae7ccc7a83842a1d1103e373a00629c00b67","user":{"login":"spotify","id":251374,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI1MTM3NA==","avatar_url":"https://avatars2.githubusercontent.com/u/251374?v=4","gravatar_id":"","url":"https://api.github.com/users/spotify","html_url":"https://github.com/spotify","followers_url":"https://api.github.com/users/spotify/followers","following_url":"https://api.github.com/users/spotify/following{/other_user}","gists_url":"https://api.github.com/users/spotify/gists{/gist_id}","starred_url":"https://api.github.com/users/spotify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spotify/subscriptions","organizations_url":"https://api.github.com/users/spotify/orgs","repos_url":"https://api.github.com/users/spotify/repos","events_url":"https://api.github.com/users/spotify/events{/privacy}","received_events_url":"https://api.github.com/users/spotify/received_events","type":"Organization","site_admin":false},"repo":{"id":5888353,"node_id":"MDEwOlJlcG9zaXRvcnk1ODg4MzUz","name":"luigi","full_name":"spotify/luigi","private":false,"owner":{"login":"spotify","id":251374,"node_id":"MDEyOk9yZ2FuaXphdGlvbjI1MTM3NA==","avatar_url":"https://avatars2.githubusercontent.com/u/251374?v=4","gravatar_id":"","url":"https://api.github.com/users/spotify","html_url":"https://github.com/spotify","followers_url":"https://api.github.com/users/spotify/followers","following_url":"https://api.github.com/users/spotify/following{/other_user}","gists_url":"https://api.github.com/users/spotify/gists{/gist_id}","starred_url":"https://api.github.com/users/spotify/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/spotify/subscriptions","organizations_url":"https://api.github.com/users/spotify/orgs","repos_url":"https://api.github.com/users/spotify/repos","events_url":"https://api.github.com/users/spotify/events{/privacy}","received_events_url":"https://api.github.com/users/spotify/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/spotify/luigi","description":"Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. ","fork":false,"url":"https://api.github.com/repos/spotify/luigi","forks_url":"https://api.github.com/repos/spotify/luigi/forks","keys_url":"https://api.github.com/repos/spotify/luigi/keys{/key_id}","collaborators_url":"https://api.github.com/repos/spotify/luigi/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/spotify/luigi/teams","hooks_url":"https://api.github.com/repos/spotify/luigi/hooks","issue_events_url":"https://api.github.com/repos/spotify/luigi/issues/events{/number}","events_url":"https://api.github.com/repos/spotify/luigi/events","assignees_url":"https://api.github.com/repos/spotify/luigi/assignees{/user}","branches_url":"https://api.github.com/repos/spotify/luigi/branches{/branch}","tags_url":"https://api.github.com/repos/spotify/luigi/tags","blobs_url":"https://api.github.com/repos/spotify/luigi/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/spotify/luigi/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/spotify/luigi/git/refs{/sha}","trees_url":"https://api.github.com/repos/spotify/luigi/git/trees{/sha}","statuses_url":"https://api.github.com/repos/spotify/luigi/statuses/{sha}","languages_url":"https://api.github.com/repos/spotify/luigi/languages","stargazers_url":"https://api.github.com/repos/spotify/luigi/stargazers","contributors_url":"https://api.github.com/repos/spotify/luigi/contributors","subscribers_url":"https://api.github.com/repos/spotify/luigi/subscribers","subscription_url":"https://api.github.com/repos/spotify/luigi/subscription","commits_url":"https://api.github.com/repos/spotify/luigi/commits{/sha}","git_commits_url":"https://api.github.com/repos/spotify/luigi/git/commits{/sha}","comments_url":"https://api.github.com/repos/spotify/luigi/comments{/number}","issue_comment_url":"https://api.github.com/repos/spotify/luigi/issues/comments{/number}","contents_url":"https://api.github.com/repos/spotify/luigi/contents/{+path}","compare_url":"https://api.github.com/repos/spotify/luigi/compare/{base}...{head}","merges_url":"https://api.github.com/repos/spotify/luigi/merges","archive_url":"https://api.github.com/repos/spotify/luigi/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/spotify/luigi/downloads","issues_url":"https://api.github.com/repos/spotify/luigi/issues{/number}","pulls_url":"https://api.github.com/repos/spotify/luigi/pulls{/number}","milestones_url":"https://api.github.com/repos/spotify/luigi/milestones{/number}","notifications_url":"https://api.github.com/repos/spotify/luigi/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/spotify/luigi/labels{/name}","releases_url":"https://api.github.com/repos/spotify/luigi/releases{/id}","deployments_url":"https://api.github.com/repos/spotify/luigi/deployments","created_at":"2012-09-20T15:06:38Z","updated_at":"2018-11-20T10:24:07Z","pushed_at":"2018-11-20T00:26:56Z","git_url":"git://github.com/spotify/luigi.git","ssh_url":"git@github.com:spotify/luigi.git","clone_url":"https://github.com/spotify/luigi.git","svn_url":"https://github.com/spotify/luigi","homepage":null,"size":9705,"stargazers_count":10456,"watchers_count":10456,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":1781,"mirror_url":null,"archived":false,"open_issues_count":84,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":1781,"open_issues":84,"watchers":10456,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2585"},"html":{"href":"https://github.com/spotify/luigi/pull/2585"},"issue":{"href":"https://api.github.com/repos/spotify/luigi/issues/2585"},"comments":{"href":"https://api.github.com/repos/spotify/luigi/issues/2585/comments"},"review_comments":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2585/comments"},"review_comment":{"href":"https://api.github.com/repos/spotify/luigi/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/spotify/luigi/pulls/2585/commits"},"statuses":{"href":"https://api.github.com/repos/spotify/luigi/statuses/40b69394e6093ff44261aec622dcab10000e7079"}},"author_association":"NONE"}}
{"actor":{"display_login":"dlstadther"}}