{"action":"created","comment":{"url":"https://api.github.com/repos/ngageoint/scale/pulls/comments/72680081","id":72680081,"diff_hunk":"@@ -4,116 +4,193 @@\n Strike\n ========================================================================================================================\n \n-Strike is the name of Scale's directory watching capability. A Strike process monitors a given directory on a Network\n-File System (NFS) for new files and ingests those files into a Scale workspace if they meet certain criteria. When a new\n-Strike process is created, a new job is created and executed to run the new process. A Strike process contains\n-configuration specifying the details of the NFS directory to monitor and how to ingest and store the files that appear\n-in the directory.\n+Strike is the name of Scale's system job that monitors and handles incoming source data files that are ingested into\n+Scale. A Strike job monitors a given workspace for new files. Scale administrators will want to create a Strike job for\n+each data feed that will be processed by Scale.\n \n-**Example Strike configuration:**\n-\n-.. code-block:: javascript\n-\n-   {\n-      \"version\": \"1.0\",\n-      \"mount\": \"host:/my/path\",\n-      \"transfer_suffix\": \"_tmp\",\n-      \"files_to_ingest\": [\n-          {\n-              \"filename_regex\": \".*h5\",\n-              \"data_types\": [],\n-              \"workspace_path\": \"/wrksp/path\",\n-              \"workspace_name\": \"raw\"\n-          }\n-      ]\n-   }\n-\n-The *mount* field specifies the NFS host and path that should be mounted in order to access the directory to be\n-monitored. The *transfer_suffix* field defines a suffix that is used on the file names to indicate that they are still\n-transferring and have not yet finished being copied into the monitored directory. The *files_to_ingest* value is a list\n-detailing the different files to ingest and how to ingest them. The *filename_regex* field defines a regular expression\n-to check against the names of newly copied files. If the expression matches a newly copied file name in the directory,\n-that file is ingested according to the other fields in the JSON object. The *data_types* field is list of strings. Any\n-file that matches the corresponding regular expression will have the data type strings \"tagged\" with the file. The data\n-type tags are used to categorize files and control which jobs and recipes they go to. The *workspace_path* specifies a\n-relative path within the workspace where each file will be stored. Three additional and dynamically named directories,\n-for the current year, month, and day, will be appended to the *workspace_path* value automatically by the Scale system\n-when a file is ingested (i.e. workspace_path/YYYY/MM/DD). The *workspace_name* field is the system name of the unique\n-workspace that should ingest the file. To see all of the options for a Strike process's configuration, please refer to\n-the Strike Configuration Specification below.\n+When a new file is copied into the monitored workspace, its file name is checked against a number rules using regular","path":"scale/docs/architecture/strike.rst","position":44,"original_position":44,"commit_id":"d2a5de2568b47e8b4abd8e0cad67cb8bd8a6d4e2","original_commit_id":"d2a5de2568b47e8b4abd8e0cad67cb8bd8a6d4e2","user":{"login":"tclarke","id":8740,"avatar_url":"https://avatars.githubusercontent.com/u/8740?v=3","gravatar_id":"","url":"https://api.github.com/users/tclarke","html_url":"https://github.com/tclarke","followers_url":"https://api.github.com/users/tclarke/followers","following_url":"https://api.github.com/users/tclarke/following{/other_user}","gists_url":"https://api.github.com/users/tclarke/gists{/gist_id}","starred_url":"https://api.github.com/users/tclarke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tclarke/subscriptions","organizations_url":"https://api.github.com/users/tclarke/orgs","repos_url":"https://api.github.com/users/tclarke/repos","events_url":"https://api.github.com/users/tclarke/events{/privacy}","received_events_url":"https://api.github.com/users/tclarke/received_events","type":"User","site_admin":false},"body":"Typo? Should probably be \"a number of rules\"","created_at":"2016-07-28T18:50:40Z","updated_at":"2016-07-28T18:50:40Z","html_url":"https://github.com/ngageoint/scale/pull/398#discussion_r72680081","pull_request_url":"https://api.github.com/repos/ngageoint/scale/pulls/398","_links":{"self":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/comments/72680081"},"html":{"href":"https://github.com/ngageoint/scale/pull/398#discussion_r72680081"},"pull_request":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/398"}}},"pull_request":{"url":"https://api.github.com/repos/ngageoint/scale/pulls/398","id":79245976,"html_url":"https://github.com/ngageoint/scale/pull/398","diff_url":"https://github.com/ngageoint/scale/pull/398.diff","patch_url":"https://github.com/ngageoint/scale/pull/398.patch","issue_url":"https://api.github.com/repos/ngageoint/scale/issues/398","number":398,"state":"open","locked":false,"title":"Issue 19 - Strike Refactor","user":{"login":"dfaller","id":15897378,"avatar_url":"https://avatars.githubusercontent.com/u/15897378?v=3","gravatar_id":"","url":"https://api.github.com/users/dfaller","html_url":"https://github.com/dfaller","followers_url":"https://api.github.com/users/dfaller/followers","following_url":"https://api.github.com/users/dfaller/following{/other_user}","gists_url":"https://api.github.com/users/dfaller/gists{/gist_id}","starred_url":"https://api.github.com/users/dfaller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfaller/subscriptions","organizations_url":"https://api.github.com/users/dfaller/orgs","repos_url":"https://api.github.com/users/dfaller/repos","events_url":"https://api.github.com/users/dfaller/events{/privacy}","received_events_url":"https://api.github.com/users/dfaller/received_events","type":"User","site_admin":false},"body":"Strike was refactored to a more abstract/general solution that does not require NFS.\r\n\r\nThere is now a \"monitor\" base class that abstracts how to watch a workspace for incoming files. The monitor and workspace details are set in the Strike configuration. The Strike configuration allows the previous file name regex matching and allows registering the new files in the monitored workspace or moving the files to a new workspace. The Strike configuration has been updated to schema version 2.0 for these changes.\r\n\r\nTwo monitors have been implemented so far:\r\n - our original directory watching monitor\r\n - an S3 monitor that polls an SQS queue for object creation events in an S3 bucket\r\n\r\nThis update removes all NFS mounting done directly by Scale and removes the need for any Scale job containers to run as privileged.","created_at":"2016-07-28T18:35:48Z","updated_at":"2016-07-28T18:50:40Z","closed_at":null,"merged_at":null,"merge_commit_sha":"def210842d1b739aacedeb4af601e8b50265b042","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/ngageoint/scale/milestones/13","html_url":"https://github.com/ngageoint/scale/milestones/Sprint%2007-18-2016","labels_url":"https://api.github.com/repos/ngageoint/scale/milestones/13/labels","id":1835127,"number":13,"title":"Sprint 07-18-2016","description":"Sprint backlog for 07-18-2016 to 07-29-2016","creator":{"login":"dfaller","id":15897378,"avatar_url":"https://avatars.githubusercontent.com/u/15897378?v=3","gravatar_id":"","url":"https://api.github.com/users/dfaller","html_url":"https://github.com/dfaller","followers_url":"https://api.github.com/users/dfaller/followers","following_url":"https://api.github.com/users/dfaller/following{/other_user}","gists_url":"https://api.github.com/users/dfaller/gists{/gist_id}","starred_url":"https://api.github.com/users/dfaller/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfaller/subscriptions","organizations_url":"https://api.github.com/users/dfaller/orgs","repos_url":"https://api.github.com/users/dfaller/repos","events_url":"https://api.github.com/users/dfaller/events{/privacy}","received_events_url":"https://api.github.com/users/dfaller/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":6,"state":"open","created_at":"2016-06-17T18:05:41Z","updated_at":"2016-07-28T18:40:13Z","due_on":"2016-07-29T04:00:00Z","closed_at":null},"commits_url":"https://api.github.com/repos/ngageoint/scale/pulls/398/commits","review_comments_url":"https://api.github.com/repos/ngageoint/scale/pulls/398/comments","review_comment_url":"https://api.github.com/repos/ngageoint/scale/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ngageoint/scale/issues/398/comments","statuses_url":"https://api.github.com/repos/ngageoint/scale/statuses/d2a5de2568b47e8b4abd8e0cad67cb8bd8a6d4e2","head":{"label":"ngageoint:issue-19","ref":"issue-19","sha":"d2a5de2568b47e8b4abd8e0cad67cb8bd8a6d4e2","user":{"login":"ngageoint","id":7175944,"avatar_url":"https://avatars.githubusercontent.com/u/7175944?v=3","gravatar_id":"","url":"https://api.github.com/users/ngageoint","html_url":"https://github.com/ngageoint","followers_url":"https://api.github.com/users/ngageoint/followers","following_url":"https://api.github.com/users/ngageoint/following{/other_user}","gists_url":"https://api.github.com/users/ngageoint/gists{/gist_id}","starred_url":"https://api.github.com/users/ngageoint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngageoint/subscriptions","organizations_url":"https://api.github.com/users/ngageoint/orgs","repos_url":"https://api.github.com/users/ngageoint/repos","events_url":"https://api.github.com/users/ngageoint/events{/privacy}","received_events_url":"https://api.github.com/users/ngageoint/received_events","type":"Organization","site_admin":false},"repo":{"id":46419785,"name":"scale","full_name":"ngageoint/scale","owner":{"login":"ngageoint","id":7175944,"avatar_url":"https://avatars.githubusercontent.com/u/7175944?v=3","gravatar_id":"","url":"https://api.github.com/users/ngageoint","html_url":"https://github.com/ngageoint","followers_url":"https://api.github.com/users/ngageoint/followers","following_url":"https://api.github.com/users/ngageoint/following{/other_user}","gists_url":"https://api.github.com/users/ngageoint/gists{/gist_id}","starred_url":"https://api.github.com/users/ngageoint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngageoint/subscriptions","organizations_url":"https://api.github.com/users/ngageoint/orgs","repos_url":"https://api.github.com/users/ngageoint/repos","events_url":"https://api.github.com/users/ngageoint/events{/privacy}","received_events_url":"https://api.github.com/users/ngageoint/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ngageoint/scale","description":"Containerized processing framework for algorithms focused on remote sensing","fork":false,"url":"https://api.github.com/repos/ngageoint/scale","forks_url":"https://api.github.com/repos/ngageoint/scale/forks","keys_url":"https://api.github.com/repos/ngageoint/scale/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ngageoint/scale/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ngageoint/scale/teams","hooks_url":"https://api.github.com/repos/ngageoint/scale/hooks","issue_events_url":"https://api.github.com/repos/ngageoint/scale/issues/events{/number}","events_url":"https://api.github.com/repos/ngageoint/scale/events","assignees_url":"https://api.github.com/repos/ngageoint/scale/assignees{/user}","branches_url":"https://api.github.com/repos/ngageoint/scale/branches{/branch}","tags_url":"https://api.github.com/repos/ngageoint/scale/tags","blobs_url":"https://api.github.com/repos/ngageoint/scale/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ngageoint/scale/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ngageoint/scale/git/refs{/sha}","trees_url":"https://api.github.com/repos/ngageoint/scale/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ngageoint/scale/statuses/{sha}","languages_url":"https://api.github.com/repos/ngageoint/scale/languages","stargazers_url":"https://api.github.com/repos/ngageoint/scale/stargazers","contributors_url":"https://api.github.com/repos/ngageoint/scale/contributors","subscribers_url":"https://api.github.com/repos/ngageoint/scale/subscribers","subscription_url":"https://api.github.com/repos/ngageoint/scale/subscription","commits_url":"https://api.github.com/repos/ngageoint/scale/commits{/sha}","git_commits_url":"https://api.github.com/repos/ngageoint/scale/git/commits{/sha}","comments_url":"https://api.github.com/repos/ngageoint/scale/comments{/number}","issue_comment_url":"https://api.github.com/repos/ngageoint/scale/issues/comments{/number}","contents_url":"https://api.github.com/repos/ngageoint/scale/contents/{+path}","compare_url":"https://api.github.com/repos/ngageoint/scale/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ngageoint/scale/merges","archive_url":"https://api.github.com/repos/ngageoint/scale/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ngageoint/scale/downloads","issues_url":"https://api.github.com/repos/ngageoint/scale/issues{/number}","pulls_url":"https://api.github.com/repos/ngageoint/scale/pulls{/number}","milestones_url":"https://api.github.com/repos/ngageoint/scale/milestones{/number}","notifications_url":"https://api.github.com/repos/ngageoint/scale/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ngageoint/scale/labels{/name}","releases_url":"https://api.github.com/repos/ngageoint/scale/releases{/id}","deployments_url":"https://api.github.com/repos/ngageoint/scale/deployments","created_at":"2015-11-18T13:16:09Z","updated_at":"2016-07-15T14:39:48Z","pushed_at":"2016-07-28T18:45:09Z","git_url":"git://github.com/ngageoint/scale.git","ssh_url":"git@github.com:ngageoint/scale.git","clone_url":"https://github.com/ngageoint/scale.git","svn_url":"https://github.com/ngageoint/scale","homepage":"http://ngageoint.github.io/scale/","size":130845,"stargazers_count":35,"watchers_count":35,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":13,"mirror_url":null,"open_issues_count":128,"forks":13,"open_issues":128,"watchers":35,"default_branch":"master"}},"base":{"label":"ngageoint:master","ref":"master","sha":"d44985728c287711b9e1a6a4cc8b95347d0bb417","user":{"login":"ngageoint","id":7175944,"avatar_url":"https://avatars.githubusercontent.com/u/7175944?v=3","gravatar_id":"","url":"https://api.github.com/users/ngageoint","html_url":"https://github.com/ngageoint","followers_url":"https://api.github.com/users/ngageoint/followers","following_url":"https://api.github.com/users/ngageoint/following{/other_user}","gists_url":"https://api.github.com/users/ngageoint/gists{/gist_id}","starred_url":"https://api.github.com/users/ngageoint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngageoint/subscriptions","organizations_url":"https://api.github.com/users/ngageoint/orgs","repos_url":"https://api.github.com/users/ngageoint/repos","events_url":"https://api.github.com/users/ngageoint/events{/privacy}","received_events_url":"https://api.github.com/users/ngageoint/received_events","type":"Organization","site_admin":false},"repo":{"id":46419785,"name":"scale","full_name":"ngageoint/scale","owner":{"login":"ngageoint","id":7175944,"avatar_url":"https://avatars.githubusercontent.com/u/7175944?v=3","gravatar_id":"","url":"https://api.github.com/users/ngageoint","html_url":"https://github.com/ngageoint","followers_url":"https://api.github.com/users/ngageoint/followers","following_url":"https://api.github.com/users/ngageoint/following{/other_user}","gists_url":"https://api.github.com/users/ngageoint/gists{/gist_id}","starred_url":"https://api.github.com/users/ngageoint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ngageoint/subscriptions","organizations_url":"https://api.github.com/users/ngageoint/orgs","repos_url":"https://api.github.com/users/ngageoint/repos","events_url":"https://api.github.com/users/ngageoint/events{/privacy}","received_events_url":"https://api.github.com/users/ngageoint/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ngageoint/scale","description":"Containerized processing framework for algorithms focused on remote sensing","fork":false,"url":"https://api.github.com/repos/ngageoint/scale","forks_url":"https://api.github.com/repos/ngageoint/scale/forks","keys_url":"https://api.github.com/repos/ngageoint/scale/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ngageoint/scale/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ngageoint/scale/teams","hooks_url":"https://api.github.com/repos/ngageoint/scale/hooks","issue_events_url":"https://api.github.com/repos/ngageoint/scale/issues/events{/number}","events_url":"https://api.github.com/repos/ngageoint/scale/events","assignees_url":"https://api.github.com/repos/ngageoint/scale/assignees{/user}","branches_url":"https://api.github.com/repos/ngageoint/scale/branches{/branch}","tags_url":"https://api.github.com/repos/ngageoint/scale/tags","blobs_url":"https://api.github.com/repos/ngageoint/scale/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ngageoint/scale/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ngageoint/scale/git/refs{/sha}","trees_url":"https://api.github.com/repos/ngageoint/scale/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ngageoint/scale/statuses/{sha}","languages_url":"https://api.github.com/repos/ngageoint/scale/languages","stargazers_url":"https://api.github.com/repos/ngageoint/scale/stargazers","contributors_url":"https://api.github.com/repos/ngageoint/scale/contributors","subscribers_url":"https://api.github.com/repos/ngageoint/scale/subscribers","subscription_url":"https://api.github.com/repos/ngageoint/scale/subscription","commits_url":"https://api.github.com/repos/ngageoint/scale/commits{/sha}","git_commits_url":"https://api.github.com/repos/ngageoint/scale/git/commits{/sha}","comments_url":"https://api.github.com/repos/ngageoint/scale/comments{/number}","issue_comment_url":"https://api.github.com/repos/ngageoint/scale/issues/comments{/number}","contents_url":"https://api.github.com/repos/ngageoint/scale/contents/{+path}","compare_url":"https://api.github.com/repos/ngageoint/scale/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ngageoint/scale/merges","archive_url":"https://api.github.com/repos/ngageoint/scale/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ngageoint/scale/downloads","issues_url":"https://api.github.com/repos/ngageoint/scale/issues{/number}","pulls_url":"https://api.github.com/repos/ngageoint/scale/pulls{/number}","milestones_url":"https://api.github.com/repos/ngageoint/scale/milestones{/number}","notifications_url":"https://api.github.com/repos/ngageoint/scale/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ngageoint/scale/labels{/name}","releases_url":"https://api.github.com/repos/ngageoint/scale/releases{/id}","deployments_url":"https://api.github.com/repos/ngageoint/scale/deployments","created_at":"2015-11-18T13:16:09Z","updated_at":"2016-07-15T14:39:48Z","pushed_at":"2016-07-28T18:45:09Z","git_url":"git://github.com/ngageoint/scale.git","ssh_url":"git@github.com:ngageoint/scale.git","clone_url":"https://github.com/ngageoint/scale.git","svn_url":"https://github.com/ngageoint/scale","homepage":"http://ngageoint.github.io/scale/","size":130845,"stargazers_count":35,"watchers_count":35,"language":"JavaScript","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":13,"mirror_url":null,"open_issues_count":128,"forks":13,"open_issues":128,"watchers":35,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/398"},"html":{"href":"https://github.com/ngageoint/scale/pull/398"},"issue":{"href":"https://api.github.com/repos/ngageoint/scale/issues/398"},"comments":{"href":"https://api.github.com/repos/ngageoint/scale/issues/398/comments"},"review_comments":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/398/comments"},"review_comment":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ngageoint/scale/pulls/398/commits"},"statuses":{"href":"https://api.github.com/repos/ngageoint/scale/statuses/d2a5de2568b47e8b4abd8e0cad67cb8bd8a6d4e2"}}}}
{"actor":{"display_login":"tclarke"}}