{"action":"created","comment":{"url":"https://api.github.com/repos/openaire/iis/pulls/comments/48717745","id":48717745,"diff_hunk":"@@ -0,0 +1,111 @@\n+# Introduction\n+\n+**Abstract**. This document describes the conventions that we use when defining Oozie workflows.\n+\n+**Data processing pipeline**. Each Oozie workflow describes an imperative implementation of a higher-level declarative data processing pipeline. The pipeline representation follows **\"Pipes an filters\"** pattern from the book by Gregor Hohpe, Bobby Woolf: \"Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions\", Addison-Wesley, 2003. The pipeline consists of workflow **nodes**, known as \"actions\" in Oozie parlance, that consume data arriving on their **input ports** and produce data to their **output ports**. Each port has a name and a type (defined by Avro schema). Output port of a producer is connected to the input port of a consumer; in order for this connection to work properly, the types of these ports have to \"match\". The data passed between the nodes is called **data store** and in practice this is simply a directory containing one or more [Avro Object Container Files][]. This document describes mostly how this high-level model is implemented in practice in Oozie workflows used in the project.\n+\n+**Data processing language**. Note that the original idea was that Oozie workflows conforming to the conventions described in this document would be automatically generated from a description of data processing pipeline. However, implementation of such description language has not been a priority and as of 2015-12-22 probably it won't be done. You can read a description of this language in a [Google doc][].\n+\n+[Google doc]: https://docs.google.com/document/d/10dfvUV4V0bT2doUTgMi8MUMVkKNATorEg4LCH7RqZaE/edit?usp=sharing\n+\n+# General naming conventions\n+\n+Use `words_separated_by_underscores` to name actions, parameters, etc. instead of, e.g. `camelCase`.\n+\n+# Ports\n+\n+## Standard ports\n+\n+### All Oozie actions except Java actions\n+\n+#### Names\n+\n+For all actions except for Java actions, **names** of properties that correspond to ports have to be prefixed properly:\n+\n+- input ports: `input_`, e.g. `input_document`;\n+- output_ports: `ouput_`, e.g. `output_person`.\n+\n+One can also use a property named simply `input` or `output`, which means that the name of the port is simply `input` or `output`, respectively. In such case, no other input or output ports, should be defined for given node.\n+\n+These rules also apply to `parameters` section in a definition of a workflow, i.e. this section might contain definitions of input and output ports of the workflow.\n+\n+#### Schemas\n+\n+In case of properties correspoding to **schemas of ports**, their names should be prefixed with `schema_input_` or `schema_output_`, depending on the type of the port. The value should be a fully qualified name of the schema. For example, take a look at the following (property name, path) pair corresponding to a property:\n+\n+    (schema_input_text, eu.dnetlib.iis.metadataextraction.schemas.DocumentText)\n+\n+\n+The properties corresponding to schemas of ports should be placed next to the properties corresponding to the ports themselves, e.g.\n+\n+    <param>input_metadata=${transformerCitationmatchingInputMetadata}</param>\n+    <param>schema_input_metadata=eu.dnetlib.iis.transformers.metadatamerger.schemas.ExtractedDocumentMetadataMergedWithOriginal</param>\n+\n+    <param>input_person=${transformerCitationmatchingInputPerson}</param>\n+    <param>schema_input_person=eu.dnetlib.iis.importer.schemas.Person</param>\n+\n+\n+Property named `schema_input` or `schema_output` corresponds to `input` or `output` port property, respectively.\n+\n+Property named `schema` means a schema corresponding to all input and output ports.\n+\n+### Java actions\n+\n+In case of Java actions, ports are given as command line arguments and prefixed properly:\n+\n+- input ports: `-I`, e.g. `-Idocument`;\n+- output ports: `-O`, e.g., `Odocument`.\n+\n+\n+## Root ports\n+\n+There is an alternative way of defining nodes and paths to data stores - so-called **root ports**. A single port is defined through two properties, one being the \"root\" and the other one being the \"leaf\". For example, in the following (property name, path) pairs of properties\n+\n+    (output_person_root, X), (person_output_name_age, Y)\n+\n+the former entry is the root while the latter is the leaf. This corresponds to the following (property name, path) pair from the standard port notation: \n+\n+    (output_person_age,  X/Y)\n+\n+This notation is necessary when using Java MapReduce workflow nodes with multiple outputs.\n+\n+# Workflow name\n+\n+The name of the workflow should correspond to the path leading to this workflow in the code repository after removing constant prefix `eu/dnetlib/iis` and removing constant suffix `oozie_app/workflow.xml`. For example, if the path to the workflow is `eu/dnetlib/iis/transformers/citationmatching/oozie_app/workflow.xml`, the entry in the Oozie XML should be\n+\n+    <workflow-app xmlns=\"uri:oozie:workflow:0.2\" name=\"transformers_citationmmatching\">.\n+\n+where the slashes in the path are replaced with underscores (`_`). This approach assures that the names of workflows are both unique and quite short.\n+\n+In case of a test workflow, its name should be created in the same way as the normal workflow but it should be prefixed additionally with `test-`, e.g. the test workflow placed at `eu/dnetlib/iis/transformers/citationmatching/sampledataproducer/oozie_app/workflow.xml` should be called `test-transformers_citationmatching_sampledataproducer`.\n+\n+# Special types of Oozie actions\n+\n+We came to use some conventions related to names of actions:\n+\n+- Prefixed with `skip-` - a node that is used to generate empty data store as an alternative to some \"real\" processing.\n+- Prefixed with `transformers_` or `transformer_` - a node that converts the data stores to a format expected by their consumers.\n+- Suffixed with `_collapser` - a node that uses \"collapser\" infrastructure to de-duplicate records from data store.\n+- Suffixed with `_union` - a node that executes a \"union\" operation on data stores.\n+\n+# Directory structure corresponding to Oozie action\n+\n+The **names of Oozie actions** should be valid HDFS file/directory names. This is because a directory named as the action is created in HDFS for most actions.\n+\n+By default, all data produced by a given workflow node is stored in a directory having the same name as the name of the workflow node. For example, for a node \"sample_processor\" that produces two output data stores with names \"processed_person\", \"processed_document\" we have the following **directory structure**:\n+\n+    sample_processor/\n+    |\n+    |- working_dir/\n+    |- processed_person/\n+\t\\- processed_document/\n+\n+\n+Here:\n+\n+- `workingDir` corresponds to a directory where the node can store results of its intermediate computations, temporary files etc. In case of this node being a subworkflow, this is the place where directories of its child workflow nodes are stored. This directory is named `working_dir` in the file system.\n+- The \"sample_processor\" directory is recreated in the `<prepare>` section of the Oozie's action. This is to ensure that the workflow can be properly restarted in case of transient errors and possibly stopped and resumed (I'm not sure about the latter two -- this has to be checked in practice).","path":"iis-workflows/docs/workflow_conventions.markdown","position":107,"original_position":107,"commit_id":"f38b3bbde0765db9f3cf2af1e2b689a2f6c214a6","original_commit_id":"f38b3bbde0765db9f3cf2af1e2b689a2f6c214a6","user":{"login":"marekhorst","id":11537389,"avatar_url":"https://avatars.githubusercontent.com/u/11537389?v=3","gravatar_id":"","url":"https://api.github.com/users/marekhorst","html_url":"https://github.com/marekhorst","followers_url":"https://api.github.com/users/marekhorst/followers","following_url":"https://api.github.com/users/marekhorst/following{/other_user}","gists_url":"https://api.github.com/users/marekhorst/gists{/gist_id}","starred_url":"https://api.github.com/users/marekhorst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marekhorst/subscriptions","organizations_url":"https://api.github.com/users/marekhorst/orgs","repos_url":"https://api.github.com/users/marekhorst/repos","events_url":"https://api.github.com/users/marekhorst/events{/privacy}","received_events_url":"https://api.github.com/users/marekhorst/received_events","type":"User","site_admin":false},"body":"Mainly because output directory may be defined by uber-workfow outside given module directory structure.","created_at":"2016-01-04T10:17:25Z","updated_at":"2016-01-04T10:17:25Z","html_url":"https://github.com/openaire/iis/pull/159#discussion_r48717745","pull_request_url":"https://api.github.com/repos/openaire/iis/pulls/159","_links":{"self":{"href":"https://api.github.com/repos/openaire/iis/pulls/comments/48717745"},"html":{"href":"https://github.com/openaire/iis/pull/159#discussion_r48717745"},"pull_request":{"href":"https://api.github.com/repos/openaire/iis/pulls/159"}}},"pull_request":{"url":"https://api.github.com/repos/openaire/iis/pulls/159","id":54405804,"html_url":"https://github.com/openaire/iis/pull/159","diff_url":"https://github.com/openaire/iis/pull/159.diff","patch_url":"https://github.com/openaire/iis/pull/159.patch","issue_url":"https://api.github.com/repos/openaire/iis/issues/159","number":159,"state":"open","locked":false,"title":"Closes #158: describe Oozie workflow conventions","user":{"login":"mkobos","id":1044091,"avatar_url":"https://avatars.githubusercontent.com/u/1044091?v=3","gravatar_id":"","url":"https://api.github.com/users/mkobos","html_url":"https://github.com/mkobos","followers_url":"https://api.github.com/users/mkobos/followers","following_url":"https://api.github.com/users/mkobos/following{/other_user}","gists_url":"https://api.github.com/users/mkobos/gists{/gist_id}","starred_url":"https://api.github.com/users/mkobos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkobos/subscriptions","organizations_url":"https://api.github.com/users/mkobos/orgs","repos_url":"https://api.github.com/users/mkobos/repos","events_url":"https://api.github.com/users/mkobos/events{/privacy}","received_events_url":"https://api.github.com/users/mkobos/received_events","type":"User","site_admin":false},"body":"A new file describing the conventions. At the same time, I removed one related by outdated file.","created_at":"2015-12-22T19:30:13Z","updated_at":"2016-01-04T10:17:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ecb3e61d0888fd8daee34a9bd4d52e8298c7fd8d","assignee":{"login":"mkobos","id":1044091,"avatar_url":"https://avatars.githubusercontent.com/u/1044091?v=3","gravatar_id":"","url":"https://api.github.com/users/mkobos","html_url":"https://github.com/mkobos","followers_url":"https://api.github.com/users/mkobos/followers","following_url":"https://api.github.com/users/mkobos/following{/other_user}","gists_url":"https://api.github.com/users/mkobos/gists{/gist_id}","starred_url":"https://api.github.com/users/mkobos/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mkobos/subscriptions","organizations_url":"https://api.github.com/users/mkobos/orgs","repos_url":"https://api.github.com/users/mkobos/repos","events_url":"https://api.github.com/users/mkobos/events{/privacy}","received_events_url":"https://api.github.com/users/mkobos/received_events","type":"User","site_admin":false},"milestone":null,"commits_url":"https://api.github.com/repos/openaire/iis/pulls/159/commits","review_comments_url":"https://api.github.com/repos/openaire/iis/pulls/159/comments","review_comment_url":"https://api.github.com/repos/openaire/iis/pulls/comments{/number}","comments_url":"https://api.github.com/repos/openaire/iis/issues/159/comments","statuses_url":"https://api.github.com/repos/openaire/iis/statuses/f38b3bbde0765db9f3cf2af1e2b689a2f6c214a6","head":{"label":"openaire:mkobos_workflow_conventions","ref":"mkobos_workflow_conventions","sha":"f38b3bbde0765db9f3cf2af1e2b689a2f6c214a6","user":{"login":"openaire","id":10127536,"avatar_url":"https://avatars.githubusercontent.com/u/10127536?v=3","gravatar_id":"","url":"https://api.github.com/users/openaire","html_url":"https://github.com/openaire","followers_url":"https://api.github.com/users/openaire/followers","following_url":"https://api.github.com/users/openaire/following{/other_user}","gists_url":"https://api.github.com/users/openaire/gists{/gist_id}","starred_url":"https://api.github.com/users/openaire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openaire/subscriptions","organizations_url":"https://api.github.com/users/openaire/orgs","repos_url":"https://api.github.com/users/openaire/repos","events_url":"https://api.github.com/users/openaire/events{/privacy}","received_events_url":"https://api.github.com/users/openaire/received_events","type":"Organization","site_admin":false},"repo":{"id":42290369,"name":"iis","full_name":"openaire/iis","owner":{"login":"openaire","id":10127536,"avatar_url":"https://avatars.githubusercontent.com/u/10127536?v=3","gravatar_id":"","url":"https://api.github.com/users/openaire","html_url":"https://github.com/openaire","followers_url":"https://api.github.com/users/openaire/followers","following_url":"https://api.github.com/users/openaire/following{/other_user}","gists_url":"https://api.github.com/users/openaire/gists{/gist_id}","starred_url":"https://api.github.com/users/openaire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openaire/subscriptions","organizations_url":"https://api.github.com/users/openaire/orgs","repos_url":"https://api.github.com/users/openaire/repos","events_url":"https://api.github.com/users/openaire/events{/privacy}","received_events_url":"https://api.github.com/users/openaire/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/openaire/iis","description":"Information Inference Service of the OpenAIRE system","fork":false,"url":"https://api.github.com/repos/openaire/iis","forks_url":"https://api.github.com/repos/openaire/iis/forks","keys_url":"https://api.github.com/repos/openaire/iis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/openaire/iis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/openaire/iis/teams","hooks_url":"https://api.github.com/repos/openaire/iis/hooks","issue_events_url":"https://api.github.com/repos/openaire/iis/issues/events{/number}","events_url":"https://api.github.com/repos/openaire/iis/events","assignees_url":"https://api.github.com/repos/openaire/iis/assignees{/user}","branches_url":"https://api.github.com/repos/openaire/iis/branches{/branch}","tags_url":"https://api.github.com/repos/openaire/iis/tags","blobs_url":"https://api.github.com/repos/openaire/iis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/openaire/iis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/openaire/iis/git/refs{/sha}","trees_url":"https://api.github.com/repos/openaire/iis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/openaire/iis/statuses/{sha}","languages_url":"https://api.github.com/repos/openaire/iis/languages","stargazers_url":"https://api.github.com/repos/openaire/iis/stargazers","contributors_url":"https://api.github.com/repos/openaire/iis/contributors","subscribers_url":"https://api.github.com/repos/openaire/iis/subscribers","subscription_url":"https://api.github.com/repos/openaire/iis/subscription","commits_url":"https://api.github.com/repos/openaire/iis/commits{/sha}","git_commits_url":"https://api.github.com/repos/openaire/iis/git/commits{/sha}","comments_url":"https://api.github.com/repos/openaire/iis/comments{/number}","issue_comment_url":"https://api.github.com/repos/openaire/iis/issues/comments{/number}","contents_url":"https://api.github.com/repos/openaire/iis/contents/{+path}","compare_url":"https://api.github.com/repos/openaire/iis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/openaire/iis/merges","archive_url":"https://api.github.com/repos/openaire/iis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/openaire/iis/downloads","issues_url":"https://api.github.com/repos/openaire/iis/issues{/number}","pulls_url":"https://api.github.com/repos/openaire/iis/pulls{/number}","milestones_url":"https://api.github.com/repos/openaire/iis/milestones{/number}","notifications_url":"https://api.github.com/repos/openaire/iis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/openaire/iis/labels{/name}","releases_url":"https://api.github.com/repos/openaire/iis/releases{/id}","created_at":"2015-09-11T05:55:08Z","updated_at":"2015-12-14T12:57:35Z","pushed_at":"2015-12-28T20:18:30Z","git_url":"git://github.com/openaire/iis.git","ssh_url":"git@github.com:openaire/iis.git","clone_url":"https://github.com/openaire/iis.git","svn_url":"https://github.com/openaire/iis","homepage":"","size":35912,"stargazers_count":5,"watchers_count":5,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":19,"forks":1,"open_issues":19,"watchers":5,"default_branch":"master"}},"base":{"label":"openaire:master","ref":"master","sha":"f0bd98a14d83adfa7d958c92477fd2a82e3f207b","user":{"login":"openaire","id":10127536,"avatar_url":"https://avatars.githubusercontent.com/u/10127536?v=3","gravatar_id":"","url":"https://api.github.com/users/openaire","html_url":"https://github.com/openaire","followers_url":"https://api.github.com/users/openaire/followers","following_url":"https://api.github.com/users/openaire/following{/other_user}","gists_url":"https://api.github.com/users/openaire/gists{/gist_id}","starred_url":"https://api.github.com/users/openaire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openaire/subscriptions","organizations_url":"https://api.github.com/users/openaire/orgs","repos_url":"https://api.github.com/users/openaire/repos","events_url":"https://api.github.com/users/openaire/events{/privacy}","received_events_url":"https://api.github.com/users/openaire/received_events","type":"Organization","site_admin":false},"repo":{"id":42290369,"name":"iis","full_name":"openaire/iis","owner":{"login":"openaire","id":10127536,"avatar_url":"https://avatars.githubusercontent.com/u/10127536?v=3","gravatar_id":"","url":"https://api.github.com/users/openaire","html_url":"https://github.com/openaire","followers_url":"https://api.github.com/users/openaire/followers","following_url":"https://api.github.com/users/openaire/following{/other_user}","gists_url":"https://api.github.com/users/openaire/gists{/gist_id}","starred_url":"https://api.github.com/users/openaire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/openaire/subscriptions","organizations_url":"https://api.github.com/users/openaire/orgs","repos_url":"https://api.github.com/users/openaire/repos","events_url":"https://api.github.com/users/openaire/events{/privacy}","received_events_url":"https://api.github.com/users/openaire/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/openaire/iis","description":"Information Inference Service of the OpenAIRE system","fork":false,"url":"https://api.github.com/repos/openaire/iis","forks_url":"https://api.github.com/repos/openaire/iis/forks","keys_url":"https://api.github.com/repos/openaire/iis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/openaire/iis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/openaire/iis/teams","hooks_url":"https://api.github.com/repos/openaire/iis/hooks","issue_events_url":"https://api.github.com/repos/openaire/iis/issues/events{/number}","events_url":"https://api.github.com/repos/openaire/iis/events","assignees_url":"https://api.github.com/repos/openaire/iis/assignees{/user}","branches_url":"https://api.github.com/repos/openaire/iis/branches{/branch}","tags_url":"https://api.github.com/repos/openaire/iis/tags","blobs_url":"https://api.github.com/repos/openaire/iis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/openaire/iis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/openaire/iis/git/refs{/sha}","trees_url":"https://api.github.com/repos/openaire/iis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/openaire/iis/statuses/{sha}","languages_url":"https://api.github.com/repos/openaire/iis/languages","stargazers_url":"https://api.github.com/repos/openaire/iis/stargazers","contributors_url":"https://api.github.com/repos/openaire/iis/contributors","subscribers_url":"https://api.github.com/repos/openaire/iis/subscribers","subscription_url":"https://api.github.com/repos/openaire/iis/subscription","commits_url":"https://api.github.com/repos/openaire/iis/commits{/sha}","git_commits_url":"https://api.github.com/repos/openaire/iis/git/commits{/sha}","comments_url":"https://api.github.com/repos/openaire/iis/comments{/number}","issue_comment_url":"https://api.github.com/repos/openaire/iis/issues/comments{/number}","contents_url":"https://api.github.com/repos/openaire/iis/contents/{+path}","compare_url":"https://api.github.com/repos/openaire/iis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/openaire/iis/merges","archive_url":"https://api.github.com/repos/openaire/iis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/openaire/iis/downloads","issues_url":"https://api.github.com/repos/openaire/iis/issues{/number}","pulls_url":"https://api.github.com/repos/openaire/iis/pulls{/number}","milestones_url":"https://api.github.com/repos/openaire/iis/milestones{/number}","notifications_url":"https://api.github.com/repos/openaire/iis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/openaire/iis/labels{/name}","releases_url":"https://api.github.com/repos/openaire/iis/releases{/id}","created_at":"2015-09-11T05:55:08Z","updated_at":"2015-12-14T12:57:35Z","pushed_at":"2015-12-28T20:18:30Z","git_url":"git://github.com/openaire/iis.git","ssh_url":"git@github.com:openaire/iis.git","clone_url":"https://github.com/openaire/iis.git","svn_url":"https://github.com/openaire/iis","homepage":"","size":35912,"stargazers_count":5,"watchers_count":5,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":19,"forks":1,"open_issues":19,"watchers":5,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/openaire/iis/pulls/159"},"html":{"href":"https://github.com/openaire/iis/pull/159"},"issue":{"href":"https://api.github.com/repos/openaire/iis/issues/159"},"comments":{"href":"https://api.github.com/repos/openaire/iis/issues/159/comments"},"review_comments":{"href":"https://api.github.com/repos/openaire/iis/pulls/159/comments"},"review_comment":{"href":"https://api.github.com/repos/openaire/iis/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/openaire/iis/pulls/159/commits"},"statuses":{"href":"https://api.github.com/repos/openaire/iis/statuses/f38b3bbde0765db9f3cf2af1e2b689a2f6c214a6"}}}}