{"action":"created","comment":{"url":"https://api.github.com/repos/hyrise/hyrise/pulls/comments/170644508","pull_request_review_id":99352952,"id":170644508,"diff_hunk":"@@ -0,0 +1,155 @@\n+#pragma once\n+\n+#include <boost/preprocessor/seq/for_each.hpp>\n+\n+#include \"all_type_variant.hpp\"\n+#include \"storage/base_value_column.hpp\"\n+#include \"storage/chunk.hpp\"\n+#include \"storage/column_iterables/base_column_iterators.hpp\"\n+\n+namespace opossum {\n+\n+// We need a boolean data type in the JitOperator, but don't want to add it to\n+// DATA_TYPE_INFO to avoid costly template instantiations.\n+// See \"all_type_variant.hpp\" for details.\n+#define JIT_DATA_TYPE_INFO ((bool, Bool, \"bool\")) DATA_TYPE_INFO\n+\n+#define JIT_VARIANT_VECTOR_MEMBER(r, d, type) \\\n+  std::vector<BOOST_PP_TUPLE_ELEM(3, 0, type)> BOOST_PP_TUPLE_ELEM(3, 1, type);\n+\n+#define JIT_VARIANT_VECTOR_RESIZE(r, d, type) BOOST_PP_TUPLE_ELEM(3, 1, type).resize(new_size);\n+\n+/* A brief overview of the type system and the way values are handled in the JitOperator:\n+ *\n+ * The JitOperator performs most of its operations on variant values, since this allows writing generic operators with\n+ * an unlimited number of variably-typed operands (without the need for excessive templating).\n+ * While this sounds costly at first, the jit engine knows the actual type of each value in advance and replaces generic\n+ * operations with specialized versions for the concrete data types.\n+ *\n+ * This entails two important restrictions:\n+ * 1) There can be no temporary (=local) variant values. This is because we support std::string as a data type.\n+ *    Strings are not POD types and they thus require memory allocations, destructors and exception\n+ *    handling. The specialization engine is not able to completely optimize this overhead away, even if no strings are\n+ *    processed in an operation.\n+ * 2) The data type of each value needs to be stored separately from the value itself. This is because we want the\n+ *    specialization engine to know about data type, but not concrete values.\n+ *\n+ * Both restrictions are enforced using the same mechanism:\n+ * All values (and other data that must not be available to the specialization engine) are encapsulated in the\n+ * JitRuntimeContext. This context is only passed to the operator after code specialization has taken place.\n+ * The JitRuntimeContext contains a vector of variant values (called tuple) which the JitOperator can access through\n+ * JitTupleValue instances.\n+ * The runtime tuple is created and destroyed outside the JitOperator's scope, so no memory management is\n+ * required from within the operator.\n+ * Each JitTupleValue encapsulates information about how to access a single value in the runtime tuple. However, the\n+ * JitTupleValues are part of the JitOperator and must thus not store a reference to the JitRuntimeContext. So while\n+ * they \"know\" how to access values in the runtime tuple, they do not have the means to do so.\n+ * Only by passing the runtime context to a JitTupleValue, a JitMaterializedValue is created.\n+ * This materialized value finally allows the operator to access a data value.\n+ */\n+\n+/* The JitVariantVector can be used in two ways:\n+ * 1) As a vector of variant values.\n+ *    Imagine you want to store a database row (aka tuple) of some table. Since each column of the table could have a\n+ *    different type, a std::vector<Variant> seems like a good choice. To store N values, the vector is resized to\n+ *    contain N slots and each value (representing one column) can be accessed by its index from [0, N).\n+ *    We do something slightly different here: Instead of one vector (where each vector element can hold any of a number\n+ *    of types), we create one strongly typed vector per data type. All of these vectors have size N, so each value\n+ *    (representing one column of the tuple) has a slot in each vector.\n+ *    Accessing the value at position P as an \"int\" will return the element at position P in the integer vector.\n+ *    There is no automatic type conversions happening here. Soring a value as \"int\" and reading it later as \"double\"\n+ *    won't work, since this accesses different memory locations in different vectors.\n+ *    This is not a problem, however, since the type of each value does not change throughout query execution.\n+ *\n+ * 2) As a variant vector.\n+ *    This data structure can also be used to store a vector of values of some unknown, but fixed type.\n+ *    Say you want to build an aggregate operator and need to store a column of aggregate values. All values\n+ *    produced will have the same type, but there is no way of knowing that type in advance.\n+ *    By adding a templated \"push\" function to the implementation below, we can add an arbitrary number of elements to\n+ *    the std::vector of that type. All other vectors will remain empty.\n+ *    This interpretation of the variant vector is not used in the code currently, but will be helpful when implementing\n+ *    further operators.\n+ */\n+class JitVariantVector {\n+ public:\n+  void resize(const size_t new_size) {\n+    BOOST_PP_SEQ_FOR_EACH(JIT_VARIANT_VECTOR_RESIZE, _, JIT_DATA_TYPE_INFO)\n+    _is_null.resize(new_size);\n+  }\n+\n+  template <typename T>\n+  T get(const size_t index) const;\n+  template <typename T>\n+  void set(const size_t index, const T value);\n+  bool is_null(const size_t index) { return _is_null[index]; }\n+  void set_is_null(const size_t index, const bool is_null) { _is_null[index] = is_null; }\n+\n+ private:\n+  BOOST_PP_SEQ_FOR_EACH(JIT_VARIANT_VECTOR_MEMBER, _, JIT_DATA_TYPE_INFO)\n+  std::vector<uint8_t> _is_null;\n+};\n+\n+// The structure encapsulates all data available to the JitOperator at runtime,\n+// but NOT during code specialization.\n+struct JitRuntimeContext {\n+  uint32_t chunk_size;\n+  ChunkOffset chunk_offset;\n+  JitVariantVector tuple;\n+  std::vector<std::shared_ptr<JitBaseColumnIterator>> inputs;\n+  std::vector<std::shared_ptr<BaseValueColumn>> outputs;\n+  std::shared_ptr<Chunk> out_chunk;\n+};\n+\n+struct JitMaterializedValue {","path":"src/lib/operators/jit_operator/jit_types.hpp","position":103,"original_position":103,"commit_id":"d32145ecc1cc2244bc1c03f1e67570dd51853d8a","original_commit_id":"d32145ecc1cc2244bc1c03f1e67570dd51853d8a","user":{"login":"mjendruk","id":2190123,"avatar_url":"https://avatars0.githubusercontent.com/u/2190123?v=4","gravatar_id":"","url":"https://api.github.com/users/mjendruk","html_url":"https://github.com/mjendruk","followers_url":"https://api.github.com/users/mjendruk/followers","following_url":"https://api.github.com/users/mjendruk/following{/other_user}","gists_url":"https://api.github.com/users/mjendruk/gists{/gist_id}","starred_url":"https://api.github.com/users/mjendruk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mjendruk/subscriptions","organizations_url":"https://api.github.com/users/mjendruk/orgs","repos_url":"https://api.github.com/users/mjendruk/repos","events_url":"https://api.github.com/users/mjendruk/events{/privacy}","received_events_url":"https://api.github.com/users/mjendruk/received_events","type":"User","site_admin":false},"body":"Could you explain the difference in usage of `JitMaterializedValue` and `JitTupleValue `?","created_at":"2018-02-26T16:12:29Z","updated_at":"2018-02-26T16:13:38Z","html_url":"https://github.com/hyrise/hyrise/pull/709#discussion_r170644508","pull_request_url":"https://api.github.com/repos/hyrise/hyrise/pulls/709","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/comments/170644508"},"html":{"href":"https://github.com/hyrise/hyrise/pull/709#discussion_r170644508"},"pull_request":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/709"}}},"pull_request":{"url":"https://api.github.com/repos/hyrise/hyrise/pulls/709","id":169839709,"html_url":"https://github.com/hyrise/hyrise/pull/709","diff_url":"https://github.com/hyrise/hyrise/pull/709.diff","patch_url":"https://github.com/hyrise/hyrise/pull/709.patch","issue_url":"https://api.github.com/repos/hyrise/hyrise/issues/709","number":709,"state":"closed","locked":false,"title":"JIT 2: types, type dispatching, basic operations","user":{"login":"jfrohnhofen","id":1701977,"avatar_url":"https://avatars3.githubusercontent.com/u/1701977?v=4","gravatar_id":"","url":"https://api.github.com/users/jfrohnhofen","html_url":"https://github.com/jfrohnhofen","followers_url":"https://api.github.com/users/jfrohnhofen/followers","following_url":"https://api.github.com/users/jfrohnhofen/following{/other_user}","gists_url":"https://api.github.com/users/jfrohnhofen/gists{/gist_id}","starred_url":"https://api.github.com/users/jfrohnhofen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jfrohnhofen/subscriptions","organizations_url":"https://api.github.com/users/jfrohnhofen/orgs","repos_url":"https://api.github.com/users/jfrohnhofen/repos","events_url":"https://api.github.com/users/jfrohnhofen/events{/privacy}","received_events_url":"https://api.github.com/users/jfrohnhofen/received_events","type":"User","site_admin":false},"body":"This PR introduces the variant types used by the jit backend, and a mechanism for dispatching types and performing computations with these variant values. I added (hopefully enough) documentation to explain the basic concepts. If something is still unclear, let me know.\r\n\r\nFor now, none of this is used anywhere in the code. But this will change in PR3, so stay tuned...","created_at":"2018-02-18T17:09:12Z","updated_at":"2018-02-26T16:13:38Z","closed_at":"2018-02-26T15:42:24Z","merged_at":"2018-02-26T15:42:24Z","merge_commit_sha":"bd229fc2b933047ba9a2cad86ae09ed1aa6a759d","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/hyrise/hyrise/pulls/709/commits","review_comments_url":"https://api.github.com/repos/hyrise/hyrise/pulls/709/comments","review_comment_url":"https://api.github.com/repos/hyrise/hyrise/pulls/comments{/number}","comments_url":"https://api.github.com/repos/hyrise/hyrise/issues/709/comments","statuses_url":"https://api.github.com/repos/hyrise/hyrise/statuses/d32145ecc1cc2244bc1c03f1e67570dd51853d8a","head":{"label":"hyrise:johannes/jit_pr_2_types","ref":"johannes/jit_pr_2_types","sha":"d32145ecc1cc2244bc1c03f1e67570dd51853d8a","user":{"login":"hyrise","id":3465095,"avatar_url":"https://avatars2.githubusercontent.com/u/3465095?v=4","gravatar_id":"","url":"https://api.github.com/users/hyrise","html_url":"https://github.com/hyrise","followers_url":"https://api.github.com/users/hyrise/followers","following_url":"https://api.github.com/users/hyrise/following{/other_user}","gists_url":"https://api.github.com/users/hyrise/gists{/gist_id}","starred_url":"https://api.github.com/users/hyrise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hyrise/subscriptions","organizations_url":"https://api.github.com/users/hyrise/orgs","repos_url":"https://api.github.com/users/hyrise/repos","events_url":"https://api.github.com/users/hyrise/events{/privacy}","received_events_url":"https://api.github.com/users/hyrise/received_events","type":"Organization","site_admin":false},"repo":{"id":87414843,"name":"hyrise","full_name":"hyrise/hyrise","owner":{"login":"hyrise","id":3465095,"avatar_url":"https://avatars2.githubusercontent.com/u/3465095?v=4","gravatar_id":"","url":"https://api.github.com/users/hyrise","html_url":"https://github.com/hyrise","followers_url":"https://api.github.com/users/hyrise/followers","following_url":"https://api.github.com/users/hyrise/following{/other_user}","gists_url":"https://api.github.com/users/hyrise/gists{/gist_id}","starred_url":"https://api.github.com/users/hyrise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hyrise/subscriptions","organizations_url":"https://api.github.com/users/hyrise/orgs","repos_url":"https://api.github.com/users/hyrise/repos","events_url":"https://api.github.com/users/hyrise/events{/privacy}","received_events_url":"https://api.github.com/users/hyrise/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hyrise/hyrise","description":"A research in-memory database.","fork":false,"url":"https://api.github.com/repos/hyrise/hyrise","forks_url":"https://api.github.com/repos/hyrise/hyrise/forks","keys_url":"https://api.github.com/repos/hyrise/hyrise/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hyrise/hyrise/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hyrise/hyrise/teams","hooks_url":"https://api.github.com/repos/hyrise/hyrise/hooks","issue_events_url":"https://api.github.com/repos/hyrise/hyrise/issues/events{/number}","events_url":"https://api.github.com/repos/hyrise/hyrise/events","assignees_url":"https://api.github.com/repos/hyrise/hyrise/assignees{/user}","branches_url":"https://api.github.com/repos/hyrise/hyrise/branches{/branch}","tags_url":"https://api.github.com/repos/hyrise/hyrise/tags","blobs_url":"https://api.github.com/repos/hyrise/hyrise/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hyrise/hyrise/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hyrise/hyrise/git/refs{/sha}","trees_url":"https://api.github.com/repos/hyrise/hyrise/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hyrise/hyrise/statuses/{sha}","languages_url":"https://api.github.com/repos/hyrise/hyrise/languages","stargazers_url":"https://api.github.com/repos/hyrise/hyrise/stargazers","contributors_url":"https://api.github.com/repos/hyrise/hyrise/contributors","subscribers_url":"https://api.github.com/repos/hyrise/hyrise/subscribers","subscription_url":"https://api.github.com/repos/hyrise/hyrise/subscription","commits_url":"https://api.github.com/repos/hyrise/hyrise/commits{/sha}","git_commits_url":"https://api.github.com/repos/hyrise/hyrise/git/commits{/sha}","comments_url":"https://api.github.com/repos/hyrise/hyrise/comments{/number}","issue_comment_url":"https://api.github.com/repos/hyrise/hyrise/issues/comments{/number}","contents_url":"https://api.github.com/repos/hyrise/hyrise/contents/{+path}","compare_url":"https://api.github.com/repos/hyrise/hyrise/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hyrise/hyrise/merges","archive_url":"https://api.github.com/repos/hyrise/hyrise/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hyrise/hyrise/downloads","issues_url":"https://api.github.com/repos/hyrise/hyrise/issues{/number}","pulls_url":"https://api.github.com/repos/hyrise/hyrise/pulls{/number}","milestones_url":"https://api.github.com/repos/hyrise/hyrise/milestones{/number}","notifications_url":"https://api.github.com/repos/hyrise/hyrise/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hyrise/hyrise/labels{/name}","releases_url":"https://api.github.com/repos/hyrise/hyrise/releases{/id}","deployments_url":"https://api.github.com/repos/hyrise/hyrise/deployments","created_at":"2017-04-06T10:03:31Z","updated_at":"2018-02-02T09:29:02Z","pushed_at":"2018-02-26T15:42:25Z","git_url":"git://github.com/hyrise/hyrise.git","ssh_url":"git@github.com:hyrise/hyrise.git","clone_url":"https://github.com/hyrise/hyrise.git","svn_url":"https://github.com/hyrise/hyrise","homepage":"https://hpi.de/plattner/projects/hyrise.html","size":12108,"stargazers_count":57,"watchers_count":57,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":19,"mirror_url":null,"archived":false,"open_issues_count":78,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":19,"open_issues":78,"watchers":57,"default_branch":"master"}},"base":{"label":"hyrise:master","ref":"master","sha":"03ce9660d8a54c1dba5590f748abac55eb7d1d9f","user":{"login":"hyrise","id":3465095,"avatar_url":"https://avatars2.githubusercontent.com/u/3465095?v=4","gravatar_id":"","url":"https://api.github.com/users/hyrise","html_url":"https://github.com/hyrise","followers_url":"https://api.github.com/users/hyrise/followers","following_url":"https://api.github.com/users/hyrise/following{/other_user}","gists_url":"https://api.github.com/users/hyrise/gists{/gist_id}","starred_url":"https://api.github.com/users/hyrise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hyrise/subscriptions","organizations_url":"https://api.github.com/users/hyrise/orgs","repos_url":"https://api.github.com/users/hyrise/repos","events_url":"https://api.github.com/users/hyrise/events{/privacy}","received_events_url":"https://api.github.com/users/hyrise/received_events","type":"Organization","site_admin":false},"repo":{"id":87414843,"name":"hyrise","full_name":"hyrise/hyrise","owner":{"login":"hyrise","id":3465095,"avatar_url":"https://avatars2.githubusercontent.com/u/3465095?v=4","gravatar_id":"","url":"https://api.github.com/users/hyrise","html_url":"https://github.com/hyrise","followers_url":"https://api.github.com/users/hyrise/followers","following_url":"https://api.github.com/users/hyrise/following{/other_user}","gists_url":"https://api.github.com/users/hyrise/gists{/gist_id}","starred_url":"https://api.github.com/users/hyrise/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hyrise/subscriptions","organizations_url":"https://api.github.com/users/hyrise/orgs","repos_url":"https://api.github.com/users/hyrise/repos","events_url":"https://api.github.com/users/hyrise/events{/privacy}","received_events_url":"https://api.github.com/users/hyrise/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/hyrise/hyrise","description":"A research in-memory database.","fork":false,"url":"https://api.github.com/repos/hyrise/hyrise","forks_url":"https://api.github.com/repos/hyrise/hyrise/forks","keys_url":"https://api.github.com/repos/hyrise/hyrise/keys{/key_id}","collaborators_url":"https://api.github.com/repos/hyrise/hyrise/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/hyrise/hyrise/teams","hooks_url":"https://api.github.com/repos/hyrise/hyrise/hooks","issue_events_url":"https://api.github.com/repos/hyrise/hyrise/issues/events{/number}","events_url":"https://api.github.com/repos/hyrise/hyrise/events","assignees_url":"https://api.github.com/repos/hyrise/hyrise/assignees{/user}","branches_url":"https://api.github.com/repos/hyrise/hyrise/branches{/branch}","tags_url":"https://api.github.com/repos/hyrise/hyrise/tags","blobs_url":"https://api.github.com/repos/hyrise/hyrise/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/hyrise/hyrise/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/hyrise/hyrise/git/refs{/sha}","trees_url":"https://api.github.com/repos/hyrise/hyrise/git/trees{/sha}","statuses_url":"https://api.github.com/repos/hyrise/hyrise/statuses/{sha}","languages_url":"https://api.github.com/repos/hyrise/hyrise/languages","stargazers_url":"https://api.github.com/repos/hyrise/hyrise/stargazers","contributors_url":"https://api.github.com/repos/hyrise/hyrise/contributors","subscribers_url":"https://api.github.com/repos/hyrise/hyrise/subscribers","subscription_url":"https://api.github.com/repos/hyrise/hyrise/subscription","commits_url":"https://api.github.com/repos/hyrise/hyrise/commits{/sha}","git_commits_url":"https://api.github.com/repos/hyrise/hyrise/git/commits{/sha}","comments_url":"https://api.github.com/repos/hyrise/hyrise/comments{/number}","issue_comment_url":"https://api.github.com/repos/hyrise/hyrise/issues/comments{/number}","contents_url":"https://api.github.com/repos/hyrise/hyrise/contents/{+path}","compare_url":"https://api.github.com/repos/hyrise/hyrise/compare/{base}...{head}","merges_url":"https://api.github.com/repos/hyrise/hyrise/merges","archive_url":"https://api.github.com/repos/hyrise/hyrise/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/hyrise/hyrise/downloads","issues_url":"https://api.github.com/repos/hyrise/hyrise/issues{/number}","pulls_url":"https://api.github.com/repos/hyrise/hyrise/pulls{/number}","milestones_url":"https://api.github.com/repos/hyrise/hyrise/milestones{/number}","notifications_url":"https://api.github.com/repos/hyrise/hyrise/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/hyrise/hyrise/labels{/name}","releases_url":"https://api.github.com/repos/hyrise/hyrise/releases{/id}","deployments_url":"https://api.github.com/repos/hyrise/hyrise/deployments","created_at":"2017-04-06T10:03:31Z","updated_at":"2018-02-02T09:29:02Z","pushed_at":"2018-02-26T15:42:25Z","git_url":"git://github.com/hyrise/hyrise.git","ssh_url":"git@github.com:hyrise/hyrise.git","clone_url":"https://github.com/hyrise/hyrise.git","svn_url":"https://github.com/hyrise/hyrise","homepage":"https://hpi.de/plattner/projects/hyrise.html","size":12108,"stargazers_count":57,"watchers_count":57,"language":"C++","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":19,"mirror_url":null,"archived":false,"open_issues_count":78,"license":{"key":"mit","name":"MIT License","spdx_id":"MIT","url":"https://api.github.com/licenses/mit"},"forks":19,"open_issues":78,"watchers":57,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/709"},"html":{"href":"https://github.com/hyrise/hyrise/pull/709"},"issue":{"href":"https://api.github.com/repos/hyrise/hyrise/issues/709"},"comments":{"href":"https://api.github.com/repos/hyrise/hyrise/issues/709/comments"},"review_comments":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/709/comments"},"review_comment":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/hyrise/hyrise/pulls/709/commits"},"statuses":{"href":"https://api.github.com/repos/hyrise/hyrise/statuses/d32145ecc1cc2244bc1c03f1e67570dd51853d8a"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"mjendruk"}}