{"action":"created","comment":{"url":"https://api.github.com/repos/citusdata/citus/pulls/comments/82177382","pull_request_review_id":3101828,"id":82177382,"diff_hunk":"@@ -0,0 +1,421 @@\n+/*-------------------------------------------------------------------------\n+ *\n+ * metadata_sync.c\n+ *\n+ * Routines for synchronizing metadata to all workers.\n+ *\n+ * Copyright (c) 2016, Citus Data, Inc.\n+ *\n+ * $Id$\n+ *\n+ *-------------------------------------------------------------------------\n+ */\n+\n+#include \"postgres.h\"\n+#include \"miscadmin.h\"\n+\n+#include <sys/stat.h>\n+#include <unistd.h>\n+\n+#include \"access/heapam.h\"\n+#include \"catalog/dependency.h\"\n+#include \"catalog/pg_foreign_server.h\"\n+#include \"distributed/citus_ruleutils.h\"\n+#include \"distributed/distribution_column.h\"\n+#include \"distributed/master_metadata_utility.h\"\n+#include \"distributed/master_protocol.h\"\n+#include \"distributed/metadata_cache.h\"\n+#include \"distributed/metadata_sync.h\"\n+#include \"distributed/multi_join_order.h\"\n+#include \"distributed/worker_manager.h\"\n+#include \"foreign/foreign.h\"\n+#include \"nodes/pg_list.h\"\n+#include \"utils/builtins.h\"\n+#include \"utils/lsyscache.h\"\n+#include \"utils/memutils.h\"\n+\n+\n+/*\n+ * ShouldSyncTableMetadata checks if a distributed table has streaming replication model\n+ * and hash distribution. In that case the distributed table is considered an MX table,\n+ * and its metadata is required to exist on the worker nodes.\n+ */\n+bool\n+ShouldSyncTableMetadata(Oid relationId)\n+{\n+\tDistTableCacheEntry *tableEntry = DistributedTableCacheEntry(relationId);\n+\tbool usesHashDistribution = (tableEntry->partitionMethod == DISTRIBUTE_BY_HASH);\n+\tbool usesStreamingReplication =\n+\t\t(tableEntry->replicationModel == REPLICATION_MODEL_STREAMING);\n+\n+\tif (usesStreamingReplication && usesHashDistribution)\n+\t{\n+\t\treturn true;\n+\t}\n+\telse\n+\t{\n+\t\treturn false;\n+\t}\n+}\n+\n+\n+/*\n+ * MetadataCreateCommands returns list of queries that are\n+ * required to create the current metadata snapshot of the node that the\n+ * function is called. The metadata snapshot commands includes the\n+ * following queries:\n+ *\n+ * (i)   Query that populates pg_dist_node table\n+ * (ii)  Queries that create the clustered tables\n+ * (iii) Queries that populate pg_dist_partition table referenced by (ii)\n+ * (iv)  Queries that populate pg_dist_shard table referenced by (iii)\n+ * (v)   Queries that populate pg_dist_shard_placement table referenced by (iv)\n+ */\n+List *\n+MetadataCreateCommands(void)\n+{\n+\tList *metadataSnapshotCommandList = NIL;\n+\tList *distributedTableList = DistributedTableList();\n+\tList *workerNodeList = WorkerNodeList();\n+\tListCell *distributedTableCell = NULL;\n+\tchar *nodeListInsertCommand = NULL;\n+\n+\t/* generate insert command for pg_dist_node table */\n+\tnodeListInsertCommand = NodeListInsertCommand(workerNodeList);\n+\tmetadataSnapshotCommandList = lappend(metadataSnapshotCommandList,\n+\t\t\t\t\t\t\t\t\t\t  nodeListInsertCommand);\n+\n+\t/* iterate over the distributed tables */\n+\tforeach(distributedTableCell, distributedTableList)\n+\t{\n+\t\tDistTableCacheEntry *cacheEntry =\n+\t\t\t(DistTableCacheEntry *) lfirst(distributedTableCell);\n+\t\tList *clusteredTableDDLEvents = NIL;\n+\t\tList *shardIntervalList = NIL;\n+\t\tList *shardCreateCommandList = NIL;\n+\t\tchar *partitionKeyColumnName = NULL;\n+\t\tOid clusteredTableId = cacheEntry->relationId;\n+\t\tchar distributionMethod = cacheEntry->partitionMethod;\n+\t\tchar *partitionKeyString = cacheEntry->partitionKeyString;\n+\n+\t\t/* add only clustered tables */\n+\t\tif (!cacheEntry->isCluster)\n+\t\t{\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* add the DDL events first */\n+\t\tpartitionKeyColumnName = ColumnNameToColumn(clusteredTableId,\n+\t\t\t\t\t\t\t\t\t\t\t\t\tpartitionKeyString);\n+\t\tclusteredTableDDLEvents = GetDistributedTableDDLEvents(clusteredTableId,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t   distributionMethod,\n+\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t   partitionKeyColumnName);\n+\t\tmetadataSnapshotCommandList = list_concat(metadataSnapshotCommandList,\n+\t\t\t\t\t\t\t\t\t\t\t\t  clusteredTableDDLEvents);\n+\n+\t\t/* add the pg_dist_shard{,placement} entries */\n+\t\tshardIntervalList = LoadShardIntervalList(clusteredTableId);\n+\t\tshardCreateCommandList = ShardListInsertCommand(shardIntervalList);\n+\n+\t\tmetadataSnapshotCommandList = list_concat(metadataSnapshotCommandList,\n+\t\t\t\t\t\t\t\t\t\t\t\t  shardCreateCommandList);\n+\t}\n+\n+\treturn metadataSnapshotCommandList;\n+}\n+\n+\n+/*\n+ * ClusterMetadataDropCommands returns list of queries that are required to\n+ * drop all the metadata of the node that are related to clustered tables.\n+ * The drop metadata snapshot commands includes the following queries:\n+ *\n+ * (i)   Queries that delete all the rows from pg_dist_node table\n+ * (ii)  Queries that drop the clustered tables and remove its references from\n+ *       the pg_dist_partition. Note that distributed relation ids are gathered\n+ *       from the worker itself to prevent dropping any non-distributed tables\n+ *       with the same name.\n+ * (iii) Queries that delete all the rows from pg_dist_shard table referenced by (ii)\n+ * (iv) Queries that delete all the rows from pg_dist_shard_placement table\n+ *      referenced by (iii)\n+ */\n+List *\n+ClusterMetadataDropCommands(void)","path":"src/backend/distributed/metadata/metadata_sync.c","position":null,"original_position":143,"commit_id":"d3e4069bf76c881b75384654b05c88d7fae95f04","original_commit_id":"6d03a8232742206cc0b82c03ee77c6c3a259e5cb","user":{"login":"aamederen","id":1406797,"avatar_url":"https://avatars.githubusercontent.com/u/1406797?v=3","gravatar_id":"","url":"https://api.github.com/users/aamederen","html_url":"https://github.com/aamederen","followers_url":"https://api.github.com/users/aamederen/followers","following_url":"https://api.github.com/users/aamederen/following{/other_user}","gists_url":"https://api.github.com/users/aamederen/gists{/gist_id}","starred_url":"https://api.github.com/users/aamederen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aamederen/subscriptions","organizations_url":"https://api.github.com/users/aamederen/orgs","repos_url":"https://api.github.com/users/aamederen/repos","events_url":"https://api.github.com/users/aamederen/events{/privacy}","received_events_url":"https://api.github.com/users/aamederen/received_events","type":"User","site_admin":false},"body":"Makes sense","created_at":"2016-10-06T12:36:01Z","updated_at":"2016-10-06T12:36:01Z","html_url":"https://github.com/citusdata/citus/pull/799#discussion_r82177382","pull_request_url":"https://api.github.com/repos/citusdata/citus/pulls/799","_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus/pulls/comments/82177382"},"html":{"href":"https://github.com/citusdata/citus/pull/799#discussion_r82177382"},"pull_request":{"href":"https://api.github.com/repos/citusdata/citus/pulls/799"}}},"pull_request":{"url":"https://api.github.com/repos/citusdata/citus/pulls/799","id":86513261,"html_url":"https://github.com/citusdata/citus/pull/799","diff_url":"https://github.com/citusdata/citus/pull/799.diff","patch_url":"https://github.com/citusdata/citus/pull/799.patch","issue_url":"https://api.github.com/repos/citusdata/citus/issues/799","number":799,"state":"open","locked":false,"title":"Add Metadata Snapshot Infrastructure","user":{"login":"aamederen","id":1406797,"avatar_url":"https://avatars.githubusercontent.com/u/1406797?v=3","gravatar_id":"","url":"https://api.github.com/users/aamederen","html_url":"https://github.com/aamederen","followers_url":"https://api.github.com/users/aamederen/followers","following_url":"https://api.github.com/users/aamederen/following{/other_user}","gists_url":"https://api.github.com/users/aamederen/gists{/gist_id}","starred_url":"https://api.github.com/users/aamederen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aamederen/subscriptions","organizations_url":"https://api.github.com/users/aamederen/orgs","repos_url":"https://api.github.com/users/aamederen/repos","events_url":"https://api.github.com/users/aamederen/events{/privacy}","received_events_url":"https://api.github.com/users/aamederen/received_events","type":"User","site_admin":false},"body":"This change adds the required infrastructure about metadata snapshot from MX\r\ncodebase into Citus, mainly metadata_sync.c and other referenced parts.","created_at":"2016-09-23T13:32:08Z","updated_at":"2016-10-06T12:36:01Z","closed_at":null,"merged_at":null,"merge_commit_sha":"defe40163890992f75067c5ebbb10f4f3cdb6762","assignee":{"login":"marcocitus","id":7127240,"avatar_url":"https://avatars.githubusercontent.com/u/7127240?v=3","gravatar_id":"","url":"https://api.github.com/users/marcocitus","html_url":"https://github.com/marcocitus","followers_url":"https://api.github.com/users/marcocitus/followers","following_url":"https://api.github.com/users/marcocitus/following{/other_user}","gists_url":"https://api.github.com/users/marcocitus/gists{/gist_id}","starred_url":"https://api.github.com/users/marcocitus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcocitus/subscriptions","organizations_url":"https://api.github.com/users/marcocitus/orgs","repos_url":"https://api.github.com/users/marcocitus/repos","events_url":"https://api.github.com/users/marcocitus/events{/privacy}","received_events_url":"https://api.github.com/users/marcocitus/received_events","type":"User","site_admin":false},"assignees":[{"login":"marcocitus","id":7127240,"avatar_url":"https://avatars.githubusercontent.com/u/7127240?v=3","gravatar_id":"","url":"https://api.github.com/users/marcocitus","html_url":"https://github.com/marcocitus","followers_url":"https://api.github.com/users/marcocitus/followers","following_url":"https://api.github.com/users/marcocitus/following{/other_user}","gists_url":"https://api.github.com/users/marcocitus/gists{/gist_id}","starred_url":"https://api.github.com/users/marcocitus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marcocitus/subscriptions","organizations_url":"https://api.github.com/users/marcocitus/orgs","repos_url":"https://api.github.com/users/marcocitus/repos","events_url":"https://api.github.com/users/marcocitus/events{/privacy}","received_events_url":"https://api.github.com/users/marcocitus/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/citusdata/citus/pulls/799/commits","review_comments_url":"https://api.github.com/repos/citusdata/citus/pulls/799/comments","review_comment_url":"https://api.github.com/repos/citusdata/citus/pulls/comments{/number}","comments_url":"https://api.github.com/repos/citusdata/citus/issues/799/comments","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/d3e4069bf76c881b75384654b05c88d7fae95f04","head":{"label":"citusdata:metadata_sync","ref":"metadata_sync","sha":"d3e4069bf76c881b75384654b05c88d7fae95f04","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":50874442,"name":"citus","full_name":"citusdata/citus","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus","description":"Scalable PostgreSQL for real-time workloads","fork":false,"url":"https://api.github.com/repos/citusdata/citus","forks_url":"https://api.github.com/repos/citusdata/citus/forks","keys_url":"https://api.github.com/repos/citusdata/citus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus/teams","hooks_url":"https://api.github.com/repos/citusdata/citus/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus/events","assignees_url":"https://api.github.com/repos/citusdata/citus/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus/tags","blobs_url":"https://api.github.com/repos/citusdata/citus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus/subscription","commits_url":"https://api.github.com/repos/citusdata/citus/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus/merges","archive_url":"https://api.github.com/repos/citusdata/citus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus/downloads","issues_url":"https://api.github.com/repos/citusdata/citus/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus/deployments","created_at":"2016-02-01T21:58:39Z","updated_at":"2016-10-06T11:06:56Z","pushed_at":"2016-10-06T12:35:21Z","git_url":"git://github.com/citusdata/citus.git","ssh_url":"git@github.com:citusdata/citus.git","clone_url":"https://github.com/citusdata/citus.git","svn_url":"https://github.com/citusdata/citus","homepage":"https://www.citusdata.com","size":3936,"stargazers_count":1618,"watchers_count":1618,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":112,"mirror_url":null,"open_issues_count":408,"forks":112,"open_issues":408,"watchers":1618,"default_branch":"master"}},"base":{"label":"citusdata:master","ref":"master","sha":"770d09b48eff53515e36686107dde6b0126e97c2","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":50874442,"name":"citus","full_name":"citusdata/citus","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars.githubusercontent.com/u/2545424?v=3","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus","description":"Scalable PostgreSQL for real-time workloads","fork":false,"url":"https://api.github.com/repos/citusdata/citus","forks_url":"https://api.github.com/repos/citusdata/citus/forks","keys_url":"https://api.github.com/repos/citusdata/citus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus/teams","hooks_url":"https://api.github.com/repos/citusdata/citus/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus/events","assignees_url":"https://api.github.com/repos/citusdata/citus/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus/tags","blobs_url":"https://api.github.com/repos/citusdata/citus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus/subscription","commits_url":"https://api.github.com/repos/citusdata/citus/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus/merges","archive_url":"https://api.github.com/repos/citusdata/citus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus/downloads","issues_url":"https://api.github.com/repos/citusdata/citus/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus/deployments","created_at":"2016-02-01T21:58:39Z","updated_at":"2016-10-06T11:06:56Z","pushed_at":"2016-10-06T12:35:21Z","git_url":"git://github.com/citusdata/citus.git","ssh_url":"git@github.com:citusdata/citus.git","clone_url":"https://github.com/citusdata/citus.git","svn_url":"https://github.com/citusdata/citus","homepage":"https://www.citusdata.com","size":3936,"stargazers_count":1618,"watchers_count":1618,"language":"C","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":112,"mirror_url":null,"open_issues_count":408,"forks":112,"open_issues":408,"watchers":1618,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus/pulls/799"},"html":{"href":"https://github.com/citusdata/citus/pull/799"},"issue":{"href":"https://api.github.com/repos/citusdata/citus/issues/799"},"comments":{"href":"https://api.github.com/repos/citusdata/citus/issues/799/comments"},"review_comments":{"href":"https://api.github.com/repos/citusdata/citus/pulls/799/comments"},"review_comment":{"href":"https://api.github.com/repos/citusdata/citus/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/citusdata/citus/pulls/799/commits"},"statuses":{"href":"https://api.github.com/repos/citusdata/citus/statuses/d3e4069bf76c881b75384654b05c88d7fae95f04"}}}}
{"actor":{"display_login":"aamederen"}}