{"action":"created","issue":{"url":"https://api.github.com/repos/scylladb/scylla/issues/3275","repository_url":"https://api.github.com/repos/scylladb/scylla","labels_url":"https://api.github.com/repos/scylladb/scylla/issues/3275/labels{/name}","comments_url":"https://api.github.com/repos/scylladb/scylla/issues/3275/comments","events_url":"https://api.github.com/repos/scylladb/scylla/issues/3275/events","html_url":"https://github.com/scylladb/scylla/issues/3275","id":304206677,"node_id":"MDU6SXNzdWUzMDQyMDY2Nzc=","number":3275,"title":"Call into view building write path when applying streaming mutations","user":{"login":"duarten","id":501746,"node_id":"MDQ6VXNlcjUwMTc0Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/501746?v=4","gravatar_id":"","url":"https://api.github.com/users/duarten","html_url":"https://github.com/duarten","followers_url":"https://api.github.com/users/duarten/followers","following_url":"https://api.github.com/users/duarten/following{/other_user}","gists_url":"https://api.github.com/users/duarten/gists{/gist_id}","starred_url":"https://api.github.com/users/duarten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duarten/subscriptions","organizations_url":"https://api.github.com/users/duarten/orgs","repos_url":"https://api.github.com/users/duarten/repos","events_url":"https://api.github.com/users/duarten/events{/privacy}","received_events_url":"https://api.github.com/users/duarten/received_events","type":"User","site_admin":false},"labels":[{"id":534960493,"node_id":"MDU6TGFiZWw1MzQ5NjA0OTM=","url":"https://api.github.com/repos/scylladb/scylla/labels/cassandra%203.x%20compatibility","name":"cassandra 3.x compatibility","color":"f9d0c4","default":false},{"id":463825172,"node_id":"MDU6TGFiZWw0NjM4MjUxNzI=","url":"https://api.github.com/repos/scylladb/scylla/labels/materialized-views","name":"materialized-views","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},"assignees":[{"login":"nyh","id":584227,"node_id":"MDQ6VXNlcjU4NDIyNw==","avatar_url":"https://avatars3.githubusercontent.com/u/584227?v=4","gravatar_id":"","url":"https://api.github.com/users/nyh","html_url":"https://github.com/nyh","followers_url":"https://api.github.com/users/nyh/followers","following_url":"https://api.github.com/users/nyh/following{/other_user}","gists_url":"https://api.github.com/users/nyh/gists{/gist_id}","starred_url":"https://api.github.com/users/nyh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nyh/subscriptions","organizations_url":"https://api.github.com/users/nyh/orgs","repos_url":"https://api.github.com/users/nyh/repos","events_url":"https://api.github.com/users/nyh/events{/privacy}","received_events_url":"https://api.github.com/users/nyh/received_events","type":"User","site_admin":false},{"login":"psarna","id":10433047,"node_id":"MDQ6VXNlcjEwNDMzMDQ3","avatar_url":"https://avatars1.githubusercontent.com/u/10433047?v=4","gravatar_id":"","url":"https://api.github.com/users/psarna","html_url":"https://github.com/psarna","followers_url":"https://api.github.com/users/psarna/followers","following_url":"https://api.github.com/users/psarna/following{/other_user}","gists_url":"https://api.github.com/users/psarna/gists{/gist_id}","starred_url":"https://api.github.com/users/psarna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/psarna/subscriptions","organizations_url":"https://api.github.com/users/psarna/orgs","repos_url":"https://api.github.com/users/psarna/repos","events_url":"https://api.github.com/users/psarna/events{/privacy}","received_events_url":"https://api.github.com/users/psarna/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/scylladb/scylla/milestones/29","html_url":"https://github.com/scylladb/scylla/milestone/29","labels_url":"https://api.github.com/repos/scylladb/scylla/milestones/29/labels","id":3262712,"node_id":"MDk6TWlsZXN0b25lMzI2MjcxMg==","number":29,"title":"2.4","description":"","creator":{"login":"dorlaor","id":1735237,"node_id":"MDQ6VXNlcjE3MzUyMzc=","avatar_url":"https://avatars0.githubusercontent.com/u/1735237?v=4","gravatar_id":"","url":"https://api.github.com/users/dorlaor","html_url":"https://github.com/dorlaor","followers_url":"https://api.github.com/users/dorlaor/followers","following_url":"https://api.github.com/users/dorlaor/following{/other_user}","gists_url":"https://api.github.com/users/dorlaor/gists{/gist_id}","starred_url":"https://api.github.com/users/dorlaor/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dorlaor/subscriptions","organizations_url":"https://api.github.com/users/dorlaor/orgs","repos_url":"https://api.github.com/users/dorlaor/repos","events_url":"https://api.github.com/users/dorlaor/events{/privacy}","received_events_url":"https://api.github.com/users/dorlaor/received_events","type":"User","site_admin":false},"open_issues":89,"closed_issues":21,"state":"open","created_at":"2018-04-12T17:05:45Z","updated_at":"2018-09-10T12:02:32Z","due_on":"2018-06-30T07:00:00Z","closed_at":null},"comments":14,"created_at":"2018-03-11T23:48:30Z","updated_at":"2018-09-12T16:33:48Z","closed_at":null,"author_association":"MEMBER","body":"*Installation details*\r\nScylla version (or git commit hash): master\r\n\r\nDuring streaming, there are cases when we should invoke the view write path. In particular, if a view has not yet finished building and we're bootstrapping a new node.\r\n\r\nWe should check whether the view is built across all base replicas responsible for the range in question. We could query just the base replica that's currently paired with view replica we will be paired with, but we may not have a ring snapshot at that point, and it may not be stable if we support multiple range movements. Better to be conservative and query all of them base nodes."},"comment":{"url":"https://api.github.com/repos/scylladb/scylla/issues/comments/420714263","html_url":"https://github.com/scylladb/scylla/issues/3275#issuecomment-420714263","issue_url":"https://api.github.com/repos/scylladb/scylla/issues/3275","id":420714263,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMDcxNDI2Mw==","user":{"login":"duarten","id":501746,"node_id":"MDQ6VXNlcjUwMTc0Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/501746?v=4","gravatar_id":"","url":"https://api.github.com/users/duarten","html_url":"https://github.com/duarten","followers_url":"https://api.github.com/users/duarten/followers","following_url":"https://api.github.com/users/duarten/following{/other_user}","gists_url":"https://api.github.com/users/duarten/gists{/gist_id}","starred_url":"https://api.github.com/users/duarten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duarten/subscriptions","organizations_url":"https://api.github.com/users/duarten/orgs","repos_url":"https://api.github.com/users/duarten/repos","events_url":"https://api.github.com/users/duarten/events{/privacy}","received_events_url":"https://api.github.com/users/duarten/received_events","type":"User","site_admin":false},"created_at":"2018-09-12T16:33:48Z","updated_at":"2018-09-12T16:33:48Z","author_association":"MEMBER","body":"On Wed, 12 Sep 2018 at 17:07, nyh <notifications@github.com> wrote:\n\n> On Wed, Sep 12, 2018 at 4:37 PM, Duarte Nunes <notifications@github.com>\n> wrote:\n>\n> > @psarna <https://github.com/psarna> / @duarten\n> > <https://github.com/duarten> - can you please detail in which cases\n> > streaming should call into view building write path (and which not) ?\n> >\n> > - bootstrap\n> > - decommission\n> > - removenode\n> > - rebuild\n> > - repair\n> >\n> > We need to call into the view building path for repair (this is because a\n> > base replica missed an updated) and for all others if the view has not\n> > finished building.\n> >\n> I think there's some confusion here given the title of this issue, whether\n> we're really talking about view *building* (i.e., the background process\n> which happens when we add a view to a base table with existing data) or\n> ordinary view writes.\n\n\nThe issue is about streaming. It also predates the view_builder and by the\n“building” in the title I mean the incremental, normal view write path.\nI’ll update it to avoid confusion.\n\n\n> The comments seem to mix the two things, sometimes\n> even in the same sentence...\n\n\nMy comment doesn't mix anything, it merely states the cases which require\nus to call into the view write path for streaming, some of which are\ndependent on the view building status.\n\n\n>\n> Regardless of view *building*, repair is a special issue which we indeed\n> need to handle (and I think we should have a separate issue on it. I was\n> sure we had, but can't find one.)\n> The question is what we expect to happen if one starts a repair only on the\n> base table - should it repair *just* the base table, or does it also do\n> something to the view table if it finds discrepancies?\n\n\nA base repair means a replica missed some mutations, which of course need\nto go through the view write path. A paired base<->view repair wouldn't\ncatch these inconsistencies.\n\n\n>\n> Cassandra compatibility requires the latter:\n> https://docs.datastax.com/en/cql/3.3/cql/cql_using/knownLimitationsMV.html\n> states that:\n> *\"Full repairs on base tables must go through the write path to generate\n> view updates\".*\n>\n> So if we want to be compatible with that, repair indeed needs to go through\n> the view write path. I wouldn't call it the \"view building\" write path\n> though - it's the full, normal, write path, which includes the view\n> updates.\n\n\nNo, it's just the view write path. We don't need to apply the base\nmutations, as those are written directly to an sstable.\n\n\n>\n> Instead of calling into the view building write path\n> >\n> > - we could transfer what is \"left\" of building the view to the new\n> > owning nodes (anyhow, we need to stop the background build and not\n> continue\n> > after the ownership of ranges has been completed).\n> >\n> >\n> This part of the discussion (which I don't understand) seems to talk about\n> a completely different issue - range movements (adding or deleting a node).\n\n\nRange movements are precisely what this issue is about, aside from base\nrepair.\n\n\n> But what is missing here?\n> If I understand correctly (but @duarte, please correct me if I'm wrong),\n> the distributed view building code already handles the case of range\n> movements - what has already been built will be streamed, unbuilt ranges\n> will be built on the new view replica as well because the view building\n>\n\nView building happens on base replicas. But no, there's no synchronization\nbetween a node learning about a view schema (for which it has no data),\nview building, and range movements. That's what this issue is meant to\naddress.\n\n\n> code also takes into account \"pending nodes\" (at least, I hope it does!).\n> New writes that happen during the build also work properly, because of the\n> \"pending nodes\" support (this I know exists because I wrote it at some\n> point, and then Duarte improved on it).\n> All in all, @duarte, I don't understand the problem that this issue claims\n> exists.\n\n\nWell, one problem you understand, which is repair. The other problem is\nview building. If node A is building view V, but then its ranges R are\nmoved from A to B, there's no guarantee those ranges have been built.\n"}}
{"actor":{"display_login":"duarten"}}