{"action":"created","issue":{"url":"https://api.github.com/repos/processone/ejabberd/issues/1320","repository_url":"https://api.github.com/repos/processone/ejabberd","labels_url":"https://api.github.com/repos/processone/ejabberd/issues/1320/labels{/name}","comments_url":"https://api.github.com/repos/processone/ejabberd/issues/1320/comments","events_url":"https://api.github.com/repos/processone/ejabberd/issues/1320/events","html_url":"https://github.com/processone/ejabberd/issues/1320","id":181116265,"number":1320,"title":"PubSub items non-retractable and ends up in other nodes","user":{"login":"mrfrostfire","id":3914832,"avatar_url":"https://avatars.githubusercontent.com/u/3914832?v=3","gravatar_id":"","url":"https://api.github.com/users/mrfrostfire","html_url":"https://github.com/mrfrostfire","followers_url":"https://api.github.com/users/mrfrostfire/followers","following_url":"https://api.github.com/users/mrfrostfire/following{/other_user}","gists_url":"https://api.github.com/users/mrfrostfire/gists{/gist_id}","starred_url":"https://api.github.com/users/mrfrostfire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrfrostfire/subscriptions","organizations_url":"https://api.github.com/users/mrfrostfire/orgs","repos_url":"https://api.github.com/users/mrfrostfire/repos","events_url":"https://api.github.com/users/mrfrostfire/events{/privacy}","received_events_url":"https://api.github.com/users/mrfrostfire/received_events","type":"User","site_admin":false},"labels":[{"id":179040816,"url":"https://api.github.com/repos/processone/ejabberd/labels/Component:Pubsub","name":"Component:Pubsub","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":{"login":"cromain","id":743187,"avatar_url":"https://avatars.githubusercontent.com/u/743187?v=3","gravatar_id":"","url":"https://api.github.com/users/cromain","html_url":"https://github.com/cromain","followers_url":"https://api.github.com/users/cromain/followers","following_url":"https://api.github.com/users/cromain/following{/other_user}","gists_url":"https://api.github.com/users/cromain/gists{/gist_id}","starred_url":"https://api.github.com/users/cromain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cromain/subscriptions","organizations_url":"https://api.github.com/users/cromain/orgs","repos_url":"https://api.github.com/users/cromain/repos","events_url":"https://api.github.com/users/cromain/events{/privacy}","received_events_url":"https://api.github.com/users/cromain/received_events","type":"User","site_admin":false},"assignees":[{"login":"cromain","id":743187,"avatar_url":"https://avatars.githubusercontent.com/u/743187?v=3","gravatar_id":"","url":"https://api.github.com/users/cromain","html_url":"https://github.com/cromain","followers_url":"https://api.github.com/users/cromain/followers","following_url":"https://api.github.com/users/cromain/following{/other_user}","gists_url":"https://api.github.com/users/cromain/gists{/gist_id}","starred_url":"https://api.github.com/users/cromain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cromain/subscriptions","organizations_url":"https://api.github.com/users/cromain/orgs","repos_url":"https://api.github.com/users/cromain/repos","events_url":"https://api.github.com/users/cromain/events{/privacy}","received_events_url":"https://api.github.com/users/cromain/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/processone/ejabberd/milestones/19","html_url":"https://github.com/processone/ejabberd/milestone/19","labels_url":"https://api.github.com/repos/processone/ejabberd/milestones/19/labels","id":2003909,"number":19,"title":"ejabberd 16.12","description":"","creator":{"login":"cromain","id":743187,"avatar_url":"https://avatars.githubusercontent.com/u/743187?v=3","gravatar_id":"","url":"https://api.github.com/users/cromain","html_url":"https://github.com/cromain","followers_url":"https://api.github.com/users/cromain/followers","following_url":"https://api.github.com/users/cromain/following{/other_user}","gists_url":"https://api.github.com/users/cromain/gists{/gist_id}","starred_url":"https://api.github.com/users/cromain/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cromain/subscriptions","organizations_url":"https://api.github.com/users/cromain/orgs","repos_url":"https://api.github.com/users/cromain/repos","events_url":"https://api.github.com/users/cromain/events{/privacy}","received_events_url":"https://api.github.com/users/cromain/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":18,"state":"open","created_at":"2016-09-14T12:28:54Z","updated_at":"2016-12-07T09:50:31Z","due_on":null,"closed_at":null},"comments":2,"created_at":"2016-10-05T10:23:19Z","updated_at":"2016-12-09T10:28:56Z","closed_at":null,"body":"> What version of ejabberd are you using?\n\n16.02, 16.09\n\n> What operating system (version) are you using?\n\nRed hat\n\n> How did you install ejabberd (source, package, distribution)?\n\nrpm\n\n> What did not work as expected? Are there error messages in the log? What\n> was the unexpected behavior? What was the expected result?\n\nWe have found that there sometimes becomes a mismatch of pubsub_items and pubsub_states, where an item exists but without a state reffering to it.\n\nThis affects the pubsub system in 2 very visible and dangerous ways.\n1. When fetching a list of items on a node, it'll return all items using the internal node, and the item will appear. But when trying to retract the item, it will return that the item doesn't exist.\n2. When the node the stray item belongs to is removed, the item will persist. Thereby, as the new node will  get the same internal id as the previous, the item will turn up on the new node when fetching the items for that node.\n\nHere's an example of fetching items for a node and trying to retract it:\n\n```\n<iq id=\"MX_19\" type=\"get\" to=\"pubsub.domain.com\" xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <items node=\"111.222.333\" />\n  </pubsub>\n</iq>\n<iq from=\"pubsub.domain.com\" to=\"xxx@yyy/zzz\" id=\"MX_19\"\n     type=\"result\" xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <items node=\"111.222.333\">\n      <item id=\"5C01FE06DF6B9\">\n        <TheObject xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n            xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns=\"\">\n            ...\n        </TheObject>\n      </item>\n    </items>\n  </pubsub>\n</iq>\n<iq id=\"MX_23\" to=\"pubsub.domain.com\" type=\"set\" xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <retract node=\"111.222.333\">\n      <item id=\"5C01FE06DF6B9\" />\n    </retract>\n  </pubsub>\n</iq>\n<iq from=\"pubsub.domain.com\" to=\"xxx@yyy/zzz\"\n      type=\"error\" id=\"MX_23\" xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <retract node=\"111.222.333\">\n      <item id=\"5C01FE06DF6B9\" />\n    </retract>\n  </pubsub>\n  <error code=\"404\" type=\"cancel\">\n    <item-not-found xmlns=\"urn:ietf:params:xml:ns:xmpp-stanzas\" />\n  </error>\n</iq>\n```\n\nAnd here is an example where a node is created, the item turns up on it, then the node is destroyed and a new one created, and the same item is returned on that node.\n\n```\n<iq from=\"pubsub.domain.com\" to=\"xxx@domain.com/yyy\" id=\"MX_4\"\n     type=\"result\" xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <items node=\"1261\">\n      <item id=\"5C379092C68B1\">\n        <AnotherObject xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n           xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns=\"\">\n            ...\n        </AnotherObject>\n      </item>\n      <item id=\"5C3778738CE25\">\n        <TheObject xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n           xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns=\"\">\n              ...\n        </TheObject>\n      </item>\n    </items>\n  </pubsub>\n</iq>\n```\n\n```\n<iq from=\"pubsub.domain.com\" to=\"xxx@domain.com/yyy\" id=\"MX_4\" type=\"result\"\n     xmlns=\"jabber:client\">\n  <pubsub xmlns=\"http://jabber.org/protocol/pubsub\">\n    <items node=\"1262\">\n      <item id=\"5C3791013CF2E\">\n        <AnotherObject xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n        xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns=\"\">\n        ...\n        </AnotherObject>\n      </item>\n      <item id=\"5C3778738CE25\">\n        <TheObject xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\"\n          xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" xmlns=\"\">\n          ...\n        </TheObject>\n      </item>\n    </items>\n  </pubsub>\n</iq>\n```\n\nAs you can see, the object with the same id turns up on both nodes\n\nThe names are fake, and the content of the objects has been replaced by ... as you probably understood.\n\nThere are no errors that we've found in the logs, and there are no errors other than the ones reported back of the items missing.\nWe started tracing the erl code for node destruction and found that it's using the states to determine what items should be removed, but when we did a dump of the mnesia db, we could see that the stray items does not have a corresponding state reference. Either the state is never created, or it was removed without removing the item.\nWe were using 16.02 and have replicated this in multiple environments, and have upgraded to 16.09 hoping the problem to go away, but we still get the stray items.\n\nAs a workaround, we will stop destroying nodes until this has been fixed.\n"},"comment":{"url":"https://api.github.com/repos/processone/ejabberd/issues/comments/265982181","html_url":"https://github.com/processone/ejabberd/issues/1320#issuecomment-265982181","issue_url":"https://api.github.com/repos/processone/ejabberd/issues/1320","id":265982181,"user":{"login":"mrfrostfire","id":3914832,"avatar_url":"https://avatars.githubusercontent.com/u/3914832?v=3","gravatar_id":"","url":"https://api.github.com/users/mrfrostfire","html_url":"https://github.com/mrfrostfire","followers_url":"https://api.github.com/users/mrfrostfire/followers","following_url":"https://api.github.com/users/mrfrostfire/following{/other_user}","gists_url":"https://api.github.com/users/mrfrostfire/gists{/gist_id}","starred_url":"https://api.github.com/users/mrfrostfire/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrfrostfire/subscriptions","organizations_url":"https://api.github.com/users/mrfrostfire/orgs","repos_url":"https://api.github.com/users/mrfrostfire/repos","events_url":"https://api.github.com/users/mrfrostfire/events{/privacy}","received_events_url":"https://api.github.com/users/mrfrostfire/received_events","type":"User","site_admin":false},"created_at":"2016-12-09T10:28:56Z","updated_at":"2016-12-09T10:28:56Z","body":"Personally I don't think that removing items from an entity on affiliation removal is a good choice.\r\nLet's say that a team is collaborating on a node, and one of the teammembers quit or gets reassigned, we still want to keep the teammembers contributions to the node. \r\n\r\nItems on a node in my opinion should stay on the node until specifically removed or node is purged/removed. Removing them on affiliationchange is a big change that may cause lots of issues to people who are using pubsub today and upgrades to the version where this is fixed."}}
{"actor":{"display_login":"mrfrostfire"}}