{"action":"created","issue":{"url":"https://api.github.com/repos/eXist-db/exist/issues/1720","repository_url":"https://api.github.com/repos/eXist-db/exist","labels_url":"https://api.github.com/repos/eXist-db/exist/issues/1720/labels{/name}","comments_url":"https://api.github.com/repos/eXist-db/exist/issues/1720/comments","events_url":"https://api.github.com/repos/eXist-db/exist/issues/1720/events","html_url":"https://github.com/eXist-db/exist/issues/1720","id":294825714,"number":1720,"title":"Frequent re-indexing required","user":{"login":"bkcsfi","id":6412502,"avatar_url":"https://avatars3.githubusercontent.com/u/6412502?v=4","gravatar_id":"","url":"https://api.github.com/users/bkcsfi","html_url":"https://github.com/bkcsfi","followers_url":"https://api.github.com/users/bkcsfi/followers","following_url":"https://api.github.com/users/bkcsfi/following{/other_user}","gists_url":"https://api.github.com/users/bkcsfi/gists{/gist_id}","starred_url":"https://api.github.com/users/bkcsfi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bkcsfi/subscriptions","organizations_url":"https://api.github.com/users/bkcsfi/orgs","repos_url":"https://api.github.com/users/bkcsfi/repos","events_url":"https://api.github.com/users/bkcsfi/events{/privacy}","received_events_url":"https://api.github.com/users/bkcsfi/received_events","type":"User","site_admin":false},"labels":[{"id":49540768,"url":"https://api.github.com/repos/eXist-db/exist/labels/bug","name":"bug","color":"fc2929","default":true},{"id":119954677,"url":"https://api.github.com/repos/eXist-db/exist/labels/investigate","name":"investigate","color":"fad8c7","default":false}],"state":"open","locked":false,"assignee":{"login":"wolfgangmm","id":607433,"avatar_url":"https://avatars0.githubusercontent.com/u/607433?v=4","gravatar_id":"","url":"https://api.github.com/users/wolfgangmm","html_url":"https://github.com/wolfgangmm","followers_url":"https://api.github.com/users/wolfgangmm/followers","following_url":"https://api.github.com/users/wolfgangmm/following{/other_user}","gists_url":"https://api.github.com/users/wolfgangmm/gists{/gist_id}","starred_url":"https://api.github.com/users/wolfgangmm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wolfgangmm/subscriptions","organizations_url":"https://api.github.com/users/wolfgangmm/orgs","repos_url":"https://api.github.com/users/wolfgangmm/repos","events_url":"https://api.github.com/users/wolfgangmm/events{/privacy}","received_events_url":"https://api.github.com/users/wolfgangmm/received_events","type":"User","site_admin":false},"assignees":[{"login":"wolfgangmm","id":607433,"avatar_url":"https://avatars0.githubusercontent.com/u/607433?v=4","gravatar_id":"","url":"https://api.github.com/users/wolfgangmm","html_url":"https://github.com/wolfgangmm","followers_url":"https://api.github.com/users/wolfgangmm/followers","following_url":"https://api.github.com/users/wolfgangmm/following{/other_user}","gists_url":"https://api.github.com/users/wolfgangmm/gists{/gist_id}","starred_url":"https://api.github.com/users/wolfgangmm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wolfgangmm/subscriptions","organizations_url":"https://api.github.com/users/wolfgangmm/orgs","repos_url":"https://api.github.com/users/wolfgangmm/repos","events_url":"https://api.github.com/users/wolfgangmm/events{/privacy}","received_events_url":"https://api.github.com/users/wolfgangmm/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/eXist-db/exist/milestones/22","html_url":"https://github.com/eXist-db/exist/milestone/22","labels_url":"https://api.github.com/repos/eXist-db/exist/milestones/22/labels","id":3116502,"number":22,"title":"eXist-4.0.1","description":"Next Hotfix Release","creator":{"login":"adamretter","id":1264057,"avatar_url":"https://avatars1.githubusercontent.com/u/1264057?v=4","gravatar_id":"","url":"https://api.github.com/users/adamretter","html_url":"https://github.com/adamretter","followers_url":"https://api.github.com/users/adamretter/followers","following_url":"https://api.github.com/users/adamretter/following{/other_user}","gists_url":"https://api.github.com/users/adamretter/gists{/gist_id}","starred_url":"https://api.github.com/users/adamretter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adamretter/subscriptions","organizations_url":"https://api.github.com/users/adamretter/orgs","repos_url":"https://api.github.com/users/adamretter/repos","events_url":"https://api.github.com/users/adamretter/events{/privacy}","received_events_url":"https://api.github.com/users/adamretter/received_events","type":"User","site_admin":false},"open_issues":38,"closed_issues":12,"state":"open","created_at":"2018-02-14T13:54:13Z","updated_at":"2018-03-05T17:33:53Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2018-02-06T16:32:31Z","updated_at":"2018-03-05T17:51:48Z","closed_at":null,"author_association":"NONE","body":"### What is the problem\r\n\r\nWe  get unexpected results from xqueries, e.g. documents or fragments thereof are not returned in xquery results when they should be.\r\n\r\n### What did you expect\r\nWe expect the existing documents fragment's to be returned by the xquery\r\n\r\n### Describe how to reproduce or add a test\r\nI am not sure how to recreate the issue. The 'fix' is to reindex the collection.\r\n\r\nAfter upgrading from 2.2 eXist to 3.6 on December 3rd, 2017 I've found that documents are becoming 'un-indexed'. That is, queries work correctly for a while, but eventually they do not return expected results until I re-index the collection that contains the documents.\r\n\r\nThis is possibly related to #1648 or #1691 \r\n\r\nIf there is a corruption issue when indexes are updated during regular document updates, then as documents are updated they may randomly be removed from the index, or subsequent queries may return the wrong results.\r\n\r\nSo far we've found that re-indexing the collection fixes the issue, for about 24 hours at most depending on update/insert volume.\r\n\r\nUnder 2.1 and older releases we had similar symptoms - queries failed to return expected results, but re-indexing \"fixed\" the issue.  In the past we had setup a cronjob to re-index our most active collections. I think this was fixed in 2.2, but I cannot do that this time because these collections are large and re-indexing takes too long.\r\n\r\nThe most recent re-index was needed for a collection that contained approximately 930 documents.\r\n\r\nthe exist.log file doesn't show any exceptions during re-indexing, nor when querying a collection that is not working correctly.\r\n\r\n### Context information\r\nWe're using the docker image evolvedbinary/exist-db   tag eXist-3.6.1\r\n(strangely monex doesn't display version information..)\r\n\r\nthe related collection.xconf is shown below\r\n\r\n```\r\n<collection xmlns=\"http://exist-db.org/collection-config/1.0\">\r\n    <index>\r\n        <fulltext default=\"none\" attributes=\"false\" alphanum=\"false\"/>\r\n        <range>\r\n            <create qname=\"invoicenumber\" type=\"xs:string\"/>\r\n            <create qname=\"due_date\" type=\"xs:string\"/>\r\n            <create qname=\"carrierid\" type=\"xs:string\"/>\r\n            <create qname=\"superseded\" type=\"xs:string\"/>\r\n            <create qname=\"supersedes\" type=\"xs:string\"/>\r\n            <create qname=\"vendorid\" type=\"xs:float\"/>\r\n            <create qname=\"state\" type=\"xs:string\"/>\r\n            <create qname=\"trackingnumber\" type=\"xs:string\"/>\r\n            <create qname=\"flags\" type=\"xs:string\"/>\r\n            <create qname=\"hash\" type=\"xs:string\"/>\r\n            <create qname=\"key\" type=\"xs:string\"/>\r\n            <create qname=\"normalized_serviceid\" type=\"xs:string\"/>\r\n            <create qname=\"shipmentid\" type=\"xs:string\"/>\r\n        </range>\r\n    </index>\r\n</collection>\r\n```"},"comment":{"url":"https://api.github.com/repos/eXist-db/exist/issues/comments/370504472","html_url":"https://github.com/eXist-db/exist/issues/1720#issuecomment-370504472","issue_url":"https://api.github.com/repos/eXist-db/exist/issues/1720","id":370504472,"user":{"login":"wolfgangmm","id":607433,"avatar_url":"https://avatars0.githubusercontent.com/u/607433?v=4","gravatar_id":"","url":"https://api.github.com/users/wolfgangmm","html_url":"https://github.com/wolfgangmm","followers_url":"https://api.github.com/users/wolfgangmm/followers","following_url":"https://api.github.com/users/wolfgangmm/following{/other_user}","gists_url":"https://api.github.com/users/wolfgangmm/gists{/gist_id}","starred_url":"https://api.github.com/users/wolfgangmm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wolfgangmm/subscriptions","organizations_url":"https://api.github.com/users/wolfgangmm/orgs","repos_url":"https://api.github.com/users/wolfgangmm/repos","events_url":"https://api.github.com/users/wolfgangmm/events{/privacy}","received_events_url":"https://api.github.com/users/wolfgangmm/received_events","type":"User","site_admin":false},"created_at":"2018-03-05T17:51:48Z","updated_at":"2018-03-05T17:51:48Z","author_association":"OWNER","body":"I managed to track down and fix one indexing issue in #1763, which leads to somewhat similar symptoms. It only applies to elements indexed by and queried through a range index though. Unfortunately your sample query above does not use any XPath filter and thus no range index, which makes me suspect the underlying cause is not in the range index, but in the structural index.\r\n\r\nDoes the issue really manifest if you run the simple query above which just lists the contents of some elements?"}}
{"actor":{"display_login":"wolfgangmm"}}