{"action":"created","issue":{"url":"https://api.github.com/repos/opencb/opencga/issues/274","repository_url":"https://api.github.com/repos/opencb/opencga","labels_url":"https://api.github.com/repos/opencb/opencga/issues/274/labels{/name}","comments_url":"https://api.github.com/repos/opencb/opencga/issues/274/comments","events_url":"https://api.github.com/repos/opencb/opencga/issues/274/events","html_url":"https://github.com/opencb/opencga/issues/274","id":135196234,"number":274,"title":"Bug in study variants using annot-ct filter","user":{"login":"Antonior26","id":5476137,"avatar_url":"https://avatars.githubusercontent.com/u/5476137?v=3","gravatar_id":"","url":"https://api.github.com/users/Antonior26","html_url":"https://github.com/Antonior26","followers_url":"https://api.github.com/users/Antonior26/followers","following_url":"https://api.github.com/users/Antonior26/following{/other_user}","gists_url":"https://api.github.com/users/Antonior26/gists{/gist_id}","starred_url":"https://api.github.com/users/Antonior26/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Antonior26/subscriptions","organizations_url":"https://api.github.com/users/Antonior26/orgs","repos_url":"https://api.github.com/users/Antonior26/repos","events_url":"https://api.github.com/users/Antonior26/events{/privacy}","received_events_url":"https://api.github.com/users/Antonior26/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-02-21T11:47:33Z","updated_at":"2016-02-22T10:20:03Z","closed_at":null,"body":"It seems there is a big bug here, annot-ct don't select all the the variants it should. surprisingly it works with it is combined with region:\r\n\r\n\r\n\r\ncurl -o reg.txt  \"http://***.***.**.**/opencga/webservices/rest/v1/studies/1453971/variants/?sid=***************&sort=true&annot-ct=SO:0001893,SO:0001574,SO:0001575,SO:0001587,SO:0001589,SO:0001578,SO:0001582,SO:0001889,SO:0001821,SO:0001822,SO:0001583,SO:0001630,SO:0001626&limit=-1&region=1:111909361-111909361\"\r\n\r\ncurl -o no_reg.txt  \"http://***.***.**.**/opencga/webservices/rest/v1/studies/1453971/variants/?sid=***************&sort=true&annot-ct=SO:0001893,SO:0001574,SO:0001575,SO:0001587,SO:0001589,SO:0001578,SO:0001582,SO:0001889,SO:0001821,SO:0001822,SO:0001583,SO:0001630,SO:0001626&limit=-1\r\n\r\n\r\nIt happened to me that the 1:111909361-111909361 variant is appear in the first case but not in the second\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/opencb/opencga/issues/comments/187108640","html_url":"https://github.com/opencb/opencga/issues/274#issuecomment-187108640","issue_url":"https://api.github.com/repos/opencb/opencga/issues/274","id":187108640,"user":{"login":"j-coll","id":4750611,"avatar_url":"https://avatars.githubusercontent.com/u/4750611?v=3","gravatar_id":"","url":"https://api.github.com/users/j-coll","html_url":"https://github.com/j-coll","followers_url":"https://api.github.com/users/j-coll/followers","following_url":"https://api.github.com/users/j-coll/following{/other_user}","gists_url":"https://api.github.com/users/j-coll/gists{/gist_id}","starred_url":"https://api.github.com/users/j-coll/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/j-coll/subscriptions","organizations_url":"https://api.github.com/users/j-coll/orgs","repos_url":"https://api.github.com/users/j-coll/repos","events_url":"https://api.github.com/users/j-coll/events{/privacy}","received_events_url":"https://api.github.com/users/j-coll/received_events","type":"User","site_admin":false},"created_at":"2016-02-22T10:20:03Z","updated_at":"2016-02-22T10:20:03Z","body":"I am pretty sure that your problem is with the automatic limit of the server. The server is not able to return all the possible variants of the server in one QueryResponse. It will kill the server (or the client), since the server has to fetch all the data of the query after sending to the client. The current default limit is set to 1000, and there is a maximum limit of 5000. This values are hard-coded until the new configuration file in yaml is finished.\r\n\r\nIn order to download large sets of variants, you need to paginate the results:\r\n\r\n - First, query the first page and ask for the total result number:\r\n```\r\nvariants?skipCount=false&skip=0&limit=2500&...\r\n >> totalNumResults=20000\r\n```\r\nIn the example the pageSize is 2500. With 20000 variants in the result, will require to split the result in 8 queries : `20000/2500=8`\r\n\r\n - variants?skip=2500&limit=2500\r\n - variants?skip=5000&limit=5000\r\n - ......\r\n - ...\r\n\r\nThis pagination can be really slow for complex queries or extra large datasets. At this moment we are implementing streaming using gRPC servers to connect directly to the internal storage server #247, but this will be extended to the main server."}}