{"action":"closed","issue":{"url":"https://api.github.com/repos/ropensci/rgbif/issues/299","repository_url":"https://api.github.com/repos/ropensci/rgbif","labels_url":"https://api.github.com/repos/ropensci/rgbif/issues/299/labels{/name}","comments_url":"https://api.github.com/repos/ropensci/rgbif/issues/299/comments","events_url":"https://api.github.com/repos/ropensci/rgbif/issues/299/events","html_url":"https://github.com/ropensci/rgbif/issues/299","id":311279463,"number":299,"title":"Suspect behavior name_lookup() or GBIF API with limit > 1000","user":{"login":"damianooldoni","id":33662631,"avatar_url":"https://avatars1.githubusercontent.com/u/33662631?v=4","gravatar_id":"","url":"https://api.github.com/users/damianooldoni","html_url":"https://github.com/damianooldoni","followers_url":"https://api.github.com/users/damianooldoni/followers","following_url":"https://api.github.com/users/damianooldoni/following{/other_user}","gists_url":"https://api.github.com/users/damianooldoni/gists{/gist_id}","starred_url":"https://api.github.com/users/damianooldoni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/damianooldoni/subscriptions","organizations_url":"https://api.github.com/users/damianooldoni/orgs","repos_url":"https://api.github.com/users/damianooldoni/repos","events_url":"https://api.github.com/users/damianooldoni/events{/privacy}","received_events_url":"https://api.github.com/users/damianooldoni/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/ropensci/rgbif/milestones/10","html_url":"https://github.com/ropensci/rgbif/milestone/10","labels_url":"https://api.github.com/repos/ropensci/rgbif/milestones/10/labels","id":1012692,"number":10,"title":"v1.0","description":"This aims to be a stable release, though not to be put into maintenance mode. With very thorough test suite, more vignettes as needed. ","creator":{"login":"sckott","id":577668,"avatar_url":"https://avatars1.githubusercontent.com/u/577668?v=4","gravatar_id":"","url":"https://api.github.com/users/sckott","html_url":"https://github.com/sckott","followers_url":"https://api.github.com/users/sckott/followers","following_url":"https://api.github.com/users/sckott/following{/other_user}","gists_url":"https://api.github.com/users/sckott/gists{/gist_id}","starred_url":"https://api.github.com/users/sckott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sckott/subscriptions","organizations_url":"https://api.github.com/users/sckott/orgs","repos_url":"https://api.github.com/users/sckott/repos","events_url":"https://api.github.com/users/sckott/events{/privacy}","received_events_url":"https://api.github.com/users/sckott/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":10,"state":"open","created_at":"2015-03-09T16:30:06Z","updated_at":"2018-04-11T16:28:51Z","due_on":"2018-04-13T07:00:00Z","closed_at":null},"comments":10,"created_at":"2018-04-04T15:28:25Z","updated_at":"2018-04-11T16:28:51Z","closed_at":"2018-04-11T16:28:50Z","author_association":"CONTRIBUTOR","body":"### Description\r\nWhile using `name_lookup()` with `limit` higher than 1000, thus using the recently implemented internal paging, I get duplicates for some `limit` values, although number of rows agree with such `limit` value.\r\n\r\n### Examples\r\nHere below the code I run, with `limit = 5000` as example:\r\n```r\r\ntest_taxon <- name_lookup(\r\n  datasetKey = \"3f8a1297-3259-4700-91fc-acc4170b27ce\",\r\n  limit = 5000,\r\n  return = \"data\")\r\nlength(test_taxon$key)\r\nlength(unique(test_taxon$key))\r\n\r\n# directly via JSON \r\nchunk1 <- jsonlite::read_json(path = \"https://api.gbif.org/v1/species/search?limit=5000&offset=0&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\", simplifyVector = TRUE)\r\nchunk2 <- jsonlite::read_json(path = \"https://api.gbif.org/v1/species/search?limit=4000&offset=1000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\", simplifyVector = TRUE)\r\nchunk3 <- jsonlite::read_json(path = \"https://api.gbif.org/v1/species/search?limit=3000&offset=2000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\", simplifyVector = TRUE)\r\nchunk4 <- jsonlite::read_json(path = \"https://api.gbif.org/v1/species/search?limit=2000&offset=3000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\", simplifyVector = TRUE)\r\nchunk5 <- jsonlite::read_json(path = \"https://api.gbif.org/v1/species/search?limit=1000&offset=4000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\", simplifyVector = TRUE)\r\n\r\n# check JSON results\r\nany(chunk1$results$key %in% chunk2$results$key)\r\nlength(chunk1$results$key[which(chunk1$results$key %in% chunk2$results$key)])\r\nany(chunk2$results$key %in% chunk3$results$key)\r\nchunk2$results$key[which(chunk2$results$key %in% chunk3$results$key)]\r\nlength(chunk2$results$key[which(chunk2$results$key %in% chunk3$results$key)])\r\nany(chunk3$results$key %in% chunk4$results$key)\r\nchunk3$results$key[which(chunk3$results$key %in% chunk4$results$key)]\r\nany(chunk4$results$key %in% chunk5$results$key)\r\nchunk4$results$key[which(chunk4$results$key %in% chunk5$results$key)]\r\nlength(chunk4$results$key[which(chunk4$results$key %in% chunk5$results$key)])\r\n\r\n```\r\nI get 4442 unique records instead of 5000. \r\nIn the table below you can see the number of unique keys for different values of `limit`:\r\n\r\nlimit | unique records\r\n--- | ---\r\n1000 | 1000\r\n1500 | 1500\r\n2000 | 1931\r\n2500 | 2387\r\n3000 | 3000\r\n3500 | 3387 \r\n4000 | 3887\r\n5000 | 4442\r\n6000 | 5545\r\n\r\nHowever, these results are not stable. Repeating the queries after a while I obtain different results. Interesting to notice that the very first queries (constantly changing `limit` value) return the correct number of records (in the case the table is related to, I started with `limit = 3000`).\r\n\r\nAfter a detailed debugging I found that:\r\n1. In each chunk of 1000 there are exactly 1000 unique records\r\n2. All the `url` in `<crul response>` seem correct. See here below the queries for `limit = 5000` (which I added in the code above):\r\n- https://api.gbif.org/v1/species/search?limit=5000&offset=0&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\r\n- https://api.gbif.org/v1/species/search?limit=4000&offset=1000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\r\n- https://api.gbif.org/v1/species/search?limit=3000&offset=2000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\r\n- https://api.gbif.org/v1/species/search?limit=2000&offset=3000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\r\n- https://api.gbif.org/v1/species/search?limit=1000&offset=4000&datasetKey=3f8a1297-3259-4700-91fc-acc4170b27ce\r\n\r\nBased on the code above, I can see that there is an overlapping among records of different chunks.\r\nAfter few minutes, i run the json queries again and I obtained surprisingly different results. For this reason, I started to think the problem is not of the paging implementation or other rgbif functionalities, but at the GBIF API side. I would like to hear your opinion on it first.\r\nTo conclude, the function `name_usage()` uses the same kind of implementation and it works perfectly, no duplicates have never found. So, I doubt the bug is in the implementation of paging, as I assumed at first glance.\r\nI hope I described clearly the problem. Thanks.\r\n\r\n<details> <summary><strong>Session Info</strong></summary>\r\n```r\r\nSession info ---------------------------------------------------------------------------------------------------------------------\r\n setting  value                       \r\n version  R version 3.4.4 (2018-03-15)\r\n system   x86_64, mingw32             \r\n ui       RStudio (1.1.442)           \r\n language (EN)                        \r\n collate  Dutch_Belgium.1252          \r\n tz       Europe/Paris                \r\n date     2018-04-04                  \r\n\r\nPackages -------------------------------------------------------------------------------------------------------------------------\r\n package    * version    date       source                         \r\n assertthat   0.2.0      2017-04-11 CRAN (R 3.4.2)                 \r\n base       * 3.4.4      2018-03-15 local                          \r\n bindr        0.1        2016-11-13 CRAN (R 3.4.2)                 \r\n bindrcpp     0.2        2017-06-17 CRAN (R 3.4.2)                 \r\n colorspace   1.3-2      2016-12-14 CRAN (R 3.4.2)                 \r\n compiler     3.4.4      2018-03-15 local                          \r\n crul         0.5.2.9100 2018-03-19 Github (ropensci/crul@8755c7c) \r\n curl         3.2        2018-03-19 local                          \r\n data.table   1.10.4-3   2017-10-27 CRAN (R 3.4.2)                 \r\n datasets   * 3.4.4      2018-03-15 local                          \r\n devtools     1.13.4     2017-11-09 CRAN (R 3.4.2)                 \r\n digest       0.6.13     2017-12-14 CRAN (R 3.4.3)                 \r\n dplyr      * 0.7.4      2017-09-28 CRAN (R 3.4.3)                 \r\n drat         0.1.4      2017-12-16 CRAN (R 3.4.3)                 \r\n fortunes     1.5-4      2016-12-29 CRAN (R 3.4.1)                 \r\n geoaxe       0.1.0      2016-02-19 CRAN (R 3.4.2)                 \r\n ggplot2      2.2.1      2016-12-30 CRAN (R 3.4.2)                 \r\n git2r        0.19.0     2017-07-19 CRAN (R 3.4.2)                 \r\n glue         1.2.0      2017-10-29 CRAN (R 3.4.2)                 \r\n graphics   * 3.4.4      2018-03-15 local                          \r\n grDevices  * 3.4.4      2018-03-15 local                          \r\n grid         3.4.4      2018-03-15 local                          \r\n gtable       0.2.0      2016-02-26 CRAN (R 3.4.2)                 \r\n httr         1.3.1      2017-08-20 CRAN (R 3.4.2)                 \r\n jsonlite   * 1.5        2017-06-01 CRAN (R 3.4.2)                 \r\n knitr        1.20       2018-02-20 CRAN (R 3.4.4)                 \r\n lattice      0.20-35    2017-03-25 CRAN (R 3.4.4)                 \r\n lazyeval   * 0.2.1      2017-10-29 CRAN (R 3.4.2)                 \r\n magrittr   * 1.5        2014-11-22 CRAN (R 3.4.2)                 \r\n memoise      1.1.0      2017-04-21 CRAN (R 3.4.2)                 \r\n methods    * 3.4.4      2018-03-15 local                          \r\n munsell      0.4.3      2016-02-13 CRAN (R 3.4.2)                 \r\n oai          0.2.2      2016-11-24 CRAN (R 3.4.2)                 \r\n pillar       1.1.0      2018-01-14 CRAN (R 3.4.3)                 \r\n pkgconfig    2.0.1      2017-03-21 CRAN (R 3.4.2)                 \r\n plyr         1.8.4      2016-06-08 CRAN (R 3.4.2)                 \r\n purrr      * 0.2.4      2017-10-18 CRAN (R 3.4.3)                 \r\n R6           2.2.2      2017-06-17 CRAN (R 3.4.2)                 \r\n Rcpp         0.12.15    2018-01-20 CRAN (R 3.4.3)                 \r\n rgbif      * 0.9.9.9412 2018-04-04 Github (ropensci/rgbif@2ebc348)\r\n rgeos        0.3-26     2017-10-31 CRAN (R 3.4.3)                 \r\n rlang        0.2.0      2018-02-20 CRAN (R 3.4.4)                 \r\n rstudioapi   0.7        2017-09-07 CRAN (R 3.4.2)                 \r\n scales       0.5.0      2017-08-24 CRAN (R 3.4.3)                 \r\n sp           1.2-7      2018-01-19 CRAN (R 3.4.4)                 \r\n stats      * 3.4.4      2018-03-15 local                          \r\n stringi      1.1.7      2018-03-12 CRAN (R 3.4.4)                 \r\n stringr    * 1.3.0      2018-02-19 CRAN (R 3.4.4)                 \r\n taxa       * 0.2.0.9117 2018-03-19 Github (ropensci/taxa@b9b108c) \r\n tibble       1.4.2      2018-01-22 CRAN (R 3.4.3)                 \r\n tidyr      * 0.8.0      2018-01-29 CRAN (R 3.4.4)                 \r\n tools        3.4.4      2018-03-15 local                          \r\n triebeard    0.3.0      2016-08-04 CRAN (R 3.4.2)                 \r\n urltools     1.7.0      2018-01-20 CRAN (R 3.4.4)                 \r\n utils      * 3.4.4      2018-03-15 local                          \r\n whisker      0.3-2      2013-04-28 CRAN (R 3.4.2)                 \r\n withr        2.1.1      2017-12-19 CRAN (R 3.4.3)                 \r\n xml2         1.2.0      2018-01-24 CRAN (R 3.4.3)                 \r\n yaml         2.1.16     2017-12-12 CRAN (R 3.4.3)  \r\n```\r\n</details>\r\n"}}
{"actor":{"display_login":"sckott"}}