{"action":"created","issue":{"url":"https://api.github.com/repos/genouest/biomaj/issues/57","repository_url":"https://api.github.com/repos/genouest/biomaj","labels_url":"https://api.github.com/repos/genouest/biomaj/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/genouest/biomaj/issues/57/comments","events_url":"https://api.github.com/repos/genouest/biomaj/issues/57/events","html_url":"https://github.com/genouest/biomaj/issues/57","id":170842439,"number":57,"title":"FTP list fails with large number of file","user":{"login":"horkko","id":9282917,"avatar_url":"https://avatars.githubusercontent.com/u/9282917?v=3","gravatar_id":"","url":"https://api.github.com/users/horkko","html_url":"https://github.com/horkko","followers_url":"https://api.github.com/users/horkko/followers","following_url":"https://api.github.com/users/horkko/following{/other_user}","gists_url":"https://api.github.com/users/horkko/gists{/gist_id}","starred_url":"https://api.github.com/users/horkko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/horkko/subscriptions","organizations_url":"https://api.github.com/users/horkko/orgs","repos_url":"https://api.github.com/users/horkko/repos","events_url":"https://api.github.com/users/horkko/events{/privacy}","received_events_url":"https://api.github.com/users/horkko/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/genouest/biomaj/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-08-12T10:24:26Z","updated_at":"2016-08-16T12:53:19Z","closed_at":null,"body":"Hi,\r\n\r\nI'm facing a problem with a bank that download a lots of files.\r\nI'm trying to get files from `Genbank WGS` (`ftp://ftp.ncbi.nlm.nih.gov/genbank/wgs`).\r\nThis directory contains around 84,000 files. Then when I run biomaj, I always get this error:\r\n```\r\n[ftp.py:list:277] Could not get errcode:(56, 'FTP response reading failed')\r\n```\r\nIt somehow mean that the ftp reponse is longer than expected to retrieve the list of files.\r\nI've try to set some options like (FTP_RESPONSE_TIME) but no success.\r\nSo my question is, do you have any clue on how to avoid such problem?\r\nThe problem is similar using `Firefox`, listing `wgs` directory ends with a blank page.\r\nHowever, using `ncftp`, command `dir` succeed but we need to wait around a minute to get the file list.\r\n\r\nThanks\r\n\r\nEmmanuel"},"comment":{"url":"https://api.github.com/repos/genouest/biomaj/issues/comments/240092381","html_url":"https://github.com/genouest/biomaj/issues/57#issuecomment-240092381","issue_url":"https://api.github.com/repos/genouest/biomaj/issues/57","id":240092381,"user":{"login":"horkko","id":9282917,"avatar_url":"https://avatars.githubusercontent.com/u/9282917?v=3","gravatar_id":"","url":"https://api.github.com/users/horkko","html_url":"https://github.com/horkko","followers_url":"https://api.github.com/users/horkko/followers","following_url":"https://api.github.com/users/horkko/following{/other_user}","gists_url":"https://api.github.com/users/horkko/gists{/gist_id}","starred_url":"https://api.github.com/users/horkko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/horkko/subscriptions","organizations_url":"https://api.github.com/users/horkko/orgs","repos_url":"https://api.github.com/users/horkko/repos","events_url":"https://api.github.com/users/horkko/events{/privacy}","received_events_url":"https://api.github.com/users/horkko/received_events","type":"User","site_admin":false},"created_at":"2016-08-16T12:53:18Z","updated_at":"2016-08-16T12:53:18Z","body":"Hi,\r\n\r\nMaybe a clue to fix this problem. Using curl option `CURLOPT_DIRLISTONLY` partially solves the problem.\r\n```\r\n...\r\n< 200 Type set to A\r\n> NLST\r\n< 150 Opening ASCII mode data connection for file list\r\n* Maxdownload = -1\r\n* Remembering we are in dir \"genbank/wgs/\"\r\n< 226 Transfer complete\r\n* Connection #0 to host ftp.ncbi.nlm.nih.gov left intact\r\n...\r\n```\r\n At least the dir listing is available, however, we fail later in the workflow as this cul option only list the directory content, is does not retrieve metadata such as permissions, date, size etc...\r\nSo, for a bank having a `release.file` set, it should not be a problem, but for bank which base its release number on date of last updated file, then we end with such error:\r\n```\r\n2016-08-16 14:47:55,917 ERROR [root][MainThread] [workflow.py:start:135] Workflow:downloadException:'year'\r\n```\r\nwhich the build of the release based on last updated files :(\r\nDoes this new option is good start to solve the problem?\r\n"}}
{"actor":{"display_login":"horkko"}}