{"action":"created","issue":{"url":"https://api.github.com/repos/mscdex/ssh2/issues/674","repository_url":"https://api.github.com/repos/mscdex/ssh2","labels_url":"https://api.github.com/repos/mscdex/ssh2/issues/674/labels{/name}","comments_url":"https://api.github.com/repos/mscdex/ssh2/issues/674/comments","events_url":"https://api.github.com/repos/mscdex/ssh2/issues/674/events","html_url":"https://github.com/mscdex/ssh2/issues/674","id":307399762,"number":674,"title":"Question: SFTP best practice for reading files in a directory one at a time?","user":{"login":"leosco","id":11724770,"avatar_url":"https://avatars0.githubusercontent.com/u/11724770?v=4","gravatar_id":"","url":"https://api.github.com/users/leosco","html_url":"https://github.com/leosco","followers_url":"https://api.github.com/users/leosco/followers","following_url":"https://api.github.com/users/leosco/following{/other_user}","gists_url":"https://api.github.com/users/leosco/gists{/gist_id}","starred_url":"https://api.github.com/users/leosco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leosco/subscriptions","organizations_url":"https://api.github.com/users/leosco/orgs","repos_url":"https://api.github.com/users/leosco/repos","events_url":"https://api.github.com/users/leosco/events{/privacy}","received_events_url":"https://api.github.com/users/leosco/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2018-03-21T20:08:24Z","updated_at":"2018-03-21T22:00:46Z","closed_at":null,"author_association":"NONE","body":"``` JavaScript\r\nimport { Client } from 'ssh2';\r\nimport _ from 'underscore';\r\n\r\n/*\r\n  Given an sftp stream, directory path, and file object,\r\n  get file contents and print them\r\n*/\r\nfunction ingestFile(stream, dir, file) {\r\n  const enc = 'utf8';\r\n  const buf = Buffer.from('', enc);\r\n  const path = `${dir}/${file.filename}`;\r\n\r\n  console.log(`> opening ${path}`);\r\n  stream.open(path, 'r', (err, handle) => {\r\n    if (err) throw err;\r\n    stream.readData(\r\n      handle,\r\n      buf,\r\n      0,\r\n      file.attrs.Stats.size,\r\n      0,\r\n      (err, bytesRead, newBuf) => {\r\n        if (err) throw err;\r\n        console.log(`> read ${bytesRead} bytes, got contents: ${newBuf.toString(enc)}`);\r\n      },\r\n    );\r\n  });\r\n}\r\n\r\nfunction importFtpOrders(servers) {\r\n  const conn = new Client();\r\n\r\n  servers.forEach((serv) => {\r\n    const cOpts = _.pick(serv, ['host', 'port', 'username', 'password', 'algorithms']);\r\n\r\n    conn.on('ready', () => {\r\n      console.log('> connection ready');\r\n      conn.sftp((err, sftp) => {\r\n        if (err) throw err;\r\n        sftp.readdir(serv.dir, (err, list) => {\r\n          if (err) throw err;\r\n          console.log(`> discovered ${list.length} files in ${serv.dir}, first file:`);\r\n          console.dir(list[0]);\r\n\r\n          list.forEach((f) => {\r\n            ingestFile(sftp, serv.dir, f);\r\n          });\r\n\r\n          conn.end();\r\n        });\r\n      });\r\n    });\r\n\r\n    conn.connect(cOpts);\r\n  });\r\n\r\n  return servers.length;\r\n}\r\n```\r\n\r\nHi there, I'm having some trouble with the above code.\r\n\r\nI'm trying to connect to a list of servers, and for each of them: iterate through the files in a directory and extract their contents into memory. These are small text files that I need to parse and finally insert into a database.\r\n\r\nI'm able to connect and list the contents of the directory, but when I try to open the file using the absolute path and `sftp.open` (or `stream.open` in my top function), it fails silently. No errors are thrown and nothing gets printed to the console. I read all the [docs](https://github.com/mscdex/ssh2-streams/blob/master/SFTPStream.md) on the sftp-stream methods and I *think* this is the way I need to do it, but I could be missing something.\r\n\r\nAny help would be greatly appreciated."},"comment":{"url":"https://api.github.com/repos/mscdex/ssh2/issues/comments/375110247","html_url":"https://github.com/mscdex/ssh2/issues/674#issuecomment-375110247","issue_url":"https://api.github.com/repos/mscdex/ssh2/issues/674","id":375110247,"user":{"login":"leosco","id":11724770,"avatar_url":"https://avatars0.githubusercontent.com/u/11724770?v=4","gravatar_id":"","url":"https://api.github.com/users/leosco","html_url":"https://github.com/leosco","followers_url":"https://api.github.com/users/leosco/followers","following_url":"https://api.github.com/users/leosco/following{/other_user}","gists_url":"https://api.github.com/users/leosco/gists{/gist_id}","starred_url":"https://api.github.com/users/leosco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leosco/subscriptions","organizations_url":"https://api.github.com/users/leosco/orgs","repos_url":"https://api.github.com/users/leosco/repos","events_url":"https://api.github.com/users/leosco/events{/privacy}","received_events_url":"https://api.github.com/users/leosco/received_events","type":"User","site_admin":false},"created_at":"2018-03-21T22:00:46Z","updated_at":"2018-03-21T22:00:46Z","author_association":"NONE","body":"Okay, @mscdex so I condensed the function to a far more basic version, just getting the first file from the first server. And I'm still getting the:\r\n``` Bash\r\nTypeError: sftp.readData is not a function\r\n```\r\n\r\nHere's the code:\r\n``` JavaScript\r\n  const conn = new Client();\r\n  const serv = servers[0];\r\n\r\n  const cOpts = _.pick(serv, ['host', 'port', 'username', 'password', 'algorithms']);\r\n\r\n  conn.on('ready', () => {\r\n    console.log('> connection ready');\r\n    conn.sftp((err, sftp) => {\r\n      sftp.readdir(serv.dir, (err, list) => {\r\n        console.log(`> discovered ${list.length} files in ${serv.dir}, first file:`);\r\n        console.dir(list[0]);\r\n\r\n        // list.forEach((f, i) => {\r\n        //   ingestFile(conn, sftp, serv.dir, f, list.length - i - 1);\r\n        // });\r\n\r\n        const file = list[0];\r\n        const path = `${serv.dir}/${file.filename}`;\r\n        console.log(`> opening ${path}...`);\r\n\r\n        sftp.open(path, 'r', (err, handle) => {\r\n          const buf = Buffer.from('', 'utf8');\r\n          sftp.readData(handle, buf, 0, file.attrs.size, 0, (err, bytesRead, newBuf) => {\r\n            console.log(newBuf.toString('utf8'));\r\n            conn.end();\r\n          });\r\n        });\r\n      });\r\n    });\r\n  });\r\n\r\n  conn.connect(cOpts);\r\n```"}}
{"actor":{"display_login":"leosco"}}