{"action":"created","issue":{"url":"https://api.github.com/repos/ncw/rclone/issues/355","repository_url":"https://api.github.com/repos/ncw/rclone","labels_url":"https://api.github.com/repos/ncw/rclone/issues/355/labels{/name}","comments_url":"https://api.github.com/repos/ncw/rclone/issues/355/comments","events_url":"https://api.github.com/repos/ncw/rclone/issues/355/events","html_url":"https://github.com/ncw/rclone/issues/355","id":133747405,"number":355,"title":"v1.27 Serious bug - rclone recursive algorithm missing some files/directories on Google drive source","user":{"login":"TimJWatts","id":4126880,"avatar_url":"https://avatars.githubusercontent.com/u/4126880?v=3","gravatar_id":"","url":"https://api.github.com/users/TimJWatts","html_url":"https://github.com/TimJWatts","followers_url":"https://api.github.com/users/TimJWatts/followers","following_url":"https://api.github.com/users/TimJWatts/following{/other_user}","gists_url":"https://api.github.com/users/TimJWatts/gists{/gist_id}","starred_url":"https://api.github.com/users/TimJWatts/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TimJWatts/subscriptions","organizations_url":"https://api.github.com/users/TimJWatts/orgs","repos_url":"https://api.github.com/users/TimJWatts/repos","events_url":"https://api.github.com/users/TimJWatts/events{/privacy}","received_events_url":"https://api.github.com/users/TimJWatts/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-02-15T15:51:04Z","updated_at":"2016-02-16T17:44:14Z","closed_at":null,"body":"I have a Drive on Google Apps for Education - and sync is not catching all the files.\r\n\r\nInvestigating further, I noticed:\r\n\r\nrclone@phoenix:/vol/kdldata/GoogleDriveBackup$ /usr/local/bin/rclone --stats=0 ls 'twatts_kdl:/Shared Drive/' | grep 'Non Pay'\r\n=========================================\r\n   180534 KDL Group Share/KDL Admin/Finances/2015-2016/KDL Non Pay Finances 2015-2016.xlsx\r\n\r\nTransferred:            0 Bytes (   0.00 kByte/s)\r\nErrors:                 0\r\nChecks:                 0\r\nTransferred:            0\r\nElapsed time:       29.5s\r\n=========================================\r\nwhich is correct:\r\n\r\nBUT\r\n\r\nrclone@phoenix:/vol/kdldata/GoogleDriveBackup$ /usr/local/bin/rclone --stats=0 ls 'twatts_kdl:/' | grep 'Non Pay'\r\n=========================================\r\nTransferred:            0 Bytes (   0.00 kByte/s)\r\nErrors:                 0\r\nChecks:                 0\r\nTransferred:            0\r\nElapsed time:       59.8s\r\n=========================================\r\n\r\nStarting a recursive listing from one level higher in the drive tree is causing certain files AND folders lower down to disappear from the source listing. The missing files/folders are consistent with what is not on the local target after a sync operation.\r\n\r\nThis suggests a problem with the recursion algorithm.\r\n\r\n\r\n\r\nNB - files are owned by various people but I have access to all.\r\n\r\nTo see how many missing files there are:\r\n\r\n/usr/local/bin/rclone --stats=0 ls 'twatts_kdl:/Shared Drive/' | wc -l\r\n===========================\r\n220\r\n===========================\r\n\r\n/usr/local/bin/rclone --stats=0 ls 'twatts_kdl:/' | grep 'Shared Drive' | wc -l\r\n===========================\r\n157\r\n===========================\r\n\r\nSo it is missing 63 files!\r\n\r\nInvestigating further, I produced a list of the missing files.\r\n\r\nEvery single missing file is a Google Doc. Many of them were created or imported by me and reside in a tree where I created every node, so that seems to rule out it getting confused over permissions.\r\n\r\nIf I select my whole Google Drive in Chrome, do a \"Download\", unpack the zipfile and look in the \"Shared Drive\" directory and run \"find -type f | wc -l\" I get 220 files, the expected number, which is a further verification that I have permissions. My account is also the GA4Education admin account.\r\n\r\nHappy to help debug this -\r\n\r\nCheers, Tim Watts"},"comment":{"url":"https://api.github.com/repos/ncw/rclone/issues/comments/184794702","html_url":"https://github.com/ncw/rclone/issues/355#issuecomment-184794702","issue_url":"https://api.github.com/repos/ncw/rclone/issues/355","id":184794702,"user":{"login":"ncw","id":536803,"avatar_url":"https://avatars.githubusercontent.com/u/536803?v=3","gravatar_id":"","url":"https://api.github.com/users/ncw","html_url":"https://github.com/ncw","followers_url":"https://api.github.com/users/ncw/followers","following_url":"https://api.github.com/users/ncw/following{/other_user}","gists_url":"https://api.github.com/users/ncw/gists{/gist_id}","starred_url":"https://api.github.com/users/ncw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ncw/subscriptions","organizations_url":"https://api.github.com/users/ncw/orgs","repos_url":"https://api.github.com/users/ncw/repos","events_url":"https://api.github.com/users/ncw/events{/privacy}","received_events_url":"https://api.github.com/users/ncw/received_events","type":"User","site_admin":false},"created_at":"2016-02-16T17:44:13Z","updated_at":"2016-02-16T17:44:13Z","body":"Great glad that has fixed it. Duplicate file names are a known gotcha in drive - I agree it wasn't a sensible design choice! \r\n\r\nThe fix is committed to master and will be in the next release. "}}