{"action":"created","comment":{"url":"https://api.github.com/repos/jinzhu/gorm/pulls/comments/84461257","pull_request_review_id":5239097,"id":84461257,"diff_hunk":"@@ -264,6 +264,58 @@ func (s *DB) Find(out interface{}, where ...interface{}) *DB {\n \treturn s.clone().NewScope(out).inlineCondition(where...).callCallbacks(s.parent.callbacks.queries).db\n }\n \n+// FindInBatches find records in batches for large tables and allow callbacks to operate\n+// on each batch\n+func (s *DB) FindInBatches(out interface{}, cb func(), where ...interface{}) *DB {\n+\tvar limit interface{}\n+\tnewDB := s.clone()\n+\tif len(newDB.search.orders) != 0 {\n+\t\t//ordering by non primary col not supported - can cause infinite loop\n+\t\tnewDB.search.orders = []interface{}{}\n+\t}\n+\tvalue := indirect(reflect.ValueOf(out))\n+\tif value.Kind() != reflect.Array && value.Kind() != reflect.Slice {\n+\t\tnewDB.AddError(ErrUnsupportedType)\n+\t\treturn newDB\n+\t}\n+\tif newDB.search.limit == -1 {\n+\t\tlimit = 1000\n+\t} else {\n+\t\tlimit = newDB.search.limit\n+\t}\n+\terr := newDB.Limit(limit).Find(out, where...).Error\n+\trecords := indirect(reflect.ValueOf(out))\n+\tfor err == nil && records.Len() > 0 {\n+\t\tcb()\n+\t\tlastRecord := records.Index(records.Len() - 1).Interface()\n+\t\tscope := s.NewScope(lastRecord)\n+\t\terr = newDB.Limit(limit).\n+\t\t\tWhere(fmt.Sprintf(\"`%v` > '%v'\", scope.PrimaryKey(), scope.PrimaryKeyValue())).","path":"main.go","position":30,"original_position":30,"commit_id":"48d0897d3e391ba6a0e3b814c9d1b546b00b4c2e","original_commit_id":"48d0897d3e391ba6a0e3b814c9d1b546b00b4c2e","user":{"login":"zardak","id":5513218,"avatar_url":"https://avatars.githubusercontent.com/u/5513218?v=3","gravatar_id":"","url":"https://api.github.com/users/zardak","html_url":"https://github.com/zardak","followers_url":"https://api.github.com/users/zardak/followers","following_url":"https://api.github.com/users/zardak/following{/other_user}","gists_url":"https://api.github.com/users/zardak/gists{/gist_id}","starred_url":"https://api.github.com/users/zardak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zardak/subscriptions","organizations_url":"https://api.github.com/users/zardak/orgs","repos_url":"https://api.github.com/users/zardak/repos","events_url":"https://api.github.com/users/zardak/events{/privacy}","received_events_url":"https://api.github.com/users/zardak/received_events","type":"User","site_admin":false},"body":"The traditional way to do batching is `OFFSET` with `LIMIT`.\r\nStill, I'm not sure whether this will get merged, because basic batching is pretty trivial to implement and it often needs to be tailored to specific use-cases.","created_at":"2016-10-21T11:10:33Z","updated_at":"2016-10-21T11:10:34Z","html_url":"https://github.com/jinzhu/gorm/pull/1235#discussion_r84461257","pull_request_url":"https://api.github.com/repos/jinzhu/gorm/pulls/1235","_links":{"self":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/comments/84461257"},"html":{"href":"https://github.com/jinzhu/gorm/pull/1235#discussion_r84461257"},"pull_request":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/1235"}}},"pull_request":{"url":"https://api.github.com/repos/jinzhu/gorm/pulls/1235","id":90248631,"html_url":"https://github.com/jinzhu/gorm/pull/1235","diff_url":"https://github.com/jinzhu/gorm/pull/1235.diff","patch_url":"https://github.com/jinzhu/gorm/pull/1235.patch","issue_url":"https://api.github.com/repos/jinzhu/gorm/issues/1235","number":1235,"state":"open","locked":false,"title":"Added FindInBatches, FindEach","user":{"login":"ronna-s","id":744636,"avatar_url":"https://avatars.githubusercontent.com/u/744636?v=3","gravatar_id":"","url":"https://api.github.com/users/ronna-s","html_url":"https://github.com/ronna-s","followers_url":"https://api.github.com/users/ronna-s/followers","following_url":"https://api.github.com/users/ronna-s/following{/other_user}","gists_url":"https://api.github.com/users/ronna-s/gists{/gist_id}","starred_url":"https://api.github.com/users/ronna-s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronna-s/subscriptions","organizations_url":"https://api.github.com/users/ronna-s/orgs","repos_url":"https://api.github.com/users/ronna-s/repos","events_url":"https://api.github.com/users/ronna-s/events{/privacy}","received_events_url":"https://api.github.com/users/ronna-s/received_events","type":"User","site_admin":false},"body":"I inquired about this in the past and in the end I implemented something on my own. I now reimplemented it to follow the code style of the project (I hope)...\r\n\r\nThe purpose is to iterate over records of a very large table without using too much memory.\r\n\r\nUsage (very similar to RoR) - full example below:\r\n\r\n```go\r\ndb.FindEach(&user, func() {\r\n\tfmt.Println(user.Name)\r\n})\r\n```\r\nwill result in:\r\n```sql\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL LIMIT 1000\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL AND ((`id` > '1000')) LIMIT 1000\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL AND ((`id` > '2000')) LIMIT 1000\r\n...\r\n```\r\nIf limit was already set, it will be used instead of defaulting to 1000. So the following:\r\n```go\r\ndb.Limit(3).FindEach(&user, func() {\r\n\tfmt.Println(user.Name)\r\n})\r\n```\r\nWill generate:\r\n```sql\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL LIMIT 3\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL AND ((`id` > '3')) LIMIT 3\r\nSELECT * FROM \"users\"  WHERE \"users\".deleted_at IS NULL AND ((`id` > '6')) LIMIT 3\r\n...\r\n```\r\n\r\nNote: I think there might be an issue with different dialects support with the explicit \r\n`Where(fmt.Sprintf(\"`%v` > '%v'\", scope.PrimaryKey(), scope.PrimaryKeyValue()))` I added to forward the iteration.\r\n\r\nLet me know what you think...\r\n\r\n```go\r\npackage main\r\n\r\nimport (\r\n\t\"github.com/jinzhu/gorm\"\r\n\t_ \"github.com/jinzhu/gorm/dialects/sqlite\"\r\n)\r\n\r\ntype Product struct {\r\n\tgorm.Model\r\n\tPrice uint\r\n}\r\n\r\nfunc main() {\r\n\tvar product Product\r\n\tvar i uint\r\n\r\n\tdb, err := gorm.Open(\"sqlite3\", \"test.db\")\r\n\tif err != nil {\r\n\t\tpanic(\"failed to connect database\")\r\n\t}\r\n\tdefer db.Close()\r\n\tdb.LogMode(true)\r\n\tdb.DropTable(&product)\r\n\t// Migrate the schema\r\n\tdb.AutoMigrate(&Product{})\r\n\tfor i = 0; i < 10000; i++ {\r\n\t\tdb.Save(&Product{Price: i})\r\n\t}\r\n\tdb.FindEach(&product, func() {\r\n\t})\r\n}\r\n\r\n```","created_at":"2016-10-20T17:48:27Z","updated_at":"2016-10-21T11:10:34Z","closed_at":null,"merged_at":null,"merge_commit_sha":"3d672cb1cdcaee98daf1844f3e9d9cd4940c1b96","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/jinzhu/gorm/pulls/1235/commits","review_comments_url":"https://api.github.com/repos/jinzhu/gorm/pulls/1235/comments","review_comment_url":"https://api.github.com/repos/jinzhu/gorm/pulls/comments{/number}","comments_url":"https://api.github.com/repos/jinzhu/gorm/issues/1235/comments","statuses_url":"https://api.github.com/repos/jinzhu/gorm/statuses/48d0897d3e391ba6a0e3b814c9d1b546b00b4c2e","head":{"label":"ronna-s:find-in-batches","ref":"find-in-batches","sha":"48d0897d3e391ba6a0e3b814c9d1b546b00b4c2e","user":{"login":"ronna-s","id":744636,"avatar_url":"https://avatars.githubusercontent.com/u/744636?v=3","gravatar_id":"","url":"https://api.github.com/users/ronna-s","html_url":"https://github.com/ronna-s","followers_url":"https://api.github.com/users/ronna-s/followers","following_url":"https://api.github.com/users/ronna-s/following{/other_user}","gists_url":"https://api.github.com/users/ronna-s/gists{/gist_id}","starred_url":"https://api.github.com/users/ronna-s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronna-s/subscriptions","organizations_url":"https://api.github.com/users/ronna-s/orgs","repos_url":"https://api.github.com/users/ronna-s/repos","events_url":"https://api.github.com/users/ronna-s/events{/privacy}","received_events_url":"https://api.github.com/users/ronna-s/received_events","type":"User","site_admin":false},"repo":{"id":71450466,"name":"gorm","full_name":"ronna-s/gorm","owner":{"login":"ronna-s","id":744636,"avatar_url":"https://avatars.githubusercontent.com/u/744636?v=3","gravatar_id":"","url":"https://api.github.com/users/ronna-s","html_url":"https://github.com/ronna-s","followers_url":"https://api.github.com/users/ronna-s/followers","following_url":"https://api.github.com/users/ronna-s/following{/other_user}","gists_url":"https://api.github.com/users/ronna-s/gists{/gist_id}","starred_url":"https://api.github.com/users/ronna-s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ronna-s/subscriptions","organizations_url":"https://api.github.com/users/ronna-s/orgs","repos_url":"https://api.github.com/users/ronna-s/repos","events_url":"https://api.github.com/users/ronna-s/events{/privacy}","received_events_url":"https://api.github.com/users/ronna-s/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ronna-s/gorm","description":"The fantastic ORM library for Golang, aims to be developer friendly","fork":true,"url":"https://api.github.com/repos/ronna-s/gorm","forks_url":"https://api.github.com/repos/ronna-s/gorm/forks","keys_url":"https://api.github.com/repos/ronna-s/gorm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ronna-s/gorm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ronna-s/gorm/teams","hooks_url":"https://api.github.com/repos/ronna-s/gorm/hooks","issue_events_url":"https://api.github.com/repos/ronna-s/gorm/issues/events{/number}","events_url":"https://api.github.com/repos/ronna-s/gorm/events","assignees_url":"https://api.github.com/repos/ronna-s/gorm/assignees{/user}","branches_url":"https://api.github.com/repos/ronna-s/gorm/branches{/branch}","tags_url":"https://api.github.com/repos/ronna-s/gorm/tags","blobs_url":"https://api.github.com/repos/ronna-s/gorm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ronna-s/gorm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ronna-s/gorm/git/refs{/sha}","trees_url":"https://api.github.com/repos/ronna-s/gorm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ronna-s/gorm/statuses/{sha}","languages_url":"https://api.github.com/repos/ronna-s/gorm/languages","stargazers_url":"https://api.github.com/repos/ronna-s/gorm/stargazers","contributors_url":"https://api.github.com/repos/ronna-s/gorm/contributors","subscribers_url":"https://api.github.com/repos/ronna-s/gorm/subscribers","subscription_url":"https://api.github.com/repos/ronna-s/gorm/subscription","commits_url":"https://api.github.com/repos/ronna-s/gorm/commits{/sha}","git_commits_url":"https://api.github.com/repos/ronna-s/gorm/git/commits{/sha}","comments_url":"https://api.github.com/repos/ronna-s/gorm/comments{/number}","issue_comment_url":"https://api.github.com/repos/ronna-s/gorm/issues/comments{/number}","contents_url":"https://api.github.com/repos/ronna-s/gorm/contents/{+path}","compare_url":"https://api.github.com/repos/ronna-s/gorm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ronna-s/gorm/merges","archive_url":"https://api.github.com/repos/ronna-s/gorm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ronna-s/gorm/downloads","issues_url":"https://api.github.com/repos/ronna-s/gorm/issues{/number}","pulls_url":"https://api.github.com/repos/ronna-s/gorm/pulls{/number}","milestones_url":"https://api.github.com/repos/ronna-s/gorm/milestones{/number}","notifications_url":"https://api.github.com/repos/ronna-s/gorm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ronna-s/gorm/labels{/name}","releases_url":"https://api.github.com/repos/ronna-s/gorm/releases{/id}","deployments_url":"https://api.github.com/repos/ronna-s/gorm/deployments","created_at":"2016-10-20T10:06:27Z","updated_at":"2016-10-20T10:06:28Z","pushed_at":"2016-10-20T21:13:11Z","git_url":"git://github.com/ronna-s/gorm.git","ssh_url":"git@github.com:ronna-s/gorm.git","clone_url":"https://github.com/ronna-s/gorm.git","svn_url":"https://github.com/ronna-s/gorm","homepage":"http://jinzhu.github.io/gorm","size":3232,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"jinzhu:master","ref":"master","sha":"a667ab842714a9fe4d53b987e876b1687d07b270","user":{"login":"jinzhu","id":6843,"avatar_url":"https://avatars.githubusercontent.com/u/6843?v=3","gravatar_id":"","url":"https://api.github.com/users/jinzhu","html_url":"https://github.com/jinzhu","followers_url":"https://api.github.com/users/jinzhu/followers","following_url":"https://api.github.com/users/jinzhu/following{/other_user}","gists_url":"https://api.github.com/users/jinzhu/gists{/gist_id}","starred_url":"https://api.github.com/users/jinzhu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinzhu/subscriptions","organizations_url":"https://api.github.com/users/jinzhu/orgs","repos_url":"https://api.github.com/users/jinzhu/repos","events_url":"https://api.github.com/users/jinzhu/events{/privacy}","received_events_url":"https://api.github.com/users/jinzhu/received_events","type":"User","site_admin":false},"repo":{"id":13855476,"name":"gorm","full_name":"jinzhu/gorm","owner":{"login":"jinzhu","id":6843,"avatar_url":"https://avatars.githubusercontent.com/u/6843?v=3","gravatar_id":"","url":"https://api.github.com/users/jinzhu","html_url":"https://github.com/jinzhu","followers_url":"https://api.github.com/users/jinzhu/followers","following_url":"https://api.github.com/users/jinzhu/following{/other_user}","gists_url":"https://api.github.com/users/jinzhu/gists{/gist_id}","starred_url":"https://api.github.com/users/jinzhu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinzhu/subscriptions","organizations_url":"https://api.github.com/users/jinzhu/orgs","repos_url":"https://api.github.com/users/jinzhu/repos","events_url":"https://api.github.com/users/jinzhu/events{/privacy}","received_events_url":"https://api.github.com/users/jinzhu/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/jinzhu/gorm","description":"The fantastic ORM library for Golang, aims to be developer friendly","fork":false,"url":"https://api.github.com/repos/jinzhu/gorm","forks_url":"https://api.github.com/repos/jinzhu/gorm/forks","keys_url":"https://api.github.com/repos/jinzhu/gorm/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jinzhu/gorm/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jinzhu/gorm/teams","hooks_url":"https://api.github.com/repos/jinzhu/gorm/hooks","issue_events_url":"https://api.github.com/repos/jinzhu/gorm/issues/events{/number}","events_url":"https://api.github.com/repos/jinzhu/gorm/events","assignees_url":"https://api.github.com/repos/jinzhu/gorm/assignees{/user}","branches_url":"https://api.github.com/repos/jinzhu/gorm/branches{/branch}","tags_url":"https://api.github.com/repos/jinzhu/gorm/tags","blobs_url":"https://api.github.com/repos/jinzhu/gorm/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jinzhu/gorm/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jinzhu/gorm/git/refs{/sha}","trees_url":"https://api.github.com/repos/jinzhu/gorm/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jinzhu/gorm/statuses/{sha}","languages_url":"https://api.github.com/repos/jinzhu/gorm/languages","stargazers_url":"https://api.github.com/repos/jinzhu/gorm/stargazers","contributors_url":"https://api.github.com/repos/jinzhu/gorm/contributors","subscribers_url":"https://api.github.com/repos/jinzhu/gorm/subscribers","subscription_url":"https://api.github.com/repos/jinzhu/gorm/subscription","commits_url":"https://api.github.com/repos/jinzhu/gorm/commits{/sha}","git_commits_url":"https://api.github.com/repos/jinzhu/gorm/git/commits{/sha}","comments_url":"https://api.github.com/repos/jinzhu/gorm/comments{/number}","issue_comment_url":"https://api.github.com/repos/jinzhu/gorm/issues/comments{/number}","contents_url":"https://api.github.com/repos/jinzhu/gorm/contents/{+path}","compare_url":"https://api.github.com/repos/jinzhu/gorm/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jinzhu/gorm/merges","archive_url":"https://api.github.com/repos/jinzhu/gorm/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jinzhu/gorm/downloads","issues_url":"https://api.github.com/repos/jinzhu/gorm/issues{/number}","pulls_url":"https://api.github.com/repos/jinzhu/gorm/pulls{/number}","milestones_url":"https://api.github.com/repos/jinzhu/gorm/milestones{/number}","notifications_url":"https://api.github.com/repos/jinzhu/gorm/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jinzhu/gorm/labels{/name}","releases_url":"https://api.github.com/repos/jinzhu/gorm/releases{/id}","deployments_url":"https://api.github.com/repos/jinzhu/gorm/deployments","created_at":"2013-10-25T08:31:38Z","updated_at":"2016-10-21T09:05:29Z","pushed_at":"2016-10-21T03:30:26Z","git_url":"git://github.com/jinzhu/gorm.git","ssh_url":"git@github.com:jinzhu/gorm.git","clone_url":"https://github.com/jinzhu/gorm.git","svn_url":"https://github.com/jinzhu/gorm","homepage":"http://jinzhu.github.io/gorm","size":3484,"stargazers_count":4479,"watchers_count":4479,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":532,"mirror_url":null,"open_issues_count":130,"forks":532,"open_issues":130,"watchers":4479,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/1235"},"html":{"href":"https://github.com/jinzhu/gorm/pull/1235"},"issue":{"href":"https://api.github.com/repos/jinzhu/gorm/issues/1235"},"comments":{"href":"https://api.github.com/repos/jinzhu/gorm/issues/1235/comments"},"review_comments":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/1235/comments"},"review_comment":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/jinzhu/gorm/pulls/1235/commits"},"statuses":{"href":"https://api.github.com/repos/jinzhu/gorm/statuses/48d0897d3e391ba6a0e3b814c9d1b546b00b4c2e"}}}}
{"actor":{"display_login":"zardak"}}