{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/27635","repository_url":"https://api.github.com/repos/golang/go","labels_url":"https://api.github.com/repos/golang/go/issues/27635/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/27635/comments","events_url":"https://api.github.com/repos/golang/go/issues/27635/events","html_url":"https://github.com/golang/go/issues/27635","id":359337553,"node_id":"MDU6SXNzdWUzNTkzMzc1NTM=","number":27635,"title":"encoding/gob: tooBig is too small","user":{"login":"bf","id":464537,"node_id":"MDQ6VXNlcjQ2NDUzNw==","avatar_url":"https://avatars0.githubusercontent.com/u/464537?v=4","gravatar_id":"","url":"https://api.github.com/users/bf","html_url":"https://github.com/bf","followers_url":"https://api.github.com/users/bf/followers","following_url":"https://api.github.com/users/bf/following{/other_user}","gists_url":"https://api.github.com/users/bf/gists{/gist_id}","starred_url":"https://api.github.com/users/bf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bf/subscriptions","organizations_url":"https://api.github.com/users/bf/orgs","repos_url":"https://api.github.com/users/bf/repos","events_url":"https://api.github.com/users/bf/events{/privacy}","received_events_url":"https://api.github.com/users/bf/received_events","type":"User","site_admin":false},"labels":[{"id":373399998,"node_id":"MDU6TGFiZWwzNzMzOTk5OTg=","url":"https://api.github.com/repos/golang/go/labels/NeedsFix","name":"NeedsFix","color":"ededed","default":false}],"state":"open","locked":false,"assignee":{"login":"robpike","id":4324516,"node_id":"MDQ6VXNlcjQzMjQ1MTY=","avatar_url":"https://avatars0.githubusercontent.com/u/4324516?v=4","gravatar_id":"","url":"https://api.github.com/users/robpike","html_url":"https://github.com/robpike","followers_url":"https://api.github.com/users/robpike/followers","following_url":"https://api.github.com/users/robpike/following{/other_user}","gists_url":"https://api.github.com/users/robpike/gists{/gist_id}","starred_url":"https://api.github.com/users/robpike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robpike/subscriptions","organizations_url":"https://api.github.com/users/robpike/orgs","repos_url":"https://api.github.com/users/robpike/repos","events_url":"https://api.github.com/users/robpike/events{/privacy}","received_events_url":"https://api.github.com/users/robpike/received_events","type":"User","site_admin":false},"assignees":[{"login":"robpike","id":4324516,"node_id":"MDQ6VXNlcjQzMjQ1MTY=","avatar_url":"https://avatars0.githubusercontent.com/u/4324516?v=4","gravatar_id":"","url":"https://api.github.com/users/robpike","html_url":"https://github.com/robpike","followers_url":"https://api.github.com/users/robpike/followers","following_url":"https://api.github.com/users/robpike/following{/other_user}","gists_url":"https://api.github.com/users/robpike/gists{/gist_id}","starred_url":"https://api.github.com/users/robpike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robpike/subscriptions","organizations_url":"https://api.github.com/users/robpike/orgs","repos_url":"https://api.github.com/users/robpike/repos","events_url":"https://api.github.com/users/robpike/events{/privacy}","received_events_url":"https://api.github.com/users/robpike/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/65","html_url":"https://github.com/golang/go/milestone/65","labels_url":"https://api.github.com/repos/golang/go/milestones/65/labels","id":3072996,"node_id":"MDk6TWlsZXN0b25lMzA3Mjk5Ng==","number":65,"title":"Go1.12","description":"","creator":{"login":"rsc","id":104030,"node_id":"MDQ6VXNlcjEwNDAzMA==","avatar_url":"https://avatars1.githubusercontent.com/u/104030?v=4","gravatar_id":"","url":"https://api.github.com/users/rsc","html_url":"https://github.com/rsc","followers_url":"https://api.github.com/users/rsc/followers","following_url":"https://api.github.com/users/rsc/following{/other_user}","gists_url":"https://api.github.com/users/rsc/gists{/gist_id}","starred_url":"https://api.github.com/users/rsc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsc/subscriptions","organizations_url":"https://api.github.com/users/rsc/orgs","repos_url":"https://api.github.com/users/rsc/repos","events_url":"https://api.github.com/users/rsc/events{/privacy}","received_events_url":"https://api.github.com/users/rsc/received_events","type":"User","site_admin":false},"open_issues":764,"closed_issues":295,"state":"open","created_at":"2018-01-29T21:17:52Z","updated_at":"2018-10-20T06:35:57Z","due_on":"2019-01-31T08:00:00Z","closed_at":null},"comments":18,"created_at":"2018-09-12T06:47:30Z","updated_at":"2018-10-22T04:43:31Z","closed_at":null,"author_association":"NONE","body":"### What version of Go are you using (`go version`)?\r\ngo version go1.10.3 linux/amd64\r\n\r\n### Does this issue reproduce with the latest release?\r\nYes\r\n\r\n### What operating system and processor architecture are you using (`go env`)?\r\narch linux with 16gb ram\r\n\r\n### What did you do?\r\nI parse large CSV files for social media analytics, creating in-memory data structures from the data. When persisting these data structures to disk using `gob`, I receive `gob: encoder: message too big` error from this code:\r\n```\r\nencoder := gob.NewEncoder(file)\r\nerr := encoder.Encode(MapData)\r\n```\r\n\r\nOne set of input data is a 4GB csv file which results in a data structure that is 200MB when saved with gob. The second set of input data is a 15GB csv file which *should* result in a 1200MB file when saved with gob, but I receive the error above.\r\n\r\nThe error is thrown at  a check if the data structure is larger than the `tooBig` constant at https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/encoder.go#L70\r\n\r\nThe tooBig constant is originally defined with a comment `TODO: Make this adjustable` here https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L17 https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L18\r\n\r\n### What did you expect to see?\r\nI expect gob to save large data structures of ~2GB that dont get close to my RAM limits of 16GB without any problems. \r\n\r\n### What did you see instead?\r\nI got the `gob: encoder: message too big` error.\r\n\r\nI fixed the error by compiling my own go and changing the tooBig check in `encoder.go` to 1<<31. Afterwards the data structure from the large 15GB CSV file was successfully saved by gob in a 1500MB gob file.\r\n\r\nTherefore I suggest this change could be done for all users, as the existing TODO comment in the golang source code suggests https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L17\r\n"},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/431739434","html_url":"https://github.com/golang/go/issues/27635#issuecomment-431739434","issue_url":"https://api.github.com/repos/golang/go/issues/27635","id":431739434,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTczOTQzNA==","user":{"login":"gopherbot","id":8566911,"node_id":"MDQ6VXNlcjg1NjY5MTE=","avatar_url":"https://avatars1.githubusercontent.com/u/8566911?v=4","gravatar_id":"","url":"https://api.github.com/users/gopherbot","html_url":"https://github.com/gopherbot","followers_url":"https://api.github.com/users/gopherbot/followers","following_url":"https://api.github.com/users/gopherbot/following{/other_user}","gists_url":"https://api.github.com/users/gopherbot/gists{/gist_id}","starred_url":"https://api.github.com/users/gopherbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gopherbot/subscriptions","organizations_url":"https://api.github.com/users/gopherbot/orgs","repos_url":"https://api.github.com/users/gopherbot/repos","events_url":"https://api.github.com/users/gopherbot/events{/privacy}","received_events_url":"https://api.github.com/users/gopherbot/received_events","type":"User","site_admin":false},"created_at":"2018-10-22T04:43:31Z","updated_at":"2018-10-22T04:43:31Z","author_association":"NONE","body":"Change https://golang.org/cl/143678 mentions this issue: `encoding/gob: increase \"tooBig\" from 1GB to 8GB on 64-bit machines`"}}
{"actor":{"display_login":"gopherbot"}}