{"action":"created","issue":{"url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/72","repository_url":"https://api.github.com/repos/GeospatialPython/pyshp","labels_url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/72/labels{/name}","comments_url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/72/comments","events_url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/72/events","html_url":"https://github.com/GeospatialPython/pyshp/issues/72","id":178603991,"number":72,"title":"Python MemeoryError exception on larage shapefile","user":{"login":"mawg","id":2999044,"avatar_url":"https://avatars.githubusercontent.com/u/2999044?v=3","gravatar_id":"","url":"https://api.github.com/users/mawg","html_url":"https://github.com/mawg","followers_url":"https://api.github.com/users/mawg/followers","following_url":"https://api.github.com/users/mawg/following{/other_user}","gists_url":"https://api.github.com/users/mawg/gists{/gist_id}","starred_url":"https://api.github.com/users/mawg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mawg/subscriptions","organizations_url":"https://api.github.com/users/mawg/orgs","repos_url":"https://api.github.com/users/mawg/repos","events_url":"https://api.github.com/users/mawg/events{/privacy}","received_events_url":"https://api.github.com/users/mawg/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-22T13:02:41Z","updated_at":"2016-09-22T13:41:34Z","closed_at":"2016-09-22T13:11:18Z","body":"[Update] please ignor/deletee this. I was running a 32 bit versoin of Python, which can only address 2gB of RAM.\r\n\r\n--------------------------------------------------------------------------------\r\n\r\nI am new to Pyshp (but not to Python) and am staring big by getting my dataset from http://edcintl.cr.usgs.gov/downloads/sciweb1/shared/topo/downloads/GMTED/GMTED_Metadata/GMTED2010_Spatial_Metadata.zip\r\n\r\nIt unpacks to 1.7gB, with the .shp file being 1.2gB.\r\n\r\nWith this code:\r\n\r\nimport shapefile\r\ndata = shapefile.Reader(\"data_file.shp\")\r\n\r\ninstantiating the Shapefile reader gives me an exception MemoryError when using Pyshp.  But I am using ony 3% of my machine's 32gB, so I don't understand it.\r\n\r\nIs there any other approach that I can take? Can process the file in chunks in Python? Or use some tool to spilt the file into chinks, then process each of them individually?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/comments/248906493","html_url":"https://github.com/GeospatialPython/pyshp/issues/72#issuecomment-248906493","issue_url":"https://api.github.com/repos/GeospatialPython/pyshp/issues/72","id":248906493,"user":{"login":"mawg","id":2999044,"avatar_url":"https://avatars.githubusercontent.com/u/2999044?v=3","gravatar_id":"","url":"https://api.github.com/users/mawg","html_url":"https://github.com/mawg","followers_url":"https://api.github.com/users/mawg/followers","following_url":"https://api.github.com/users/mawg/following{/other_user}","gists_url":"https://api.github.com/users/mawg/gists{/gist_id}","starred_url":"https://api.github.com/users/mawg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mawg/subscriptions","organizations_url":"https://api.github.com/users/mawg/orgs","repos_url":"https://api.github.com/users/mawg/repos","events_url":"https://api.github.com/users/mawg/events{/privacy}","received_events_url":"https://api.github.com/users/mawg/received_events","type":"User","site_admin":false},"created_at":"2016-09-22T13:41:34Z","updated_at":"2016-09-22T13:41:34Z","body":"Pardon me for being dumb.  Do you want me to post a question there and answer it? I already have one on S.O, at  http://stackoverflow.com/questions/39639579/memory-error-on-large-shapefile-in-python/39639735#39639735, but not on GIS StackExchange"}}
{"actor":{"display_login":"mawg"}}