{"action":"created","issue":{"url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/87","labels_url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/87/labels{/name}","comments_url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/87/comments","events_url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/87/events","html_url":"https://github.com/havardgulldahl/jottalib/issues/87","id":129176020,"number":87,"title":"Parsing big XML files with \"lxml.objectify.fromstring\" returns an error","user":{"login":"antonhagg","id":16721156,"avatar_url":"https://avatars.githubusercontent.com/u/16721156?v=3","gravatar_id":"","url":"https://api.github.com/users/antonhagg","html_url":"https://github.com/antonhagg","followers_url":"https://api.github.com/users/antonhagg/followers","following_url":"https://api.github.com/users/antonhagg/following{/other_user}","gists_url":"https://api.github.com/users/antonhagg/gists{/gist_id}","starred_url":"https://api.github.com/users/antonhagg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antonhagg/subscriptions","organizations_url":"https://api.github.com/users/antonhagg/orgs","repos_url":"https://api.github.com/users/antonhagg/repos","events_url":"https://api.github.com/users/antonhagg/events{/privacy}","received_events_url":"https://api.github.com/users/antonhagg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-27T15:24:25Z","updated_at":"2016-01-28T13:26:43Z","closed_at":null,"body":"This is mainly related to #78 where an xml- file can grow quite big (in my case its around 500 mb and contains 779917 files and 90361 folders). But I guess this could happen otherwise too.\r\n\r\nAnyway, there is an option to use a custom parser with the option \"huge_tree\" (http://stackoverflow.com/questions/11850345/using-python-lxml-etree-for-huge-xml-files). Would this be an option or is there another way of parsing large xml-files, for example in chunks?\r\n\r\n```\r\nreply: 'HTTP/1.1 200 OK\\r\\n'\r\nheader: Date: Wed, 27 Jan 2016 14:13:59 GMT\r\nheader: Accept-Ranges: bytes\r\nheader: path_list_total_files: 779917\r\nheader: path_list_total_folders: 90361\r\nheader: Content-Type: text/xml\r\nheader: Transfer-Encoding: chunked\r\nheader: Server: Jetty(8.1.4.v20120524)\r\nDEBUG:requests.packages.urllib3.connectionpool:\"GET /jfs/XX/Jotta/Sync/Backup2?mode=list HTTP/1.1\" 200 None\r\nTraceback (most recent call last):\r\n  File \"C:\\Python27\\Scripts\\jotta-download-script.py\", line 9, in <module>\r\n    load_entry_point('jottalib==0.4.1.post1', 'console_scripts', 'jotta-download')()\r\n  File \"c:\\python27\\lib\\site-packages\\jottalib\\cli.py\", line 258, in download\r\n    fileTree = remote_object.filedirlist().tree #Download the folder tree\r\n  File \"c:\\python27\\lib\\site-packages\\jottalib\\JFS.py\", line 304, in filedirlist\r\n    return self.jfs.getObject(url)\r\n  File \"c:\\python27\\lib\\site-packages\\jottalib\\JFS.py\", line 851, in getObject\r\n    o = self.get(url)\r\n  File \"c:\\python27\\lib\\site-packages\\jottalib\\JFS.py\", line 839, in get\r\n    o = lxml.objectify.fromstring(self.raw(url))\r\n  File \"src/lxml/lxml.objectify.pyx\", line 1801, in lxml.objectify.fromstring (src\\lxml\\lxml.objectify.c:26755)\r\n  File \"src/lxml/lxml.etree.pyx\", line 3213, in lxml.etree.fromstring (src\\lxml\\lxml.etree.c:82934)\r\n  File \"src/lxml/parser.pxi\", line 1819, in lxml.etree._parseMemoryDocument (src\\lxml\\lxml.etree.c:124533)\r\n  File \"src/lxml/parser.pxi\", line 1707, in lxml.etree._parseDoc (src\\lxml\\lxml.etree.c:123074)\r\n  File \"src/lxml/parser.pxi\", line 1079, in lxml.etree._BaseParser._parseDoc (src\\lxml\\lxml.etree.c:117114)\r\n  File \"src/lxml/parser.pxi\", line 573, in lxml.etree._ParserContext._handleParseResultDoc (src\\lxml\\lxml.etree.c:110510)\r\n  File \"src/lxml/parser.pxi\", line 683, in lxml.etree._handleParseResult (src\\lxml\\lxml.etree.c:112276)\r\n  File \"src/lxml/parser.pxi\", line 624, in lxml.etree._raiseParseError (src\\lxml\\lxml.etree.c:111367)\r\nlxml.etree.XMLSyntaxError: None\r\n```"},"comment":{"url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/comments/176185006","html_url":"https://github.com/havardgulldahl/jottalib/issues/87#issuecomment-176185006","issue_url":"https://api.github.com/repos/havardgulldahl/jottalib/issues/87","id":176185006,"user":{"login":"antonhagg","id":16721156,"avatar_url":"https://avatars.githubusercontent.com/u/16721156?v=3","gravatar_id":"","url":"https://api.github.com/users/antonhagg","html_url":"https://github.com/antonhagg","followers_url":"https://api.github.com/users/antonhagg/followers","following_url":"https://api.github.com/users/antonhagg/following{/other_user}","gists_url":"https://api.github.com/users/antonhagg/gists{/gist_id}","starred_url":"https://api.github.com/users/antonhagg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antonhagg/subscriptions","organizations_url":"https://api.github.com/users/antonhagg/orgs","repos_url":"https://api.github.com/users/antonhagg/repos","events_url":"https://api.github.com/users/antonhagg/events{/privacy}","received_events_url":"https://api.github.com/users/antonhagg/received_events","type":"User","site_admin":false},"created_at":"2016-01-28T13:26:43Z","updated_at":"2016-01-28T13:26:43Z","body":"So I have found a workaround for this by first writing the xml to a file and then reading it into memory. This means that it doesn't have to have them both in memory at the same time. Is this an acceptable solution?\r\n\r\n```\r\ndef get(self, url):\r\n        'Make a GET request for url and return the response content as a generic lxml object'    \r\n        url = self.escapeUrl(url)\r\n        if \"?mode=list\" in url: #Check if we are requested a full tree of the directory\r\n            if os.path.exists('temp.xml'):\r\n                os.remove('temp.xml') \r\n            with open(\"temp.xml\", \"w\") as text_file:\r\n                text_file.write(self.raw(url))\r\n            o = lxml.objectify.parse(\"temp.xml\")\r\n            o = o.getroot()\r\n            if os.path.exists('temp.xml'):\r\n               os.remove('temp.xml') \r\n        else:\r\n            o = lxml.objectify.fromstring(self.raw(url))\r\n        if o.tag == 'error':\r\n            JFSError.raiseError(o, url)\r\n        return o\r\n```"}}