{"action":"created","issue":{"url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/167","repository_url":"https://api.github.com/repos/OCA/l10n-netherlands","labels_url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/167/labels{/name}","comments_url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/167/comments","events_url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/167/events","html_url":"https://github.com/OCA/l10n-netherlands/pull/167","id":322992297,"number":167,"title":"auditfile/[IMP] Memory footprint, speed and disk space","user":{"login":"StefanRijnhart","id":1033124,"avatar_url":"https://avatars2.githubusercontent.com/u/1033124?v=4","gravatar_id":"","url":"https://api.github.com/users/StefanRijnhart","html_url":"https://github.com/StefanRijnhart","followers_url":"https://api.github.com/users/StefanRijnhart/followers","following_url":"https://api.github.com/users/StefanRijnhart/following{/other_user}","gists_url":"https://api.github.com/users/StefanRijnhart/gists{/gist_id}","starred_url":"https://api.github.com/users/StefanRijnhart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/StefanRijnhart/subscriptions","organizations_url":"https://api.github.com/users/StefanRijnhart/orgs","repos_url":"https://api.github.com/users/StefanRijnhart/repos","events_url":"https://api.github.com/users/StefanRijnhart/events{/privacy}","received_events_url":"https://api.github.com/users/StefanRijnhart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/OCA/l10n-netherlands/milestones/2","html_url":"https://github.com/OCA/l10n-netherlands/milestone/2","labels_url":"https://api.github.com/repos/OCA/l10n-netherlands/milestones/2/labels","id":1614466,"number":2,"title":"8.0","description":"","creator":{"login":"hbrunn","id":2563186,"avatar_url":"https://avatars2.githubusercontent.com/u/2563186?v=4","gravatar_id":"","url":"https://api.github.com/users/hbrunn","html_url":"https://github.com/hbrunn","followers_url":"https://api.github.com/users/hbrunn/followers","following_url":"https://api.github.com/users/hbrunn/following{/other_user}","gists_url":"https://api.github.com/users/hbrunn/gists{/gist_id}","starred_url":"https://api.github.com/users/hbrunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hbrunn/subscriptions","organizations_url":"https://api.github.com/users/hbrunn/orgs","repos_url":"https://api.github.com/users/hbrunn/repos","events_url":"https://api.github.com/users/hbrunn/events{/privacy}","received_events_url":"https://api.github.com/users/hbrunn/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":22,"state":"open","created_at":"2016-03-01T09:21:28Z","updated_at":"2018-05-14T21:41:55Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2018-05-14T21:41:55Z","updated_at":"2018-05-14T22:43:24Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/OCA/l10n-netherlands/pulls/167","html_url":"https://github.com/OCA/l10n-netherlands/pull/167","diff_url":"https://github.com/OCA/l10n-netherlands/pull/167.diff","patch_url":"https://github.com/OCA/l10n-netherlands/pull/167.patch"},"body":"Generating and storing a 70MB auditfile takes 8 minutes for qweb, but for\r\nlxml.etree to reconstruct in order to apply a namespace takes 3 hours and 1GB\r\nof extra memory.\r\n\r\nFurther optimization is achieved by sticking to the Odoo prefetch size when\r\nchunking record sets.\r\n\r\nAs XML compresses very well, the auditfile is now stored in the Odoo database\r\nin compressed format for easier handling and saving disk space."},"comment":{"url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/comments/388986177","html_url":"https://github.com/OCA/l10n-netherlands/pull/167#issuecomment-388986177","issue_url":"https://api.github.com/repos/OCA/l10n-netherlands/issues/167","id":388986177,"user":{"login":"hbrunn","id":2563186,"avatar_url":"https://avatars2.githubusercontent.com/u/2563186?v=4","gravatar_id":"","url":"https://api.github.com/users/hbrunn","html_url":"https://github.com/hbrunn","followers_url":"https://api.github.com/users/hbrunn/followers","following_url":"https://api.github.com/users/hbrunn/following{/other_user}","gists_url":"https://api.github.com/users/hbrunn/gists{/gist_id}","starred_url":"https://api.github.com/users/hbrunn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hbrunn/subscriptions","organizations_url":"https://api.github.com/users/hbrunn/orgs","repos_url":"https://api.github.com/users/hbrunn/repos","events_url":"https://api.github.com/users/hbrunn/events{/privacy}","received_events_url":"https://api.github.com/users/hbrunn/received_events","type":"User","site_admin":false},"created_at":"2018-05-14T22:43:24Z","updated_at":"2018-05-14T22:43:24Z","author_association":"MEMBER","body":"I'm a bit embarrassed to read this code. But well, that's part of the game, we hate our own code a few years after the fact. The huge memory footprint stems from my bad understanding of lxml back then. We can just construct a generator like the one I use there, but feed that directly to the validator, starting with our handcrafted node and then feed the children from a generator that reads the qweb doc from a generator. then we should have a constant memory footprint without string manipulations."}}
{"actor":{"display_login":"hbrunn"}}