{"action":"created","issue":{"url":"https://api.github.com/repos/pmaupin/pdfrw/issues/67","repository_url":"https://api.github.com/repos/pmaupin/pdfrw","labels_url":"https://api.github.com/repos/pmaupin/pdfrw/issues/67/labels{/name}","comments_url":"https://api.github.com/repos/pmaupin/pdfrw/issues/67/comments","events_url":"https://api.github.com/repos/pmaupin/pdfrw/issues/67/events","html_url":"https://github.com/pmaupin/pdfrw/issues/67","id":187871661,"number":67,"title":"Inline images not handled","user":{"login":"divergentdave","id":181772,"avatar_url":"https://avatars.githubusercontent.com/u/181772?v=3","gravatar_id":"","url":"https://api.github.com/users/divergentdave","html_url":"https://github.com/divergentdave","followers_url":"https://api.github.com/users/divergentdave/followers","following_url":"https://api.github.com/users/divergentdave/following{/other_user}","gists_url":"https://api.github.com/users/divergentdave/gists{/gist_id}","starred_url":"https://api.github.com/users/divergentdave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/divergentdave/subscriptions","organizations_url":"https://api.github.com/users/divergentdave/orgs","repos_url":"https://api.github.com/users/divergentdave/repos","events_url":"https://api.github.com/users/divergentdave/events{/privacy}","received_events_url":"https://api.github.com/users/divergentdave/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-08T00:46:29Z","updated_at":"2016-11-08T04:56:02Z","closed_at":null,"body":"Inline images, as described at section 4.8.6 of the PDF 1.7 reference, turn content streams on their head by putting raw image data in the middle of normal PDF objects. It seems the tokenizer doesn't handle inline images at present, so the image data gets parsed into nonsense operators/operands. If there are left and right angle brackets in the image data, one of the tokens will be an invalid hex-encoded string, which will raise an assertion when you try to `decode()` it.\r\n\r\n[I've started work on this in my fork](https://github.com/divergentdave/pdfrw/tree/inline-image), but I'm wondering if the image data should be returned as a different data type or object. I haven't looked at PdfWriter yet either.\r\n\r\nI filed a PR over at pmaupin/static_pdfs#1 to add a file with inline images to the test files. If you run the script below on that file, you should get an AssertionError with a message of '<\\x00\\x00>'.\r\n\r\n```\r\nimport sys\r\n\r\nimport pdfrw\r\n\r\n\r\nwith open(sys.argv[1], \"rb\") as f:\r\n    doc = pdfrw.PdfReader(f)\r\n    for page in doc.pages:\r\n        if isinstance(page.Contents, pdfrw.PdfArray):\r\n            contents = list(page.Contents)\r\n        else:\r\n            contents = [page.Contents]\r\n        pdfrw.uncompress.uncompress(contents)\r\n        for content in contents:\r\n            if content is None:\r\n                continue\r\n            for token in pdfrw.PdfTokens(content.stream):\r\n                if isinstance(token, pdfrw.PdfString):\r\n                    token.decode()\r\n```"},"comment":{"url":"https://api.github.com/repos/pmaupin/pdfrw/issues/comments/259045760","html_url":"https://github.com/pmaupin/pdfrw/issues/67#issuecomment-259045760","issue_url":"https://api.github.com/repos/pmaupin/pdfrw/issues/67","id":259045760,"user":{"login":"pmaupin","id":2379062,"avatar_url":"https://avatars.githubusercontent.com/u/2379062?v=3","gravatar_id":"","url":"https://api.github.com/users/pmaupin","html_url":"https://github.com/pmaupin","followers_url":"https://api.github.com/users/pmaupin/followers","following_url":"https://api.github.com/users/pmaupin/following{/other_user}","gists_url":"https://api.github.com/users/pmaupin/gists{/gist_id}","starred_url":"https://api.github.com/users/pmaupin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmaupin/subscriptions","organizations_url":"https://api.github.com/users/pmaupin/orgs","repos_url":"https://api.github.com/users/pmaupin/repos","events_url":"https://api.github.com/users/pmaupin/events{/privacy}","received_events_url":"https://api.github.com/users/pmaupin/received_events","type":"User","site_admin":false},"created_at":"2016-11-08T04:56:02Z","updated_at":"2016-11-08T04:56:02Z","body":"The core of pdfrw uses the tokenizer quite heavily for parsing PDF objects, but never actually parses into content streams (although there is a simple example in rl2/examples that does use the tokenizer to parse into content streams, and I know of several users who have written their own similar content stream parsers on top of the tokenizer).  The tokenizer was designed to go fast and to be cheaply restartable when upper level code needs to parse into something special.\r\n\r\nEssentially, if you are parsing the tokens in a content stream, when you encounter the BI token, you should read the tokenizer object's `floc` attribute to determine the location of the image data, extract the image data yourself and calculate its size, and then restart the tokenizer to the EI token by setting the `floc` attribute.    For an example of this, look for `floc` in pdfreader.py, especially readstream(), which has a very similar problem -- it finds the start and end of content streams.\r\n\r\nOtherwise, if you modify the tokenizer as you propose, you'll be slowing down all PDF reading for all users, and that would make me sad.\r\n\r\nPlease take a look at pdfreader, and then let me know if this makes sense, or if you can articulate a reason why you really need to modify the tokenizer for this special case.\r\n\r\nThanks,\r\nPat\r\n\r\nP.S. I'm not sure where pdfwriter figures into this -- when you hand it a content stream, it figures you know what you're doing, and outputs it.\r\n\r\nP.P.S I also don't think that a grep for EI is going to give you long-term joy; there is no guarantee that some form of image data won't have that 16 bit pattern 'EI' embedded in it somewhere.  The right answer is to actually figure out how much space the image data occupies by using the data in the dictionary keys, and that could involve several different kinds of decompression (which are not yet supported by pdfrw core).\r\n"}}
{"actor":{"display_login":"pmaupin"}}