{"action":"created","issue":{"url":"https://api.github.com/repos/gawel/pyquery/issues/199","repository_url":"https://api.github.com/repos/gawel/pyquery","labels_url":"https://api.github.com/repos/gawel/pyquery/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/gawel/pyquery/issues/199/comments","events_url":"https://api.github.com/repos/gawel/pyquery/issues/199/events","html_url":"https://github.com/gawel/pyquery/issues/199","id":381877462,"node_id":"MDU6SXNzdWUzODE4Nzc0NjI=","number":199,"title":"Error when parsing XHTML","user":{"login":"laggardkernel","id":12206611,"node_id":"MDQ6VXNlcjEyMjA2NjEx","avatar_url":"https://avatars3.githubusercontent.com/u/12206611?v=4","gravatar_id":"","url":"https://api.github.com/users/laggardkernel","html_url":"https://github.com/laggardkernel","followers_url":"https://api.github.com/users/laggardkernel/followers","following_url":"https://api.github.com/users/laggardkernel/following{/other_user}","gists_url":"https://api.github.com/users/laggardkernel/gists{/gist_id}","starred_url":"https://api.github.com/users/laggardkernel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laggardkernel/subscriptions","organizations_url":"https://api.github.com/users/laggardkernel/orgs","repos_url":"https://api.github.com/users/laggardkernel/repos","events_url":"https://api.github.com/users/laggardkernel/events{/privacy}","received_events_url":"https://api.github.com/users/laggardkernel/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-11-17T16:22:53Z","updated_at":"2018-11-20T10:59:11Z","closed_at":null,"author_association":"NONE","body":"A problem occurred when I was parsing an XHML file from index page of pipenv doc: https://pipenv.readthedocs.io/en/latest/\r\n\r\nHere's part of the codes I'm using:\r\n\r\n```python\r\ndef remove_pics(filename):\r\n    doc = pq(filename=filename, parser=\"html\")\r\n    remove_list = [\r\n        \"div.section > h1 + img\",\r\n        \"div.section > h1 ~ a.image-reference\",\r\n        \"body div.footer + a\",\r\n    ]\r\n    for item in remove_list:\r\n        temp = doc.find(item)\r\n        if temp:\r\n            print(temp)\r\n            temp.remove()\r\n\r\n    with open(filename, \"w+\") as f:\r\n        f.write(str(doc))\r\n```\r\n\r\nBasically, I'm trying to remove the badges from the XHTML file with the help of `html` parser.\r\n\r\nThe original header and `<!DOCTYPE>` are displayed below:\r\n\r\n```html\r\n<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\r\n  \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\r\n\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n  <head>\r\n    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\r\n    <title>Pipenv: Python Dev Workflow for Humans &#8212; pipenv 2018.11.14 documentation</title>\r\n    <link rel=\"stylesheet\" href=\"_static/alabaster.css\" type=\"text/css\" />\r\n    <link rel=\"stylesheet\" href=\"_static/pygments.css\" type=\"text/css\" />\r\n    <link rel=\"stylesheet\" href=\"_static/custom.css\" type=\"text/css\" />\r\n    <script type=\"text/javascript\">\r\n      var DOCUMENTATION_OPTIONS = {\r\n        URL_ROOT:    './',\r\n        VERSION:     '2018.11.14',\r\n        COLLAPSE_INDEX: false,\r\n        FILE_SUFFIX: '.html',\r\n        HAS_SOURCE:  true,\r\n        SOURCELINK_SUFFIX: '.txt'\r\n      };\r\n    </script>\r\n    <script type=\"text/javascript\" src=\"_static/jquery.js\"></script>\r\n    <script type=\"text/javascript\" src=\"_static/underscore.js\"></script>\r\n    <script type=\"text/javascript\" src=\"_static/doctools.js\"></script>\r\n```\r\n\r\nAfter running the script once, badges were removed successfully. I noticed the `<!DOCTYPE>` was omitted by ` f.write(str(doc))`:\r\n\r\n```html\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\" xmlns=\"http://www.w3.org/1999/xhtml\">\r\n  <head>\r\n    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\" />\r\n    <title>Pipenv: Python Dev Workflow for Humans — pipenv 2018.11.14 documentation</title>\r\n    <link rel=\"stylesheet\" href=\"_static/alabaster.css\" type=\"text/css\" />\r\n    <link rel=\"stylesheet\" href=\"_static/pygments.css\" type=\"text/css\" />\r\n    <link rel=\"stylesheet\" href=\"_static/custom.css\" type=\"text/css\" />\r\n    <script type=\"text/javascript\">\r\n      var DOCUMENTATION_OPTIONS = {\r\n        URL_ROOT:    './',\r\n        VERSION:     '2018.11.14',\r\n        COLLAPSE_INDEX: false,\r\n        FILE_SUFFIX: '.html',\r\n        HAS_SOURCE:  true,\r\n        SOURCELINK_SUFFIX: '.txt'\r\n      };\r\n    </script>\r\n    <script type=\"text/javascript\" src=\"_static/jquery.js\"></script>\r\n    <script type=\"text/javascript\" src=\"_static/underscore.js\"></script>\r\n    <script type=\"text/javascript\" src=\"_static/doctools.js\"></script>\r\n```\r\n\r\nThe more confusing thing is, after running the script again (the 2nd time), of course there's no badges to be removed, the style of the XHTML file was changed once more. `<html>` attribute `xmlns` was omitted, `<script></script>` tag was changed as `<script/>`:\r\n\r\n```html\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n  <head>\r\n    <meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\"/>\r\n    <title>Pipenv: Python Dev Workflow for Humans — pipenv 2018.11.14 documentation</title>\r\n    <link rel=\"stylesheet\" href=\"_static/alabaster.css\" type=\"text/css\"/>\r\n    <link rel=\"stylesheet\" href=\"_static/pygments.css\" type=\"text/css\"/>\r\n    <link rel=\"stylesheet\" href=\"_static/custom.css\" type=\"text/css\"/>\r\n    <script type=\"text/javascript\">\r\n      var DOCUMENTATION_OPTIONS = {\r\n        URL_ROOT:    './',\r\n        VERSION:     '2018.11.14',\r\n        COLLAPSE_INDEX: false,\r\n        FILE_SUFFIX: '.html',\r\n        HAS_SOURCE:  true,\r\n        SOURCELINK_SUFFIX: '.txt'\r\n      };\r\n    </script>\r\n    <script type=\"text/javascript\" src=\"_static/jquery.js\"/>\r\n    <script type=\"text/javascript\" src=\"_static/underscore.js\"/>\r\n    <script type=\"text/javascript\" src=\"_static/doctools.js\"/>\r\n```\r\n\r\nObviously, the 2nd running of the script made the XHTML invalid. I couldn't figure out what's wrong. Is this caused by the parser `paser='html'`, or my wrong use of pyquery to modify local XHTML file by ` f.write(str(doc))`?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/gawel/pyquery/issues/comments/440232880","html_url":"https://github.com/gawel/pyquery/issues/199#issuecomment-440232880","issue_url":"https://api.github.com/repos/gawel/pyquery/issues/199","id":440232880,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDIzMjg4MA==","user":{"login":"laggardkernel","id":12206611,"node_id":"MDQ6VXNlcjEyMjA2NjEx","avatar_url":"https://avatars3.githubusercontent.com/u/12206611?v=4","gravatar_id":"","url":"https://api.github.com/users/laggardkernel","html_url":"https://github.com/laggardkernel","followers_url":"https://api.github.com/users/laggardkernel/followers","following_url":"https://api.github.com/users/laggardkernel/following{/other_user}","gists_url":"https://api.github.com/users/laggardkernel/gists{/gist_id}","starred_url":"https://api.github.com/users/laggardkernel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laggardkernel/subscriptions","organizations_url":"https://api.github.com/users/laggardkernel/orgs","repos_url":"https://api.github.com/users/laggardkernel/repos","events_url":"https://api.github.com/users/laggardkernel/events{/privacy}","received_events_url":"https://api.github.com/users/laggardkernel/received_events","type":"User","site_admin":false},"created_at":"2018-11-20T10:59:11Z","updated_at":"2018-11-20T10:59:11Z","author_association":"NONE","body":"@gawel Thanks for ur help. `doc.outerHtml()` do solve my problem, it ensures tags closed correctly in the XHTML file. I don't care about the dropping of `<!DOCTYPE>` thing very much. \r\n\r\nTo find out why `.outerHtml()` makes a difference, I did some digging in the source of `pyquery.pyquery`.\r\n\r\n```python\r\n    def __str__(self):\r\n        encoding = str if PY3k else None\r\n        return ''.join([etree.tostring(e, encoding=encoding) for e in self])\r\n\r\n...\r\n\r\n    @with_camel_case_alias\r\n    def outer_html(self, method=\"html\"):\r\n        \"\"\"Get the html representation of the first selected element::\r\n\r\n            >>> d = PyQuery('<div><span class=\"red\">toto</span> rocks</div>')\r\n            >>> print(d('span'))\r\n            <span class=\"red\">toto</span> rocks\r\n            >>> print(d('span').outer_html())\r\n            <span class=\"red\">toto</span>\r\n            >>> print(d('span').outerHtml())\r\n            <span class=\"red\">toto</span>\r\n\r\n            >>> S = PyQuery('<p>Only <b>me</b> & myself</p>')\r\n            >>> print(S('b').outer_html())\r\n            <b>me</b>\r\n\r\n        ..\r\n        \"\"\"\r\n\r\n        if not self:\r\n            return None\r\n        e0 = self[0]\r\n        if e0.tail:\r\n            e0 = deepcopy(e0)\r\n            e0.tail = ''\r\n        return etree.tostring(e0, encoding=text_type, method=method)\r\n```\r\n\r\nIt seems `etree.tostring(..., method='html')` in `.outerHtml()` helped me get a valid XHTML cause there's only one object in my code. Then I consolidated my guessing with a looking into source of `lxml`:\r\n\r\n```shell\r\nIn [6]: help(etree.tostring)\r\nHelp on cython_function_or_method in module lxml.etree:\r\n\r\ntostring(element_or_tree, *, encoding=None, method='xml', xml_declaration=None, pretty_print=False, with_tail=True, standalone=None, doctype=None, exclusive=False, with_comments=True, inclusive_ns_prefixes=None)\r\n...\r\n```\r\n\r\nAs you can see, `lxml.etree.tostring()` convert objects as `xml` strings by default. Is is better that we keep the content as what is parsed as, in my case I parsed the content as `html`, when trying convert the content as strings in `__str__()`?"}}
{"actor":{"display_login":"laggardkernel"}}