{"action":"created","issue":{"url":"https://api.github.com/repos/barrust/mediawiki/issues/42","repository_url":"https://api.github.com/repos/barrust/mediawiki","labels_url":"https://api.github.com/repos/barrust/mediawiki/issues/42/labels{/name}","comments_url":"https://api.github.com/repos/barrust/mediawiki/issues/42/comments","events_url":"https://api.github.com/repos/barrust/mediawiki/issues/42/events","html_url":"https://github.com/barrust/mediawiki/issues/42","id":299756936,"number":42,"title":"Method `page.sections` return html stuff in some cases","user":{"login":"nagash91","id":173719,"avatar_url":"https://avatars1.githubusercontent.com/u/173719?v=4","gravatar_id":"","url":"https://api.github.com/users/nagash91","html_url":"https://github.com/nagash91","followers_url":"https://api.github.com/users/nagash91/followers","following_url":"https://api.github.com/users/nagash91/following{/other_user}","gists_url":"https://api.github.com/users/nagash91/gists{/gist_id}","starred_url":"https://api.github.com/users/nagash91/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nagash91/subscriptions","organizations_url":"https://api.github.com/users/nagash91/orgs","repos_url":"https://api.github.com/users/nagash91/repos","events_url":"https://api.github.com/users/nagash91/events{/privacy}","received_events_url":"https://api.github.com/users/nagash91/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-02-23T15:52:10Z","updated_at":"2018-02-23T21:32:30Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\nI'm using this library to get textual descriptions for classes in the CUB 2011 dataset.\r\n\r\nFor each class of the 200 bird classes in the CUB dataset, I get the relative wikipedia page and look at the sections with the property `page.sections`.\r\nIn some cases I get html codes inside the sections, for example:\r\n\r\n```\r\nfrom mediawiki import MediaWiki\r\nwikipedia = MediaWiki()\r\npage = wikipedia.page('Pied billed Grebe')\r\nprint(page.sections)\r\n```\r\noutput:\r\n`[u'Taxonomy and name', u'Subspecies<sup>&#91;8&#93;</sup>', u'Description', u'Vocalization', u'Distribution and habitat', u'Behaviour', u'Breeding', u'Diet', u'Threats', u'In culture', u'Status', u'References', u'External links']`\r\n\r\nThen, if I use the `page.section(str)` method with the string `u'Subspecies<sup>&#91;8&#93;</sup>'`:\r\n```\r\nprint(page.section(page.sections[1]))\r\n```\r\noutput: ```None```\r\n\r\nThe correct string to find the object with the method `page.sections(str)` is simply `'Subspecies'`.\r\n\r\nI actually managed to fix this issue implementing this method:\r\n\r\n```\r\ndef fixed_sections(page_content, verbose=False):\r\n    sections = []\r\n    import re\r\n    section_regexp = r'\\n==* .* ==*\\n' # '== {STUFF_NOT_\\n} =='\r\n    found_obj = re.findall( section_regexp, page.content)\r\n    \r\n    if found_obj is not None:\r\n        for obj in found_obj:\r\n            obj = obj.lstrip('\\n= ').rstrip(' =\\n')\r\n            sections.append(obj)\r\n            if verbose: print(\"Found section: {}\".format(obj))\r\n    return sections\r\n\r\ncorrect_sections  = fixed_sections(page.content)\r\nprint(correct_sections)\r\nprint(page.section(correct_sections[1]))\r\n```\r\n\r\nWith this code I get the correct output, i.e. the content of the section (sub-section in this case):\r\n\r\n```\r\n[u'Taxonomy and name', u'Subspecies', u'Description', u'Vocalization', u'Distribution and habitat', u'Behaviour', u'Breeding', u'Diet', u'Threats', u'In culture', u'Status', u'References', u'External links']\r\nP. p. podiceps, (Linnaeus, 1758), North America to Panama & Cuba.\r\nP. p. antillarum, (Bangs, 1913), Greater & Lesser Antilles.\r\nP. p. antarcticus, (Lesson, 1842), South America to central Chile & Argentina.\r\n```\r\n\r\n\r\nThis fix works for me, but it require to execute a reg-exp for each page, so maybe is not optimal.\r\n"},"comment":{"url":"https://api.github.com/repos/barrust/mediawiki/issues/comments/368144470","html_url":"https://github.com/barrust/mediawiki/issues/42#issuecomment-368144470","issue_url":"https://api.github.com/repos/barrust/mediawiki/issues/42","id":368144470,"user":{"login":"barrust","id":3161814,"avatar_url":"https://avatars2.githubusercontent.com/u/3161814?v=4","gravatar_id":"","url":"https://api.github.com/users/barrust","html_url":"https://github.com/barrust","followers_url":"https://api.github.com/users/barrust/followers","following_url":"https://api.github.com/users/barrust/following{/other_user}","gists_url":"https://api.github.com/users/barrust/gists{/gist_id}","starred_url":"https://api.github.com/users/barrust/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/barrust/subscriptions","organizations_url":"https://api.github.com/users/barrust/orgs","repos_url":"https://api.github.com/users/barrust/repos","events_url":"https://api.github.com/users/barrust/events{/privacy}","received_events_url":"https://api.github.com/users/barrust/received_events","type":"User","site_admin":false},"created_at":"2018-02-23T21:32:30Z","updated_at":"2018-02-23T21:32:30Z","author_association":"OWNER","body":"I think 2 or three test will fail once this is changed. If you submit a PR and they are failing, I can help fix them! "}}
{"actor":{"display_login":"barrust"}}