{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/2367","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/2367/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/2367/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/2367/events","html_url":"https://github.com/scrapy/scrapy/issues/2367","id":185844443,"number":2367,"title":"Extracting text in custom tags use Scrapy","user":{"login":"hzxie","id":1730504,"avatar_url":"https://avatars.githubusercontent.com/u/1730504?v=3","gravatar_id":"","url":"https://api.github.com/users/hzxie","html_url":"https://github.com/hzxie","followers_url":"https://api.github.com/users/hzxie/followers","following_url":"https://api.github.com/users/hzxie/following{/other_user}","gists_url":"https://api.github.com/users/hzxie/gists{/gist_id}","starred_url":"https://api.github.com/users/hzxie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hzxie/subscriptions","organizations_url":"https://api.github.com/users/hzxie/orgs","repos_url":"https://api.github.com/users/hzxie/repos","events_url":"https://api.github.com/users/hzxie/events{/privacy}","received_events_url":"https://api.github.com/users/hzxie/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-10-28T05:53:47Z","updated_at":"2016-10-28T11:18:06Z","closed_at":"2016-10-28T08:54:48Z","body":"This is a [web page](https://www.ncbi.nlm.nih.gov/pubmed/15721472) for a paper. There're some useful information to extract.\r\n\r\nI want to extract title, author and abstract of this paper. So I wrote following code:\r\n\r\n    class PublicationSpider(scrapy.Spider):\r\n        name = \"publications\"\r\n        start_urls = [\r\n            'https://www.ncbi.nlm.nih.gov/pubmed/15721472',\r\n        ]\r\n\r\n        def parse(self, response):\r\n            for publication in response.css('div.rprt.abstract'):\r\n                yield {\r\n                    'title': publication.css('h1::text').extract_first(),\r\n                    'author': publication.css('div.auths > a::text').extract(),\r\n                    'abstract': publication.css('div.abstr abstracttext::text').extract(),\r\n                    'doi': publication.css('div.aux a::text').extract_first(),\r\n                }\r\n\r\nUnfortunately, the code above could not return the right content. It seems that the `abstracttext` element cannot be recognized by Scrapy.\r\n\r\nI tried to replace `abstracttext` with `h4`, and the script went well.\r\n\r\nSo how can I extract the content in `abstracttext`?\r\n\r\nSee more: http://stackoverflow.com/questions/40241637/extracting-text-in-custom-tags-use-scrapy/"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/256896250","html_url":"https://github.com/scrapy/scrapy/issues/2367#issuecomment-256896250","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/2367","id":256896250,"user":{"login":"redapple","id":886296,"avatar_url":"https://avatars.githubusercontent.com/u/886296?v=3","gravatar_id":"","url":"https://api.github.com/users/redapple","html_url":"https://github.com/redapple","followers_url":"https://api.github.com/users/redapple/followers","following_url":"https://api.github.com/users/redapple/following{/other_user}","gists_url":"https://api.github.com/users/redapple/gists{/gist_id}","starred_url":"https://api.github.com/users/redapple/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redapple/subscriptions","organizations_url":"https://api.github.com/users/redapple/orgs","repos_url":"https://api.github.com/users/redapple/repos","events_url":"https://api.github.com/users/redapple/events{/privacy}","received_events_url":"https://api.github.com/users/redapple/received_events","type":"User","site_admin":false},"created_at":"2016-10-28T11:18:05Z","updated_at":"2016-10-28T11:18:05Z","body":"@hzxie , well it ran fine for me, same code as you, http://pastebin.com/atp34920 (as I commented on StackOverflow)\r\nSo it's not reproducible with the code you supplied.\r\nThere's probably something specific do your setup. You'll need to provide use with a reproducible faulty example showing a bug in scrapy. See https://docs.scrapy.org/en/latest/contributing.html#reporting-bugs\r\nShare you console logs, `scrapy version -v` which will show your system, python version and more."}}
{"actor":{"display_login":"redapple"}}