{"action":"created","issue":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/3502","repository_url":"https://api.github.com/repos/scrapy/scrapy","labels_url":"https://api.github.com/repos/scrapy/scrapy/issues/3502/labels{/name}","comments_url":"https://api.github.com/repos/scrapy/scrapy/issues/3502/comments","events_url":"https://api.github.com/repos/scrapy/scrapy/issues/3502/events","html_url":"https://github.com/scrapy/scrapy/issues/3502","id":383163654,"node_id":"MDU6SXNzdWUzODMxNjM2NTQ=","number":3502,"title":"How to deal with redirects to a bookmark within a page (911 error)","user":{"login":"theresearchant","id":43101253,"node_id":"MDQ6VXNlcjQzMTAxMjUz","avatar_url":"https://avatars0.githubusercontent.com/u/43101253?v=4","gravatar_id":"","url":"https://api.github.com/users/theresearchant","html_url":"https://github.com/theresearchant","followers_url":"https://api.github.com/users/theresearchant/followers","following_url":"https://api.github.com/users/theresearchant/following{/other_user}","gists_url":"https://api.github.com/users/theresearchant/gists{/gist_id}","starred_url":"https://api.github.com/users/theresearchant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/theresearchant/subscriptions","organizations_url":"https://api.github.com/users/theresearchant/orgs","repos_url":"https://api.github.com/users/theresearchant/repos","events_url":"https://api.github.com/users/theresearchant/events{/privacy}","received_events_url":"https://api.github.com/users/theresearchant/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-11-21T15:13:42Z","updated_at":"2018-11-23T17:03:37Z","closed_at":"2018-11-23T17:03:37Z","author_association":"NONE","body":"I am very new to programming, so apologies if this is a rookie issue. I am a researcher, and I've been building spiders to allow me to crawl specific search results of IGN, the gaming forum. The first spider collects each entry in the search results, along with URLs, and then the second spider crawls each of those URLs for the content. \r\n\r\nThe problem is that IGN redirects URLs associated with a specific post to a new URL that incorporates a #bookmark at the end of the address. This allows the visitor to the page to jump directly down to the post in question, but I want my spider to crawl over the entire thread. In addition, my spider ends up with a (911) error after the redirect and returns no data. The only data retrieved is from any search results that linked directly to a thread rather than a post.\r\n\r\nI am absolutely stumped and confused, so any help would amazing! Both spiders are attached below.\r\n\r\n[broforceIGN_spider.txt](https://github.com/scrapy/scrapy/files/2604725/broforceIGN_spider.txt)\r\n[broforceIGN2_spider.txt](https://github.com/scrapy/scrapy/files/2604726/broforceIGN2_spider.txt)\r\n"},"comment":{"url":"https://api.github.com/repos/scrapy/scrapy/issues/comments/441287525","html_url":"https://github.com/scrapy/scrapy/issues/3502#issuecomment-441287525","issue_url":"https://api.github.com/repos/scrapy/scrapy/issues/3502","id":441287525,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTI4NzUyNQ==","user":{"login":"theresearchant","id":43101253,"node_id":"MDQ6VXNlcjQzMTAxMjUz","avatar_url":"https://avatars0.githubusercontent.com/u/43101253?v=4","gravatar_id":"","url":"https://api.github.com/users/theresearchant","html_url":"https://github.com/theresearchant","followers_url":"https://api.github.com/users/theresearchant/followers","following_url":"https://api.github.com/users/theresearchant/following{/other_user}","gists_url":"https://api.github.com/users/theresearchant/gists{/gist_id}","starred_url":"https://api.github.com/users/theresearchant/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/theresearchant/subscriptions","organizations_url":"https://api.github.com/users/theresearchant/orgs","repos_url":"https://api.github.com/users/theresearchant/repos","events_url":"https://api.github.com/users/theresearchant/events{/privacy}","received_events_url":"https://api.github.com/users/theresearchant/received_events","type":"User","site_admin":false},"created_at":"2018-11-23T17:03:37Z","updated_at":"2018-11-23T17:03:37Z","author_association":"NONE","body":"> I believe the preferred way to ask specific questions like these is [Stack Overflow](https://stackoverflow.com/tags/scrapy).\r\n> \r\n> See https://doc.scrapy.org/en/latest/index.html#getting-help\r\n\r\nThank you! Will post over there."}}
{"actor":{"display_login":"theresearchant"}}