title: How to scrape content rendered in popup window with javascript: links using scrapy
body: <p>I'm trying to use scrapy to get content rendered only after a javascript: link is clicked.  As the links don't appear to follow a systematic numbering scheme, I don't know how to</p>
  
  <p>1 - activate a javascript: link to expand a collapsed panel</p>
  
  <p>2 - activate a (now visible) javascript: link to cause the popup to be rendered so that its content (the abstract) can be scraped</p>
  
  <p>The site <a href="https://b-com.mci-group.com/EventProgramme/EHA19.aspx" rel="nofollow noreferrer">https://b-com.mci-group.com/EventProgramme/EHA19.aspx</a> contains links to abstracts that will be presented at a conference I plan to attend.  The site's export to PDF is buggy, in that it duplicates a lot of data at PDF generation time.  Rather than dealing with the bug, I turned to scrapy only to realize that I'm in over my head.  I've read:</p>
  
  <p><a href="https://stackoverflow.com/questions/8550114/can-scrapy-be-used-to-scrape-dynamic-content-from-websites-that-are-using-ajax">Can scrapy be used to scrape dynamic content from websites that are using AJAX?</a></p>
  
  <p>and</p>
  
  <p><a href="https://stackoverflow.com/questions/16499161/how-to-scrape-coupon-code-of-coupon-site-coupon-code-comes-on-clicking-button">How to scrape coupon code of coupon site (coupon code comes on clicking button)</a></p>
  
  <p>But I don't think I'm able to connect the dots.  I've also seen mentions to Selenium, but am not sure that I must resort to that.</p>
  
  <p>I have made little progress, and wonder if I can get a push in the right direction, with the following information in hand:</p>
  
  <p>In order to make the POST request that will expand the collapsed panel (item 1 above) I have a traced that the on-page JS javascript:ShowCollapsiblePanel(116114,1695,44,191); will result in a POST request to TARGETURLOFWEBSITE/EventSessionAjaxService/GetSessionDetailsHtml with payload:</p>
  
  <p>{"eventSessionID":116114,"eventSessionWebSiteSetupViewID":191}</p>
  
  <p>The parameters for eventSessionID and eventSessionWebSiteSetupViewID are clearly in the javascript:ShowCollapsiblePanel text.</p>
  
  <p>How do I use scrapy to iterate over all of the links of form javascript:ShowCollapsiblePanel?  I tried to use SgmlLinkExtractor, but that didn't return any of the javascript:ShowCollapsiblePanel() links - I suspect that they don't meet the criteria for "links".</p>
  
  <p>UPDATE</p>
  
  <p>Making progress, I've found that SgmlLinkExtractor is not the right way to go, and the much simpler:</p>
  
  <p>sel.xpath('//a[contains(@href, "javascript:ShowCollapsiblePanel")]').re('((\d+)\,(\d+)\,(\d+)\,(\d+)')</p>
  
  <p>in scrapy console returns me all of the numeric parameters for each javascript:ShowCollapsiblePanel() (of course, right now they are all in one long string, but I'm just messing around in the console).</p>
  
  <p>The next step will be to take the first javascript:ShowCollapsiblePanel() and generate the POST request and analyze the response to see if the response contains what I see when I click the link in the browser.</p>