title: Scrapy - how to manage cookies/sessions
body: <p>I'm a bit confused as to how cookies work with Scrapy, and how you manage those cookies.</p>
  
  <p>This is basically a simplified version of what I'm trying to do:
  <img src="https://i.stack.imgur.com/nn517.png" alt="enter image description here"></p>
  
  <hr>
  
  <h2>The way the website works:</h2>
  
  <p>When you visit the website you get a session cookie.</p>
  
  <p>When you make a search, the website remembers what you searched for, so when you do something like going to the next page of results, it knows the search it is dealing with.</p>
  
  <hr>
  
  <h2>My script:</h2>
  
  <p>My spider has a start url of searchpage_url</p>
  
  <p>The searchpage is requested by <code>parse()</code> and the search form response gets passed to <code>search_generator()</code></p>
  
  <p><code>search_generator()</code> then <code>yield</code>s lots of search requests using <code>FormRequest</code> and the search form response.</p>
  
  <p>Each of those FormRequests, and subsequent child requests need to have it's own session, so needs to have it's own individual cookiejar and it's own session cookie.</p>
  
  <hr>
  
  <p>I've seen the section of the docs that talks about a meta option that stops cookies from being merged. What does that actually mean? Does it mean the spider that makes the request will have its own cookiejar for the rest of its life?</p>
  
  <p>If the cookies are then on a per Spider level, then how does it work when multiple spiders are spawned? Is it possible to make only the first request generator spawn new spiders and make sure that from then on only that spider deals with future requests?</p>
  
  <p>I assume I have to disable multiple concurrent requests.. otherwise one spider would be making multiple searches under the same session cookie, and future requests will only relate to the most recent search made?</p>
  
  <p>I'm confused, any clarification would be greatly received!</p>
  
  <hr>
  
  <h2>EDIT:</h2>
  
  <p>Another options I've just thought of is managing the session cookie completely manually, and passing it from one request to the other.</p>
  
  <p>I suppose that would mean disabling cookies.. and then grabbing the session cookie from the search response, and passing it along to each subsequent request.</p>
  
  <p>Is this what you should do in this situation?</p>