title: building scrapy spiders into my own program, i don't want to call scrapy from command line)
body: <p>in a similar vein to this question:
  <a href="https://stackoverflow.com/questions/10943745/running-multiple-spiders-in-scrapy">stackoverflow: running-multiple-spiders-in-scrapy</a></p>
  
  <p>I am wondering, can I run a entire scrapy project from within another python program? Lets just say I wanted to build a entire program that required scraping several different sites, and I build entire scrapy projects for each site. </p>
  
  <p>instead of running from command line as a one of, I want to run these spiders and acquire the information from them.</p>
  
  <p>I can use mongoDB in  python ok, and I can already build scrapy projects that contain spiders, but now just merging it all into one application.</p>
  
  <p>I want to run the application once, and have the ability to control multiple spiders from my own program </p>
  
  <p>Why do this? well this application may also connect to other sites using a API and needs to compare results from the API site to the scraped site in real time. I don't want to ever have to call scrapy from the command line, its all meant to be self contained.</p>
  
  <p>(I have been asking lots of questions about scraping recently, because I am trying to find the right solution to build in)</p>
  
  <p>Thanks :)</p>