title: Is it a good idea to apply ML libraries on pandas data frame?
body: <p>i am building a cognitive miner AI Bot. where My Bot has two task , one is train and other is predict.i'm using some/few ML functionalities. so here i have lots of documents(~200,000 docs) which i'm training. and then in predicting for a query, i'm following some steps to find most accurate matched document(by looking score, confidence on each document) from training. and some known functions i'm using finding like TF-IDF,n-gram,cosine-similarity of each tokens available in asked query. for doing this i am using core python , python third-party libraries,NoSQL database for keeping training data. </p>
  
  <p>NOTE: all performance improvement taken care using core python as much as possible. (please don't give suggestion to use Elastic Search or python whoosh because i just want to use my silly code for another decade.:) )</p>
  
  <p>I'm facing Performance issue. like to do score it is taking 2-3 seconds time. which is not good.i want that result should come in some milliseconds.</p>
  
  <p>SO my question to you, if i use pandas , and try to apply all above functionality to it, will it give better performance ? or numpy matrix calculation will give better performance ? </p>
  
  <p>so here i don't think code required to be paste. i just need experienced peoples view on my problem. and of course keeping in mind solution should be scalable. </p>