title: Storing Packet Capture Files In Database
body: <p>I am trying to store packet capture files into a database, and I am running into problems with performance. </p>
  
  <p>Here are some of my goals (in order of priority)
  1) Store individual packet frames in database and be able to search for them using at least their time stamp
  2) Store packet captures as they are received (Let's say, at a rate of at least a few hundred packets per second)
  3) Filter and search through packets for patterns (size, duration, tcp or udp, etc.)</p>
  
  <p>At first, I thought that I would be best off using a Time Series database like InfluxDB, but I then realized that I would need to search on fields other than time, making this less effective.  Then, I tried parsing all of the fields into a packet dump xml file (pdml) using tshark and then storing them into a NoSQL database (elasticsearch), but this proved to be exceptionally slow. </p>
  
  <p>Here is my question: 
  1) What are some efficient database options for storing packet capture data?  Am I better off with SQL or NoSQL database?</p>
  
  <p>2) How can I save an individual packet to a database? I can't find a way to get individual packets from tshark, bro, snort, or other packet analyzers from the command line?</p>
  
  <p>3) Are there any other options for parsing the packets to a useful datbase format?  I was thinking of extracting fewer fields using tshark; I suppose I don't need all of them.</p>
  
  <p>Any help, thoughts, or ideas would be great.  Thanks.</p>