title: Probabilistic Counting in Python
body: <p>I have a 50gb txt file of random strings , out of which I want to count the number of occurrences of a substring in that file.. many times, for different <strong>not predefined random</strong> substrings.</p>
  
  <p>I was wondering if there is another way to approach the problem.</p>
  
  <p><strong>probabilistic way</strong></p>
  
  <p>Something like a bloom filter , but instead of  probabilistic membership check, we could have <strong>probabilistic counting</strong>. That data structure would be used for <strong>count estimations</strong>.</p>
  
  <p><strong>Other statistical method(?)</strong></p>
  
  <p>Any dummy method that I could use to estimate the number of occurrences of a string in a text file ? Open to alternatives.</p>
  
  <p>It would be nice if it could be done in &lt;= logarithmic time as I will be doing the same task a lot of times.</p>