<p>You could use BiEntropy, Trientropy or their addition TriBientropy to compute the entropy of your pickle files. The algorithms are described on www.arxiv.org and BiEntropy has been implemented with test harnesses on Github. BiEntropy has been tested positevely on large raw binary files</p>
Compute entropy of a pickle file
<p>I'm working on the <a href="https://www.unb.ca/cic/datasets/vpn.html" rel="nofollow noreferrer">ISCXVPN2016 dataset</a>, it consists of some pcap files (each pcap is captured traffic of a specific app such as skype, youtube, etc.) and I have converted them to pickle files and then write them into a text file using code below:</p>

<pre><code>pkl = open("AIMchat2.pcapng.pickle", "rb")
with open('file.txt', 'w') as f:
    for Item in pkl:
        f.write('%s\n' %Item)
</code></pre>

<p>file.txt:</p>

<blockquote>
  <p>b'\x80\x03]q\x00(cnumpy.core.multiarray\n'
  b'_reconstruct\n'
  b'q\x01cnumpy\n'
  b'ndarray\n'
  b'q\x02K\x00\x85q\x03C\x01bq\x04\x87q\x05Rq\x06(K\x01K\x9d\x85q\x07cnumpy\n'
  b'dtype\n'
  b'q\x08X\x02\x00\x00\x00u1q\tK\x00K\x01\x87q\n'
  b'Rq\x0b(K\x03X\x01\x00\x00\x00|q\x0cNNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00tq\rb\x89C\x9dE\x00\x00\x9dU\xbc@\x00\x80\x06\xd7\xc9\x83\xca\xf0W@\x0c\x18\xa74I\x01\xbb\t].\xc8\xf3*\xc51P\x18\xfa[)j\x00\x00\x17\x03\x02\x00p\x14\x90\xccY|\xa3\x7f\xd1\x12\xe2\xb4.U9)\xf20\xf1{\xbd\x1d\xa3W\x0c\x19\xc2\xf0\x8c\x0b\x8c\x86\x16\x99\xd8:\x19\xb0G\xe7\xb2\xf4\x9d\x82\x8e&amp;a\x04\xf2\xa2\x8e\xce\xa4b\xcc\xfb\xe4\xd0\xde\x89eUU]\x1e\xfeF\x9bv\x88\xf4\xf3\xdc\x8f\xde\xa6Kk1q`\x94]\x13\xd7|\xa3\x16\xce\xcc\x1b\xa7\x10\xc5\xbd\x00\xe8M\x8b\x05v\x95\xa3\x8c\xd0\x83\xc1\xf1\x12\xee\x9f\xefmq\x0etq\x0fbh\x01h\x02K\x00\x85q\x10h\x04\x87q\x11Rq\x12(K\x01K.\x85q\x13h\x0b\x89C.E\x00\x00</p>
</blockquote>

<p>My question is how I can compute the entropy of each pickle file?</p>

<p>(I have updated the question)</p>