title: Brutally slow Apache Avro performance in Python, different results when encoding to messages vs. files
body: <p>So following the answer here: <a href="https://stackoverflow.com/questions/23614588/encode-an-object-with-avro-to-a-byte-array-in-python">Encode an object with Avro to a byte array in Python</a> I am able to send messages through ZeroMQ - but the performance is <em>brutally slow.</em></p>
  
  <p>This is to be expected since the Avro Python implementation is pure Python and we see similar performance comments from the author(s) of FastAvro. AFAIK, FastAvro cannot be used to generate a message for use with a message queue, it is geared to writing to files.</p>
  
  <p>So back to the link above, I'm curious to know if this method is not more complicated than it actually needs to be - it seems strange that the Avro DatumWriter cannot be natively used to create something suitable for messaging. </p>
  
  <p>This leads me to my final point (and reason for my suspicion). When I use the standard example from the <a href="https://avro.apache.org/docs/current/gettingstartedpython.html" rel="nofollow noreferrer">Getting Started with Avro (Python)</a> example, I can stream one of my binary files to the .avro file and it comes in around 5.8MB. When I used the message method to encode it as a byte array, that ends up with a total message size of 11MB. Why is there such a huge discrepancy between these methods? Presumably they would be quite similar...</p>
  
  <p>Please note that I've removed the deflate codec from the writer example to ensure it's an apples-to-apples comparison. When deflate is enabled, the size is just 2.8MB.</p>