title: Data Modeling with Kafka? Topics and Partitions
body: <p>One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?".</p>
  
  <p>I've read and watched some introductory materials. In particular, take, for example, <a href="http://www.systemswemake.com/papers/kafka">Kafka: a Distributed Messaging System for Log Processing</a>, which writes:</p>
  
  <ul>
  <li>"a Topic is the container with which messages are associated"</li>
  <li>"the smallest unit of parallelism is the partition of a topic. This implies that all messages that ... belong to a particular partition of a topic will be consumed by a consumer in a consumer group."</li>
  </ul>
  
  <p>Knowing this, what would be a good example that illustrates how to use topics and partitions? When should something be a topic? When should something be a partition?</p>
  
  <p>As an example, let's say my (Clojure) data looks like:</p>
  
  <pre><code>{:user-id 101 :viewed "/page1.html" :at #inst "2013-04-12T23:20:50.22Z"}
  {:user-id 102 :viewed "/page2.html" :at #inst "2013-04-12T23:20:55.50Z"}
  </code></pre>
  
  <p>Should the topic be based on <code>user-id</code>? <code>viewed</code>? <code>at</code>? What about the partition?</p>
  
  <p>How do I decide?</p>