Being somewhat new to Clojure I can't seem to figure out how to do something that seems like it should be simple. I just can't see it. I have a seq of vectors. Let's say each vector has two values representing customer number and invoice number and each of the vectors represents a sale of an item. So it would look something like this:

([ 100 2000 ] [ 100 2000 ] [ 101 2001 ] [ 100 2002 ])

I want to count the number of unique customers and unique invoices. So the example should produce the vector

[ 2 3 ]

In Java or another imperative language I would loop over each one of the vectors in the seq, add the customer number and invoice number to a set then count the number of values in each set and return it. I can't see the functional way to do this.

Thanks for the help.

EDIT: I should have specified in my original question that the seq of vectors is in the 10's of millions and actually has more than just two values. So I want to only go through the seq one time and calculate these unique counts (and some sums as well) on that one run through the seq.


In Clojure you can do it almost the same way - first call distinct to get unique values and then use count to count results:

(def vectors (list [ 100 2000 ] [ 100 2000 ] [ 101 2001 ] [ 100 2002 ]))
(defn count-unique [coll] 
   (count (distinct coll)))
(def result [(count-unique (map first vectors)) (count-unique (map second vectors))])

Note that here you first get list of first and second elements of vectors (map first/second vectors) and then operate on each separately, thus iterating over collection twice. If performance does matter, you may do same thing with iteration (see loop form or tail recursion) and sets, just like you would do in Java. To further improve performance you can also use transients. Though for beginner like you I would recommend first way with distinct.

UPD. Here's version with loop:

(defn count-unique-vec [coll]
  (loop [coll coll, e1 (transient #{}), e2 (transient #{})]
    (cond (empty? coll) [(count (persistent! e1)) (count (persistent! e2))]
          :else (recur (rest coll)
                       (conj! e1 (first (first coll)))
                       (conj! e2 (second (first coll)))))))
(count-unique-vec vectors)    ==> [2 3]

As you can see, no need in atoms or something like that. First, you pass state to each next iteration (recur call). Second, you use transients to use temporary mutable collections (read more on transients for details) and thus avoid creation of new object each time.

UPD2. Here's version with reduce for extended question (with price):

(defn count-with-price
  "Takes input of form ([customer invoice price] [customer invoice price] ...)  
   and produces vector of 3 elements, where 1st and 2nd are counts of unique    
   customers and invoices and 3rd is total sum of all prices"
  (let [[custs invs total]
        (reduce (fn [[custs invs total] [cust inv price]]
                  [(conj! custs cust) (conj! invs inv) (+ total price)])
            [(transient #{}) (transient #{}) 0]
    [(count (persistent! custs)) (count (persistent! invs)) total]))

Here we hold intermediate results in a vector [custs invs total], unpack, process and pack them back to a vector each time. As you can see, implementing such nontrivial logic with reduce is harder (both to write and read) and requires even more code (in looped version it is enough to add one more parameter for price to loop args). So I agree with @ammaloy that for simpler cases reduce is better, but more complex things require more low-level constructs, such as loop/recur pair.

  • Thanks. Performance does matter very much since the collection of transactions is in the 10's of millions. Does using the loop form then require the use of atoms or something like that to maintain state between each iteration of the loop? That's the part that's tripping me up I think. – Dave Kincaid Jul 11 '12 at 12:57
  • @DaveKincaid: see my update. Note, however, that time complexity of all solutions is the same, so their running time will differ only by (probably quite small) constant multiplier. – ffriend Jul 11 '12 at 13:13
  • This is excellent! Thanks. After posting my question and seeing your first response. I went off and experimented a little bit. Here is what I came up with. I wonder if you could help me understand what the differences might be between your approach and this one. (let [ customer-set (atom #{}) invoice-set (atom #{})] (doseq [ [customer invoice] txn ] (swap! customer-set conj customer) (swap! invoice-set conj invoice)) [ (count (deref customer-set)) (count (deref invoice-set)) ]) – Dave Kincaid Jul 11 '12 at 13:17
  • 1
    First, your approach is imperative, and since Clojure is mostly functional language, in more sophisticated cases you may got into minor issues. It's always better to use language main paradigm, just because there are more tools for programming in its style. Second, you use sync primitives, which are completely unnecessary here: in functional languages you use recursion instead of explicit loops (like while in Java) and change the state when passing vars to next recur step (see args to recur in my example). Also sync may be quite expensive for the system. The rest is more or less the same. – ffriend Jul 11 '12 at 13:44
  • thanks for the explanation. That's exactly what I was trying to understand. – Dave Kincaid Jul 11 '12 at 13:53

As is often the case when consuming a sequence, reduce is nicer than loop here. You can just do:

(map count (reduce (partial map conj) 
                   [#{} #{}]

Or, if you're really into transients:

(map (comp count persistent!)
     (reduce (partial map conj!) 
             (repeatedly 2 #(transient #{}))

Both of these solutions traverse the input only once, and they take much less code than the loop/recur solution.

  • That is very nice! Let me throw a wrinkle into it though. Add a third element to each of the vectors that is the price. Now produce a vector that includes the counts as before but also adds on the sum of the prices. Can that be done in a similar clean way? – Dave Kincaid Jul 11 '12 at 21:20
  • It's of course possible, and reduce will still be the best approach, but I'm not going to write it myself :P. – amalloy Jul 11 '12 at 22:21
  • I took a stab at this. Tell me how crazy this. I create a map of functions (def f-map {0 count 1 count 2 (partial reduce +)}) then use map-indexed to run each function on the corresponding function from f-map. Like this: (map-indexed #((get f-map %1) %2) (reduce (partial map conj) [#[] #[] []] txn)) – Dave Kincaid Jul 11 '12 at 23:58
  • It turns out that this solution blows the stack with too much data. And "too much" turns out to not be that much. I'm getting StackOverflowError using it with thousands of rows. – Dave Kincaid Jul 13 '12 at 19:59
  • Oh, good point. You could fix that with (comp doall (partial map conj)). – amalloy Jul 13 '12 at 21:52

Or you could use sets to handle the de-duping for you since sets can have a max of one of any specific value.

(def vectors '([100 2000] [100 2000] [101 2001] [100 2002]))    
[(count (into #{} (map first vectors)))  (count (into #{} (map second vectors)))]

Here's a nice way to do this with a map and higher order functions:

  (comp count set list) 
  [[ 100 2000 ] [ 100 2000 ] [ 101 2001 ] [ 100 2002 ]])

=> (2 3)

Also other solutions to the nice above mentioned ones:

(map (comp count distinct vector) [ 100 2000 ] [ 100 2000 ] [ 101 2001 ] [ 100 2002 ])

Other written with thread-last macro:

(->> '([100 2000] [100 2000] [101 2001] [100 2002]) (apply map vector) (map distinct) (map count))

both return (2 3).

Your Answer

