Take the 2-minute tour ×

We have a log correlation program which process a stream of logs coming in. There are about 10 different type of log messages which can come from various servers. These individual logs describe various facets of a single transaction that happened on the server. Each log entry has a unique transaction identification number which ties together all logs belonging to the transaction. Depending on the transaction the number of logs needed to fully describe a transaction may vary between 4 and 6. The log entries can come in any order. Log correlation of one transaction is deemed complete when all logs describing that particular transaction has been received.

There is also a requirement that multiple transactions needs to be coalesced into single logical transaction. This is triggered when a specific log entry with 2 transaction identification is received.

We have the program ready and running fine, but we need to document this in a standard format so that all complex conditions are made explicit and a new member to the team can understand the logic of correlation without reading through the code.

State diagrams do not cut as the logs can come in any order (and some logs are even optional) and that will cause the number of states to explode. I know that Complex Event processing (CEP) is a well established discipline and will have a solution to the problem I describe. There may be even more elegant solution which I am not even aware of.

What language/tool would SO community suggest for such a documentation need.

Thank You.

share|improve this question
add comment

1 Answer

In some sense, the fact that you are asking this question means that your software is probably not quite done yet -- the source code should be the single best place to learn what complex set of conditions applies to the various coalescing rules, but because it isn't, that means writing and maintaining the rules is more complicated than it should be.

Something I learned ages ago from The Pragmatic Programmer is the importance of writing "little languages" that solve problems. Rather than hard-coding a list of conditions in code, most programmers would figure out some way to put the list into an array, and just check all the conditions in the array. The Pragmatic Programmer approach suggests to take that one step further and write a little language that describes a list of conditions, and use that to populate the in-memory array from a user-accessible file.

I think the same approach would make sense here: invent a language that you can use to describe the complex relationships between the various logs and implement the parser for that language. Re-write portions of your code to use the parser. Yes, your current code solves your problems today, but it might be too difficult to modify without expert knowledge of your system. You want administrators and new team members to be able to easily modify the rules as log messages rarely stay the same for long, so you should make it as easy as possible to read, modify, and verify the rules.

Language Implementation Patterns may be a good introduction to writing your own little languages -- it is written with ANTLR as the parser-generator of choice, but much of what is covered is useful with other parser-generators as well.

share|improve this answer
Thanks for the tips. As you have pointed out, the software is done for the current state of inputs, but if we can document this preferably using a diagram, it will be easy to get a big picture of what is happening when we need to take a look this at a later stage. –  Rajkumar S Dec 5 '11 at 7:50
Your tip about writing a new language to solve this is a great idea, and is definitely worth taking a look. –  Rajkumar S Dec 5 '11 at 7:52
Accepting this answer for now, but more answers are always welcome! –  Rajkumar S Dec 13 '11 at 11:25
You might want to hold off marking my answer "accepted" -- I do like the reputation points :) but I recognize that my answer doesn't begin to address the full range of your question -- and someone else might do better if they knew you still wanted an answer. :) –  sarnold Dec 13 '11 at 23:42
Thanks :) Accepted your suggestion. –  Rajkumar S Dec 14 '11 at 6:46
add comment

Your Answer


