Take the 2-minute tour ×

My employer has a CustomLog piped handler that reduces and preps the static web content machine access logs into a format that is kosher for being dumped into hadoop/hive. Unfortunately it hinges on a regex to convert the log format along with somewhat convoluted logic to deal with the worst the internet can throw at it.

The fix is to replace the current script and make the entire pipeline more durable. One idea I had was to do away with the regex part and do something like:

LogFormat "%v:%p\t%h\t%l\t%u\t%t\t\"%r\"\t%>s\t%O\t\"%{Referer}i\"\t\"%{User-Agent}i\"" vhost_combined_tabs

The trick is that EVERY field would be tab delimited. So that in a script language, a simple split on "\t" could break the access log down reliably. Even if one of the fields returns as "", the tab would maintain record integrity.

The problem is, I couldn't find anything on google that mentions doing this. To me it seems intuitive to break the work down from the source instead of having a script deal with the problem. Am I missing something here?

share|improve this question
The last sentence of your first para ends mid-sentence: "In the last two weeks I". –  Mark Wagner Oct 19 '10 at 20:52
@embobo Thanks, was going to elaborate that the current implementation has hung my companies entire static web array in the last few weeks so I have to replace it and want to improve on it. –  David Oct 19 '10 at 20:57
add comment

1 Answer

up vote 2 down vote accepted

I don't think you are missing anything, the obvious issue is that none of the unquoted data fields must have any possibility of containing the separator character. If any quoted data fields might ever contain your separator character then you'll have to use a less efficient method to subsequently separate the fields.

Tab ought to be relatively safe




share|improve this answer
Thanks! I figure for the edge cases where a query string does contain a tab character somehow, I'd throw the entire line away ( hoping this only happens 1x in say a million records ). –  David Oct 21 '10 at 0:42
add comment

Your Answer


