I just volunteered at work for a project that involves creating a program to edit log files on a server. By edit I mean extract useful information to another location and remove lines of text that is searched for to have certain text identifiers.

My problem is that I don't know the most efficient way to do this.

Is it best to use a scripting language like python? or should I create shell files?

I plan to have cron run the job every night.

closed as primarily opinion-based by Gilles, jasonwryan, Hauke Laging, slm, Anthon Jun 28 '13 at 4:03


  • awk was made for this. But if you happen to be familiar with Python it would make more sense to use Python, of course. The question is not only "which tool solves this task best" but also "how useful it is for me later to have learnt this or that"... – Hauke Laging Jun 28 '13 at 1:04
  • What is generating these log files? Apache, or something else? – slm Jun 28 '13 at 1:56

Personally, I'd use Perl. That's because I'm familiar with Perl, and so is the rest of the company I work for. Our software is largely in Perl.

But, that's not really an answer for you. You need to consider:

  • How trivial is this problem? If its real trivial ("grab the lines containing FOO"), you can just use grep.
  • Is it likely to remain trivial? If its likely to fairly quickly grow non-trivial, then you may just want to skip the trivial solution stage, because you'll just have to re-do it next week.

Let's say its not trivial, then:

  • What languages are you familiar with? Production code isn't really a great target for learning a new language.
  • What languages are your coworkers familiar with? They're going to have to maintain your program in the future.
  • Of your choices, which would be easiest to write the program in? To maintain it?
  • Keep in mind which languages have libraries that'd help you out. Make sure to check.

If it turns out that you and all your coworkers know Java, but not Perl, Python, or Ruby then Java might be the right answer—even though Java is, taken alone, less suited to the task. Same if they all know PHP, or even C++.

PS: Editing logs is somewhat weird. You might get more help here if you describe the problem you're trying to solve.

  • I like this answer, after considering I agree the efficiency of the tool is not the most important thing, it is its use and support-ability over time. Many tools can do the same job nearly as well as the next. Also, my use of the word logs is not totally correct, I should redefine that word to program ouputs – Quinma Jun 28 '13 at 15:45

Your description "extract useful information to another location and remove lines of text that is searched for to have certain text identifiers" just screams "Perl" to me. Depending on the log file format (I've heard of JSON log files, and there's certainly CSV log files out there) a Perl program may not even require any non-standard modules. You won't even be stretching the limits of Perl, like you might with a PHP program.

Since you're running this hypothetical program from cron, you will probably have a bash or sh script around it, to set up environment variables and so forth.
