Take the 2-minute tour ×

I frequently find myself writing small programs to parse text files (typically CSV files) and find that I always fall back on writing these parsers in Java simply because that's the language I'm most familiar with.

But since this sort of task keeps coming up, what language would people recommend I use for this kind of task? I'm happy to invest the time in learning a new language if it would save me time in the long run whenever I write this sorts of thing.

I'm not concerned at how quickly the language itself can run/parse files. What I want is a language that makes writing ad hock parsers easy/quick. I'm suspect Java is not the most efficient language for me to be doing thing kind of thing in.

Python? Ruby? Something else? (Please, don't let it be Perl)

share|improve this question

closed as not constructive by Matt Ball, Mark Elliot, IfLoop, dmckee, Daniel Vandersluis Sep 27 '10 at 4:26


... what's wrong with Perl? –  fennec Sep 27 '10 at 3:26
This is so subjective; "best" entirely depends on your needs and the context. What languages do you already know? What do your parsers need to interface with? On what platform(s)? Who else will need to use your code? –  Matt Ball Sep 27 '10 at 3:28
By "best" I meant easiest and quickest to write functional code in. I primarily work in a Windows environment, and I'm the only one who will need to use these scripts, so it doesn't have to be pretty. –  nedlud Sep 27 '10 at 3:45
Try Ruby. its got the features of Perl plus Python. –  ghostdog74 Sep 27 '10 at 4:28
You can try Scala. Part of its standard library is dedicated to text parsing. The parsing rules are written in a syntax close to BNF. –  barjak Sep 27 '10 at 6:28
show 1 more comment

6 Answers

up vote 2 down vote accepted

I have worked with Java and also Python and Python is the easiest to read through files and also output files. Python also has PLY (Python Lex-Yacc) which is very easy to parse even complicated files.

share|improve this answer
add comment

I'd strongly recommend that you do most simple text parsing with a combination of sed awk and bash

If your needs extend beyond the capabilities of these dedicated text processing tools, find a scripting language you are most comfortable with, Ruby or Python suit most, but don't be dismissive of Perl, it was designed originally to process text and does it quickly and powerfully, the CPAN library is (literally) awesome too.

A great deal of text processing can be done with a simple Bash script.. e.g.

cat file | while read a b c; do 
    #process ...likely: echo "${a//search/replace} $b $c"; # etc...

However, you should probably post some examples of the types of text parsing problems you generally face, to get truly useful answers.

Update : for CSV use case.

Assuming a bash shell (zsh for example will do this differently) from the command line without creating a shell script.

Let's assume for this example file.csv looks like this...

john doe, 2010-09-20, male, 090-555-1234
jane doe, 2010-09-30, female, 080-555-4321


cat file.csv | while IFS=, read name date sex number; do echo "name: ${name}\ndate: ${date}\nsex: ${sex}\nnum: ${number}\n";  done

would produce:

name: john doe
date: 2010-09-20
sex: male
number: 090-555-1234

name: jane doe
date: 2010-09-30
sex: female
number: 080-555-4321

Let's break that single line up...

cat file.csv | 
  while IFS=, read name date sex number;  #use IFS to split the incoming stream into comma separated sets of 4 parameters.
    # name, date, sex, number.
      #access the parameters (safely!) inside quotes with the ${param} syntax.

Bash has a fairly rich parameter expansion syntax ( @see http://www.gnu.org/software/bash/manual/bashref.html#Shell-Parameter-Expansion ) that will let you do search / replace and a variety of other simple operations on the fields in your csv records.

Problems will need to be fairly complex before you need to use a scripting language. For example, grep can filter results (or more often the incoming file before processing), sort and uniq will do common sorting and de-duping. tr can do things like remove or squeeze whitespace or replace specific chars.

The power of the basic unix commands to process text should be understood, they will save you many hours of time trying to things which have been done many times before.

Once you know how each tool works you can quickly utilize unix tools and it's pipeline to solve most problems.

Additional note

I often use Bash to process text directly from the clipboard, (I'm assuming cygwin in your case.)

so cat file.csv would be replaced by cat /dev/clipboard in the example one-liner.

While and read

read in the example is not a specific parameter of the while command, it simply reads input from the incoming pipe and allows you to split it into arbitrary parameters... see more info here. http://ss64.com/bash/read.html


In cases where you have text delimited at specific columns you can use the unix cut command to split the incoming line at the required column numbers. @see http://compute.cnr.berkeley.edu/cgi-bin/man-cgi?cut+1

Problems with CSV format.

Since the CSV format is a fairly loose specification, it's worth noting Toad's point about CSV's with records which include delimiter chars or newlines in their records.

In these cases, Bash is inadequate on it's own, and a scripting language with a useful CSV library is a better option, but don't forget that you can simplify your script to just process the input and provide a suitable output, which you can sort / grep etc. The choice is yours of course, but beware of reinventing the wheel, finding the right tools for a specific problem comes with experience, and is also dependent on your preferred runtime conditions.

Good CSV libraries for popular scripting languages.

share|improve this answer
useless cat example. –  ghostdog74 Sep 27 '10 at 4:27
The point is you can do a great deal of things with a text file without even invoking a scripting language or a tool like sed, awk etc. ... Without knowing what type of problems the OP has to solve, most specific examples would be useless. This scriptlet does nothing of course, and assumes some knowledge on the OP's part. But it's a gimme for more info. I'm open, as are most, to requests for more specific responses. But thanks for your insightful comment. –  EmacsFodder Sep 27 '10 at 4:38
The most recent file I processed like this, comes from a web based event registration form. Users can select which day they want to attend (Monday through Friday). In the db that stores the registrations, the days were put in a single field. The user then wanted to count how many attendees had registered for a Monday and asked I separate the values into separate fields in the CSV dump of the db. –  nedlud Sep 27 '10 at 4:50
(oops. hit enter too soon.) So the script just extracts the value for the days field and separates the days into individual fields in the CSV, creating 4 new columns (one for each day) in the process. If a registrant is not attending on a particular day, the field is left blank. –  nedlud Sep 27 '10 at 4:53
Ok, I'm adding an example... –  EmacsFodder Sep 27 '10 at 5:20
show 5 more comments

If you're willing to check out a new (to you!) language, you might want to give SNOBOL a try.

It might be overkill for simple CSV files and the like, but since its pattern matching paradigm is closer to context-free grammars than regular expressions, parsers tend to be simple and transparent. Contrast this to extended regular expressions, which can be coerced into parsing some context-free constructs, but can quickly explode into incomprehensible tangles of metacharacters and multiple levels of quoting and escaping, more closely resembling line noise than code.

share|improve this answer
Ok, I wouldn't recommend SNOBOL to anyone but... +1 for mentioning it. Haven't programmed in it for 15 years or so :) –  RHSeeger Sep 27 '10 at 17:55
@RHSeeger: What can I say..I was exposed to it at an impressionable age. :-) Too bad it never really caught on...it was one of my more enjoyable language learning experiences, which is why it stuck with me, I guess. –  Jim Lewis Sep 27 '10 at 18:14
add comment

The best language is clearly HyperTalk, Example:

share|improve this answer
+1 clearly the best! –  ergosys Sep 27 '10 at 4:14
add comment

The BEST option would just be on what you're most comfortable with. Each language varies from another, but in general, they can all do what parsing and what not. I personally copy and paste from Excel into TextEdit, and use Regular Expressions to mass edit. If it's super complicated, I use PHP because that's the programming language I feel the best at.

If you really don't want to use Java though, I suggest that you use PHP. Some advantages are:

  • Not having to deal with variable types (that much)
  • Easy file management functions i.e. $file_contents=file_get_contents('filename');
  • Great documentation

You could also try out Python, which I hear is great, but I haven't really used.

share|improve this answer
add comment

I typically use Python for text parsing. It has a full set of useful string manipulation feature, regular expressions, simple file i/o, and it's very quick to whip up a program for what I need. There's a pretty nice tutorial, too.

I think everyone can agree that it is certainly not Malbolge.

share|improve this answer
add comment
