Take the 2-minute tour ×

I work for an organization that does a lot of work with government data. We have a couple of different projects where we've abstracted out common text search/manipulation operations into reusable libraries, for things like standardizing the way politicians' names are displayed (e.g., transforming "MCDONALD, BOB (R-VA)" into "Bob McDonald (R-VA)"), or finding legal citations in text (e.g., finding a reference to (e.g., finding occurrences of things like "1 U.S.C. 7" in text, determining that it's a US Code citation, and returning a structure that says it's referring to section 1 of title 7). These are relatively simple operations, and lots of collaborators in our space would like to use them, but we end up having to pick a language in which to implement each (the former is in Python; the latter, Javascript), and we freeze out potential consumers/contributors who work in different languages and don't want to resort to hacks like shelling out to a node process to handle their text. This all seems like a shame because what we're expressing is so simple, and ought, one would think, to be pretty easy to share.

What would be ideal would be a tiny DSL that could express a few basic text processing operations: regular expression search/replace, a few list-processing operations like map and filter, and the ability to store stuff in JSON-ish data structures (maps and lists), and a mechanism to either translate this DSL into or allow it to be consumed from the actual higher-level languages we and our collaborators want to work with (Python, JS, Ruby, and PHP are probably the main ones). Does anything like this exist?

I've considered building one myself... maybe a declarative thing on top of something like YAML, or maybe a tiny subset of Scheme or Lua, or maybe something entirely invented for this purpose. But I wanted to see if anything was already out there first.

share|improve this question
Probably ANTLR is useful to you. antlr.org/wiki/display/ANTLR3/… and antlr.org –  KeesDijk Mar 29 '13 at 20:24
add comment

3 Answers

The best language I'm aware of specific to text search and processing is awk. If awk doesn't meet your needs, it's likely nothing will unless you create it yourself.

However, if you do need to make your own, you don't need to start completely from scratch for each language. You can use a tool like antlr that can be exported to various languages, or build it in one language and use the respective native interfaces to access it from other languages.

share|improve this answer
Yes, awk comes conceptually close to what I'm looking for. It's a little daunting for people that aren't familiar with it, though, so I'm nervous about using it in a context where I'm trying to encourage collaboration. Antlr is a great tip, though. –  Andrew Pendleton Mar 29 '13 at 21:23
add comment

Assuming terseness isn't a requirement of your DSL, you may consider defining a JSON or XML document structure and avoid inventing a new language. This makes parsing in python, perl, and other environments far easier, and they only have to understand the structure, not the syntax.

share|improve this answer
add comment

You don't need to write a DSL for this.

Text processing/manipulation is a task virtually every programming project has to deal with in some way or another, and as a result virtually all popular programming languages have text processing capabilities.

Perl, Python, C#, etc can do all of the text processing things you've thought of, plus the additional aspects you haven't thought of yet.

Just use your higher level language.

share|improve this answer
Perhaps I should be more concrete. This: github.com/unitedstates/citation/blob/master/citations/usc.js is the source for our US Code citation extractor. It has a couple of regular expressions, plus a small amount of logic, like collapsing a couple of ways of expressing section ranges into a "range: true" property on the output object. I want that entire kernel of functionality to be expressable in a way such that it can be directly consumed from JS, Python, Ruby, and PHP. Obviously each of those languages can do that on its own, but the goal is to avoid reinventing the wheel for each. –  Andrew Pendleton Mar 29 '13 at 19:58
add comment

Your Answer


