title: Data structure for multi-language dictionary?
body: <p><strong>One-line summary:</strong> suggest optimal (lookup-speed/compactness) data structure(s) for a multi-lingual dictionary representing primarily Indo-European languages (list at bottom).</p>
  
  <p>Say you want to build some data structure(s) to implement a multi-language dictionary for let's say the top-N (N~40) European languages on the internet, ranking choice of language <a href="http://en.wikipedia.org/wiki/Global_Internet_usage" rel="nofollow noreferrer">by number of webpages</a> (rough list of languages given at bottom of this question).
  The aim is to store the working vocabulary of each language (i.e. 25,000 words for English etc.) Proper nouns excluded. Not sure whether we store plurals, verb conjugations, prefixes etc., or add language-specific rules on how these are formed from noun singulars or verb stems.
  Also your choice on how we encode and handle accents, diphthongs and language-specific special characters e.g. maybe where possible we <a href="https://stackoverflow.com/questions/1057789/translation-table-for-all-world-languages">transliterate</a> things (e.g. <a href="http://en.wikipedia.org/wiki/Romanize#Overview_and_summary" rel="nofollow noreferrer">Romanize</a>  German ß as 'ss', then add a rule to convert it). Obviously if you choose to use 40-100 characters and a trie, there are way too many branches and most of them are empty.</p>
  
  <p><strong>Task definition:</strong> Whatever data structure(s) you use, you must do both of the following:</p>
  
  <ol>
  <li><strong>The main operation in lookup is to quickly get an indication 'Yes this is a valid word in languages A,B and F but not C,D or E'.</strong> So, if N=40 languages, your structure quickly returns 40 Booleans.</li>
  <li><strong>The secondary operation is to return some pointer/object for that word (and all its variants) for each language</strong> (or null if it was invalid). This pointer/object could be user-defined e.g. the Part-of-Speech and dictionary definition/thesaurus similes/list of translations into the other languages/... It could be language-specific or language-independent e.g. a shared definition of <em>pizza</em>)</li>
  </ol>
  
  <p>And the <strong>main metric for efficiency is a tradeoff of a) compactness (across all N languages) and b) lookup speed</strong>. Insertion time not important. The compactness constraint excludes memory-wasteful approaches like <em>"keep a separate hash for each word"</em> or <em>"keep a separate for each language, and each word within that language"</em>.</p>
  
  <p>So:</p>
  
  <ol>
  <li>What are the possible data structures, how do they rank on the
  lookup speed/compactness curve?</li>
  <li>Do you have a unified structure for all N languages, or partition e.g. the Germanic languages into one sub-structure, Slavic into
  another etc? or just N separate structures (which would allow you to
  Huffman-encode )?</li>
  <li>What representation do you use for characters, accents and language-specific special characters?</li>
  <li>Ideally, give link to algorithm or code, esp. Python or else C.  -</li>
  </ol>
  
  <p>(I checked SO and there have been related questions but not this exact question. Certainly not looking for a SQL database. One 2000 paper which might be useful: <a href="http://arxiv.org/ftp/cs/papers/0006/0006032.pdf" rel="nofollow noreferrer">"Estimation of English and non-English Language Use on the WWW" - Grefenstette &amp; Nioche</a>. And one <a href="http://www.dmoz.org/Reference/Dictionaries/World_Languages/Multilingual/" rel="nofollow noreferrer">list of multi-language dictionaries</a>)
  Resources: two online multi-language dictionaries are <a href="http://www.interglot.com/" rel="nofollow noreferrer">Interglot (en/ge/nl/fr/sp/se)</a> and <a href="http://lookwayup.com/free/" rel="nofollow noreferrer">LookWayUp (en&lt;->fr/ge/sp/nl/pt)</a>.</p>
  
  <hr>
  
  <p>Languages to include:</p>
  
  <p>Probably mainly <a href="http://en.wikipedia.org/wiki/Language_family" rel="nofollow noreferrer">Indo-European languages</a> for simplicity: English, French, Spanish, German, Italian, Swedish + Albanian, Czech, Danish, Dutch, Estonian, Finnish, Hungarian, Icelandic, Latvian, Lithuanian, Norwegian, Polish, Portuguese, Romanian, Russian, Serbo Croat, Slovak, Slovenian + Breton, Catalan, Corsican, Esperanto, Gaelic, Welsh</p>
  
  <p>Probably include Russian, Slavic, Turkish, exclude Arabic, Hebrew, Iranian, Indian etc. Maybe include Malay family too. Tell me what's achievable.</p>