typo3-media.com: Latest TYPO3 Blog News http://www.typo3-media.com/ Latest News of the TYPO3 Blog en typo3-media.com: Latest TYPO3 Blog News http://www.typo3-media.com/fileadmin/tt_news_article.gif http://www.typo3-media.com/ Latest News of the TYPO3 Blog TYPO3 - get.content.right http://blogs.law.harvard.edu/tech/rss Mon, 03 Dec 2012 23:08:00 +0100 Boost recent documents in SOLR http://www.typo3-media.com/blog/solr-recip-boosting.html Finding the correct boosting function to boost recent documents here and here for example. I found it hard to find the correct values for the recip function - and so I wrote a little helper where you specify two points and get the correct parameters to use in the recip function: Click here to calculate recip paramaters for solr]]> news Mon, 03 Dec 2012 23:08:00 +0100 Find top caches - Cachemgm 2 http://www.typo3-media.com/blog/monitor-cache-activity.html Follow up post - introducing 2 more features in the cachemgm extension.
  • Use Tx_Cachemgm_Cache_Frontend_LogablePhpFrontend to also log PHP Caches
  • Use the new cli command "cachemgm_top" to watch the top caches
  • Find top cache identifiers

    Once you have setuped the Logable Cache Frontends you can simply call: cli_dispatch.phpsh cachemgm_top To see a refreshing list of the top cache activity. All filter arguments do also apply here - so you can use: cli_dispatch.phpsh cachemgm_top --filterAction=MISS to only see the top cache misses. It gives you something like this:   Read in the next post for what this can be useful... ]]>
    news Mon, 09 Apr 2012 18:10:00 +0200
    Don‘t slow applications with caches http://www.typo3-media.com/blog/cachemgm-cache-log.html Ok the title is a bit provocative - it could have also been something like „select your caches...
    TYPO3 comes with a flexible Caching Framework, backported from FLOW3. The key concept is, that you easily can use caches by selecing a frontend and a backend. With the frontend you decide what you want to save (most likely a variable) and with the backend you decide where to save it.

    What the framework does not provide out of the box is some statistics for the caches. But this can easily be achieved by using a cache frontend that is able to provide some logs. Read on if you want to know more :-)

    New functionalities in the „cachemgm“ extension.

    Maybe some know this extension already - it was initiated by Kasper and offers functions to check and analyse the cached pages.

    In the current version on forge this extension comes with two additional functionalities to check the caches offered by the caching framework:

    Overview of the configured caches:

    This module should be self explaining, it simply shows all configured caches and you can check further details.

    Analyse your caches with the cache log:

    The extension comes with an extended VariableFrontend. In addition to the standard frontend it logs all operations for potential evaluations. Inspired by varnish, the logging is happening as fast as possible by using an OS message queue or shared memory. This way it shouldn‘t have significant impact on cache times. But it needs PHP shared memory functionalities compiled into PHP.

    To use this frontend you need to configure it in your localconf.php like this:

    $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_object']['frontend'] = 'Tx_Cachemgm_Cache_Frontend_LogableVariableFrontend';

    Then of course you need some tools to read the log (since the log is not persited). You can use the cachemgm_log cli command for this:

    php cli_dispatch.phpsh cachemgm_log
    And you should get something like this:

    And after terminating the log reader ( Ctrl+C) - you should get a summary like this:

    By using the parameters „--cache“ and „--filterUrl“ you can limit the logs that are evaluated to a certain cache and/or request url.

    Dependency Injection Caching Analysis:

    Lets take a small example: A website with an extbase USER_INT object. In this case there are around  150 objects that need to be created by the DI-Container per request.What do you think is faster? Using the „NullBackend“ or the default „DbBackend“ for the „extbase_object“ cache?I did a (really small and not representative) benchmark to that Url (20 requests) with running cachemgm_log - and here are the results:
    Using Database Backend I got the following result
    Time per request:       2223.411 [ms] (mean)

    And the cachemgm_log showed:

    Cache "extbase_object":
     get method:
       Average GET-Hit time:935ns
       Overall Hit Time:2.8sec
     has method:
       Overall Time:3.35sec
     set method:
        Sucess Writes:
        Failed Writes:0
        Overall Write Time:0ns

    Using no cache at all I got the following result:
    Time per request:       2147.321 [ms] (mean)

    And the cachemgm_log showed:

    Cache "extbase_object":
     get method:
       Overall Hit Time:0ns
     has method:
       Overall Time:954ms
     set method:
        Sucess Writes:2823
        Failed Writes:42
        Average Write time:364ns
        Overall Write Time:1.03sec
        Sucess Writes:
        Failed Writes:0
        Overall Write Time:0ns

    To summarize this: Using the database cache backend we had a total of over 6 seconds spend for reading something out of the cache. The overhead of using the Nullbackend was only 2 seconds (some of this might only because of the cache log actions). That means for the above test szenario its slower to read the informations from cache than to build the information from scretch.

    This is a general downside of the Database Cache Backend - when used for high frequented caches! Especially when used on high traffic sides or even worse on load balanced environments with a single database this effect could be much bigger.

    What this shows is, that selecting and benchmarking different Caches is crucial for a high performance application. And of course there are more numbers to watch at - than only the cache times.

    Inside TYPO3 development Thu, 05 Apr 2012 15:48:00 +0200
    TYPO3 Caching and the cHash http://www.typo3-media.com/blog/chash-caching-typo3.html Explains TYPO3 caching and the need for a flexible cHash calculation cHash? There is a pretty old but nice article from Kasper, that explains the purpose of cHash[1]. To summarize this: If a cHash (or cache Hash) is part of the current URL, and if that cHash is correct, TYPO3 generates a cache entry for the generated content. The cHash is only correct if TYPO3 itself generated the URL. So with the help of the cHash TYPO3 can cache multiple variants of a page - depending on its request parameters. This is used for example to cache all the detail views of your news records.

    What is the goal of caching

    If we try to look at the problem of caching a bit more generic we can find the following goals:
    1. Content that can be cached should be cached, and we want as less cache entries as possible. To cache content means its likely that the generated content is valid for a certain amount of visitors.
    2. Parts of the content that should not be cached should not be cached, but the rest of the page should be cached. For example you might have dynamic elements like a shopping basket or a username.
    3. We never want uncached page rendering.
    Lets look at this 3 things more in detail:

    1) Cache generated pages in TYPO3

    Since the content on a page can differ depending of many things, TYPO3 Caches seperate variants based on:
    • Diffrent usergroups
    • the different states of all typoscript conditions. (This depends on your typoscript setup)
    • Diffrent values of predefined Core parameters if they are valid (id, type, MP)
    • Based on other incoming parameters if the cHash matches. This last point is important to understand - since this is what you need to make your extension output cacheable the way you want.

    2) Have dynamic content

    TYPO3 has the concept to support dynamic elements on a page.
    It works in a way, that the page itself is cached, but in the cache there can be Placeholders that get replaced before the content is send to the user. This can be achieved using _INT Typoscript Objects like USER_INT or COA_INT. Extbase supports uncached actions - but at the end it uses the same concept - since the extbase converts itself to USER_INT if an uncached action is called. There are third party extensions that modifies the _INT behaviour - so that the dynamic elements are loaded by ajax calls on the client side. Which is a good approach and gives additional performance - if you can rely on enabled javascript.

    3) Avoid uncached contend:

    That is easy. Just set the disableNoCacheParameter setting in your localconf.php to 1 and the pageNotFoundOnCHashError also to 1.

    cHash details

    Imagine the following Use-Cases: News Single Plugin: You have a single view that displays some data based on a "tx_news_pi1[uid]" parameter.  
    So this is straight forward: If you have a valid cHash for that parameter TYPO3 caches a separate variant of this page (e.g. id=12&tx_plugin_pi1[uid]&cHash=<correct_news_cHash> )

    Search Plugin: Its a different story for a search plugin: You have a search that searches for things based on a query: You are not allowed to have a cHash Parameter then. You want this for two reasons: 1) You dont want to cache every search 2) The user enters the searchword - and its hard to generate the correct cHash. Your URLs need to look like this:
       id=12& tx_search_pi1[query]=test
       id=12& tx_search_pi1[query]=test2 TYPO3 will then have only one Cache entry for that page. But your plugin will be a USER_INT and therefore can evaluate this parameter each time and displays the correct search.

    current cHash problems and limitations

    It is not possible to have this behaviours mixed on one page:  If you have elements on your page that should be cached based on some parameters but you still want to have elements that needs to be uncached. And there are many use-cases for this. To come back to the two examples, imagine you have the search and the news plugin on one page: Url:  id=12&tx_search_pi1[query]=test&tx_plugin_pi1[uid]=1&cHash= <correct_news_cHash> => will not work since the cHash is not valid anymore. (It triggers 404 - or delivers uncached page - depending on your settings) Url: id=12& tx_search_pi1[query]=test&tx_plugin_pi1[uid]=1
    Will not work since with a missing cHash TYPO3 might deliver the wrong cached content for the news extension.

    If you manage to gerenate this urls, it will work but you pollute the Cache Table with new entries for every tx_search_pi1 parameter

    Also the current extbase uncached actions solution is another example for the same problem. Since extbase replaces USER to USER_INT on the fly if an uncached action is called it needs to have the cHash Parameter set. Which results in a new Cache entry for every Parameter combination of your uncached action. This is not what you want.

    A third aspect of that problem is, that parameters like "L" or other Parameters that might be part of some conditions are used for cHash calculation also. That can cause tricky bugs and uncached pages also. Its not required that parameters, that are evaluated in typoscript conditions require a cHash.

    The solution: Flexible cHash calculation

    The cHash should (and have to) be only calculated and evaluated for parameters that are used in cachable plugins (or actions).  

    For the above example that means: Url: id=12&tx_search_pi1[query]=foo&tx_plugin_pi1[uid]=1&cHash=<correct_news_cHash>
    and Url: id=12&tx_search_pi1[query]=bar&tx_plugin_pi1[uid]=1&cHash=<correct_news_cHash>
    should both result in a cache hit.

     should throw an error (404) since its a wrong cHash

    should throw an error since the cHash is missing but required for tx_plugin_pi1[uid] Thanks to all the reviewers - especially Tolleiv - the improvement made it into the Core [2] - and you now have the possibility to control which parameters should be used for cHash logic in the install tool: cHashExcludedParameters: The the given parameters will be ignored in the cHash calculation. Example: L,tx_search_pi1[query]
    cHashOnlyForParameters: If set only the given parameters will be evaluat
    ed in the cHash calculation. Example: tx_news_pi1[uid]. Only use this if you are sure to know all relevant parameters. cHashRequiredParameters: Configure Parameters that require a cHash. If no cHash is given but one of the parameters are set, then TYPO3 triggers the configured cHash Error behaviour cHashExcludedParametersIfEmpty: Configure Parameters that are only relevant for the chash if there's an associated value available. And asterisk "*" can be used to
    skip all empty parameters.

    news Fri, 02 Mar 2012 22:58:00 +0100
    SOLR Search Request Handlers explained http://www.typo3-media.com/blog/solr-search-request-handlers.html With SOLR you can execute complex queries over your indexed documents. Like with other software the... Like with other software the possibilities are grown over time and there are many different configurations and parameters that could be used in order to specify a query in SOLR.

    Request Handlers

    <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" /> That means you can reach this handler by calling <your_solr_url>/update 
    If the name is not starting with "/" you can by default call the request handler with the path select and the parameter qt like this: "/select&qt=standard"... These kind of requestHandlers are normally reserved for handlers that searches for something (the so called "search Handlers").  You can have a look at your solrconf.xml for more details regarding the configured handlers or call the plugin handler like this: /admin/plugins  (if it is configured of course). You can check the solrconfig.xml examples here: solrconfig.xml in solr SVN and the solrconfig.xml included in the TYPO3 solr extension (which is used in the examples below)

    Search Handlers

    So we can define the search handlers as SOLR request handlers that returns a list of results to you. Often they have names that do not start with "/" and therefore are used by specifiing the qt parameter in the url like ".../select&qt=<searchhandlername>" The search Handlers normally uses different components that do parts of the work. For example there are components for Querying, Facetting, Highlighting etc. All the search Handlers should understand the so called common query parameters [3]. The main ones are:
    • sort: For defining the sorting
    • start, rows - to define offest and result count
    • defType - specify the query parser

    And additionally the used components "understand" his parameters. Please also note, that some of these parameters (and all other parameters) can be configured with default values in the solrconf.xml.

    Query Parsers:

    The Solr.SearchHandler can be configured to use different query Parsers for "translating" the value of the "q" parameter to the correct Lucene Query. Therefore the query parser is one of the most important parts for the search handlers - and understanding whats happening behind the scenes is useful. Unless configured otherwise the query parser used is the standard lucene query parser. There are other parsers that are often used - like dismax and edismax.  All have its pro and cons. The parser that is used is defined with the defType parameter (either explicit or implicit by using a configured search handler). Another way of specifying the parser that should be used is the LocalParams[1] syntax. We will have a look at some examples later. Here is the full list of query parsers,  that allows you to build really fancy querys:
    • boost - to boost a query
    • frange - can be used to speed up range queries
    • field - simple field query useful in filter querys
    • raw / term - create raw term query from input

    Query Parsers at a closer look

    the lucene query parser (standard)


    Lets have a look at some first basic example:
    /select?q=forum&rows=10&qt=standard&wt=standard&debugQuery=on Please note 2 details here: first the debugQuery=on is used which includes the result of the query parsing together woth ranking details. And second the lucene parser is used by setting the requestHandler to standard ( query type: qt=standard ). The "standard" search request handler  is normaly defined to use the lucene parser, because of the following snippet in the solrconfig.xml. (It might be different for your installation) <requestHandler name="standard" class="solr.SearchHandler">
        <!-- default values for query parameters -->
         <lst name="defaults">
         <arr name="last-components">
    /select?q=forum&rows=10&wt=standard&debugQuery=on&defType=lucene The resulting parsed query of this example is: "content:forum" meaning it will return all documents with the term "forum" in the content field.

    Here is another example:
    /select?q=forum billing&rows=10&defType=lucene&wt=standard&debugQuery=on With the resulting Query: "+content:forum +content:billing" - note that both terms are marked as obsolete. Examples - search in different fields:
    q=content:forum title:billing
    q=content:forum OR title:billing
    Resulting Query: "content:forum title:billing"

    q=+content:nokia +content:prepaid^10 title:billing
    Resulting Query: "+content:nokia content:prepaid^10.0 title:billing" Guess what this query does? It will find all documents that match the term "nokia" - and it will additional boost up documents that matches the term "prepaid" in the content and "billing" in the title. Where the term prepaid is 10 times more relevant for the score calculation, but still documents are listed that don't have the term "prepaid" in it, since the term is not marked as obsolete. Multiple terms in one field can also be written like this:
    q=title:(nokia handy)
    Resulting Query: "+title:nokia +title:hallo
    " Hooks and function queries:
    An example query using _val_:
    q=forum _val_:"{!func}id"


    To summarize the pros of the lucene parser:
    • two hooks that allows more complex use cases

    The dismax query parser

    Lets see some examples:

        <lst name="defaults">

    Another way of forcing the dismax query parser would have been to use LocalParams Syntax like this: q={!dismax}forum Lets look at a second example (again only the q parameter is shown)
    q=forum +must -dont


    qf=content^2 title^10

    The dismax parser comes with a couple of parameters that helps to translate the human query to a relevant lucene query [4] - lets have a closer look at some of them. The mm (minimum should match) Parameter:
    To make it more concrete look at this example:
    /select?q=nokia sony panasonic&rows=10&qt=dismax&debugQuery=on&mm=2&qf=content^10 title^20 This will result in the parsed query:
        (content:nokia^10.0 | title:nokia^20.0)
        (content:sony^10.0 | title:sony^20.0)
        (content:panasonic^10.0 | title:panasonic^20.0)
      • pf - Phrase Fields (e.g. content^2)
      • ps - Phrase sloop (e.g. 15 )
    For more details see [4] So the big advantage of the dismax parser is, that you have a good scoring calculation based on normal queries. (much better than a simple lucene parser result)

    Boost Query Parsers:

    q={!boost b=<the boost function query>}<any other query> Note that the boost query itself set with the local param "b" and that the boost query needs always another "normal" querystring as well. For example q={!boost b=id}forum
    will result in the parsed Query: "boost(content:forum,str(id))" The result means, that all documents that matches "forum" in the content field will be returned and additionally boosted by the id field (which is not very useful of course but serves as an example). You also see that the lucene parser is used for the appended query string "forum" - since this is the default parser. If you want to use the boost query together with a dismax query you can do this like this:

    Function Query Parser

    The last query parser for this article is the function query parser. This parser allows to use functions in your query - the normal use case is to influence the scoring in a way you would like to have it (e.g. boosting newer documents...) Lets look at a simple example: /select?q={!func}id&rows=10&debugQuery=on This results in this query: str(id) It will simply find all documents (a function query itself do not filter) and the scoring equals the fieldvalue of the field id. Another example is this: /select?q={!func}product(2,price_f)&debugQuery=on&fl=id,score,price_f&fq=id:2 The resulting query is: "product(const(1.0),sfloat(app_price_f))" Please note that we used the parameter "fq" to filter for the docuemnt with the id 2 and we also displayed the "score" field: So this query will give us the doubled price of the document in the score field. Imagine you can use this to calculate and return fancy stuff for documents (like term frequency etc). Typical usage of function query
    • lucene parser: using the _val_ hook
    • dismax parser: using the _val_ hook or the bf parameter
    Example dismax and bf usage:
    /select?q=nokia&debugQuery=on&qf=content title&qt=dismax&bf=id Will result in: "+(content:nokia | title:nokia) () str(id)"

    Combining dismax and lucene

    With the above knowlege its easy to combine the two parsers in different ways: Lucene in dismax:
    Using the bq parameter of dismax to add lucene query: /select?q=car&debugQuery=on&qf=content&qt=dismax&bq=pagetype:app Resulting query: "+(content:car) () pagetype:app" Dismax in lucene: Using the _query_ hook for lucene parser to add a dismax query: /select?q=pagetype:app _query_:"{!dismax%20qf=content}car"&debugQuery=on&qt=standard Resulting query: "+pagetype:app +(+(content:car) ())"

    The fq (filter query) parameter

    Its suggested to use the filter queries (like the name says) for facet flltering - the caching feature is more helpful for complex query parts. The advantage is, that the filter query uses the filterCache from SOLR. It works something like this:

    Search components

    • SpellCheckComponent - advanced spell checking for the query
    news development Mon, 23 Jan 2012 08:23:00 +0100
    JavaScript device detection http://www.typo3-media.com/blog/javascript-device-detection.html For some recent project we required some device specific logic. After searching for javascript... https://github.com/danielpoe/DeviceDetection Please fell free to use and improve it with further checks and user agends. The lib comes with some basic QUnit test cases that can also be extended with the time. Example Usage:
    var detection = DeviceDetection(); 
    news Sat, 21 Jan 2012 16:07:00 +0100
    Sessions with Extbase http://www.typo3-media.com/blog/sessions-extbase-typo3.html Sessions are used to save informations that are bound to the current website user. In most cases... class BasketController {
    public function addItemAction(Product $product) {
    $basket = $this->userSession->getBasket();
    } If you have a look at the attached UML scetch: That means there is a Session Class in the Domain Layer of your application, that should be of scope singleton and defines a well defined Interface for your Domain. (In this example to get the basket Instance, that acts as Aggregate Root for the Items in the basket). For the actual storage of the Objects in the Session the Persitence Implementation is part of your applications system Layer. Here you can find a sample SessionStorage Implementation that works for TYPO3 4.x:
    class Tx_Extkey_System_Session_SessionStorage implements t3lib_Singleton {

    const SESSIONNAMESPACE = 'tx_extkey';

    * Returns the object stored in the user´s session
    * @param string $key
    * @return Object the stored object
    public function get($key) {
    $sessionData = $this->getFrontendUser()->getKey('ses', self::SESSIONNAMESPACE.$key);
    if ($sessionData == '') {
    throw new LogicException('No value for key found in session '.$key);
    return $sessionData;

    * checks if object is stored in the user´s session
    * @param string $key
    * @return boolean
    public function has($key) {
    if ($sessionData == '') {
    return false;
    return true;

    * Writes something to storage
    * @param string $key
    * @param string $value
    * @return void
    public function set($key,$value) {
    $this->getFrontendUser()->setKey('ses', self::SESSIONNAMESPACE.$key, $value);

    * Writes a object to the session if the key is empty it used the classname
    * @param object $object
    * @param string $key
    * @return void
    public function storeObject($object,$key=null) {
    if (is_null($key)) {
    $key = get_class($object);
    return $this->set($key,serialize($object));

    * Writes something to storage
    * @param string $key
    * @return object
    public function getObject($key) {
    return unserialize($this->get($key));

    * Cleans up the session: removes the stored object from the PHP session
    * @param string $key
    * @return void
    public function clean($key) {
    $this->getFrontendUser()->setKey('ses', self::SESSIONNAMESPACE.$key, NULL);

    * Gets a frontend user which is taken from the global registry or as fallback from TSFE->fe_user.
    * @return ux_tslib_feUserAuth The current extended frontend user object
    * @throws LogicException
    protected function getFrontendUser() {
    if ($GLOBALS ['TSFE']->fe_user) {
    return $GLOBALS ['TSFE']->fe_user;
    throw new LogicException ( 'No Frontentuser found in session!' );
    And finally this is the sample code for the UserSession Class:
    class Tx_Extkey_Domain_UserSession implements t3lib_Singleton {
    * @var Tx_Extkey_System_Session_SessionStorage
    private $sessionStorage;


    public function __construct(Tx_Extkey_System_Session_SessionStorage $sessionStorage) {
    $this->sessionStorage = $sessionStorage;

    public function getBasket() {
    if ($this->sessionStorage->has('Basket')) {
    return $this->sessionStorage->getObject('Basket');
    else {
    return $this->objectManager->create('Basket');

    public function saveBasket(Basket $basket) {


    Serialisation drawbacks..

    The objects are stored in the User Session as serialized string. This can cause several problems:
    • Not all associated framework objects are serializable. For example you will have problems serializing domain entities (too big, lost connection to persitence manager...)
    • Cyclic relations will cause problems
    • The serialized version can simple be too big
    Thats where you need to think what data should be stored in your serialized objects. PHP5 gives you the possibility to clean up with your data in the __sleep method and to reconstitute them in the __wakeup method. This is where you:
    • can store ids instead of entitys and try to get them with a repository on wakeup
    • throw injected singletons away and try to get them back on __wakeup

    Dependency Injection

    There are two ways of injecting objects after deserialization:
    1. in the __wakeup method using the Extbase ObjectManager (t3lib_div::makeInstance('Tx_Extbase_Object_ObjectManager')
    2. explicit In the UserSession Class like:
    	public function getBasket() {
    $basket = $this->sessionStorage->getObject('Basket');
    else {

    In FLOW3

    Well in FLOW3 you don't need to deal with a "SessionStorage" Implementation and think about __sleep and __wakeup... this is all handled by the Framework and the @scope session  annotation. See Roberts Post about this: http://robertlemke.de/blog/posts/2010/08/19/session-handling-and-object-serialization ]]>
    pattern Mon, 21 Nov 2011 21:05:00 +0100
    Finding more with SOLR http://www.typo3-media.com/blog/solr-noun-expansion.html For a recent project we launched we had to deal with very different quality of the indexed... For example: If someone searches for "Autonavigation" it should find documents with "car navigation" and "auto navigationsgerät". There are the follwoing Issues to solve here:
    1. You need to split the word "Autonavigation" into "Auto" and "navigation"
    2. You need to search for the translated, synonyms and untranslated tokens.
    3. Be fuzzy enough
    4. Be relevant

    Using DictionaryCompoundWordTokenFilterFactory to split nouns

    A stable way to split words by meaningful subwords you can use the solr.DictionaryCompoundWordTokenFilterFactory Filter. This Filter uses a dictionary of words and splits tokens by detected words:
    <!-- 1 split subwords english nouns -->
    <filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/german-common-nouns.txt" minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>
    <!-- 2 split subwords german nouns -->
    <filter class="solr.DictionaryCompoundWordTokenFilterFactory" dictionary="wordlists/english-common-nouns.txt" minWordSize="5" minSubwordSize="4" maxSubwordSize="15" onlyLongestMatch="true"/>

    Using SynonymFilterFactory to translate

    <filter class="solr.SynonymFilterFactory" synonyms="wordlists/translationsgroups-english-german-nouns.txt" ignoreCase="true" expand="true"/>
    The file has synonymgroups like: "auto,car" - so that the search uses always the english and german version of a noun

    Using the correct Stemmer

    Try out different stemmer and analyse the results using the SOLR admin->analyse GUI. For german the Stemmer order (from aggressive to unaggressive) should be like this for example:
    • GermanStemFilterFactory
    • SnowballPorterFilterFactory (language German)
    • GermanLightStemFilterFactory
    • GermanMinimalStemFilterFactory

    Be relevant

    Using a aggressive stemmer and the explained token expansions you will have more hits when you search. But there is the risk to find documents that are not so relevant (the old recall to precision problem). Therefore I prefer to have a "text" field that is configured less aggressive and dont uses language expansion. In addition you could configure a "expandedtext" field, that uses the described configuration. When doing search you need to place a query that searches in both fields, but having a higher boost set to the "text" field.
    <fieldType name="expandedtext" class="solr.TextField" positionIncrementGap="100">
    <analyzer type="index">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.HyphenatedWordsFilterFactory"/>
    <!-- Case insensitive stop word removal.
    add enablePositionIncrements=true in both the index and query
    analyzers to leave a 'gap' for more accurate phrase queries.
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.StopFilterFactory"
    <filter class="solr.StopFilterFactory"
    <!-- 1 split subwords english nouns -->
    <!-- 2 split subwords german nouns -->
    <!-- 3 expand english words to include the german translation -->
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.GermanStemFilterFactory"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
    <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
    <filter class="solr.GermanStemFilterFactory"/>
    <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>


    Actually finding and creating the correct dictionaries ad synonymlists takes most of the time. Here are the lists that I created based on the links at the end of the article. The translation was created using google translate.
    • list of common english nouns
    • list of common german nouns (Substantive)
    • noun synonym list with german, english translation groups

    Links for wordlists

    Open Source Wordlists:
    http://wordlist.sourceforge.net/ Dictionaries that can be used by open office:
    http://wiki.services.openoffice.org/wiki/Dictionaries List of most used words in different languages:

    German synonyms:

    Other lists of lists:

    SOLR related links

    List of all Filters:

    Other Links

    Nice blog around text technologies:
    http://www.texttechnologies.com/?pagename=about "Institute der Deutschen Sprache": http://www.ids-mannheim.de/kl/projekte/methoden/derewo.html Solr suggestion of Stemmers:
    http://wiki.apache.org/solr/LanguageAnalysis Open Source text processing software GATE:

    Open Source linguistic text processing LingPipe:
    http://alias-i.com/lingpipe/ Semantic Indexing
    news Sat, 08 Oct 2011 23:15:00 +0200
    Need for speed: TYPO3 Website-Caching http://www.typo3-media.com/blog/website-caching-login.html Caching is always a tricky thing and a good caching strategy always depends on the use cases of the... Website Caching Levels Of course there are tons of possible ways to cache a content on a website - I only want to look at the most promising cache levels:

    How dynamic is my website?

    1. based on time (news articles)

    Decide on a cache strategie

    1 Use Cache-Control where possible

    The best you can do is allow client side caching. In TYPO3 this is pretty easy, you just need to set: config.sendCacheHeaders=1 The precondition is, that the page is cacheable (no user INT). TYPO3 then sends proper Cache-Control Headers.The Expire time is depending on the settings in the page: You can control the expire time for all pages in the page properties (Cache expire). (Default is always one day or you set the default with config.cache_period ). Also you can set the expire to end at midnight with: config.cache_clearAtMidnight =1 

    Problem 1: "USER_INT":

      plugin.tx_felogin_pi1 = USER 
      [usergroup = *]
      plugin.tx_felogin_pi1 = USER_INT

      [globalVar = GP:aoe_solr|sword=]

    Problem 2: Login and User specific content


        [globalString = _SERVER|HTTPS=on]


    2. Use Varnish

    If you only have Cache-Headers, TYPO3 still needs to deliver the page for every new user. By adding another layer of cache even this can be avoided. Fabrizio wrote a nice blog article on setting up Nginx + Varnish: http://www.fabrizio-branca.de/nginx-varnish-apache-magento-typo3.html

    Problem 1: Cookies


    Problem 2: rsaauth and PHPSESSION cookie

    If you dont use https for your frontend login, TYPO3 offers a way to still secure your login. This is done with the help of the extension "rsaauth" and can be activated by setting the securityLevel to "rsa". However this extension is not functional with client side caching - so please use https and deactivate rsa... A recommended login configuration is: $TYPO3_CONF_VARS['FE']['loginSecurityLevel'] = 'normal'; and have "saltedpasswd" installed and activated in extension settings. (For the backend you should use lockSLL ) TYPO3 might still send PHPSESSION cookies - they can be dropped in varnish. ( see also Bug: http://forge.typo3.org/issues/29927 )

    ESI - Edge Side Includes?

    ESI is a technique from varnish to get dynamic content in a cached page. The extension moc_varnish has support for this (rewriting USER_INT to ESI Includes). But in fact I think ESI don't has  advantages:
    Inside TYPO3 Sat, 17 Sep 2011 23:03:00 +0200
    TYPO3 Localisation with TemplaVoila http://www.typo3-media.com/blog/localisation-secrets.html This article should give a compact summary of a possible best practice multilanguage configuration... Precondition Install the extension "languagevisibility" (see links below). This extension is required for the full localisation power. You may also want to have a look at the "l10n" extension that adds mass translation support.

    Website Languages

    Add the "website language" records under the TYPO3 page root and configure the desired fallback order. (typically you want to have at least "default" in you fallback order)


    mod.SHARED { 		   
    defaultLanguageFlag = us
    defaultLanguageLabel = English - US
    mod.web_txtemplavoilaM1.enableLocalizationLinkForFCEs = 1
    mod.web_txtemplavoilaM1.hideCopyForTranslation >


     config {
    sys_language_mode = ignore
    sys_language_overlay = hideNonTranslated
    sys_language_uid = 0
    locale_all = en_US.utf8
    [globalVar = GP:L = 1]
    config {
    sys_language_uid = 1
    language = de
    locale_all = de_DE.utf8

    TemplaVoila Localisation Modi

    TemplaVoila has 4 localisation modi:
    1. "Disabled" ( langDisabled = 1 )
    2. "Inlinetranslation" ( langChildren = 1 )
    3. "Inlinetranslation-seperate" ( langChildren = 0 )
    4. "databaseoverlay" ( langDatabaseOverlay =1 ) *
    (* the mode "databaseoverlay" is added by the extension "languagevisibility" )

    Container FCEs

    You should use the mode "disabled" for all FCEs that act as pure containers:
    <meta type="“array”" />
    This setup in the DS tells TYPO3 that this FCE should not be translated. And it tells TemplaVoila, that this FCE should not be edited and that the default value for the language field should be "All".

    Content FCEs and mixed FCEs

    The typical FCEs have fields with some content, that needs to be localised. You should use the mode "databaseoverlay". (Inlinetranslation support has two big disadvantages: 1. No concurrent work on languageversions in workspaces and 2. bad usability)
    <meta type="“array”" />
    If your FCE has translatable content and also container fields that allows for further nesting you need to configure the container fields in a way that it always uses the default language content:
    <field_content type="array">
    <TCEforms type="array">
    If your FCE has translatable content that is not required, you may want to fall back to the default language if this field has no content. This is useful for Images for example:
    <field_image type="array">
    <TCEforms type="array">
    (* the "l10n_mode" support for flexforms  is added by the extension "languagevisibility" )

    page DS

    <meta type="“array”" />
    This configuration enables the inline localisation for any TemplaVoila page property. (In addition you have the possibility the maintain a completely own contentelement structure on pages for certain languages)

    Special Attention

        • don't forget to check your correct UTF8 setup. (If TYPO3 version is below 4.5 you need to set "forceCharset = utf-8" and maybe also setDBInit )
        • TYPO3 uses the language ISO code as key for inlinetranslation in flexforms. If you have multiple localizations per language you need to add additional pseudo ISO codes. ( Read more in a related article "multilanguage websites with same language")
        Inside TYPO3 Wed, 13 Jul 2011 21:08:00 +0200