text: Q:
  
  Is there a way parse log messages using rsyslog config and transform them to structured messages?
  
  I am trying to parse log messages and transform them to structured messages using rsyslog. Is there a way support such operation with rsyslog config? I have not yet explored the option to write custom parser or message modification plugin for this. 
  I found template list properties which can do some of it. Is there a way to do the following?
  
  Map 2 fields to single output name. Ex: "__ts": "2018-09-20 10:18:56.363" (first 2 fields in example below). Would not use regex here as I am looking for a solution that does not depend on value of the fields. Ex: the two fields could be two strings or some other values not just dates.
  Extract what is left in msg after extracting all known fields based on position. Ex: "msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN". 
  Is there a way to use local variables to hold the values of fields from msg and use the variables in templates? 
  
  Example Log message:
  2018-09-20 10:18:56.363  INFO --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN
  1. rsyslog config template definition
  template(name="structure-log-format" type="list") {
    constant(value="{")
  
    # This only extracts the first field with value 2018-09-20.
    # TODO: What is a way to map first 2 fields to map to __ts field? 
    property(outname="__ts" name="msg" field.number="1" field.delimiter="32" format="jsonf") constant(value=", ")
  
    constant(value="\"event\":[{")
      constant(value="\"payload\":{")
          property(outname="_log_" name="syslogtag" format="jsonf") constant(value=", ")
          property(outname="__loglvl" name="msg" field.number="4" field.delimiter="32" format="jsonf") constant(value=", ")
          property(outname="__thread" name="msg" field.number="7" field.delimiter="32" format="jsonf") constant(value=", ")
          property(outname="__key1" name="msg" field.number="8" field.delimiter="32" format="jsonf") constant(value=", ")
          # The following setting will include full message value starting from "2018-09-20 ... DOWN"
          # TODO: What is a way to only include message starting from "Unregistering ... DOWN"?
          property(name="msg" format="jsonf" droplastlf="on" )
      constant(value="}")
  constant(value="}]} \n")
  
  }
  2. Expected result: 
  {
     "__ts": "2018-09-20 10:18:56.363",
     "event": [
         {
            "payload": {
               "_log_": "catalina",
               "__loglvl": "INFO",
               "__thread": "Thread-68",
               "__key1": "x.y.z.key1Value",
               "msg": "Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
            }
         }
       ]
  }
  
  3. Actual result:
  {
     "__ts": "2018-09-20",
     "event": [
         {
            "payload": {
               "_log_": "catalina",
               "__loglvl": "INFO",
               "__thread": "Thread-68",
               "__key1": "x.y.z.key1Value",
               "msg": "2018-09-20 10:18:56.363  INFO 2144 --- [Thread-68] x.y.z.key1Value Unregistering application nameOfAnApiHere with someOtherName with status DOWN"
            }
         }
       ]
  }
  
  Thank you.
  
  A:
  
  You can also use regular expressions to match parts of a message. For example, replace your outname="__ts" property with:
   property(outname="__ts" name="msg" 
    regex.expression="([^ ]+ +[^ ]+)" 
    regex.type="ERE" 
    regex.submatch="1" format="jsonf")
  
  Here the extended regular expression (ERE) looks for not-a-space ([^ ]) one or more of them (+), followed by a space or more, and another not-a-space. These 2 words are captured as a submatch by the () and you select this one, counting from 1.  The result should be as you want.
  You can similarly use a regex for the second requirement, either by counting "words" and spaces again, or some more precise other match.  Here the regex skips 6 words by putting a repeat count {6} after the word-and-spaces pattern, then captures the rest (.*). Since there are 2 sets of (), the submatch to keep is now 2, not 1:
   property(name="msg" 
    regex.expression="([^ ]+ +){6}(.*)" 
    regex.type="ERE" 
    regex.submatch="2" format="jsonf" droplastlf="on" )
  
  