title: spring boot custom rolling log file
body: <p>We have a spring boot application, and use logback with rolling files to create json and standard log files.</p>
  
  <p>We use filebeat to scrape the log.json files and push to the elk server</p>
  
  <p>I would like to add an  op.json  custom json rolling log file in the log directory.   This is to collect information on operations called and perf info.</p>
  
  <p>Basily, I just want to  use a class that manage rolling files.    In the class, there will be a logOp function that accects a java object, and serializes it, added some standard log fields to the json output.</p>
  
  <p>Questions:
  1) {Not linkly, but would be nice} is there functionality in logback that will allow me to created a named logger that will append a string (serialized JSON in my case) to the end of op.json log file, and implement rolling logger functionality</p>
  
  <p>2) if this is not available through logback (which I would understand).  Is there a java package that manages appending a string to a rolling file?   I could just create an instance of this "Appender" and add strings to it.   This would be like a standanone Rolling file appender in logback</p>
  
  <p>My Goal with this is to create a simple JSON file that can either be directly viewed, and/or scraped with filebeat and sent to logstash for creating dashboards for operation performance.</p>
  
  <p>-------------------- Configururation for  spring-boot-logger.log and spring-boot-logger.json  files {for log.debug("my log message},  Includes sleuth  ----------</p>
  
  <p>
  </p>
  
  <pre><code>&lt;include resource="org/springframework/boot/logging/logback/defaults.xml"/&gt;
  
  &lt;springProperty scope="context" name="site" source="ci2.application"/&gt;
  &lt;springProperty scope="context" name="app" source="spring.application.name"/&gt;
  
  
  &lt;property name="LOGS" value="./logs" /&gt;
  
  
  &lt;!-- You can override this to have a custom pattern --&gt;
  &lt;property name="CONSOLE_LOG_PATTERN"
            value="%clr(%d{yyyy-MM-dd HH:mm:ss.SSS}){faint} %clr(${LOG_LEVEL_PATTERN:-%5p}) %clr(${PID:- }){magenta} %clr(---){faint} %clr([%15.15t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n${LOG_EXCEPTION_CONVERSION_WORD:-%wEx}"/&gt;
  
  &lt;!-- Appender to log to console --&gt;
  &lt;appender name="Console" class="ch.qos.logback.core.ConsoleAppender"&gt;
      &lt;filter class="ch.qos.logback.classic.filter.ThresholdFilter"&gt;
          &lt;!-- Minimum logging level to be presented in the console logs--&gt;
      &lt;/filter&gt;
      &lt;encoder&gt;
          &lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
          &lt;charset&gt;utf8&lt;/charset&gt;
      &lt;/encoder&gt;
  &lt;/appender&gt;
  
  
  
  &lt;appender name="RollingFile"
      class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
      &lt;file&gt;${LOGS}/spring-boot-logger.log&lt;/file&gt;
      &lt;encoder&gt;
          &lt;pattern&gt;${CONSOLE_LOG_PATTERN}&lt;/pattern&gt;
          &lt;charset&gt;utf8&lt;/charset&gt;
      &lt;/encoder&gt;
      &lt;rollingPolicy
          class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
          &lt;!-- rollover daily and when the file reaches 10 MegaBytes --&gt;
          &lt;fileNamePattern&gt;${LOGS}/archived/spring-boot-logger-%d{yyyy-MM-dd}.%i.log
          &lt;/fileNamePattern&gt;
          &lt;timeBasedFileNamingAndTriggeringPolicy
              class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"&gt;
              &lt;maxFileSize&gt;10MB&lt;/maxFileSize&gt;
          &lt;/timeBasedFileNamingAndTriggeringPolicy&gt;
      &lt;/rollingPolicy&gt;
  &lt;/appender&gt;
  
  
  &lt;appender name="Json"
            class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
      &lt;file&gt;${LOGS}/spring-boot-logger.json&lt;/file&gt;
      &lt;encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"&gt;
          &lt;providers&gt;
              &lt;timestamp&gt;
                  &lt;timeZone&gt;UTC&lt;/timeZone&gt;
              &lt;/timestamp&gt;
              &lt;pattern&gt;
                  &lt;pattern&gt;
                      {
                      "severity": "%level",
                      "service": "${app:-}",
                      "trace": "%X{X-B3-TraceId:-}",
                      "span": "%X{X-B3-SpanId:-}",
                      "parent": "%X{X-B3-ParentSpanId:-}",
                      "exportable": "%X{X-Span-Export:-}",
                      "pid": "${PID:-}",
                      "thread": "%thread",
                      "class": "%logger{40}",
                      "rest": "%message",
                      "schema":"ci2",
                      "group":"devops",
                      "site": "${site}"
  
                      }
                  &lt;/pattern&gt;
              &lt;/pattern&gt;
          &lt;/providers&gt;
      &lt;/encoder&gt;  
      &lt;rollingPolicy
          class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
          &lt;!-- rollover daily and when the file reaches 10 MegaBytes --&gt;
          &lt;fileNamePattern&gt;${LOGS}/archived/spring-boot-logger-%d{yyyy-MM-dd}.%i.json
          &lt;/fileNamePattern&gt;
          &lt;timeBasedFileNamingAndTriggeringPolicy
              class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"&gt;
              &lt;maxFileSize&gt;10MB&lt;/maxFileSize&gt;
          &lt;/timeBasedFileNamingAndTriggeringPolicy&gt;
      &lt;/rollingPolicy&gt;
  
  &lt;/appender&gt;
  
  &lt;!-- LOG everything at INFO level --&gt;
  &lt;root level="warn"&gt;
      &lt;appender-ref ref="RollingFile" /&gt;
      &lt;appender-ref ref="Console" /&gt;
      &lt;appender-ref ref="Json" /&gt;
  &lt;/root&gt;
  
  &lt;!-- LOG everything at INFO level --&gt;
  &lt;root name="org.springframework" level="info"&gt;
      &lt;appender-ref ref="RollingFile" /&gt;
      &lt;appender-ref ref="Console" /&gt;
      &lt;appender-ref ref="Json" /&gt;
  &lt;/root&gt;
  
  
  &lt;!-- LOG "com.siemens.pl*" at INFO level --&gt;
  &lt;logger name="com.siemens.pl" level="info" additivity="false"&gt;
      &lt;appender-ref ref="RollingFile" /&gt;
      &lt;appender-ref ref="Console" /&gt;
      &lt;appender-ref ref="Json" /&gt;
  &lt;/logger&gt;
  </code></pre>
  
  <p></p>
  
  <p>-----------------  End logback config ---------------------------------</p>
  
  <p>I am trying to add dynamicly adding values to the json log.</p>
  
  <p>Something like</p>
  
  <p>1)  Highest level support, add values from map as JSON entries in generated json</p>
  
  <pre><code> something like
  
  
  
      Map myValues = new HashMap&lt;String,String&gt; ();
      myValues.put("startTime",startTime);
      myValues.put("endTime",endTime);
      myValues.put("op",operation);
      myValues.put("user",user);
      // ideal usage format, would add map as "key":"value"  to json output
      log.debug(myValues);
  </code></pre>
  
  <p>2) If appending a user generated sting to a logfile (with nothing added by logging), I could generate the log string (json myKeys + stander logging), and then append</p>
  
  <pre><code>    Map myValues = new HashMap&lt;String,String&gt; ();
      myValues.put("startTime",startTime);
      myValues.put("endTime",endTime);
      myValues.put("op",operation);
      myValues.put("user",user);
  
      #Generate JSON String that includes standard logging fields and dynamic
      # user fileds passed in myValues map
      String myLogStringToAppendToLogFile = generateLogString(myValues);
  
  
      #Ideal op would accept string and append user provide string to end of
      #logfile with no modification
      log.debug("RawString", myLogStringToAppendToLogFile );
      log.debug(myValues);
  </code></pre>
  
  <hr>
  
  <p>The above would be the most simple,  I am not sure if there is an API on a logger that would accept a map and add to log</p>
  
  <p>Another option would for the logger add a fixed set of properties to the json file, and the the property value could be passed in on the user log line.</p>
  
  <p>For example, from the logback def above
         -------------------</p>
  
  <pre><code>&lt;appender name="Json"
            class="ch.qos.logback.core.rolling.RollingFileAppender"&gt;
      &lt;file&gt;${LOGS}/spring-boot-logger.json&lt;/file&gt;
      &lt;encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"&gt;
          &lt;providers&gt;
              &lt;timestamp&gt;
                  &lt;timeZone&gt;UTC&lt;/timeZone&gt;
              &lt;/timestamp&gt;
              &lt;pattern&gt;
                  &lt;pattern&gt;
                      {
                      "severity": "%level",
                      "service": "${app:-}",
                      "trace": "%X{X-B3-TraceId:-}",
                      "span": "%X{X-B3-SpanId:-}",
                      "parent": "%X{X-B3-ParentSpanId:-}",
                      "exportable": "%X{X-Span-Export:-}",
                      "pid": "${PID:-}",
                      "thread": "%thread",
                      "class": "%logger{40}",
                      "rest": "%message",
                      "schema":"ci2",
                      "group":"devops",
                      "site": "${site}"
  
                      }
                  &lt;/pattern&gt;
              &lt;/pattern&gt;
          &lt;/providers&gt;
      &lt;/encoder&gt;  
      &lt;rollingPolicy
          class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"&gt;
          &lt;!-- rollover daily and when the file reaches 10 MegaBytes --&gt;
          &lt;fileNamePattern&gt;${LOGS}/archived/spring-boot-logger-%d{yyyy-MM-dd}.%i.json
          &lt;/fileNamePattern&gt;
          &lt;timeBasedFileNamingAndTriggeringPolicy
              class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"&gt;
              &lt;maxFileSize&gt;10MB&lt;/maxFileSize&gt;
          &lt;/timeBasedFileNamingAndTriggeringPolicy&gt;
      &lt;/rollingPolicy&gt;
  
  &lt;/appender&gt;
  -----------------------
  
  In this appender def, the &lt;pattern&gt; is used to add values to the json def for sleuth.    
  
  How can I define entries in this pattern and pass in the dynamic value to the logger.  It looks possible and sleuth may do it with  
  
                       "trace": "%X{X-B3-TraceId:-}",
                      "span": "%X{X-B3-SpanId:-}",
                      "parent": "%X{X-B3-ParentSpanId:-}",
                      "exportable": "%X{X-Span-Export:-}",
  </code></pre>
  
  <p>Another API that would work is that the logger would accept a string and append the user provided string to the end of the log file</p>