title: Can not get to work Fluentbit multi-line parser in k8s env
body: <p>My project is deployed in k8s environment and we are using fluent bit to send logs to ES. I need to send java stacktrace as one document. Therefore I have used fluent bit multi-line parser but I cannot get it work.</p>
  <p><strong>Approach 1:</strong></p>
  <p>As per lot of tutorials and documentations I configured fluent bit as follows.</p>
  <pre><code>[INPUT]
    Name              tail
    Tag               kube.test.*
    Path              /var/log/containers/*.log
    DB                /var/log/test.db
    Mem_Buf_Limit     50MB
    Refresh_Interval  10
    Multiline         On
    Parser_Firstline  multine_parser_first_line
  
  [PARSER]
    Name        multine_parser_first_line
    Format      regex
    Regex       /^(?&lt;time&gt;(\d)+(-\d+)+(\S)+\W(\S)+)(?&lt;message&gt;.*)/
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Keep On
  </code></pre>
  <p><strong>Approach 2:</strong></p>
  <p>As per answer in <a href="https://stackoverflow.com/questions/51645296/fluentbit-with-mycat-multiline-parsing">Fluentbit with mycat multiline parsing</a> used two parsers</p>
  <pre><code>[INPUT]
    Name              tail
    Tag               kube.test.*
    Path              /var/log/containers/*.log
    DB                /var/log/test.db
    Mem_Buf_Limit     50MB
    Refresh_Interval  10
    Multiline         On
    Parser_Firstline  multine_parser_first_line
    Parser_1          error_log_parser
  
  [PARSER]
    Name        multine_parser_first_line
    Format      regex
    Regex       (\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2}).(\d{3})
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Keep On
  
  [PARSER]
    Name        error_log_parser
    Format      regex
    Regex       \n(?m)^.*?Exception.*(?:[\r\n]+^\s*at .*)+\n
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L
    Time_Keep On
  </code></pre>
  <p>Following is my log:</p>
  <pre><code>    2021-07-07 16:46:46.720 DEBUG [auth-service,0f6d420997b2b169,8ed349d5c074f8a9,false] [ADMIN,admin@testing.com,] 1 --- [nio-8080-exec-6] .m.m.a.ExceptionHandlerExceptionResolver : Invoking @ExceptionHandler method: public org.springframework.http.ResponseEntity&lt;com.testing.fluentbit.exceptions.PlatformException&gt; com.testing.platform.common.core.web.ApiExceptionHandler.handlePlatformException(com.testing.fluentbit.exceptions.PlatformException)
      2021-07-07 16:46:46.720  WARN [auth-service,0f6d420997b2b169,8ed349d5c074f8a9,false] [ADMIN,admin@testing.com,] 1 --- [nio-8080-exec-6] c.l.p.c.core.web.ApiExceptionHandler     : Request resulted in platform error com.testing.fluentbit.exceptions.ResourceNotFoundException:avatar.not.found. Stacktrace is available in debug level
      2021-07-07 16:46:46.721 DEBUG [auth-service,0f6d420997b2b169,8ed349d5c074f8a9,false] [ADMIN,admin@testing.com,] 1 --- [nio-8080-exec-6] c.l.p.c.core.web.ApiExceptionHandler     : Platform Exception Handler - com.testing.fluentbit.exceptions.ResourceNotFoundException : avatar.not.found =&gt; {value=null}
  
      com.testing.fluentbit.exceptions.ResourceNotFoundException: avatar.not.found
          at com.testing.platform.auth.services.AvatarService.getAvatarByIndividualId(AvatarService.java:163)
          at com.testing.platform.auth.api.v1.LoginResource.getAvatarByIndividualId(LoginResource.java:541)
          at com.testing.platform.auth.api.v1.LoginResource$$FastClassBySpringCGLIB$$9a81467a.invoke(&lt;generated&gt;)
          at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
          at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
          at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:69)
          at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
          at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
          at com.testing.platform.auth.api.v1.LoginResource$$EnhancerBySpringCGLIB$$208fb4d9.getAvatarByIndividualId(&lt;generated&gt;)
          at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
          at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
          at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
          at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)
          at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
          at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
          at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
          at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
          at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
          at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
          at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at com.testing.platform.common.core.web.filters.ContextLoggingFilter.doFilter(ContextLoggingFilter.java:45)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.security.oauth2.client.filter.OAuth2ClientContextFilter.doFilter(OAuth2ClientContextFilter.java:60)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
          at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
          at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:66)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
          at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
          at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
          at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
          at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.cloud.sleuth.instrument.web.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.java:48)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at brave.servlet.TracingFilter.doFilter(TracingFilter.java:86)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
          at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
          at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
          at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
          at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
          at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
          at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
          at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
          at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
          at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
          at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
          at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)
          at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
          at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)
          at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
          at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
          at java.lang.Thread.run(Thread.java:748)
  
      2021-07-07 16:46:46.722 DEBUG [auth-service,0f6d420997b2b169,8ed349d5c074f8a9,false] [ADMIN,admin@testing.com,] 1 --- [nio-8080-exec-6] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Written [com.testing.fluentbit.exceptions.ResourceNotFoundException: avatar.not.found] as &quot;application/json&quot; using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@5cd6719d]
      2021-07-07 16:46:46.722  WARN [auth-service,0f6d420997b2b169,8ed349d5c074f8a9,false] [ADMIN,admin@testing.com,] 1 --- [nio-8080-exec-6] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [com.testing.fluentbit.exceptions.ResourceNotFoundException: avatar.not.found]
  </code></pre>
  <p>My first line multi line parser regex matches the log line. However I cannot get this to work. Any help is much appreciated. I am using fluentbit amazon/aws-for-fluent-bit:2.15.1</p>
  <p><strong>EDIT</strong>
  Approach 1 parser works locally with ES.</p>