I've installed ElasticSearch 1.4.1 on Ubuntu machine.

Its logs are in the default location:


When I run the ElasticSearch, after a while the log files grows and it become difficult to handle them.

The logs are already in a rotating file mode. Meaning, every day there is a new log file.

I want to configure the logs to be zipped (e.g. = file.log.zip ,currently they're not).

I also know that I can do it in the logging.yml file but I don't know how to do it.

Can someone help me with that?


After some digging (also in the ElasticSearch source code) I've found an answer. :)

Usually, when you're using a software like ElasticSearch and want it to be in production use, you're thinking that a basic function like logging is all taking care of. Sadly, in the current version (1.4 and prior) this is not the case.

ElasticSearch is using log4j as its logging mechanism.

log4j is doing the following:

  1. Add logs to a log file
  2. Rotate the log file when it is bigger then CONFIGURED_VALUE

If there is a heavy use in ElasticSearch, the logs are piling up and eventually filling your entire storage.

The answer to that is adding the following:

  1. Zip the old rotated log file
  2. Make sure that the total log files will not be over than CONFIGURED_VALUE

The answer to that is that there is another API that called log4j-extensions and it extends log4j capabilities and zip the rotating log files.

Sadly, it will be available only on the next version 1.5 or in the master branch for the crazy people among us who compile ElasticSearch from source code. (see log4j rollingPolicy support).

BUT, there is a simpler solution:


If the ElasticSearch is running on a Linux OS, you can use logrotate daemon. (see what is log rotate Understanding logrotate utility)

You need to do the following:

  1. Reset the log configuration
  2. Create a new file for handling log files

Reset the log configuration

sudo vi /etc/elasticsearch/logging.yml

Change the following in logging.yml

# Mark the dailyRollingFile Appender
#  file:
#    type: dailyRollingFile
#    file: ${path.logs}/${cluster.name}.log
#    datePattern: "'.'yyyy-MM-dd"
#    layout:
#      type: pattern

# Add the file Appender
  type: file
  datePattern: "'.'yyyy-MM-dd"
    type: pattern

Create a new file for handling log files

sudo vi /etc/logrotate.d/elasticsearch

Add the following to the logrotate file:

/var/log/elasticsearch/*.log {
    rotate 100
    size 50M
    create 644 elasticsearch elasticsearch

After that restart ElasticSearch

sudo service elasticsearch stop
sudo service elasticsearch start

This way you'll limit the total log files storage into 5GB (rotate 100 * 50M).

Obviously, you can configure it as you see fit.

  • datePattern pertains to the rolling schedule for the DailyRollingFileAppender, which you are getting rid of. I believe you should remove that line whenever switching to FileAppender. – Larry Silverman Jul 13 '15 at 16:45
  • Is there no need for a post-rotate command? That is, does log4j log properly after the files are rotated? – ColinM Oct 7 '15 at 15:18
  • Can anyone confirm that the log4j-extensions are in the 1.5+ versions now and that you don't need to use logrotate to truncate old log files? I would like to use native log4j if possible. Thanks! – Alf47 Nov 23 '15 at 15:16

I would disagree with your answer slightly. First, I would recommend against using zip, if you use gzip instead you can still read the files from the command line without extracting them first. Second, while logrotation is a handy tool, log4j should be able to handle it's own log rotation natively. The following config should work nicely:

    type: org.apache.log4j.rolling.RollingFileAppender
    rollingPolicy: org.apache.log4j.rolling.TimeBasedRollingPolicy
    rollingPolicy.FileNamePattern: ${path.logs}/${cluster.name}.log.%d{yyyy-MM-dd}.gz
      type: pattern
      conversionPattern: "%d{ISO8601}[%-5p][%-25c] %m%n"

Your Answer

