title: Backup repository size is much bigger than indices size
body: <p>I have an Elasticsearch backup per hour, after about 370 times of backup (about 15 days) , my backup repository is more than 15G !!! But the total indices size is just about 500M !! Elasticsearch is incremental backup, but 15G VS 500M , the difference is so huge ! I wonder whether it is normal with so big different size between indices and backup repository ? </p>
  
  <p>Is it caused by my frequent backup (hourly) ? I use the hourly backup in cluster 1 and hourly restore in cluster 2 to keep two ES clusters data same real time .</p>
  
  <p>I am using Elasticsearch backup API to backup</p>
  
  <ol>
  <li><p>setting repo: 
  curl -XPUT "<a href="http://IP:9200/_snapshot/backup">http://IP:9200/_snapshot/backup</a>" -d "{\"type\": \"fs\",\"settings\": {\"compress\" : true,\"location\": \"backup\"}}"</p></li>
  <li><p>backup : 
  CURTIME=<code>date +"%Y-%m-%d %H:%M:%S"</code>
  BKTIME=${CURTIME//[- :]/}
  curl -XPUT "<a href="http://IP:9200/_snapshot/backup/snapshot_">http://IP:9200/<em>snapshot/backup/snapshot</em></a>$BKTIME?wait_for_completion=true"</p></li>
  </ol>
  
  <p>My Elasticsearch settings : 2 nodes , 12 shard/node , 2 indices , fs type of backup to store snapshots to NAS</p>
  
  <p>in Elasticsearch data directory , the indices size :</p>
  
  <p>node1 indices size : 
  [root@esnode1 indices]$ du -sh</p>
  
  <p>307M .</p>
  
  <p>node2 indices size
  [root@esnode2 indices]$ du -sh</p>
  
  <p>238M .</p>
  
  <p>[root@esnode1 indices]$ du -lh</p>
  
  <p>8.0K    ./index1/10/translog
  8.0K    ./index1/10/_state
  2.9M    ./index1/10/index
  2.9M    ./index1/10
  12K     ./index1/5/translog
  8.0K    ./index1/5/_state
  1.5M    ./index1/5/index
  1.5M    ./index1/5
  8.0K    ./index1/4/translog
  8.0K    ./index1/4/_state
  2.9M    ./index1/4/index
  2.9M    ./index1/4
  8.0K    ./index1/_state
  8.0K    ./index1/7/translog
  8.0K    ./index1/7/_state
  2.9M    ./index1/7/index
  2.9M    ./index1/7
  8.0K    ./index1/1/translog
  8.0K    ./index1/1/_state
  2.9M    ./index1/1/index
  2.9M    ./index1/1
  8.0K    ./index1/2/translog
  8.0K    ./index1/2/_state
  2.9M    ./index1/2/index
  2.9M    ./index1/2
  8.0K    ./index1/6/translog
  8.0K    ./index1/6/_state
  3.0M    ./index1/6/index
  3.0M    ./index1/6
  8.0K    ./index1/0/translog
  8.0K    ./index1/0/_state
  1.5M    ./index1/0/index
  1.5M    ./index1/0
  8.0K    ./index1/8/translog
  8.0K    ./index1/8/_state
  1.5M    ./index1/8/index
  1.5M    ./index1/8
  8.0K    ./index1/11/translog
  8.0K    ./index1/11/_state
  2.9M    ./index1/11/index
  2.9M    ./index1/11
  12K     ./index1/9/translog
  8.0K    ./index1/9/_state
  3.0M    ./index1/9/index
  3.0M    ./index1/9
  8.0K    ./index1/3/translog
  8.0K    ./index1/3/_state
  3.0M    ./index1/3/index
  3.0M    ./index1/3
  31M     ./index1
  16K     ./index2/10/translog
  8.0K    ./index2/10/_state
  16M     ./index2/10/index
  16M     ./index2/10
  36K     ./index2/5/translog
  8.0K    ./index2/5/_state
  43M     ./index2/5/index
  43M     ./index2/5
  20K     ./index2/4/translog
  8.0K    ./index2/4/_state
  17M     ./index2/4/index
  18M     ./index2/4
  8.0K    ./index2/_state
  40K     ./index2/7/translog
  8.0K    ./index2/7/_state
  32M     ./index2/7/index
  32M     ./index2/7
  68K     ./index2/1/translog
  8.0K    ./index2/1/_state
  21M     ./index2/1/index
  21M     ./index2/1
  64K     ./index2/2/translog
  8.0K    ./index2/2/_state
  19M     ./index2/2/index
  19M     ./index2/2
  116K    ./index2/6/translog
  8.0K    ./index2/6/_state
  22M     ./index2/6/index
  22M     ./index2/6
  24K     ./index2/0/translog
  8.0K    ./index2/0/_state
  17M     ./index2/0/index
  17M     ./index2/0
  128K    ./index2/8/translog
  8.0K    ./index2/8/_state
  34M     ./index2/8/index
  34M     ./index2/8
  72K     ./index2/11/translog
  8.0K    ./index2/11/_state
  20M     ./index2/11/index
  20M     ./index2/11
  88K     ./index2/9/translog
  8.0K    ./index2/9/_state
  22M     ./index2/9/index
  22M     ./index2/9
  76K     ./index2/3/translog
  8.0K    ./index2/3/_state
  16M     ./index2/3/index
  16M     ./index2/3
  277M    ./index2
  307M    .</p>
  
  <p>in backup repository , the size :
  [root@esnode1 backup]$ du -lh</p>
  
  <p>114M ./backup/indices/index1/0</p>
  
  <p>112M ./backup/indices/index1/5</p>
  
  <p>114M ./backup/indices/index1/11</p>
  
  <p>114M ./backup/indices/index1/10</p>
  
  <p>111M ./backup/indices/index1/8</p>
  
  <p>116M ./backup/indices/index1/4</p>
  
  <p>120M ./backup/indices/index1/9</p>
  
  <p>118M ./backup/indices/index1/3</p>
  
  <p>114M ./backup/indices/index1/2</p>
  
  <p>115M ./backup/indices/index1/7</p>
  
  <p>115M ./backup/indices/index1/1</p>
  
  <p>112M ./backup/indices/index1/6</p>
  
  <p>1.4G ./backup/indices/index1</p>
  
  <p>747M ./backup/indices/index2/0</p>
  
  <p>1.6G ./backup/indices/index2/5</p>
  
  <p>887M ./backup/indices/index2/11</p>
  
  <p>743M ./backup/indices/index2/10</p>
  
  <p>2.1G ./backup/indices/index2/8</p>
  
  <p>801M ./backup/indices/index2/4</p>
  
  <p>1.3G ./backup/indices/index2/9</p>
  
  <p>878M ./backup/indices/index2/3</p>
  
  <p>951M ./backup/indices/index2/2</p>
  
  <p>1.2G ./backup/indices/index2/7</p>
  
  <p>953M ./backup/indices/index2/1</p>
  
  <p>943M ./backup/indices/index2/6</p>
  
  <p>13G ./backup/indices/index2</p>
  
  <p>15G ./backup/indices</p>
  
  <p>15G ./backup</p>
  
  <p>1.1M ./backuplogs</p>
  
  <p>15G .</p>
  
  <p>======
  <a href="https://www.elastic.co/blog/introducing-snapshot-restore" rel="nofollow">https://www.elastic.co/blog/introducing-snapshot-restore</a>
  Both backup and restore operations are incremental, which means that only files that changed since the last snapshot will be copied into the repository or restored into an index. <strong>Incremental snapshots allow performing the snapshot operation as frequently as needed without too much disk space overhead.</strong> Users can now easily create a snapshot before upgrade or a risky change in the cluster and quickly rollback to the previous index state if things go wrong. The snapshot/restore mechanism can be also used to synchronize data between a “hot” cluster and a remote, “cold” backup cluster in a different geographic region for fast disaster recovery.</p>
  
  <p>From the above , my case is really an issue , Can anyone help me ? Thanks in advance !</p>