{"action":"created","issue":{"url":"https://api.github.com/repos/aol/moloch/issues/501","repository_url":"https://api.github.com/repos/aol/moloch","labels_url":"https://api.github.com/repos/aol/moloch/issues/501/labels{/name}","comments_url":"https://api.github.com/repos/aol/moloch/issues/501/comments","events_url":"https://api.github.com/repos/aol/moloch/issues/501/events","html_url":"https://github.com/aol/moloch/issues/501","id":165102402,"number":501,"title":"Incorrect number of indices being removed with db maint - w/hourly indices ","user":{"login":"jbagel2","id":8388709,"avatar_url":"https://avatars.githubusercontent.com/u/8388709?v=3","gravatar_id":"","url":"https://api.github.com/users/jbagel2","html_url":"https://github.com/jbagel2","followers_url":"https://api.github.com/users/jbagel2/followers","following_url":"https://api.github.com/users/jbagel2/following{/other_user}","gists_url":"https://api.github.com/users/jbagel2/gists{/gist_id}","starred_url":"https://api.github.com/users/jbagel2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbagel2/subscriptions","organizations_url":"https://api.github.com/users/jbagel2/orgs","repos_url":"https://api.github.com/users/jbagel2/repos","events_url":"https://api.github.com/users/jbagel2/events{/privacy}","received_events_url":"https://api.github.com/users/jbagel2/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-12T15:00:29Z","updated_at":"2016-07-22T23:06:12Z","closed_at":"2016-07-12T23:09:55Z","body":"**Elasticsearch version**: 2.3.3\r\n\r\n**Moloch version**: v0.14.2\r\n\r\n**OS name and version**: CentOS Linux release 7.2.1511\r\n\r\n**Provide logs, stack traces and steps to reproduce**:\r\n\r\nevery morning looking at the data in the viewer the spi data is only available starting at 4:00am, so at 8:00am I only have about 4 hours worth of spi data. Through the day the spi data will build without issue. By the end of the day the I still have SPI data all the way back to 4:00am that same morning.\r\n\r\nI currently have the db expire job set up to run every 2 hours to reduce the time to completion as the system is always under pretty heavy load. Was also testing if running more frequently would reduce the number of indices needing to be deleted each time. Noticing the expire logs the number of indices being deleted is not what I would expect when passing in 20 hours as the retaining number, and it checking every 2 hours. It seems once the time hits about 6:00pm, the number of indices being deleted seems to increase each check until it peaks at deleting 5 every 2 hours (which doesn't make sense to me). It then drops back down starting at about 6:00am. Now maybe this is just me not understanding how to properly configure the db script. If so please let me know what I need to do to correct.\r\n```\r\ndb expire script:\r\n------------\r\nESHOSTPORT=myremoteESServer:9200\r\nRETAINNUMHOURS=20\r\necho \"-------------------\" >> /srv/moloch/db/daily.log\r\necho \"$(date)\" >> /srv/moloch/db/daily.log\r\n/srv/moloch/db/db.pl $ESHOSTPORT expire hourly $RETAINNUMHOURS\r\n-------------\r\n\r\nhere is snippet of the db expire job logs:\r\n\r\nMon Jul 11 14:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 0 indices, optimizing 10\r\n-------------------\r\nMon Jul 11 16:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 0 indices, optimizing 12\r\n-------------------\r\nMon Jul 11 18:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 1 indices, optimizing 13\r\n-------------------\r\nMon Jul 11 20:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 4 indices, optimizing 11\r\n-------------------\r\nMon Jul 11 22:00:02 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 5 indices, optimizing 9\r\n-------------------\r\nTue Jul 12 00:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 5 indices, optimizing 7\r\n-------------------\r\nTue Jul 12 02:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 5 indices, optimizing 5\r\n-------------------\r\nTue Jul 12 04:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 5 indices, optimizing 3\r\n-------------------\r\nTue Jul 12 06:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 3 indices, optimizing 3\r\n-------------------\r\nTue Jul 12 08:00:01 MDT 2016\r\nOptimizing Admin Indices\r\n\r\nExpiring 1 indices, optimizing 4\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/aol/moloch/issues/comments/234677945","html_url":"https://github.com/aol/moloch/issues/501#issuecomment-234677945","issue_url":"https://api.github.com/repos/aol/moloch/issues/501","id":234677945,"user":{"login":"jbagel2","id":8388709,"avatar_url":"https://avatars.githubusercontent.com/u/8388709?v=3","gravatar_id":"","url":"https://api.github.com/users/jbagel2","html_url":"https://github.com/jbagel2","followers_url":"https://api.github.com/users/jbagel2/followers","following_url":"https://api.github.com/users/jbagel2/following{/other_user}","gists_url":"https://api.github.com/users/jbagel2/gists{/gist_id}","starred_url":"https://api.github.com/users/jbagel2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbagel2/subscriptions","organizations_url":"https://api.github.com/users/jbagel2/orgs","repos_url":"https://api.github.com/users/jbagel2/repos","events_url":"https://api.github.com/users/jbagel2/events{/privacy}","received_events_url":"https://api.github.com/users/jbagel2/received_events","type":"User","site_admin":false},"created_at":"2016-07-22T23:06:12Z","updated_at":"2016-07-22T23:06:12Z","body":"That fixed part of it, it still seemed to be ignoring the int value being passed in and locked to 12 indexes.  See the below fixes, that took care of all the issues for me. mktime seemed to be what was mucking up the @startTime value. Also the subtraction was trying to happen to the scalar not the array, and before the scalar was declared anyway. I would have just created a pull request but my fork is a little behind.\r\n\r\n----------------------------------------------------\r\nindex 03f70c2..a10506e 100755\r\n--- a/db/db.pl\r\n+++ b/db/db.pl\r\n@@ -39,6 +39,7 @@ use HTTP::Request::Common;\r\n use LWP::UserAgent;\r\n use JSON;\r\n use Data::Dumper;\r\n+use Time::Local;\r\n use POSIX;\r\n use strict;\r\n \r\n@@ -1988,19 +1989,19 @@ if ($ARGV[1] =~ /^users-?import$/) {\r\n \r\n     my @startTime = gmtime;\r\n     if ($ARGV[2] eq \"hourly\") {\r\n-        $startTime[2] -= int($ARGV[3]);\r\n+        @startTime[2] -= int($ARGV[3]);\r\n     } elsif ($ARGV[2] eq \"daily\") {\r\n-        $startTime[3] -= int($ARGV[3]);\r\n+        @startTime[3] -= int($ARGV[3]);\r\n     } elsif ($ARGV[2] eq \"weekly\") {\r\n-        $startTime[3] -= 7*int($ARGV[3]);\r\n+        @startTime[3] -= 7*int($ARGV[3]);\r\n     } elsif ($ARGV[2] eq \"monthly\") {\r\n-        $startTime[4] -= int($ARGV[3]);\r\n+        @startTime[4] -= int($ARGV[3]);\r\n     }\r\n \r\n     parseArgs(4);\r\n \r\n     my $optimizecnt = 0;\r\n-    my $startTime = mktime(@startTime);\r\n+    my $startTime = timegm(@startTime);\r\n     while ($startTime <= $endTime) {\r\n         my $iname = time2index($ARGV[2], $startTime);\r\n         if (exists $indices->{$iname}) {\r\n"}}
{"actor":{"display_login":"jbagel2"}}