title: Using an HDFS Sink and rollInterval in Flume-ng to batch up 90 seconds of log information
body: <p>I am trying to use Flume-ng to grab 90 seconds of log information and put it into a file in HDFS.  I have flume working to look at the log file via an exec and tail however it is creating a file every 5 seconds instead of what I am trying to configure as every 90 seconds.  </p>
  
  <p>My flume.conf is as follows:</p>
  
  <pre><code># example.conf: A single-node Flume configuration                                                                                                                  
  # Name the components on this agent                                                                                                                                
  agent1.sources = source1
  agent1.sinks = sink1
  agent1.channels = channel1
  
  # Describe/configure source1                                                                                                                                       
  agent1.sources.source1.type = exec
  agent1.sources.source1.command = tail -f /home/cloudera/LogCreator/fortune_log.log
  
  # Describe sink1                                                                                                                                                   
  agent1.sinks.sink1.type = hdfs
  agent1.sinks.sink1.hdfs.path = hdfs://localhost/flume/logtest/
  agent1.sinks.sink1.hdfs.filePrefix = LogCreateTest
  # this parameter seems to be getting overridden                                                                                                                    
  agent1.sinks.sink1.hdfs.rollInterval=90
  agent1.sinks.sink1.hdfs.rollSize=0
  agent1.sinks.sink1.hdfs.hdfs.rollCount = 0
  
  # Use a channel which buffers events in memory                                                                                                                     
  agent1.channels.channel1.type = memory
  
  # Bind the source and sink to the channel                                                                                                                          
  agent1.sources.source1.channels = channel1
  agent1.sinks.sink1.channel = channel1
  </code></pre>
  
  <p>I am attempting to control the file size by the parameter - <strong>agent1.sinks.sink1.hdfs.rollInterval=90</strong>.</p>
  
  <p>Running this config produces:</p>
  
  <pre><code>13/01/03 09:43:02 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/etc/flume-ng/conf/flume.conf
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Processing:sink1
  13/01/03 09:43:02 INFO conf.FlumeConfiguration: Added sinks: sink1 Agent: agent1
  13/01/03 09:43:03 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration  for agents: [agent1]
  13/01/03 09:43:03 INFO properties.PropertiesFileConfigurationProvider: Creating channels
  13/01/03 09:43:03 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: CHANNEL, name: channel1, registered successfully.
  13/01/03 09:43:03 INFO properties.PropertiesFileConfigurationProvider: created channel channel1
  13/01/03 09:43:03 INFO sink.DefaultSinkFactory: Creating instance of sink: sink1, type: hdfs
  13/01/03 09:43:03 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false
  13/01/03 09:43:03 INFO instrumentation.MonitoredCounterGroup: Monitoried counter group for type: SINK, name: sink1, registered successfully.
  13/01/03 09:43:03 INFO nodemanager.DefaultLogicalNodeManager: Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:source1,state:IDLE} }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@1a50ca0c counterGroup:{ name:null counters:{} } }} channels:{channel1=org.apache.flume.channel.MemoryChannel{name: channel1}} }
  13/01/03 09:43:03 INFO nodemanager.DefaultLogicalNodeManager: Starting Channel channel1
  13/01/03 09:43:03 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: channel1 started
  13/01/03 09:43:03 INFO nodemanager.DefaultLogicalNodeManager: Starting Sink sink1
  13/01/03 09:43:03 INFO nodemanager.DefaultLogicalNodeManager: Starting Source source1
  13/01/03 09:43:03 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: sink1 started
  13/01/03 09:43:03 INFO source.ExecSource: Exec source starting with command:tail -f /home/cloudera/LogCreator/fortune_log.log
  13/01/03 09:43:07 INFO hdfs.BucketWriter: Creating hdfs://localhost/flume/logtest//LogCreateTest.1357224186506.tmp
  13/01/03 09:43:08 INFO hdfs.BucketWriter: Renaming hdfs://localhost/flume/logtest/LogCreateTest.1357224186506.tmp to hdfs://localhost/flume/logtest/LogCreateTest.1357224186506
  13/01/03 09:43:08 INFO hdfs.BucketWriter: Creating hdfs://localhost/flume/logtest//LogCreateTest.1357224186507.tmp
  13/01/03 09:43:12 INFO hdfs.BucketWriter: Renaming hdfs://localhost/flume/logtest/LogCreateTest.1357224186507.tmp to hdfs://localhost/flume/logtest/LogCreateTest.1357224186507
  13/01/03 09:43:12 INFO hdfs.BucketWriter: Creating hdfs://localhost/flume/logtest//LogCreateTest.1357224186508.tmp
  13/01/03 09:43:12 INFO hdfs.BucketWriter: Renaming hdfs://localhost/flume/logtest/LogCreateTest.1357224186508.tmp to hdfs://localhost/flume/logtest/LogCreateTest.1357224186508
  13/01/03 09:43:12 INFO hdfs.BucketWriter: Creating hdfs://localhost/flume/logtest//LogCreateTest.1357224186509.tmp
  13/01/03 09:43:18 INFO hdfs.BucketWriter: Renaming hdfs://localhost/flume/logtest/LogCreateTest.1357224186509.tmp to hdfs://localhost/flume/logtest/LogCreateTest.1357224186509
  13/01/03 09:43:18 INFO hdfs.BucketWriter: Creating hdfs://localhost/flume/logtest//LogCreateTest.1357224186510.tmp
  13/01/03 09:43:18 INFO hdfs.BucketWriter: Renaming hdfs://localhost/flume/logtest/LogCreateTest.1357224186510.tmp to hdfs://localhost/flume/logtest/LogCreateTest.1357224186510
  </code></pre>
  
  <p>As you can see by the timestamps it is creating a file about every 5 seconds or so.  This creates to many small files.</p>
  
  <p>I would like to be able to create the file on a larger time interval (90 seconds).  </p>