title: logstash - splitting an event into two, based on an included array
body: <p>I'm working with logstash for the first time, and I'm trying to take JSON reports from amavisd-new in for searching and analysis. Amavisd-new is able to write the json logging to redis, and I have everything importing perfectly, and have started learning my way through all this.</p>
  
  <p>But I have one issue - the format of the JSON report from amavis looks like the following - note that "recipients" has an array, with one entry for each recipient.</p>
  
  <p>I'd like to split the entire event into two - one for each recipient, leaving all the other fields the same, but replacing the "action", "ccat_main", "queued_as", etc fields from each recipient array member into the main even.</p>
  
  <p>The idea would be that one incoming event with two recipients would result in two separate log events in logstash - one for each person.</p>
  
  <p>I've looked at split for events, but I'm not seeing how to do this - I can't seem to find any appropriate examples anywhere.</p>
  
  <p>So, for real-word examples, given this:</p>
  
  <pre><code> {
      "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
      "time_unix" =&gt; 1399368587.048,
      "time_iso_week_date" =&gt; "2014-W19-2",
      "partition" =&gt; "19",
      "type" =&gt; "amavis",
      "host" =&gt; "mailer.example.net",
      "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
      "recipients" =&gt; [
        { "action" =&gt; "PASS",
          "ccat_main" =&gt; "Clean",
          "queued_as" =&gt; "3gNFyR4Mfjzc3",
          "rcpt_is_local" =&gt; false,
          "rcpt_to" =&gt; "recip2@example.org",
          "smtp_code" =&gt; "250",
          "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
          "spam_score" =&gt; -2.0
        },
        { "action" =&gt; "PASS",
          "ccat_main" =&gt; "Clean",
          "mail_id_related" =&gt; "men7HTERZaOF",
          "penpals_age" =&gt; 1114599,
          "queued_as" =&gt; "3gNFyR4n6Lzc4",
          "rcpt_is_local" =&gt; true,
          "rcpt_to" =&gt; "recip1@example.net",
          "smtp_code" =&gt; "250",
          "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
          "spam_score" =&gt; -5.272
        }
      ],
      "smtp_code"  =&gt; ["250"],
    }
  </code></pre>
  
  <p>I'd like to end up with two different events, like these:</p>
  
  <pre><code>  {
      "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
      "time_unix" =&gt; 1399368587.048,
      "time_iso_week_date" =&gt; "2014-W19-2",
      "partition" =&gt; "19",
      "type" =&gt; "amavis",
      "host" =&gt; "mailer.example.net",
      "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
      "action" =&gt; "PASS",
      "ccat_main" =&gt; "Clean",
      "queued_as" =&gt; "3gNFyR4Mfjzc3",
      "rcpt_is_local" =&gt; false,
      "rcpt_to" =&gt; "recip2@example.org",
      "smtp_code" =&gt; "250",
      "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
      "spam_score" =&gt; -2.0
      "smtp_code"  =&gt; ["250"],
    }
  </code></pre>
  
  <p>and</p>
  
  <pre><code>  {
      "@timestamp" =&gt; "2014-05-06T09:29:47.048Z",
      "time_unix" =&gt; 1399368587.048,
      "time_iso_week_date" =&gt; "2014-W19-2",
      "partition" =&gt; "19",
      "type" =&gt; "amavis",
      "host" =&gt; "mailer.example.net",
      "queued_as" =&gt; ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
      "recipients" =&gt; [
      "action" =&gt; "PASS",
      "ccat_main" =&gt; "Clean",
      "mail_id_related" =&gt; "men7HTERZaOF",
      "penpals_age" =&gt; 1114599,
      "queued_as" =&gt; "3gNFyR4n6Lzc4",
      "rcpt_is_local" =&gt; true,
      "rcpt_to" =&gt; "recip1@example.net",
      "smtp_code" =&gt; "250",
      "smtp_response" =&gt; "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
      "spam_score" =&gt; -5.272
      "smtp_code"  =&gt; ["250"],
    }
  </code></pre>
  
  <p><strong>EDIT:</strong></p>
  
  <p>Okay, I simply used the split filter - I should have seen that.  But there's one thing that's confusing me.</p>
  
  <p>When there's a single recipient, it passes the block right through - the result in kibana looks like:</p>
  
  <pre><code>recipients      {
    "action": "PASS",
    "bypass_banned_checks": true,
    "bypass_spam_checks": true,
    "ccat_main": "Clean",
    "queued_as": "3qv7Km4Ybpz14Kyh",
    "rcpt_is_local": true,
    "rcpt_to": "user@domain.com",
    "rid": "552213780",
    "smtp_code": "250",
    "smtp_response": "250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Km4Ybpz14Kyh"
  }
  </code></pre>
  
  <p>But when there are 2 or more recipients, the new events each look like this, with their appropriate info:</p>
  
  <pre><code>recipients.action       PASS
  recipients.ccat_main        CleanTag
  recipients.queued_as        3qv7Ly4Pqvz4wyS
  recipients.rcpt_is_local        true
  recipients.rcpt_to      user@domain.com
  recipients.rid      552278239
  recipients.smtp_code        250
  recipients.smtp_response        250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Ly4Pqvz4wyS
  recipients.whitelisted      true
  </code></pre>
  
  <p>Why the difference between the two?  I think I'd prefer to keep the recipients field as a hash of values, so what's the best way to get make the split events consistent with the single event?</p>