title: How to insert data into druid via tranquility
body: <p>By following tutorial at <a href="http://druid.io/docs/latest/tutorials/tutorial-loading-streaming-data.html" rel="nofollow noreferrer">http://druid.io/docs/latest/tutorials/tutorial-loading-streaming-data.html</a> , I was able to insert data into druid via Kafka console</p>
  
  <h2>Kafka console</h2>
  
  <p>The spec file looks as following</p>
  
  <p><strong>examples/indexing/wikipedia.spec</strong></p>
  
  <pre><code>[
    {
      "dataSchema" : {
        "dataSource" : "wikipedia",
        "parser" : {
          "type" : "string",
          "parseSpec" : {
            "format" : "json",
            "timestampSpec" : {
              "column" : "timestamp",
              "format" : "auto"
            },
            "dimensionsSpec" : {
              "dimensions": ["page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city"],
              "dimensionExclusions" : [],
              "spatialDimensions" : []
            }
          }
        },
        "metricsSpec" : [{
          "type" : "count",
          "name" : "count"
        }, {
          "type" : "doubleSum",
          "name" : "added",
          "fieldName" : "added"
        }, {
          "type" : "doubleSum",
          "name" : "deleted",
          "fieldName" : "deleted"
        }, {
          "type" : "doubleSum",
          "name" : "delta",
          "fieldName" : "delta"
        }],
        "granularitySpec" : {
          "type" : "uniform",
          "segmentGranularity" : "DAY",
          "queryGranularity" : "NONE"
        }
      },
      "ioConfig" : {
        "type" : "realtime",
        "firehose": {
          "type": "kafka-0.8",
          "consumerProps": {
            "zookeeper.connect": "localhost:2181",
            "zookeeper.connection.timeout.ms" : "15000",
            "zookeeper.session.timeout.ms" : "15000",
            "zookeeper.sync.time.ms" : "5000",
            "group.id": "druid-example",
            "fetch.message.max.bytes" : "1048586",
            "auto.offset.reset": "largest",
            "auto.commit.enable": "false"
          },
          "feed": "wikipedia"
        },
        "plumber": {
          "type": "realtime"
        }
      },
      "tuningConfig": {
        "type" : "realtime",
        "maxRowsInMemory": 500000,
        "intermediatePersistPeriod": "PT10m",
        "windowPeriod": "PT10m",
        "basePersistDirectory": "\/tmp\/realtime\/basePersist",
        "rejectionPolicy": {
          "type": "messageTime"
        }
      }
    }
  ]
  </code></pre>
  
  <p>I start <strong>realtime</strong> via</p>
  
  <pre><code>java -Xmx512m -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Ddruid.realtime.specFile=examples/indexing/wikipedia.spec -classpath config/_common:config/realtime:lib/* io.druid.cli.Main server realtime
  </code></pre>
  
  <p>In Kafka console, I paste and enter the following</p>
  
  <pre><code>{"timestamp": "2013-08-10T01:02:33Z", "page": "Good Bye", "language" : "en", "user" : "catty", "unpatrolled" : "true", "newPage" : "true", "robot": "false", "anonymous": "false", "namespace":"article", "continent":"North America", "country":"United States", "region":"Bay Area", "city":"San Francisco", "added": 57, "deleted": 200, "delta": -143}
  </code></pre>
  
  <p>Then I tend to perform query by creating <code>select.json</code> and run <code>curl -X POST 'http://localhost:8084/druid/v2/?pretty' -H 'content-type: application/json'  -d @select.json</code></p>
  
  <p><strong>select.json</strong></p>
  
  <pre><code> {
     "queryType": "select",
     "dataSource": "wikipedia",
     "dimensions":[],
     "metrics":[],
     "granularity": "all",
     "intervals": [
       "2000-01-01/2020-01-02"
     ],
  
     "filter" : {"type":"and",
          "fields" : [
                  { "type": "selector", "dimension": "user", "value": "catty" }
          ]
     },
  
     "pagingSpec":{"pagingIdentifiers": {}, "threshold":500}
   }
  </code></pre>
  
  <p>I was able to get the following result.</p>
  
  <pre><code>[ {
    "timestamp" : "2013-08-10T01:02:33.000Z",
    "result" : {
      "pagingIdentifiers" : {
        "wikipedia_2013-08-10T00:00:00.000Z_2013-08-11T00:00:00.000Z_2013-08-10T00:00:00.000Z" : 0
      },
      "events" : [ {
        "segmentId" : "wikipedia_2013-08-10T00:00:00.000Z_2013-08-11T00:00:00.000Z_2013-08-10T00:00:00.000Z",
        "offset" : 0,
        "event" : {
          "timestamp" : "2013-08-10T01:02:33.000Z",
          "continent" : "North America",
          "robot" : "false",
          "country" : "United States",
          "city" : "San Francisco",
          "newPage" : "true",
          "unpatrolled" : "true",
          "namespace" : "article",
          "anonymous" : "false",
          "language" : "en",
          "page" : "Good Bye",
          "region" : "Bay Area",
          "user" : "catty",
          "deleted" : 200.0,
          "added" : 57.0,
          "count" : 1,
          "delta" : -143.0
        }
      } ]
    }
  } ]
  </code></pre>
  
  <p>It seem that I had setup Druid correctly.</p>
  
  <p>Now, I would like to insert data via HTTP endpoint. According to <a href="https://stackoverflow.com/questions/31630369/how-realtime-data-input-to-druid">How realtime data input to Druid?</a>, it seems like recommended way is to use <code>tranquility</code></p>
  
  <h2>tranquility</h2>
  
  <p>I have indexing service started via</p>
  
  <pre><code>java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath config/_common:config/overlord:lib/*: io.druid.cli.Main server overlord
  </code></pre>
  
  <p><strong>conf/server.json</strong> looks like</p>
  
  <pre><code>{
     "dataSources" : [
        {
           "spec" : {
              "dataSchema" : {
                  "dataSource" : "wikipedia",
                  "parser" : {
                      "type" : "string",
                      "parseSpec" : {
                        "format" : "json",
                        "timestampSpec" : {
                          "column" : "timestamp",
                          "format" : "auto"
                        },
                        "dimensionsSpec" : {
                          "dimensions": ["page","language","user","unpatrolled","newPage","robot","anonymous","namespace","continent","country","region","city"],
                          "dimensionExclusions" : [],
                          "spatialDimensions" : []
                        }
                      }
                  },
                  "metricsSpec" : [{
                      "type" : "count",
                      "name" : "count"
                  }, {
                      "type" : "doubleSum",
                      "name" : "added",
                      "fieldName" : "added"
                  }, {
                      "type" : "doubleSum",
                      "name" : "deleted",
                      "fieldName" : "deleted"
                  }, {
                      "type" : "doubleSum",
                      "name" : "delta",
                      "fieldName" : "delta"
                  }],
                  "granularitySpec" : {
                      "type" : "uniform",
                      "segmentGranularity" : "DAY",
                      "queryGranularity" : "NONE"
                  }
              },
              "tuningConfig" : {
                 "windowPeriod" : "PT10M",
                 "type" : "realtime",
                 "intermediatePersistPeriod" : "PT10M",
                 "maxRowsInMemory" : "100000"
              }
           },
           "properties" : {
              "task.partitions" : "1",
              "task.replicants" : "1"
           }
        }
     ],
     "properties" : {
        "zookeeper.connect" : "localhost",
        "http.port" : "8200",
        "http.threads" : "8"
     }
  }
  </code></pre>
  
  <p>Then, I start the server using</p>
  
  <pre><code>bin/tranquility server -configFile conf/server.json
  </code></pre>
  
  <p>I perform post to <a href="http://xx.xxx.xxx.xxx:8200/v1/post/wikipedia" rel="nofollow noreferrer">http://xx.xxx.xxx.xxx:8200/v1/post/wikipedia</a>, with <code>content-type</code> equals <code>application/json</code></p>
  
  <pre><code>{"timestamp": "2013-08-10T01:02:33Z", "page": "Selamat Pagi", "language" : "en", "user" : "catty", "unpatrolled" : "true", "newPage" : "true", "robot": "false", "anonymous": "false", "namespace":"article", "continent":"North America", "country":"United States", "region":"Bay Area", "city":"San Francisco", "added": 57, "deleted": 200, "delta": -143}
  </code></pre>
  
  <p>I get the the following respond</p>
  
  <pre><code>{"result":{"received":1,"sent":0}}
  </code></pre>
  
  <p><strong><em>It seems that tranquility has received our data, but failed to send it to druid!</em></strong></p>
  
  <p>I try to run <code>curl -X POST 'http://localhost:8084/druid/v2/?pretty' -H 'content-type: application/json'  -d @select.json</code>, but doesn't get the output I inserted via tranquility.</p>
  
  <p>Any idea why? Thanks.</p>