title: Understanding response time costs
body: <p>I'm new in Golang and trying to write simple web-server with json response. One of endpoints is for database access. With static-like response (such as any static string or simple data conversion) average response time is about 0s (&lt;1ms). If I add some database request, average time increases up to 250ms and not depending on function content. So I'm trying to understand the timing of this function.</p>
  
  <p>What I've already know:</p>
  
  <ol>
  <li>Query to db costs about 40-50ms</li>
  <li>Datetime transformation costs >1ms</li>
  <li><code>json.Marshal</code> costs &lt;1ms</li>
  <li><code>string(json.Marshal)</code> &lt;1ms</li>
  </ol>
  
  <p>All my problem is in <code>rows.Next()</code>:</p>
  
  <ol start="5">
  <li>all inside <code>rows.Next()</code> costs &lt;1ms separately</li>
  <li>Each <code>row.Next()</code> procedure cost depends on row length and it is about 10-50ms. </li>
  </ol>
  
  <p>If I understand <code>db.Query()</code> gets all rows in one request and keeps it in memory providing virtual cursor as iterator ( <code>.Next()</code>). Why does it cost so much?</p>
  
  <ol start="7">
  <li>All <code>row.Next()</code> data processing costs about 180ms. With 40ms <code>db.Query</code> together it approx equals to total response time (~240ms)</li>
  <li>If I comment all <code>row.Next()</code> cycle and just make <code>db.Query</code> total response still costs 200-240ms</li>
  </ol>
  
  <p>Packages I use:</p>
  
  <pre><code>import (
      "database/sql"
      "encoding/json"
      "fmt"
      "log"
      "os"
      "strconv"
      "time"
      "github.com/kshvakov/clickhouse"
      routing "github.com/qiangxue/fasthttp-routing"
      "github.com/valyala/fasthttp"
  )
  </code></pre>
  
  <pre class="lang-golang prettyprint-override"><code>func getRecs(client string, dt string) string {
      dtT := transformDate(&amp;dt)
      rows, err := db.Query(getRecsRequest(&amp;client, &amp;dtT))
  
      if err != nil {
          log.Fatal(err)
      }
      defer rows.Close()
  
      start := time.Now()
  
      got := []databaseRows{}
      elapsed := time.Since(start)
      for rows.Next() {
          elapsed = time.Since(start)
          log.Printf("Next took %s", elapsed)
          var r databaseRows
          err := rows.Scan(&amp;r.ItemIds, &amp;r.DtRecommendation, &amp;r.Number)
          if err != nil {
              log.Fatal(err)
          }
          got = append(got, r)
          start = time.Now()
      }
  
      jsonData, err := json.Marshal(&amp;got)
      return string(jsonData)
  }
  </code></pre>
  
  <pre class="lang-golang prettyprint-override"><code>func getRecs(client string, dt string) string {
      dtT := transformDate(&amp;dt)
      rows, err := db.Query(getRecsRequest(&amp;client, &amp;dtT))
  
      if err != nil {
          log.Fatal(err)
      }
      defer rows.Close()
  
      // start := time.Now()
  
      got := "[]databaseRows{}"
      // elapsed := time.Since(start)
      // for rows.Next() {
      //  elapsed = time.Since(start)
      //  log.Printf("Next took %s", elapsed)
      //  var r databaseRows
      //  err := rows.Scan(&amp;r.ItemIds, &amp;r.DtRecommendation, &amp;r.Number)
      //  if err != nil {
      //      log.Fatal(err)
      //  }
      //  got = append(got, r)
      //  start = time.Now()
      // }
  
      jsonData, err := json.Marshal(&amp;got)
      return string(jsonData)
  }
  </code></pre>
  
  <p>Can I expect web-server response time is about database request time?</p>
  
  <p>UPDATE. </p>
  
  <p>I've tried "github.com/jmoiron/sqlx" with "github.com/kshvakov/clickhouse" and result was the same; only syntax changed.</p>
  
  <p>I also tried "github.com/mailru/go-clickhouse" with "database/sql". It provides access to HTTP interface to clickhouse. I got all rows in one requets as expected, but it tooks about 240ms for the http request and about 0ms for data processing.. So I am looking for opportunity to keep tcp connection opened to be able get all nessasery rows in one request and work with my program side memory buffer (with some buffer limit in combnation with tcp/socket buffer). </p>
  
  <p>UPDATE 2.</p>
  
  <p>Ok, I've tried native clickhouse client (and other such DBeaver) and approx 250ms is the limit for this request. But it may be only clickhouse native tcp interface limitation.
  But what is the 40-50ms db.Query() operation, which returns no rows. Is the time for query on server side? Or just query string check?</p>