title: HHVM fastcgi + Nginx performance fluctuations
body: <p>Currently we started using HHVM in a production envirionment and so far almost all results are pretty inpressive. Our overall transaction rate is greatly improved compared to PHP-FPM with APC. Almost all requests are under 500ms however every couple of requests (5 to 10 or so) results in a request time of 2 to even 5 seconds.</p>
  
  <p>The page requested does not seem to make any difference and also requesting the same page over and over again will trigger this behavior within a couple of requests.</p>
  
  <p>We are running HHVM in server mode with the following command line options:</p>
  
  <pre><code>/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true
  </code></pre>
  
  <p>We are running nginx for the webserver with these relevant configurations (I am sorry if i forgot something importand here).</p>
  
  <pre><code>fastcgi_buffer_size 128k;
  fastcgi_buffers 256 16k;
  fastcgi_busy_buffers_size 256k;
  fastcgi_temp_file_write_size 256k;
  fastcgi_read_timeout 240;
  fastcgi_intercept_errors on;
  </code></pre>
  
  <p>The server has 128GB of memory and 24 cores (hyperthreading so actually 12).</p>
  
  <p>We did a fair bit of searching on <a href="https://github.com/facebook/hhvm/wiki/Runtime-options" rel="nofollow">https://github.com/facebook/hhvm/wiki/Runtime-options</a> however most options aren't explained very well so I have no idea what they do and testing them in a production environment is a bit scary.</p>
  
  <p>Had anybody here had a similar problem or could maybe point me in a direction with some of the HHVM options I would be very grateful.</p>
  
  <p>The HHVM version used is from <a href="http://www.hop5.in/yum/el6/" rel="nofollow">http://www.hop5.in/yum/el6/</a></p>
  
  <pre><code>HipHop VM 3.0.1 (rel)
  Compiler: 
  Repo schema: e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
  </code></pre>
  
  <p>And the the /etc/hhvm/config.hdf</p>
  
  <pre><code>Log {
    Level = Warning
    AlwaysLogUnhandledExceptions = true
    RuntimeErrorReportingLevel = 8191
  }
  
  MySQL {
    TypedResults = false
  }
  </code></pre>
  
  <p>We are using supervisord to start HHVM so here the supervisor config as wel:</p>
  
  <pre><code>[program:hhvm]
  stopasgroup=true
  killasgroup=true
  command=/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true 
  user=admin
  stdout_logfile=/var/log/hhvm/admin.log
  stderr_logfile=/var/log/hhvm/admin.error.log
  directory=/home/admin
  umask=000
  </code></pre>
  
  <p>There is a php.ini in /etc/hhvm/php.ini but the content is empty. Also the pages tried all do some database connectivity but this is usually very minimal. For a complete picture also the my.cnf. The mysql version used is percona</p>
  
  <pre><code>[mysql]
  
  # CLIENT #
  port                           = 3306
  socket                         = /var/lib/mysql/mysql.sock
  
  [mysqld]
  
  # GENERAL #
  user                           = mysql
  default-storage-engine         = InnoDB
  socket                         = /var/lib/mysql/mysql.sock
  pid-file                       = /var/lib/mysql/mysql.pid
  
  [mysqld]
  
  # MyISAM #
  key-buffer-size                = 32M
  myisam-recover                 = FORCE,BACKUP
  
  # SAFETY #
  max-allowed-packet             = 128M
  max-connect-errors             = 1000000
  
  # DATA STORAGE #
  datadir                        = /var/lib/mysql/
  
  # BINARY LOGGING #
  log-bin                        = /var/lib/mysql/mysql-bin
  expire-logs-days               = 14
  sync-binlog                    = 1
  
  # CACHES AND LIMITS #
  tmp-table-size                 = 128M
  max-heap-table-size            = 256M
  query-cache-size               = 10G
  max-connections                = 1000
  thread-cache-size              = 100
  open-files-limit               = 65535
  table-definition-cache         = 4096
  table-open-cache               = 4000
  join-buffer-size               = 1M
  
  # INNODB #
  innodb-flush-method            = O_DIRECT
  innodb-log-files-in-group      = 2
  innodb-log-file-size           = 512M
  innodb-flush-log-at-trx-commit = 1
  innodb-file-per-table          = 1
  innodb-buffer-pool-size        = 73G
  
  # LOGGING #
  log-error                      = /var/lib/mysql/mysql-error.log
  log-queries-not-using-indexes  = 1
  slow-query-log                 = 1
  slow-query-log-file            = /var/lib/mysql/mysql-slow.log
  </code></pre>
  
  <p>Mysql version:</p>
  
  <pre><code>innodb_version  5.6.17-65.0
  protocol_version    10
  slave_type_conversions  
  version 5.6.17-65.0-56-log
  version_comment Percona Server (GPL), Release 65.0, Revision 587
  version_compile_machine x86_64
  version_compile_os  Linux
  </code></pre>