title: REST API Response time increases under load [Tomcat]
body: <p>We have one REST API (GET) which can be called by lot of mobile users at the same time. Our current user base is around 300k but is expected to grow around 1 million.</p>
  
  <p>The API is simple. It makes 3 parallel requests using Akka, and returns the combined result. The main code looks like this:</p>
  
  <pre><code>        Future&lt;List&lt;CardDTO&gt;&gt; pnrFuture = null;
          Future&lt;List&lt;CardDTO&gt;&gt; newsFuture = null;
  
          ExecutionContext ec = ExecutionContexts.fromExecutorService(executor);
  
          final List&lt;CardDTO&gt; combinedDTOs = new ArrayList&lt;CardDTO&gt;();
  
          // Array list of futures
          List&lt;Future&lt;List&lt;CardDTO&gt;&gt;&gt; futures = new ArrayList&lt;Future&lt;List&lt;CardDTO&gt;&gt;&gt;();
  
          futures.add(future(new PNRFuture(pnrService, userId), ec));
          futures.add(future(new NewsFuture(newsService, userId), ec));
          futures.add(future(new SettingsFuture(userPreferenceManager, userId), ec));
  
          Future&lt;Iterable&lt;List&lt;CardDTO&gt;&gt;&gt; futuresSequence = sequence(futures, ec);
  
          // combine the cards
          Future&lt;List&lt;CardDTO&gt;&gt; futureSum =  futuresSequence.map(
                  new Mapper&lt;Iterable&lt;List&lt;CardDTO&gt;&gt;, List&lt;CardDTO&gt;&gt;() {
                      @Override
                      public List&lt;CardDTO&gt; apply(Iterable&lt;List&lt;CardDTO&gt;&gt; allDTOs) {
  
                          for (List&lt;CardDTO&gt; cardDTOs : allDTOs) {
  
                              if(cardDTOs!=null)
                                  combinedDTOs.addAll(cardDTOs);
  
                          }
  
                          Collections.sort(combinedDTOs);
  
                          return combinedDTOs;
                      }
                  }
          );
  
          Await.result(futureSum, Duration.Inf());
  
          return combinedDTOs;
  </code></pre>
  
  <p>The 3 futures are simple select statements from a MY SQL database which execute under a milliseconds. We are using Spring + Hibernate here.</p>
  
  <p>The whole API takes 50ms to return the result on an average.</p>
  
  <p>Now, while we were doing the performance testing with 3 servers, we came to conclusion that after around 200 requests / seconds, the response time of API starts increasing linearly. It goes as high as 3 -5 seconds under load. The surprising part is that CPU usage revolves around 20% at that time and there is nothing significant going on in JVM Memory. The memory usage is around 700 MB. We have 16 GB</p>
  
  <p>I am unable to find where the bottleneck is. How can I scale this API to at least 1000 requests / sec. I am at least looking for pointers on where to start. I have explored tools like <code>top</code> , <code>visualvm</code>, but do not find anything alarming.</p>
  
  <p>This our JVM settings on Java 7</p>
  
  <blockquote>
    <p>export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true -server -Xms4g
    -Xmx16g -XX:MaxPermSize=1g -XX:PermSize=512m -XX:MaxNewSize=4g -XX:NewSize=512m -XX:SurvivorRatio=16 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:MaxTenuringThreshold=0 -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSParallelRemarkEnabled -XX:+UseCMSInitiatingOccupancyOnly -XX:ParallelGCThreads=12 -XX:LargePageSizeInBytes=256m -Dspring.profiles.active=staging -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9899 -Djava.rmi.server.hostname=$HOSTNAME -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"</p>
  </blockquote>
  
  <p>I have read these questions and it seems that its a general trend. Would switching to some other framework like node.js or Erlang help?</p>
  
  <p><a href="https://stackoverflow.com/questions/31059089/response-time-increases-as-concurrency-increases-in-java">Response time Increases as concurrency increases in Java</a></p>
  
  <p><a href="https://stackoverflow.com/questions/26030851/tomcat-response-time-is-increasing-as-concurrency-is-increased-in-apache-bench">Tomcat response time is increasing as concurrency is increased in apache bench</a></p>