title: Slow multi-threaded java app: is this due to access to static object?
body: <p>FIRST, AN INTRO...</p>
  
  <p>I have a set of classes that inherit from the same class <code>Feature</code> but are different from each other in that each of them uses different information for its computation. So here's one example:</p>
  
  <pre><code>public class FeatureA extends Feature 
  {
      private MyTableA table = null;
  
      public FeatureA(final String fName, final MyTableA table) {
          super(fName);
          this.table = table;
      }
  
      public Double compute(String input) 
      {
          return table.computeProduct(String input);
      }
  }
  
  public class MyTableA {
  
      private static  HashMap&lt;String,Integer&gt; map1 = null;    
      private static  HashMap&lt;String,Double&gt; map2 = null;
  
      public MyTableA(final String serFilePath)
      {
          // map1 and map2 are deserizlied here from serFilePath
      }
  
      public Double computeProduct(String input)
      {
          Integer val1 = map1.get(input);
          Double val2 = map2.get(input);
  
          Double res = val1 * val2;
  
          return res;
      }
  </code></pre>
  
  <p>So <code>FeatureA</code> depends on an object <code>MyTableA</code> and that object is deserialized when my app is loaded.</p>
  
  <p>This app is multithreaded, with separate threads processing different input data tokens in the same way. For each token, I compute feature values, including <code>FeatureA</code>. <code>MyTableA</code> is the same for all data tokens (as a trivial example, a feature could be someone's phone number looked up by their name in a single global phone book.</p>
  
  <p>NOW, THE PROBLEM...</p>
  
  <p>I have been adding more and more features to my system and each new one added a tiny bit of extra processing time. I added one more and allof a sudden my program runs a lot slower. Like 10 times slower than a version without it. I've been trying to pinpoint the source of the problem without much luck.</p>
  
  <p>QUESTION: is it possible that the slowdown is caused by multiple threads trying to <code>map1.get(input)</code> and <code>map2.get(input)</code> from the same static objects? If so, how could I fix this?</p>
  
  <p><strong>UPDATE</strong>: The app has a server and client components with the multithreaded server getting data tokens from multiple copies of the client. In fact, there are 3 RedHat machines that are being used: 1 copy of the server and about 100 copies of the client run on each of the machines. Clients from all 3 machines send data to all 3 servers (so clients on machine1 do not just send data to server1). I have used the profiler like this: <code>java -agentlib:hprof=cpu=samples ...</code> Is this the right way to profile in this case?</p>
  
  <p>Below are the top 5 traces for the "Fast" case, i.e. with fewer features, and the "Slow" case, i.e. with more features. <code>java.io.FileInputStream.read0</code> is part of deserialization of a bunch of maps at start.</p>
  
  <pre><code>Server1_Fast:rank   self  accum   count trace method
  Server1_Fast-   1 73.21% 73.21%  405018 300953 java.net.PlainSocketImpl.socketAccept
  Server1_Fast-   2  5.63% 78.84%   31145 300698 java.io.FileInputStream.read0
  Server1_Fast-   3  4.33% 83.16%   23928 301032 java.lang.UNIXProcess.waitForProcessExit
  Server1_Fast-   4  4.28% 87.45%   23685 301031 java.io.FileInputStream.readBytes
  Server1_Fast-   5  4.08% 91.53%   22570 301001 java.net.SocketInputStream.socketRead0
  --
  Server1_Slow:rank   self  accum   count trace method
  Server1_Slow-   1 43.66% 43.66%  374607 301136 java.lang.UNIXProcess.forkAndExec
  Server1_Slow-   2 23.38% 67.04%  200653 301130 java.io.FileInputStream.readBytes
  Server1_Slow-   3  9.74% 76.78%   83571 301058 java.net.PlainSocketImpl.socketAccept
  Server1_Slow-   4  7.44% 84.23%   63876 301131 java.lang.UNIXProcess.waitForProcessExit
  Server1_Slow-   5  3.70% 87.92%   31711 300690 java.io.FileInputStream.read0
  --
  Server2_Fast:rank   self  accum   count trace method
  Server2_Fast-   1 76.35% 76.35%  427397 300917 java.net.PlainSocketImpl.socketAccept
  Server2_Fast-   2  5.21% 81.57%   29183 300690 java.io.FileInputStream.read0
  Server2_Fast-   3  4.23% 85.80%   23689 300965 java.net.SocketInputStream.socketRead0
  Server2_Fast-   4  4.12% 89.92%   23083 300691 java.io.FileInputStream.readBytes
  Server2_Fast-   5  3.09% 93.02%   17320 300993 java.lang.UNIXProcess.waitForProcessExit
  --
  Server2_Slow:rank   self  accum   count trace method
  Server2_Slow-   1 50.19% 50.19%  173210 301024 java.net.PlainSocketImpl.socketAccept
  Server2_Slow-   2  9.10% 59.28%   31391 300686 java.io.FileInputStream.read0
  Server2_Slow-   3  6.81% 66.09%   23507 300687 java.io.FileInputStream.readBytes
  Server2_Slow-   4  5.44% 71.54%   18789 301094 java.lang.UNIXProcess.waitForProcessExit
  Server2_Slow-   5  5.38% 76.92%   18571 301093 java.io.FileInputStream.readBytes
  --
  Server3_Fast:rank   self  accum   count trace method
  Server3_Fast-   1 73.38% 73.38%  410860 300954 java.net.PlainSocketImpl.socketAccept
  Server3_Fast-   2  6.81% 80.20%   38134 300692 java.io.FileInputStream.read0
  Server3_Fast-   3  4.95% 85.14%   27700 300693 java.io.FileInputStream.readBytes
  Server3_Fast-   4  3.76% 88.91%   21071 301038 java.lang.UNIXProcess.waitForProcessExit
  Server3_Fast-   5  3.75% 92.65%   20974 301037 java.io.FileInputStream.readBytes
  --
  Server3_Slow:rank   self  accum   count trace method
  Server3_Slow-   1 48.32% 48.32%  166867 301048 java.net.PlainSocketImpl.socketAccept
  Server3_Slow-   2 10.62% 58.94%   36686 300693 java.io.FileInputStream.read0
  Server3_Slow-   3  8.19% 67.13%   28280 300690 java.io.FileInputStream.readBytes
  Server3_Slow-   4  5.22% 72.35%   18022 301119 java.lang.UNIXProcess.waitForProcessExit
  Server3_Slow-   5  5.06% 77.41%   17464 301118 java.io.FileInputStream.readBytes
  </code></pre>
  
  <p>Does this info shed any light on the problem?</p>