title: RMI performance too slow
body: <p>I have been working on a  project which is about data copy from one db to another. Written in java6. And working on distributed machines. We have 9 server machines. One is the main node which i have my main control module working on which gets copy requests and distributes the job to other 8 machines.</p>
  
  <p>It was started to be coded with JMS to distribute tasks to that 8 machine before.There was an Apache Active MQ server working on another computer. But i saw it was not much appropriate and the machines need to be more tightly coupled as it was causing some code overhead and it was necessary to return a response message for all the messages sent to slave machines. I decided to change the interconnection between main node and other 8 slave machines and coded that with RMI.</p>
  
  <p>I coded an RMI server for slave machines and client for main node machine. Then created threads on main node to trigger distributed-tasks at slave machines.</p>
  
  <p>The thing is performance decreased drastically. Normally i was able to copy about 6GB of data in about 6 mins from one specific db to another. Now it takes more than 1 and a half hour to copy 9GB of data. It used to consume lots of CPU when the task was executing on slave machines. I was  observing CPU usage ratio more than 90%. Now it never uses more than 15%. </p>
  
  <p>What i need is to understand what is causing this performance decrease. What should i do ? Should i use a troubleshooting tool ?</p>
  
  <p>Edit------------</p>
  
  <p>Ok i create a slave module instance on my laptop only, and sent 16 tasks to work on it and profiled CPU using jvisualvm.The results are in the pic <a href="http://i.imgur.com/jyRgd.png" rel="nofollow">CPU Profiler result</a>.</p>
  
  <p>The control methods in the package (e.g. failTaskIfAbort(), performSanityCheck(), etc) were also present when i was using JMS for machine communication. It makes me think, RMI threads are somehow low priority or something. </p>
  
  <p>I also uploaded the nps file exported from jvisualvm. You can get it here: <a href="http://dl.dropbox.com/u/17633663/profilersnapshot.nps" rel="nofollow">exported profile result</a></p>