title: R and makePSOCKcluter EC2 socketConnection
body: <p>I would like to be able to connect my Mac to my EC2 instances to carry out parallel processing on AWS via the <code>parallel</code> package using <code>makePSOCKcluster</code> or <code>makeSOCKCluster</code>. </p>
  
  <p>At the moment my attempt leaves R 'hanging' so I have adapted the the <code>makePSOCKcluster</code>, and some of its sub-routines such that some of its outputs can be seen by adding a <code>-v</code> option to the ssh. I think I have managed to deal with the password-less ssh login, but am getting stuck at the the <code>socketConnection</code> part which I think is causing problems.</p>
  
  <p>I have tried associating Elastic IPs and using those as the IP addresses to no avail, have also tried adjusting the security groups so that it includes the default port that makePSOCKcluster uses to no avail either...In the latter case, I did not use the ports argument and used the default port option of 10187 to which it says:</p>
  
  <pre><code>Error in socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  : 
    cannot open the connection
  In addition: Warning message:
  In socketConnection("localhost", port = port, server = TRUE, blocking = TRUE,  :
    port 10187 cannot be opened
  </code></pre>
  
  <p>Looking on a few hpc mailing list questions this seemed to be an issue related to being windows...but given I am connecting from a Mac I don't think I fall under this category.</p>
  
  <p>The <code>hosts</code> object is just the public DNS that is provided after launching an EC2 instance.</p>
  
  <p>The below is my current attempt having adapted some of the <code>makePSOCKcluster</code> function to <code>makePSOCKcluster1</code> they should be roughly the same.</p>
  
  <p>I have specified the <code>rscript</code> parameter to match that which would be expected on an ubuntu instance, and I have specified the <code>ubuntu</code> as the username to login as on the ubuntu EC2 instance.</p>
  
  <pre><code>makePSOCKcluster1 &lt;- function (names, ...) {
      if (is.numeric(names)) 
          names &lt;- rep("localhost", names[1])
      options &lt;- parallel:::addClusterOptions(parallel:::defaultClusterOptions, list(...))
      cl &lt;- vector("list", length(names))
      for (i in seq_along(cl)) cl[[i]] &lt;- newPSOCKnode1(names[[i]], 
          options = options, rank = i)
      class(cl) &lt;- c("SOCKcluster", "cluster")
      cl
  }
  
  
  
  newPSOCKnode1 &lt;- function (machine = "localhost", ..., options = parallel:::defaultClusterOptions, 
                             rank) 
  {
      options &lt;- options
      if (is.list(machine)) {
          options &lt;- options
          machine &lt;- machine$host
      }
      outfile &lt;- parallel:::getClusterOption("outfile", options)
      master &lt;- if (machine == "localhost") 
          "localhost"
      else parallel:::getClusterOption("master", options)
      port &lt;- parallel:::getClusterOption("port", options)
      manual &lt;- parallel:::getClusterOption("manual", options)
      timeout &lt;- parallel:::getClusterOption("timeout", options)
      methods &lt;- parallel:::getClusterOption("methods", options)
      useXDR &lt;- parallel:::getClusterOption("useXDR", options)
      env &lt;- paste("MASTER=", master, " PORT=", port, " OUT=", 
                   outfile, " TIMEOUT=", timeout, " METHODS=", methods, 
                   " XDR=", useXDR, sep = "")
      arg &lt;- "parallel:::.slaveRSOCK()"
      rscript &lt;- if (parallel:::getClusterOption("homogeneous", options)) {
          shQuote(parallel:::getClusterOption("rscript", options))
      }
      else "Rscript"
      cmd &lt;- paste(rscript, "-e", shQuote(arg), env)
      renice &lt;- parallel:::getClusterOption("renice", options)
      if (!is.na(renice) &amp;&amp; renice) 
          cmd &lt;- sprintf("nice +%d %s", as.integer(renice), cmd)
      if (manual) {
          cat("Manually start worker on", machine, "with\n    ", 
              cmd, "\n")
          flush.console()
      }
      else {
          if (machine != "localhost") {
              rshcmd &lt;- parallel:::getClusterOption("rshcmd", options)
              user &lt;- parallel:::getClusterOption("user", options)
              cmd &lt;- shQuote(cmd)
              cmd &lt;- paste(rshcmd, "-v -l", user, machine, cmd)
              print(cmd)
          }
          if (.Platform$OS.type == "windows") {
              system(cmd, wait = FALSE, input = "")
          }
          else system(cmd, wait = FALSE)
      }
      print("ssh done!!! about to start socketConnection....")
      con &lt;- socketConnection("localhost", port = port, server = TRUE, 
                              blocking = TRUE, open = "a+b", timeout = timeout)
      print("socketConnection complete!!!")
      structure(list(con = con, host = machine, rank = rank), class = if (useXDR) 
          "SOCKnode"
                else "SOCK0node")
  }
  
  
  
   hosts &lt;- c("ec2-xxx-xx-xxx-xxxx.zone.compute.amazonaws.com","ec2-xx-xxx-xxx-xxx.zone.compute.amazonaws.com")
   # the code to try and connect to the actual EC2 instance...
   cl1 &lt;- makePSOCKcluster1(hosts, user="ubuntu", rscript="/usr/lib/R/bin/Rscript", port=8787)
  
  
  
  [1] "ssh -v -l ubuntu ec2-xxxxxxxxxxx.zone.compute.amazonaws.com \"'/usr/lib/R/bin/Rscript' -e 'parallel:::.slaveRSOCK()' MASTER=local.machine.name PORT=8787 OUT=/dev/null TIMEOUT=2592000 METHODS=TRUE XDR=TRUE\""
  [1] "ssh done!!! about to start socketConnection...."
  OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
  debug1: Reading configuration data /etc/ssh_config
  debug1: Connecting to ec2-xxxxxxxxxxx.zone.compute.amazonaws.com [xx.xxx.xx.x.x] port 22.
  debug1: Connection established.
  debug1: identity file /Users/username/.ssh/identity type -1
  debug1: identity file /Users/username/.ssh/id_rsa type 1
  debug1: identity file /Users/username/.ssh/id_dsa type 2
  debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
  debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
  debug1: Enabling compatibility mode for protocol 2.0
  debug1: Local version string SSH-2.0-OpenSSH_5.2
  debug1: SSH2_MSG_KEXINIT sent
  debug1: SSH2_MSG_KEXINIT received
  debug1: kex: server-&gt;client aes128-ctr hmac-md5 none
  debug1: kex: client-&gt;server aes128-ctr hmac-md5 none
  debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024&lt;1024&lt;8192) sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
  debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
  debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
  Warning: Permanently added 'ec2-xx-xx-xxx-xxx-xx.ap-southeast-1.compute.amazonaws.com,xx.xxx.xxx.xx.x' (RSA) to the list of known hosts.
  debug1: ssh_rsa_verify: signature correct
  debug1: SSH2_MSG_NEWKEYS sent
  debug1: expecting SSH2_MSG_NEWKEYS
  debug1: SSH2_MSG_NEWKEYS received
  debug1: SSH2_MSG_SERVICE_REQUEST sent
  debug1: SSH2_MSG_SERVICE_ACCEPT received
  debug1: Authentications that can continue: publickey
  debug1: Next authentication method: publickey
  debug1: Offering public key: /Users/username/.ssh/id_rsa
  debug1: Server accepts key: pkalg ssh-rsa blen 277
  debug1: Authentication succeeded (publickey).
  debug1: channel 0: new [client-session]
  debug1: Requesting no-more-sessions@openssh.com
  debug1: Entering interactive session.
  debug1: Sending command: '/usr/lib/R/bin/Rscript' -e 'parallel:::.slaveRSOCK()' MASTER=local.machine.name PORT=8787 OUT=/dev/null TIMEOUT=2592000 METHODS=TRUE XDR=TRUE
  debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
  debug1: channel 0: free: client-session, nchannels 1
  debug1: fd 0 clearing O_NONBLOCK
  debug1: fd 1 clearing O_NONBLOCK
  debug1: fd 2 clearing O_NONBLOCK
  Transferred: sent 2352, received 2400 bytes, in 20.0 seconds
  Bytes per second: sent 117.5, received 119.9
  debug1: Exit status 1
  </code></pre>
  
  <p>I am using a Mac on OS X 10.6.8 to connect to a Debian Ubunutu instance. If there are better ways to connect to the EC2 instances to carry out parallel processing that people know of that would also be extremely useful.</p>
  
  <p>The end goal is to use <code>foreach</code> to carry out this processing once the cluster has been registered.</p>
  
  <p>Also as a side question I was wondering, what are the speed/processing pros and cons associated with carrying out a process in parallel vs using MPI? or some other method?</p>
  
  <p>Thanks in advance!</p>
  
  <p><strong>EDIT</strong> I have managed to get the <code>makePSOCKcluster</code> to work if starting from a separate EC2 instance, and the <code>parLapply</code> function works, and I can even register using <code>registerDoParallel(cl1)</code> where <code>cl1</code> is the cluster object, but for some reason <code>foreach</code>... <code>%dopar%</code> does not work...giving the error:</p>
  
  <pre><code>Error in serialize(data, node$con) : error writing to connection
  </code></pre>
  
  <p>or</p>
  
  <pre><code>Error in unserialize(node$con) : error reading from connection
  </code></pre>
  
  <p>the connections seem ok when looking at <code>showConnections()</code> with this as the following output:</p>
  
  <pre><code>&gt; showConnections()
    description                                                 class      mode  text     isopen   can read can write
  3 "&lt;-ip-xx-xxxx-x-xxx.zone.compute.internal:10187"   "sockconn" "a+b" "binary" "opened" "yes"    "yes"    
  4 "&lt;-ip-yy-yyyy-y-yyyy.zone.compute.internal:10187" "sockconn" "a+b" "binary" "opened" "yes"    "yes"    
  5 "&lt;-ip-zz-zzzz-z-zzzz.zone.compute.internal:10187"  "sockconn" "a+b" "binary" "opened" "yes"    "yes"    
  &gt; 
  </code></pre>
  
  <p>where <code>x</code>,<code>y</code> &amp; <code>z</code> represent the different ip addresses....The <code>foreach</code> examples come directly out out the examples given in the help files of <code>foreach</code> and furthermore some of the <code>clusterCall</code>/<code>clusterExport</code>/<code>clusterEvalQ</code> functions from the <code>parallel</code> package do not work either... giving a similar error message as before....</p>
  
  <p>I would still like to be able to connect from a Mac...but would also still like to be able to use foreach to carry out parallel processing...hope the extra info helps</p>