title: Issues with PvP communication via TCP
body: <p>I have just started working with TCP (and all associated libraries) due to the need to implement communication between two processes over an internet connection. My code works but it is very slow compared to what I (perhaps due to lack of experience) would expect given the network latency and bandwidth. Also, I'm sure there are many other things wrong with the code, which is using the UNIX socket API. I would prefer not to use big libraries (such as Boost) for my project unless there is a very good reason.</p>
  <p>I include a minimal working example. It is rather long despite my best efforts to shorten it. However, I think most of the problems should be in the first file (tcp_helpers.h) which is only used by the client and server main programs in a fairly obvious way. The functions there are not fully optimized but I find it hard to believe that is the problem, rather <strong>likely are some fundamental flaws in the logic</strong>.</p>
  <p>I also want to ask some <strong>questions</strong> relevant to the problem:</p>
  <ol>
  <li>For network performance, should I worry about using IPv4 vs IPv6? Could it be that my network dislikes the use of IPv4 somehow and penalized performance?</li>
  <li>Since the Socket API emulates a stream, I would think it does not matter if you call send() multiple times on smaller chunks of data or once on a big chunk. But perhaps it does matter and doing it with smaller chunks (I call send for my custom protocol header and the data separately each time) leads to issues?</li>
  <li>Suppose that two parties communicate over a network doing work on the received data before sending their next message (as is done in my example). If the two processes take x amount of time on localhost to finish, they should never take longer than (2*x + (network overhead)) on the real network, right? If x is small, making the computations (i.e. work before sending next message) go faster will not help, right?</li>
  <li>My example program takes about 4ms when running on localhost and &gt;0.7 seconds when running on the local (university) network I'm using. The local network has ping times (measured with <code>ping</code>) of ( min/avg/max/mdev [ms] = 4.36 / 97.6 / 405. / 86.3 ) and a bandwidth (measured with <code>iperf</code>) of ~70Mbit/s. When runnin the example program on the network I get (measured with <code>wireshark</code> filtering on the port in question) 190 packets with an average throughput of 172kB/s and average packet size ~726 Bytes. Is this realistic? To me it seems like my program should be much faster given these network parameters, despite the fairly high ping time.</li>
  <li>Looking at the actual network traffic generated by the example program, I started thinking about all the &quot;features&quot; of TCP that are done under the hood. I read somewhere that many programs use several sockets at the same time &quot;to gain speed&quot;. Could this help here, for example using two sockets, each for just one-way communication? In particular, maybe somehow reducing the number of ack packets could help performance?</li>
  <li>The way I'm writing messages/headers as structs has (at least) two big problems that I already know. First, I do not enforce network byte order. If one communicating party uses big-endian and the other little-endian, this program will not work. Furthermore, due to struct padding (see <a href="http://www.catb.org/esr/structure-packing/" rel="nofollow noreferrer">catb.org/esr/structure-packing/</a>), the sizes of the structs may vary between implementations or compilers, which would also break my program. I could add something like (for gcc) <code>__attribute__((__packed__))</code> to the structs but that would make it very compiler specific and perhaps even lead to inefficiency. Are there standard ways of dealing with this issue (I've seen something about aligning manually)? (Maybe I'm looking for the wrong keywords.)</li>
  </ol>
  <pre><code>// tcp_helpers.h. // NOTE: Using this code is very ill-advised.
  #include &lt;iostream&gt;
  #include &lt;string&gt;
  #include &lt;sstream&gt;
  #include &lt;vector&gt;
  #include &lt;unistd.h&gt;  // POSIX specific
  #include &lt;sys/socket.h&gt; // POSIX specific
  #include &lt;netinet/in.h&gt; // POSIX specific
  #include &lt;arpa/inet.h&gt; // POSIX specific
  #include &lt;cerrno&gt;  // for checking socket error messages
  #include &lt;cstdint&gt; // for fixed length integer types
  
  //////////////////// PROFILING ///////////////////
  #include &lt;chrono&gt;
  static auto start = std::chrono::high_resolution_clock::now();
  void print_now(const std::string &amp;message) {
      auto t2 = std::chrono::high_resolution_clock::now();
      std::chrono::duration&lt;double&gt; time_span = t2 - start;
      std::cout &lt;&lt; time_span.count() &lt;&lt; &quot;: &quot; &lt;&lt; message &lt;&lt; std::endl;
  }
  //////////////////// PROFILING ///////////////////
  
  struct TCPMessageHeader {
      uint8_t protocol_name[4];
      uint32_t message_bytes;
  };
  
  struct ServerSends {
      uint16_t a;
      uint32_t b;
      uint32_t c;
  };
  
  typedef uint8_t ClientSends;
  
  namespace TCP_Helpers {
      template&lt;typename NakedStruct&gt;
      void send_full_message(int fd, TCPMessageHeader header_to_send, const std::vector&lt;NakedStruct&gt; &amp;structs_to_send) {
          print_now(&quot;Begin send_full_message&quot;);
          if (header_to_send.message_bytes != sizeof(NakedStruct) * structs_to_send.size()) {
              throw std::runtime_error(&quot;Struct vector's size does not match the size claimed by message header&quot;);
          }
          int bytes_to_send = sizeof(header_to_send);
          int send_retval;
          while (bytes_to_send != 0) {
              send_retval = send(fd, &amp;header_to_send, sizeof(header_to_send), 0);
              if (send_retval == -1) {
                  int errsv = errno;  // from errno.h
                  std::stringstream s;
                  s &lt;&lt; &quot;Sending data failed (locally). Errno:&quot; &lt;&lt; errsv &lt;&lt; &quot; while sending header.&quot;;
                  throw std::runtime_error(&quot;Sending data failed (locally)&quot;);
              }
              bytes_to_send -= send_retval;
          }
          bytes_to_send = header_to_send.message_bytes;
          while (bytes_to_send != 0) {
              send_retval = send(fd, &amp;structs_to_send[0], sizeof(NakedStruct) * structs_to_send.size(), 0);
              if (send_retval == -1) {
                  int errsv = errno;  // from errno.h
                  std::stringstream s;
                  s &lt;&lt; &quot;Sending data failed (locally). Errno:&quot; &lt;&lt; errsv &lt;&lt;
                    &quot; while sending data of size &quot; &lt;&lt; header_to_send.message_bytes &lt;&lt; &quot;.&quot;;
                  throw std::runtime_error(s.str());
              }
              bytes_to_send -= send_retval;
          }
          print_now(&quot;end send_full_message.&quot;);
      }
  
      template&lt;typename NakedStruct&gt;
      std::vector&lt;NakedStruct&gt; receive_structs(int fd, uint32_t bytes_to_read) {
          print_now(&quot;Begin receive_structs&quot;);
          unsigned long num_structs_to_read;
          // ensure expected message is non-zero length and a multiple of the SingleBlockParityRequest struct
          if (bytes_to_read &gt; 0 &amp;&amp; bytes_to_read % sizeof(NakedStruct) == 0) {
              num_structs_to_read = bytes_to_read / sizeof(NakedStruct);
          } else {
              std::stringstream s;
              s &lt;&lt; &quot;Message length (bytes_to_read = &quot; &lt;&lt; bytes_to_read &lt;&lt;
                &quot; ) specified in header does not divide into required stuct size (&quot; &lt;&lt; sizeof(NakedStruct) &lt;&lt; &quot;).&quot;;
              throw std::runtime_error(s.str());
          }
          // vector must have size &gt; 0 for the following pointer arithmetic to work 
          // (this method must check this in above code).
          std::vector&lt;NakedStruct&gt; received_data(num_structs_to_read);
          int valread;
          while (bytes_to_read &gt; 0)  // todo need to include some sort of timeout?!
          {
              valread = read(fd,
                             ((uint8_t *) (&amp;received_data[0])) +
                             (num_structs_to_read * sizeof(NakedStruct) - bytes_to_read),
                             bytes_to_read);
              if (valread == -1) {
                  throw std::runtime_error(&quot;Reading from socket file descriptor failed&quot;);
              } else {
                  bytes_to_read -= valread;
              }
          }
          print_now(&quot;End receive_structs&quot;);
          return received_data;
      }
  
      void send_header(int fd, TCPMessageHeader header_to_send) {
          print_now(&quot;Start send_header&quot;);
          int bytes_to_send = sizeof(header_to_send);
          int send_retval;
          while (bytes_to_send != 0) {
              send_retval = send(fd, &amp;header_to_send, sizeof(header_to_send), 0);
              if (send_retval == -1) {
                  int errsv = errno;  // from errno.h
                  std::stringstream s;
                  s &lt;&lt; &quot;Sending data failed (locally). Errno:&quot; &lt;&lt; errsv &lt;&lt; &quot; while sending (lone) header.&quot;;
                  throw std::runtime_error(s.str());
              }
              bytes_to_send -= send_retval;
          }
          print_now(&quot;End send_header&quot;);
      }
  
      TCPMessageHeader receive_header(int fd) {
          print_now(&quot;Start receive_header (calls receive_structs)&quot;);
          TCPMessageHeader retval = receive_structs&lt;TCPMessageHeader&gt;(fd, sizeof(TCPMessageHeader)).at(0);
          print_now(&quot;End receive_header (calls receive_structs)&quot;);
          return retval;
      }
  }
  
  </code></pre>
  <pre><code>// main_server.cpp
  #include &quot;tcp_helpers.h&quot;
  
  int init_server(int port) {
      int server_fd;
      int new_socket;
      struct sockaddr_in address{};
      int opt = 1;
      int addrlen = sizeof(address);
      // Creating socket file descriptor
      if ((server_fd = socket(AF_INET, SOCK_STREAM, 0)) == 0) {
          throw std::runtime_error(&quot;socket creation failed\n&quot;);
      }
  
      if (setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR | SO_REUSEPORT, &amp;opt, sizeof(opt))) {
          throw std::runtime_error(&quot;failed to set socket options&quot;);
      }
      address.sin_family = AF_INET;
      address.sin_addr.s_addr = INADDR_ANY;
      address.sin_port = htons(port);
      // Forcefully attaching socket to the port
      if (bind(server_fd, (struct sockaddr *) &amp;address, sizeof(address)) &lt; 0) {
          throw std::runtime_error(&quot;bind failed&quot;);
      }
      if (listen(server_fd, 3) &lt; 0) {
          throw std::runtime_error(&quot;listen failed&quot;);
      }
      if ((new_socket = accept(server_fd, (struct sockaddr *) &amp;address, (socklen_t *) &amp;addrlen)) &lt; 0) {
          throw std::runtime_error(&quot;accept failed&quot;);
      }
      if (close(server_fd)) // don't need to listen for any more tcp connections (PvP connection).
          throw std::runtime_error(&quot;closing server socket failed&quot;);
      return new_socket;
  }
  
  int main() {
      int port = 20000;
      int socket_fd = init_server(port);
      while (true) {
          TCPMessageHeader rcv_header = TCP_Helpers::receive_header(socket_fd);
          if (rcv_header.protocol_name[0] == 0)   // using first byte of header name as signal to end
              break;
          // receive message
          auto rcv_message = TCP_Helpers::receive_structs&lt;ClientSends&gt;(socket_fd, rcv_header.message_bytes);
          for (ClientSends ex : rcv_message) // example &quot;use&quot; of the received data that takes a bit of time.
              std::cout &lt;&lt;  static_cast&lt;int&gt;(ex) &lt;&lt; &quot; &quot;;
          std::cout &lt;&lt; std::endl &lt;&lt; std::endl;
  
          // send a &quot;response&quot; containing 1000 structs of zeros
          auto bunch_of_zeros = std::vector&lt;ServerSends&gt;(500);
          TCPMessageHeader send_header{&quot;abc&quot;, 500 * sizeof(ServerSends)};
          TCP_Helpers::send_full_message(socket_fd, send_header, bunch_of_zeros);
  
      }
      exit(EXIT_SUCCESS);
  }
  </code></pre>
  <pre><code>// main_client.cpp
  #include &quot;tcp_helpers.h&quot;
  
  int init_client(const std::string &amp;ip_address, int port) {
      int sock_fd;
      struct sockaddr_in serv_addr{};
  
      if ((sock_fd = socket(AF_INET, SOCK_STREAM, 0)) &lt; 0) {
          throw std::runtime_error(&quot;TCP Socket creation failed\n&quot;);
      }
      serv_addr.sin_family = AF_INET;
      serv_addr.sin_port = htons(port);
      // Convert IPv4 address from text to binary form
      if (inet_pton(AF_INET, ip_address.c_str(), &amp;serv_addr.sin_addr) &lt;= 0) {
          throw std::runtime_error(&quot;Invalid address/ Address not supported for TCP connection\n&quot;);
      }
      if (connect(sock_fd, (struct sockaddr *) &amp;serv_addr, sizeof(serv_addr)) &lt; 0) {
          throw std::runtime_error(&quot;Failed to connect to server.\n&quot;);
      }
      return sock_fd;
  }
  
  int main() {
      // establish connection to server and get socket file descriptor.
      int port = 20000;
      int socket_fd = init_client(&quot;127.0.0.1&quot;, port);
      for (int i = 0; i &lt; 20; ++i) {  // repeat sending and receiving random data
          // send a message containing 200 structs of zeros
          auto bunch_of_zeros = std::vector&lt;ClientSends&gt;(250);
          TCPMessageHeader send_header{&quot;abc&quot;, 250 * sizeof(ClientSends)};
          TCP_Helpers::send_full_message(socket_fd, send_header, bunch_of_zeros);
  
          // receive response
          TCPMessageHeader rcv_header = TCP_Helpers::receive_header(socket_fd);
          auto rcv_message = TCP_Helpers::receive_structs&lt;ServerSends&gt;(socket_fd, rcv_header.message_bytes);
          for (ServerSends ex : rcv_message) // example &quot;use&quot; of the received data that takes a bit of time.
              std::cout &lt;&lt; ex.a &lt;&lt; ex.b &lt;&lt; ex.c &lt;&lt; &quot; &quot;;
          std::cout &lt;&lt; std::endl &lt;&lt; std::endl;
      }
      auto end_header = TCPMessageHeader{}; // initialized all fields to zero. (First byte of name == 0) is &quot;end&quot; signal.
      TCP_Helpers::send_header(socket_fd, end_header);
      exit(EXIT_SUCCESS);
  }
  </code></pre>