Take the 2-minute tour ×

I am fairly new to C and writing a TCP server, and was wondering how to handle recv()'s from a client who will send commands that the server will respond to. For the sake of this question, let's just say header is 1st byte, command identifier is 2nd byte, and payload length is 3rd byte, followed by the payload (if any).

What is the best way to recv() this data? I was thinking to call recv() to read in the first 3 bytes into the buffer, check to make sure header and command identifiers are valid, then check payload length and call recv() again with payload length as length and add this to the back of the aforementioned buffer. Reading Beej's networking article (particularly this section here: http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html#sonofdataencap), however, he advises to use "an array big enough for two [max length] packets" to handle situations such as getting some of the next packet.

What is the best way to handle these types of recv()'s? Basic question, but I would like to implement it efficiently, handling all cases that can arise. Thanks in advance.

share|improve this question
add comment

4 Answers

up vote 3 down vote accepted

The method that Beej is alluding to, and AlastairG mentions, works something like this:

For each concurrent connection, you maintain a buffer of read-but-not-yet-processed data. (This is the buffer that Beej suggests sizing to twice the maximum packet length). Obviously, the buffer starts off empty:

unsigned char recv_buffer[BUF_SIZE];
size_t recv_len = 0;

Whenever your socket is readable, read into the remaining space in the buffer, then immediately try and process what you have:

result = recv(sock, recv_buffer + recv_len, BUF_SIZE - recv_len, 0);

if (result > 0) {
    recv_len += result;
    process_buffer(recv_buffer, &recv_len);

The process_buffer() will try and process the data in the buffer as a packet. If the buffer doesn't contain a full packet yet, it just returns - otherwise, it processes the data and removes it from the buffer. So for you example protocol, it would look something like:

void process_buffer(unsigned char *buffer, size_t *len)
    while (*len >= 3) {
        /* We have at least 3 bytes, so we have the payload length */

        unsigned payload_len = buffer[2];

        if (*len < 3 + payload_len) {
            /* Too short - haven't recieved whole payload yet */

        /* OK - execute command */
        do_command(buffer[0], buffer[1], payload_len, &buffer[3]);

        /* Now shuffle the remaining data in the buffer back to the start */
        *len -= 3 + payload_len;
        if (*len > 0)
            memmove(buffer, buffer + 3 + payload_len, *len);

(The do_command() function would check for a valid header and command byte).

This kind of technique ends up being necessary, because any recv() can return a short length - with your proposed method, what happens if your payload length is 500, but the next recv() only returns you 400 bytes? You'll have to save those 400 bytes until the next time the socket becomes readable anyway.

When you handle multiple concurrent clients, you simply have one recv_buffer and recv_len per client, and stuff them into a per-client struct (which likely contains other things too - like the client's socket, perhaps their source address, current state etc.).

share|improve this answer
+1 for short reads! I'd add more +s but am not allowed. The data may have got split across a packet boundary and not yet arrived. Or the program may have received a signal. –  Zan Lynx Dec 3 '10 at 1:51
caf, thank you! Code always helps a lot. Beej's explanation, as well as Alastair's makes much more sense now. I'm sure this will help others on stack as well. I will implement something very similar to this for my recv's. One question though, on the line: result = recv(sock, recv_buffer + recv_len, BUF_SIZE - recv_len, 0); you are not looping correct? So for example sake, if BUF_SIZE is 1024, you attempt to do a recv of 1024 bytes and process whatever comes in the buffer, even if it's one byte? Also, what is the performance hit of memmove on the buffer if I have lots of commands to process? –  Jack Dec 3 '10 at 2:15
@Jack: Right - the looping happens around that whole recv(); process_buffer(); segment (presumably with a select() in there to block). process_buffer(); gets called even in the case of only having a single byte, because I feel it's better to keep all the smarts for "is this a complete message" in one place (in this case, process_buffer() would drop out straight away, and you'd end up back in select(), waiting for more bytes). –  caf Dec 3 '10 at 2:33
cont... The performance of memmove() will only be an issue if you typically receive large numbers of commands in a single recv(), because memmove() will be called for each complete command processed (in the common case, where there's no trailing data after a command, memmove() won't ever be called). You can of course improve process_buffer() to do only a single memmove(), but I felt that the simpler code would be easier to get the concept across. –  caf Dec 3 '10 at 2:35
Thanks caf :) I will try to implement something similar to this and will post back if I run into any other issues. I like the fact that you put all the smarts in a separate method (process_buffer) and can simply call recv() and then process_buffer() from the top level file. This code was very helpful as I am still learning the intricacies of C and TCP/IP in general (I come from a Java background so this is hard transition for me), so I appreciate it. –  Jack Dec 3 '10 at 2:58
show 4 more comments

Nice question. How perfect do you want to go? For an all singing all dancing solution, use asynchronous sockets, read all the data you can whenever you can, and whenever you get new data call some data processing function on the buffer.

This allows you to do big reads. If you get a lot of commands pipelined you could potentially process them without having to wait on the socket again, thus increasing performance and response time.

Do something similar on the write. That is the command processing function writes to a buffer. If there is data in the buffer then when checking sockets (select or poll) check for writeability and write as much as you can, remembering to only remove the bytes actually written from the buffer.

Circular buffers work well in such situations.

There are lighter simpler solutions. However this one is a good one. Remember that a server might get multiple connections and packets can be split. If you read from a socket into a buffer only to find you don;t have the data for a complete command, what do you do with the data you have already read? Where do you store it? If you store it in a buffer associated with that connection, then you might as well go the whole hog and just read into the buffer as described above in the first place.

This solution also avoids having to spawn a separate thread for each connection - you can handle any number of connections without any real problems. Spawning a thread per connection is an unnecessary waste of system resources - except in certain circumstances where multiple threads would be recommended anyway, and for that you can simply have worker threads to execute such blocking tasks while keeping the socket handling single threaded.

Basically I agree with what you say Beej says, but don't read tiddly bits at a time. Read big chunks at a time. Writing a socket server like this, learning and designing as I went along based on a tiny bit of socket experience and man pages, was one of the most fun projects I have ever worked on, and very educational.

share|improve this answer
Thanks AlastairG. I'm basically trying to keep it as simple as possible and just handle the most common situations such as reading into the next packet, or reading a partial packet. I'm actually using select to monitor the client socket for incoming data (commands in this case) so I don't think it's a matter of asynchronous sockets. Would my method of reading in 3 bytes, then the payload not work? The reason I did not want to implement what Beej said is because it's difficult to understand without actual code / pseudocode. Thanks for the advice though :) –  Jack Dec 2 '10 at 16:14
For the writes / sends after I process the incoming data, I was simply going to use a variation of Beej's sendall method since I know the length of the packet .. shown here: beej.us/guide/bgnet/output/html/multipage/advanced.html#sendall. I was more concerned about the reads. –  Jack Dec 2 '10 at 16:15
My comment about write was just for completeness - in case someone else comes along wanting to know how to write a really good socket server. One thing I will say is that I've tried writing socket servers that are small and simple but you very quickly find that you just need to do it the way I describe. Not always though. Reading 3 bytes will work, but why do it that way? –  AlastairG Dec 2 '10 at 16:33
Given that you need to write the following data to a buffer in case you get a partial packet, you may as well just read as much as you can, even if that includes part or all of the next command. You will, ultimately, get simpler code as well as better performance. –  AlastairG Dec 2 '10 at 16:33
I have to go home now. There should be plenty of examples of buffered socket reading on the internet. I also recommend asynchronous sockets even if they aren't strictly speaking necessary because you are using select(). The main reason is so that you never end up blocking on a write, essential if you are handling multiple sockets on one thread. Socket programming can get very complicated very easily. The solution I outline is the basis for a phenomenally robust server. It may be better for you to do it your way, encounter the problems and ovecome them yourself. It's a rewarding journey. –  AlastairG Dec 2 '10 at 16:35
add comment

The solution Alastair describes is the best in terms of performance. FYI - asynchronous programming is also known as event-driven programming. In other words, you wait for data to come on the socket, read it into a buffer, process what/when you can, and repeat. Your application can do other things between reading data and processing it.

A couple more links that I found helpful doing something very similar:

The second is a great library to help implement all of this.

As for using a buffer and reading as much as you can, it's another performance thing. Bulk reads are better, less system calls (reads). You process the data in the buffer when you decide you have enough to process, but making sure to process only one of your "packets" (the one you described with 3-byte header) at a time and not destroy other data in the buffer.

share|improve this answer
Thank you loan, this helps. When you say "read as much as you can" for better performance, do you mean simply doing one recv with MAX_BUF_SIZE as the length? (i.e. #define MAX_BUF_SIZE 1024 ... unsigned char buf[MAX_BUF_SIZE] ... recv(sockfd, buf, MAX_BUF_SIZE, 0))? Or did you mean something else? Anyway, I guess I am doing this in an event-driven programming fashion as Alastair described. The reason I wanted to read 3 bytes first and then the rest of the payload is that I then know the exact length of the packet to pass into recv on the second call. –  Jack Dec 2 '10 at 21:17
@Jack: 1024 is thinking small. On a gigabit network you could easily end up needing megabytes of buffers for maximum bandwidth. –  Zan Lynx Dec 3 '10 at 1:49
Understood, for the sake of my program, however, I probably don't need anything that large. It is definitely something I will consider though, thanks! –  Jack Dec 3 '10 at 2:17
You probably won't be reading MAX_BUF_SIZE every time. If you've had a previous partial command read into the buffer, you will have less space left in the buffer. You need to track how much data is in the buffer to work out how much space is left. There are two approaches here. Circular buffers, which may require a double read when you cross the end and loop back to the start, or simply shift the data with memmove after processing some bytes. given that the buffer needs to be quite big, and also that you will need one per connection, you should malloc them rather than statically allocate them. –  AlastairG Dec 3 '10 at 9:41
Your buffer only needs to be as large as the minimum amount of data that can be processed. For example, it needs to be at least the size of the largest header, or command, or argument, etc. Otherwise, the buffer would fill up and you can't process incomplete data: deadlock. –  Ioan Dec 6 '10 at 13:43
show 1 more comment

There are basically two assumptions if you are using multiple connection then the best way to handle multiple connection (whether listening socket, readfd or writefd) is with select/poll/epoll. You can use either of these based upon your requirement.

on your second query how to handle multiple recv() this practice can be used: whenever the data is arrived just peek the header(it should be of fixed length and format as you have described).

    buff_header = (char*) malloc(HEADER_LENGTH);
    count =  recv(sock_fd, buff_header, HEADER_LENGTH, MSG_PEEK);
    /*MSG_PEEK if you want to use the header later other wise you can set it to zero
      and read the buffer from queue and the logic for the code written below would
      be changed accordingly*/

by this you got you header and you can verify the parameter and also extract the full msg length. After getting the full msg length just recieve the full msg

    buffer =(char*) malloc(msg_length);
        count = recv(sock_fd, buffer, msg_length, 0);

so in this way you need not to take any array having some fixed length and you can easily implement your logic.

share|improve this answer
add comment

Your Answer


