user3819: Every time I design a serial protocol to be used between two arduinos, I feel a bit like I'm reinventing a wheel. I wonder if there are any best practices or patterns people follow. This question is less about the actual code, but more about the format of the messages. 
For example, if I wanted to tell an arduino to flash it's first LED 3 times I might send:
^L1,F3\n


'^': starts a new command
'L': Defines the command, (L: target this command to an LED)
'1': Target the first LED
',': Command line separator, new value in this message to follow
'F': Flash sub-command
'3': 3 times (Flash the LED three times)
'\n': End the command

Thoughts? How do you usually approach writing a new serial protocol? What about if I wanted to send a query from arduino 1 to arduino 2 and then receive a response?

user25701: If you're into standards, then you may take a look at ASN.1 / BER TLV encoding. ASN.1 is a language used to describe data structures, made specifically for communication. BER is a TLV method of encoding the data structured using ASN.1. Problem is that ASN.1 encoding may be tricky at best. Creating a full fledged ASN.1 compiler is a project in itself (and a particularly tricky one at that, think months).

It's probably better to keep just the TLV structure. TLV basically consists of three elements: a tag, a length and a value field. The tag defines the type of the data (text string, octet string, integer etc.) and the length the length of the value.
In BER the T also denotes if the value is a set of TLV structures itself (a constructed node) or directly a value (a primitive node). That way you can create a tree in binary, much like XML (but without the XML overhead).
Example:
TT LL VV
02 01 FF

is an integer (tag 02) with a length of the value of 1 (length 01) and value -1 (value FF). In ASN.1 / BER the integers are sign big endian numbers, but you can of course use your own format.
TT LL (TT LL VV, TT LL VV VV)
30 07  02 01 FF  02 02 00 FF

is a sequence (a list) with length 7 containing two integers, one with value -1 and one with value 255. The two integer encodings together make up the value of the sequence.
You can simply toss this into an online decoder too, aint that nice?

You can also use indefinite length in BER which will allow you to stream data. In that case you do need to parse your tree correctly though. I'd consider it an advanced topic, you need to know about breadth first and depth first parsing, for one.

Using a TLV scheme basically allows you to think of any kind of data structure and encode it. ASN.1 goes a lot further than that, giving you unique identifiers (OID's), choices (a lot like C-unions), includes of other ASN.1 structures etc. etc. but that may be overkill for your project. Probably the best well known ASN.1-defined structures today are the certificates used by your browser.

user1585: I do not have any formal expertise on serial protocols, but I've used them quite a few times, and more or less settled on this scheme:
(Packet header)(ID byte)(data)(fletcher16 checksum)(Packet Footer)
I usually make the header 2 bytes and the Footer 1 byte. My parser will dump everything when it sees a new packet header, and attempt to parse the message if it sees a footer. If the checksum fails, it will not ditch the message, but continue adding until the footer character is found and a checksum succeeds. That way the footer only needs to be one byte since collisions don't disrupt the message.
The ID is arbitrary, sometimes with the length of the data section being the bottom nibble (4 bits). A second length bit could be used but I normally don't bother since the length does not need to be known to parse correctly, so seeing the right length come through for a given ID is just more confirmation that the message was correct.
The fletcher16 checksum is a 2 byte checksum with nearly the same quality as CRC but is much easier to implement. some details here. The code can be as simple as this: 
for(int i=0; i < bufSize; i++ ){
   sum1 = (sum1 + buffer[i]) % 255;
   sum2 = (sum2 + sum1) % 255;
}
uint16_t checksum = (((uint16_t)sum1)<<8) | sum2;

I've also used a call and respone system for critical messages, Where the PC will send a message every 500ms or so until it gets an OK message with a checksum of the whole original message as data (including the original checksum). 
This scheme is, of course, not well suited for being typed into a terminal like your example would be.  Your protocol seems pretty good for being limited to ASCII and I'm sure is easier for a quick project that you want to be able to directly read and send messages. For larger projects it is nice to have the density of a binary protocol and the security of a checksum.

user1566: There are a lot of ways to write a serial protocol depending on what functionality you might want and how much error checking you need.
Some of the common things you see in point to point protocols are:
End of message
The simplest ASCII protocols just have an end of message character sequence, often \r or \n as this is what gets printed when the enter key is hit. Binary protocols might use 0x03 or some other common byte.
Start of message
The problem with just having end of message is that you don't know what other bytes have already been received when you send your message. These bytes would then be prefixed to the message and cause it to be interpreted wrongly. For example, if the Arduino just woke from sleep there might be some garbage in the serial buffer. To get around this you have a start of message sequence. In your example, ^,  in binary protocols often 0x02
Error Checking
If the message can be corrupted we need some error checking. This could be a checksum or a CRC error or something else.
Escape Characters
It could be that the checksum adds to a control character, such as the 'start of message' or 'end of message' byte, or the message contains a value equal to a control character. The solution is to introduce an escape character. The escape character is placed before a modified control character so that the actual control character is not present. E.g. if a start character is 0x02, using the escape character 0x10 we can send the value 0x02 in the message as the byte pair 0x10 0x12 (byte XOR control character)
Packet number
If a message is corrupted we could request a resend with a nack or retry message, but if multiple messages have been sent then only the latest message can be resent. Instead the packet can be given a number that rolls over after a certain number of messages. For example, if this number is 16, the transmitting device can store the last 16 messages send and if any were corrupted the receiving device can request a resend using the packet number. 
Length
Often in binary protocols you see a length byte which tells the receiving device how many characters are in the message. This adds another level of error checking as if the correct number of bytes were not received then there was an error.
Arduino specific
When coming up with a protocol for Arduino the first consideration is how reliable is the communications channel. If you are sending over most wireless mediums, XBee, WiFi, etc, there is already built in error checking and retries and thus no point in putting these in your protocol. If you are sending over RS422 for a couple of kilometres then it will be necessary. The things I would include are the start of message and end of message characters, as you have. My typical implementation looks something like:
>messageType,data1,data2,…,dataN\n
Delimiting the data parts with a comma allows for easy parsing, and the message is sent using ASCII. ASCII protocols are great because you can type messages into the serial monitor.
If you want a binary protocol, maybe to shorten the message sizes, you will have to implement escaping if a data byte can be the same as a control byte. Binary control characters are better for systems where the full spectrum of error checking and retries is desired. The payload can still be ASCII if desired.

3819: <p>Every time I design a serial protocol to be used between two arduinos, I feel a bit like I'm reinventing a wheel. I wonder if there are any best practices or patterns people follow. This question is less about the actual code, but more about the format of the messages. </p>

<p>For example, if I wanted to tell an arduino to flash it's first LED 3 times I might send:</p>

<pre><code>^L1,F3\n
</code></pre>

<ul>
<li>'^': starts a new command</li>
<li>'L': Defines the command, (L: target this command to an LED)</li>
<li>'1': Target the first LED</li>
<li>',': Command line separator, new value in this message to follow</li>
<li>'F': Flash sub-command</li>
<li>'3': 3 times (Flash the LED three times)</li>
<li>'\n': End the command</li>
</ul>

<p>Thoughts? How do you usually approach writing a new serial protocol? What about if I wanted to send a query from arduino 1 to arduino 2 and then receive a response?</p>
 25701: <p>If you're into standards, then you may take a look at ASN.1 / BER TLV encoding. ASN.1 is a language used to describe data structures, made specifically for communication. BER is a TLV method of encoding the data structured using ASN.1. Problem is that ASN.1 encoding may be tricky at best. Creating a full fledged ASN.1 compiler is a project in itself (and a particularly tricky one at that, think <em>months</em>).</p>

<hr>

<p>It's probably better to keep just the TLV structure. TLV basically consists of three elements: a tag, a length and a value field. The tag defines the type of the data (text string, octet string, integer etc.) and the length the length <em>of the value</em>.</p>

<p>In BER the T also denotes if the value is a set of TLV structures itself (a constructed node) or directly a value (a primitive node). That way you can create a tree in binary, much like XML (but without the XML overhead).</p>

<p>Example:</p>

<pre><code>TT LL VV
02 01 FF
</code></pre>

<p>is an integer (tag <code>02</code>) with a length of the value of 1 (length <code>01</code>) and value -1 (value <code>FF</code>). In ASN.1 / BER the integers are sign big endian numbers, but you can of course use your own format.</p>

<pre><code>TT LL (TT LL VV, TT LL VV VV)
30 07  02 01 FF  02 02 00 FF
</code></pre>

<p>is a sequence (a list) with length 7 containing <em>two</em> integers, one with value -1 and one with value 255. The two integer encodings together make up the value of the sequence.</p>

<p>You can simply toss this into an online decoder too, aint that nice?</p>

<hr>

<p>You can also use indefinite length in BER which will allow you to stream data. In that case you do need to parse your tree correctly though. I'd consider it an advanced topic, you need to know about breadth first and depth first parsing, for one.</p>

<hr>

<p>Using a TLV scheme basically allows you to think of any kind of data structure and encode it. ASN.1 goes a lot further than that, giving you unique identifiers (OID's), choices (a lot like C-unions), includes of other ASN.1 structures etc. etc. but that may be overkill for your project. Probably the best well known ASN.1-defined structures today are the certificates used by your browser.</p>
 1585: <p>I do not have any formal expertise on serial protocols, but I've used them quite a few times, and more or less settled on this scheme:</p>

<p>(Packet header)(ID byte)(data)(fletcher16 checksum)(Packet Footer)</p>

<p>I usually make the header 2 bytes and the Footer 1 byte. My parser will dump everything when it sees a new packet header, and attempt to parse the message if it sees a footer. If the checksum fails, it will not ditch the message, but continue adding until the footer character is found and a checksum succeeds. That way the footer only needs to be one byte since collisions don't disrupt the message.</p>

<p>The ID is arbitrary, sometimes with the length of the data section being the bottom nibble (4 bits). A second length bit could be used but I normally don't bother since the length does not need to be known to parse correctly, so seeing the right length come through for a given ID is just more confirmation that the message was correct.</p>

<p>The fletcher16 checksum is a 2 byte checksum with nearly the same quality as CRC but is much easier to implement. some details <a href="https://en.wikipedia.org/wiki/Fletcher&#39;s_checksum">here</a>. The code can be as simple as this: </p>

<pre><code>for(int i=0; i &lt; bufSize; i++ ){
   sum1 = (sum1 + buffer[i]) % 255;
   sum2 = (sum2 + sum1) % 255;
}
uint16_t checksum = (((uint16_t)sum1)&lt;&lt;8) | sum2;
</code></pre>

<p>I've also used a call and respone system for critical messages, Where the PC will send a message every 500ms or so until it gets an OK message with a checksum of the whole original message as data (including the original checksum). </p>

<p>This scheme is, of course, not well suited for being typed into a terminal like your example would be.  Your protocol seems pretty good for being limited to ASCII and I'm sure is easier for a quick project that you want to be able to directly read and send messages. For larger projects it is nice to have the density of a binary protocol and the security of a checksum.</p>
 1566: <p>There are a lot of ways to write a serial protocol depending on what functionality you might want and how much error checking you need.</p>

<p>Some of the common things you see in point to point protocols are:</p>

<p><strong>End of message</strong></p>

<p>The simplest ASCII protocols just have an end of message character sequence, often <code>\r</code> or <code>\n</code> as this is what gets printed when the enter key is hit. Binary protocols might use <code>0x03</code> or some other common byte.</p>

<p><strong>Start of message</strong></p>

<p>The problem with just having end of message is that you don't know what other bytes have already been received when you send your message. These bytes would then be prefixed to the message and cause it to be interpreted wrongly. For example, if the Arduino just woke from sleep there might be some garbage in the serial buffer. To get around this you have a start of message sequence. In your example, <code>^</code>,  in binary protocols often <code>0x02</code></p>

<p><strong>Error Checking</strong></p>

<p>If the message can be corrupted we need some error checking. This could be a checksum or a CRC error or something else.</p>

<p><strong>Escape Characters</strong></p>

<p>It could be that the checksum adds to a control character, such as the 'start of message' or 'end of message' byte, or the message contains a value equal to a control character. The solution is to introduce an escape character. The escape character is placed before a <em>modified</em> control character so that the actual control character is not present. E.g. if a start character is 0x02, using the escape character 0x10 we can send the <em>value</em> 0x02 in the message as the byte pair 0x10 0x12 (byte XOR control character)</p>

<p><strong>Packet number</strong></p>

<p>If a message is corrupted we could request a resend with a nack or retry message, but if multiple messages have been sent then only the latest message can be resent. Instead the packet can be given a number that rolls over after a certain number of messages. For example, if this number is 16, the transmitting device can store the last 16 messages send and if any were corrupted the receiving device can request a resend using the packet number. </p>

<p><strong>Length</strong></p>

<p>Often in binary protocols you see a length byte which tells the receiving device how many characters are in the message. This adds another level of error checking as if the correct number of bytes were not received then there was an error.</p>

<p><strong><em>Arduino specific</em></strong></p>

<p>When coming up with a protocol for Arduino the first consideration is how reliable is the communications channel. If you are sending over most wireless mediums, XBee, WiFi, etc, there is already built in error checking and retries and thus no point in putting these in your protocol. If you are sending over RS422 for a couple of kilometres then it will be necessary. The things I would include are the start of message and end of message characters, as you have. My typical implementation looks something like:</p>

<p><code>&gt;messageType,data1,data2,…,dataN\n</code></p>

<p>Delimiting the data parts with a comma allows for easy parsing, and the message is sent using ASCII. ASCII protocols are great because you can type messages into the serial monitor.</p>

<p>If you want a binary protocol, maybe to shorten the message sizes, you will have to implement escaping if a data byte can be the same as a control byte. Binary control characters are better for systems where the full spectrum of error checking and retries is desired. The payload can still be ASCII if desired.</p>
