user20373: I'm running into a bizarre problem. I've got a custom board using an Atmega328P, running off the internal oscillator. From Atmel studio, I've successfully uploaded the standard optiboot bootloader (optiboot_atmega328.hex) and burned the appropriate fuses (hfuse = 0xDE, lfuse = 0xE2, extended_fuse = 0x05). As per an arduino forum post, I added the following entry to boards.txt:

    atmega328_384_8.name=ATmega328 Optiboot @ 38,400baud w/ 8MHz Int. RC Osc.
atmega328_384_8.upload.protocol=arduino
atmega328_384_8.upload.maximum_size=30720
atmega328_384_8.upload.speed=57600 
atmega328_384_8.upload.tool=avrdude 

atmega328_384_8.bootloader.low_fuses=0xE2
atmega328_384_8.bootloader.high_fuses=0xDE
atmega328_384_8.bootloader.extended_fuses=0x05
atmega328_384_8.bootloader.path=optiboot
atmega328_384_8.bootloader.file=optiboot_atmega328.hex
atmega328_384_8.bootloader.unlock_bits=0x3F
atmega328_384_8.bootloader.lock_bits=0x0F

atmega328_384_8.build.mcu=atmega328p
atmega328_384_8.build.f_cpu=8000000L
atmega328_384_8.build.core=arduino
atmega328_384_8.build.variant=standard

Note here, I've modified the upload speed to 57600 baud to "fool" it into the right upload speed given an 8MHz oscillator.
I can now successfully upload sketches from the Arduino IDE (1.6.3) on the lab windows machine, but when I try from my Mac (various IDE versions have been tried), I get the following protocol error:
avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0
Here's the full trace:
/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/bin/avrdude -C/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/etc/avrdude.conf -v -patmega328p -carduino -P/dev/cu.usbserial-AH02F0KI -b57600 -D -Uflash:w:/var/folders/db/hw2v9r6j7qj_w9gh3qkhh_580000gr/T/build53eb1895270295360d41549eaf6e0624.tmp/sketch_apr19a.ino.hex:i 

avrdude: Version 6.0.1, compiled on Apr 14 2015 at 16:30:25
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/etc/avrdude.conf"
         User configuration file is "/Users/npisenti/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/cu.usbserial-AH02F0KI
         Using Programmer              : arduino
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00
         Hardware Version: -1538883332
         Firmware Version: 6291456.-1538883332

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0
         Vtarget         : 322122165.6 V
         Varef           : 275608114.4 V
         Oscillator      : 0.001 Hz
         SCK period      : 368336.6 us


avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00
avrdude: stk500_initialize(): (a) protocol error, expect=0x14, resp=0xe0
avrdude: initialization failed, rc=-1
         Double check connections and try again, or use -F to override
         this check.

avrdude: stk500_disable(): protocol error, expect=0x14, resp=0xe0

avrdude done.  Thank you.

Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.

On Windows, the "successful" full trace from avrdude is:
Sketch uses 1,974 bytes (6%) of program storage space. Maximum is 30,720 bytes.
Global variables use 192 bytes of dynamic memory.
C:\arduino-1.6.3\hardware\tools\avr/bin/avrdude -CC:\arduino-1.6.3\hardware\tools\avr/etc/avrdude.conf -v -patmega328p -carduino -PCOM10 -b57600 -D -Uflash:w:C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:i 

avrdude: Version 6.0.1, compiled on Mar 30 2015 at 14:56:06
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "C:\arduino-1.6.3\hardware\tools\avr/etc/avrdude.conf"

         Using Port                    : COM10
         Using Programmer              : arduino
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino
         Hardware Version: 3
         Firmware Version: 4.4
         Vtarget         : 0.3 V
         Varef           : 0.3 V
         Oscillator      : 28.800 kHz
         SCK period      : 3.3 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e950f
avrdude: reading input file "C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex"
avrdude: writing flash (1974 bytes):

Writing | ################################################## | 100% 0.77s

avrdude: 1974 bytes of flash written
avrdude: verifying flash memory against C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:
avrdude: load data flash data from input file C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:
avrdude: input file C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex contains 1974 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.86s

avrdude: verifying ...
avrdude: 1974 bytes of flash verified

avrdude done.  Thank you.

In both cases, I'm using the FTDI friend from Adafruit; at first I figured it must be a driver issue, but I've installed the latest FTDI drivers and doesn't seem to help. Besides, I can successfully get text dumped from the ucontroller onto the serial port from a sketch previously uploaded on the windows machine.
Any help would be greatly appreciated!

user58887: I managed to figure this out after hours of searching. I wanted to flash the new Nano bootloader to my Nano, but it would fail with the same error as you.
For some reason, after Arduino 1.8.5, AVR boards with "Arduino as ISP" were changed so that they no longer work for me.
The solution for me was to revert programmers.txt to what it was on 1.8.5, i.e. change the arduinoasisp lines to:
arduinoasisp.protocol=stk500v1
arduinoasisp.speed=19200
arduinoasisp.program.protocol=stk500v1

This uses the stk500v1 protocol instead of the arduino protocol (you can see the -carduino option in your command line) and makes everything work for me.

20373: <p>I'm running into a bizarre problem. I've got a custom board using an Atmega328P, running off the internal oscillator. From Atmel studio, I've successfully uploaded the standard optiboot bootloader (<code>optiboot_atmega328.hex</code>) and burned the appropriate fuses (<code>hfuse = 0xDE</code>, <code>lfuse = 0xE2</code>, <code>extended_fuse = 0x05</code>). As per an arduino forum post, I added the following entry to <code>boards.txt</code>:

    atmega328_384_8.name=ATmega328 Optiboot @ 38,400baud w/ 8MHz Int. RC Osc.</p>

<pre class="lang-cpp prettyprint-override"><code>atmega328_384_8.upload.protocol=arduino
atmega328_384_8.upload.maximum_size=30720
atmega328_384_8.upload.speed=57600 
atmega328_384_8.upload.tool=avrdude 

atmega328_384_8.bootloader.low_fuses=0xE2
atmega328_384_8.bootloader.high_fuses=0xDE
atmega328_384_8.bootloader.extended_fuses=0x05
atmega328_384_8.bootloader.path=optiboot
atmega328_384_8.bootloader.file=optiboot_atmega328.hex
atmega328_384_8.bootloader.unlock_bits=0x3F
atmega328_384_8.bootloader.lock_bits=0x0F

atmega328_384_8.build.mcu=atmega328p
atmega328_384_8.build.f_cpu=8000000L
atmega328_384_8.build.core=arduino
atmega328_384_8.build.variant=standard
</code></pre>

<p>Note here, I've modified the upload speed to 57600 baud to "fool" it into the right upload speed given an 8MHz oscillator.</p>

<p>I can now successfully upload sketches from the Arduino IDE (1.6.3) on the lab windows machine, but when I try from my Mac (various IDE versions have been tried), I get the following protocol error:</p>

<p><code>avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0</code></p>

<p>Here's the full trace:</p>

<pre class="lang-cpp prettyprint-override"><code>/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/bin/avrdude -C/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/etc/avrdude.conf -v -patmega328p -carduino -P/dev/cu.usbserial-AH02F0KI -b57600 -D -Uflash:w:/var/folders/db/hw2v9r6j7qj_w9gh3qkhh_580000gr/T/build53eb1895270295360d41549eaf6e0624.tmp/sketch_apr19a.ino.hex:i 

avrdude: Version 6.0.1, compiled on Apr 14 2015 at 16:30:25
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/Applications/Arduino-1.6.8.app/Contents/Java/hardware/tools/avr/etc/avrdude.conf"
         User configuration file is "/Users/npisenti/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/cu.usbserial-AH02F0KI
         Using Programmer              : arduino
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00
         Hardware Version: -1538883332
         Firmware Version: 6291456.-1538883332

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0
         Vtarget         : 322122165.6 V
         Varef           : 275608114.4 V
         Oscillator      : 0.001 Hz
         SCK period      : 368336.6 us


avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xe0

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x00
avrdude: stk500_initialize(): (a) protocol error, expect=0x14, resp=0xe0
avrdude: initialization failed, rc=-1
         Double check connections and try again, or use -F to override
         this check.

avrdude: stk500_disable(): protocol error, expect=0x14, resp=0xe0

avrdude done.  Thank you.

Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.
</code></pre>

<p>On Windows, the "successful" full trace from avrdude is:</p>

<pre class="lang-cpp prettyprint-override"><code>Sketch uses 1,974 bytes (6%) of program storage space. Maximum is 30,720 bytes.
Global variables use 192 bytes of dynamic memory.
C:\arduino-1.6.3\hardware\tools\avr/bin/avrdude -CC:\arduino-1.6.3\hardware\tools\avr/etc/avrdude.conf -v -patmega328p -carduino -PCOM10 -b57600 -D -Uflash:w:C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:i 

avrdude: Version 6.0.1, compiled on Mar 30 2015 at 14:56:06
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "C:\arduino-1.6.3\hardware\tools\avr/etc/avrdude.conf"

         Using Port                    : COM10
         Using Programmer              : arduino
         Overriding Baud Rate          : 57600
         AVR Part                      : ATmega328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino
         Hardware Version: 3
         Firmware Version: 4.4
         Vtarget         : 0.3 V
         Varef           : 0.3 V
         Oscillator      : 28.800 kHz
         SCK period      : 3.3 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e950f
avrdude: reading input file "C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex"
avrdude: writing flash (1974 bytes):

Writing | ################################################## | 100% 0.77s

avrdude: 1974 bytes of flash written
avrdude: verifying flash memory against C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:
avrdude: load data flash data from input file C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex:
avrdude: input file C:\Users\GENERA~1\AppData\Local\Temp\build1097250116669309732.tmp/sketch_apr19a.cpp.hex contains 1974 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 0.86s

avrdude: verifying ...
avrdude: 1974 bytes of flash verified

avrdude done.  Thank you.
</code></pre>

<p>In both cases, I'm using the FTDI friend from Adafruit; at first I figured it must be a driver issue, but I've installed the latest FTDI drivers and doesn't seem to help. Besides, I can successfully get text dumped from the ucontroller onto the serial port from a sketch previously uploaded on the windows machine.</p>

<p>Any help would be greatly appreciated!</p>
 58887: <p>I managed to figure this out after hours of searching. I wanted to flash the new Nano bootloader to my Nano, but it would fail with the same error as you.</p>

<p>For some reason, after Arduino 1.8.5, AVR boards with "Arduino as ISP" were changed so that they no longer work for me.</p>

<p>The solution for me was to revert <code>programmers.txt</code> to what it was on 1.8.5, i.e. change the <code>arduinoasisp</code> lines to:</p>

<pre><code>arduinoasisp.protocol=stk500v1
arduinoasisp.speed=19200
arduinoasisp.program.protocol=stk500v1
</code></pre>

<p>This uses the stk500v1 protocol instead of the arduino protocol (you can see the <code>-carduino</code> option in your command line) and makes everything work for me.</p>
