user136: Nearly all modern Arduino boards use the Optiboot bootloader. Compared to the ATmegaboot used before, this is smaller in flash terms (by about 1.5KB) and faster in upload speed (115.2k vs 57.6k) and starts your sketch quicker (the delay whilst waiting after reset is shorter).
Are there any alternatives to the Optiboot bootloader? Specifically, are there any smaller (in flash terms) boot loaders? What differences do they have?

user220: 
Are there any alternatives to the Optiboot bootloader?

Err... not really. There's actually a [unofficial] modification to Optiboot to make it faster when uploading.

To install the high baud rate variations…

Ensure the Arduino IDE is not running
Download the 7-Zip archive using the link above
Extract the three files to the following directory: {Arduino Installation Root}\hardware\arduino\bootloaders\optiboot\
Open the boards.txt file located in the following directory: {Arduino Installation Root}\hardware\arduino\

Find This section inside it:
uno.name=Arduino Uno
uno.upload.protocol=arduino
uno.upload.maximum_size=32256
uno.upload.speed=115200
uno.bootloader.low_fuses=0xff
uno.bootloader.high_fuses=0xde
uno.bootloader.extended_fuses=0x05
uno.bootloader.path=optiboot
uno.bootloader.file=optiboot_atmega328.hex
uno.bootloader.unlock_bits=0x3F
uno.bootloader.lock_bits=0x0F
uno.build.mcu=atmega328p
uno.build.f_cpu=16000000L
uno.build.core=arduino
uno.build.variant=standard

Add the following section…
uno_uf.name=Arduino Uno / Upload Fast
uno_uf.upload.protocol=arduino
uno_uf.upload.maximum_size=32256
uno_uf.upload.speed=250000
uno_uf.bootloader.low_fuses=0xff
uno_uf.bootloader.high_fuses=0xde
uno_uf.bootloader.extended_fuses=0x05
uno_uf.bootloader.path=optiboot
uno_uf.bootloader.file=optiboot_atmega328_250000.hex
uno_uf.bootloader.unlock_bits=0x3F
uno_uf.bootloader.lock_bits=0x0F
uno_uf.build.mcu=atmega328p
uno_uf.build.f_cpu=16000000L
uno_uf.build.core=arduino
uno_uf.build.variant=standard


Change the two highlighted values to the baud rate you plan to use:    250000, 500000, or 1000000 Save and close the boards.txt file
Using your favourite ISP programmer, install the new bootloader. Don’t forget to: Ensure the Arduino Uno / Upload Fast board is selected (Tools / Board), If appropriate, ensure the Serial Port of
  your programmer is selected (Tools / Serial Port), Ensure the
  programmer is selected (Tools / Programmer), After installing the
  bootloader, change the Serial Port to the serial port of your Arduino
  Uno.


Above text slightly modified for compatibility with site. Original post

Besides that, and the ATmegaboot bootloader, you can't really find any alternative. I would imagine that if Arduino loads it on all the Unos ever produced, it's pretty optimized. The above hack speeds up upload time, but it doesn't reduce the size of the bootloader. You can also remove the wait time on the bootloader so it has a faster startup.
If you're that concerned about maximising flash see: How to really shrink a sketch and this answer [on on What can I do if I run out of Flash memory or SRAM?].

user15295: TinySafeBoot is an alternative for ATtiny and ATmega devices.

Compatible to plenty of ATtinys and ATmegas (see below)
Read/Write access to Flash and EEPROM.
All IOs available for serial communication
One-wire, also with multiple AVRs on a single line
Autobauding, rugged protocol.
Activated by Reset + Password (optional)
Password and Timeout user changeable anytime.
"Emergency Erase" to overcome lost Password - kills all data but leaves bootloader intact.
ATtinys: ~ 550 bytes, 
ATmegas: < 512 bytes (!)
TSB for ATtiny can update itself.
The Sourcecode is free Software (GPL)


user844: Optiboot is about the smallest bootloader available.
But, there is a much more powerful (with inbuilt console for hardware testing, RAM and Flash contents checking, etc) bootloader with a more developed and robust stk500v2 communications protocol between avrdude and the AVR bootloader.
I've used it on the ATmega2560 and on the ATmega1284p. It works for both 3 byte and 2 byte program counter MCU.
I have a STK500v2 Bootloader version on Sourceforge.
This bootloader is based the Peter Fleury STK500v2 bootloader, updated by Mark Sproul. I've made edits to have it supporting with the Goldilocks (an ATmega1284p) board.
The key feature is to use the STK500v2 protocol for uploading the Flash image to the Arduino. This protocol is more robust and gives better error capabilities than the avr109 protocol. As an aside the avr109 protocol is called "Arduino" in the Arduino world, because it toggles DTR (to reset the board and automatically enter the bootloader), and the STK500v2 protocol is called "Wiring" in the Arduino world, because of the same reason.
The Fleury/Sproul STK500v2 bootloader is designed to work with the ATmega2560 which has a 3 byte program register, because of the larger Flash size. But, it also supports normal 2 byte boards (everything else really).
For boards with plenty of Flash (like Goldilocks 1284p), there is a hardware monitor that can be used to query Flash, SRAM and hardware; you can check connections without additional code, and check the actual contents of Flash and EEPROM. Because the 1284p supports 2 USART, I added the ability to have debugging on USART1, so that the upload process via USART0 isn't disturbed.

136: <p>Nearly all modern Arduino boards use the <a href="https://code.google.com/p/optiboot/">Optiboot bootloader</a>. Compared to the ATmegaboot used before, this is smaller in flash terms (by about 1.5KB) and faster in upload speed (115.2k vs 57.6k) and starts your sketch quicker (the delay whilst waiting after reset is shorter).</p>

<p>Are there any alternatives to the Optiboot bootloader? Specifically, are there any smaller (in flash terms) boot loaders? What differences do they have?</p>
 220: <blockquote>
  <p>Are there any alternatives to the Optiboot bootloader?</p>
</blockquote>

<p>Err... not really. There's actually <strong>a [unofficial] <a href="http://zygomorphic.com/arduino-tiny/?attachment_id=257" rel="nofollow noreferrer">modification to Optiboot to make it faster when uploading</a>.</strong></p>

<blockquote>
  <p>To install the high baud rate variations…</p>
  
  <ul>
  <li>Ensure the Arduino IDE is not running</li>
  <li>Download the 7-Zip archive using the link above</li>
  <li>Extract the three files to the following directory: <code>{Arduino Installation Root}\hardware\arduino\bootloaders\optiboot\</code></li>
  <li>Open the boards.txt file located in the following directory: <code>{Arduino Installation Root}\hardware\arduino\</code></li>
  </ul>
  
  <p>Find This section inside it:</p>

<pre><code>uno.name=Arduino Uno
uno.upload.protocol=arduino
uno.upload.maximum_size=32256
uno.upload.speed=115200
uno.bootloader.low_fuses=0xff
uno.bootloader.high_fuses=0xde
uno.bootloader.extended_fuses=0x05
uno.bootloader.path=optiboot
uno.bootloader.file=optiboot_atmega328.hex
uno.bootloader.unlock_bits=0x3F
uno.bootloader.lock_bits=0x0F
uno.build.mcu=atmega328p
uno.build.f_cpu=16000000L
uno.build.core=arduino
uno.build.variant=standard
</code></pre>
  
  <p>Add the following section…</p>

<pre><code>uno_uf.name=Arduino Uno / Upload Fast
uno_uf.upload.protocol=arduino
uno_uf.upload.maximum_size=32256
uno_uf.upload.speed=250000
uno_uf.bootloader.low_fuses=0xff
uno_uf.bootloader.high_fuses=0xde
uno_uf.bootloader.extended_fuses=0x05
uno_uf.bootloader.path=optiboot
uno_uf.bootloader.file=optiboot_atmega328_250000.hex
uno_uf.bootloader.unlock_bits=0x3F
uno_uf.bootloader.lock_bits=0x0F
uno_uf.build.mcu=atmega328p
uno_uf.build.f_cpu=16000000L
uno_uf.build.core=arduino
uno_uf.build.variant=standard
</code></pre>
  
  <ul>
  <li>Change the two highlighted values to the baud rate you plan to use:    250000, 500000, or 1000000 Save and close the boards.txt file</li>
  <li>Using your favourite ISP programmer, install the new bootloader. <em>Don’t forget to: Ensure the Arduino Uno / Upload Fast board is selected (Tools / Board)</em>, <em>If appropriate, ensure the Serial Port of
  your programmer is selected (Tools / Serial Port)</em>, <em>Ensure the
  programmer is selected (Tools / Programmer)</em>, <em>After installing the
  bootloader, change the Serial Port to the serial port of your Arduino
  Uno</em>.</li>
  </ul>
</blockquote>

<p>Above text slightly modified for compatibility with site. <em><a href="http://zygomorphic.com/arduino-tiny/?page_id=249" rel="nofollow noreferrer">Original post</a></em></p>

<hr>

<p>Besides that, and the ATmegaboot bootloader, you can't really find any alternative. I would imagine that if Arduino loads it on all the Unos ever produced, it's pretty optimized. The above hack speeds up upload time, but it doesn't reduce the size of the bootloader. You can also <a href="http://learn.adafruit.com/arduino-tips-tricks-and-techniques/bootloader" rel="nofollow noreferrer">remove the wait time on the bootloader</a> so it has a faster startup.</p>

<p>If you're that concerned about maximising flash see: <em><a href="https://arduino.stackexchange.com/questions/193/how-to-really-shrink-a-sketch">How to really shrink a sketch</a></em> and <a href="https://arduino.stackexchange.com/a/222/220">this answer</a> [on on <em><a href="https://arduino.stackexchange.com/questions/221/what-can-i-do-if-i-run-out-of-flash-memory-or-sram">What can I do if I run out of Flash memory or SRAM?</a></em>].</p>
 15295: <p><a href="http://jtxp.org/tech/tinysafeboot_en.htm" rel="nofollow">TinySafeBoot</a> is an alternative for ATtiny and ATmega devices.</p>

<ul>
<li><p>Compatible to plenty of ATtinys and ATmegas (see below)</p></li>
<li><p>Read/Write access to Flash and EEPROM.</p></li>
<li><p>All IOs available for serial communication</p></li>
<li><p>One-wire, also with multiple AVRs on a single line</p></li>
<li><p>Autobauding, rugged protocol.</p></li>
<li><p>Activated by Reset + Password (optional)</p></li>
<li><p>Password and Timeout user changeable anytime.</p></li>
<li><p>"Emergency Erase" to overcome lost Password - kills all data but leaves bootloader intact.</p></li>
<li><p>ATtinys: ~ 550 bytes, </p></li>
<li><p>ATmegas: &lt; 512 bytes (!)</p></li>
<li><p>TSB for ATtiny can update itself.</p></li>
<li><p>The Sourcecode is free Software (GPL)</p></li>
</ul>
 844: <p>Optiboot is about the smallest bootloader available.</p>

<p>But, there is a much more powerful (with inbuilt console for hardware testing, RAM and Flash contents checking, etc) bootloader with a more developed and robust stk500v2 communications protocol between avrdude and the AVR bootloader.</p>

<p>I've used it on the ATmega2560 and on the ATmega1284p. It works for both 3 byte and 2 byte program counter MCU.</p>

<p>I have a <a href="https://sourceforge.net/projects/avrfreertos/files/AVRstk500v2_bootloader/" rel="noreferrer" title="STK500v2 Bootloader">STK500v2 Bootloader</a> version on <a href="https://sourceforge.net/projects/avrfreertos/" rel="noreferrer" title="AVRfreeRTOS">Sourceforge</a>.</p>

<p>This bootloader is based the Peter Fleury STK500v2 bootloader, updated by Mark Sproul. I've made edits to have it supporting with the Goldilocks (an ATmega1284p) board.</p>

<p>The key feature is to use the <strong>STK500v2</strong> protocol for uploading the Flash image to the Arduino. This protocol is more robust and gives better error capabilities than the <strong>avr109</strong> protocol. As an aside the avr109 protocol is called "Arduino" in the Arduino world, because it toggles DTR (to reset the board and automatically enter the bootloader), and the STK500v2 protocol is called "Wiring" in the Arduino world, because of the same reason.</p>

<p>The Fleury/Sproul STK500v2 bootloader is designed to work with the ATmega2560 which has a 3 byte program register, because of the larger Flash size. But, it also supports normal 2 byte boards (everything else really).</p>

<p>For boards with plenty of Flash (like Goldilocks 1284p), there is a <strong>hardware monitor</strong> that can be used to query Flash, SRAM and hardware; you can check connections without additional code, and check the actual contents of Flash and EEPROM. Because the 1284p supports 2 USART, I added the ability to have debugging on <strong>USART1</strong>, so that the upload process via <strong>USART0</strong> isn't disturbed.</p>
