When Napoleon was defeated and exiled the first time Lallemand joined the army of Louis XVIII. In 1815 he and his brother Henri tried to lead a rebellion against the Bourbon government but were arrested. When Napoleon returned from Elba to start the Hundred Days, he released them and gave them both commands in the Imperial Guard—Henri commanded the Foot Artillery Regiment and Charles the "Chasseurs à Cheval"—and the brevet rank of "Général de Division". After the Battle of Waterloo, Lallemand accompanied Napoleon to Rochefort, where Napoleon surrendered. Lallemand tried to follow Napoleon into exile but the British refused that and imprisoned him in Malta for two months before he escaped.