For the Overland Campaign of 1864, Smith was assigned by Grant to command the XVIII Corps in Maj. Gen. Benjamin Butler's Army of the James, which he led in the Battle of Cold Harbor and the first operations against Petersburg. Smith's corps and a division of black troops (under Edward W. Hinks) were ordered to take the city. Remembering the debacle at Cold Harbor, Smith performed exhaustive reconnaissance. Determining that the section of the defensive line was manned primarily by artillery, he ordered an attack. However, the attack was delayed and in the meantime he became apprehensive about a rumor circulating that Lee was about to arrive. He lost his nerve, perhaps because of the formidable character of the Confederate works or perhaps because of a recurring bout with malaria, but his hesitation may have lost him the opportunity to shorten the war by nearly a year. On July 19, 1864, he was relieved from command of the XVIII Corps and he spent the remainder of the war on "special duty."