He was the second son of John Wallace (1712–1805) of Cessnock, Glasgow and Kelly in Ayrshire, by his third wife, Janet, third daughter of Robert Colquhoun of the island of St. Christopher. His father was a West India merchant in Glasgow, who amassed a large fortune and became proprietor of some major estates. The eldest son was Sir James Maxwell Wallace. By the father's will Robert Wallace received the estate of Kelly and part of the West Indian property, and was known by the designation of Wallace of Kelly.