Lord Rokeby was born Matthew Robinson in a large family from Yorkshire. His parents were Matthew Robinson (1694–1778) of Edgely and West Layton, Yorkshire, and Elizabeth, daughter of Robert Drake of Cambridge; his sisters included the novelists Elizabeth Montagu and Sarah Scott. Of his six brothers, Thomas the second was known as a legal writer, William the fifth as a cleric, and the youngest, Charles, as a Member of Parliament. His father inherited property in the neighbourhood of Rokeby from his great-uncle Matthew Robinson, rector of Burneston; and his mother inherited the Mount Morris estate at Monks Horton, near Hythe, Kent, from her brother Morris Drake Morris, which Rokeby in turn succeeded to in 1746.