Robert was probably the eldest of Henry's many illegitimate children. He was born before his father's accession to the English throne, either during the reign of his grandfather William the Conqueror or his uncle William Rufus. He is sometimes and erroneously designated as a son of Nest, daughter of Rhys ap Tewdwr, last king of Deheubarth, although his mother has been identified as a member of "the Gay or Gayt family of north Oxfordshire", possibly a daughter of Rainald Gay (fl. 1086) of Hampton Gay and Northbrook Gay in Oxfordshire. Rainald had known issue Robert Gay of Hampton (died c. 1138) and Stephen Gay of Northbrook (died after 1154). A number of Oxfordshire women feature as the mothers of Robert's siblings.