Breckinridge returned to Kentucky, accepting the pastorate of First Presbyterian Church of Lexington. His return to Kentucky was also motivated by a growing fondness for his cousin, Virginia Hart Shelby, who had cared for two of his children during his stay in Pennsylvania. Virginia was the widow of Alfred Shelby, the son of Isaac Shelby, who was twice governor of Kentucky. Their written exchanges included love poems from Robert and concerned questions from Virginia about the wisdom of engaging in a relationship. Despite being advised by her sisters to avoid the marriage and her own wavering on the issue, the two were married in April 1847. They had three children, only one of whom survived to adulthood. Disagreements among the children of both partners' previous marriages exacerbated an already tense union, which almost ended in divorce in September 1856. Robert managed to reconcile with his wife, and they remained together until Virginia's death in 1859.