John William "Jack" Swilling was born on April 1, 1830, at Red House Plantation, Anderson, South Carolina, to George Swilling and Margaret Farrar Swilling, the eighth of their 10 children. George Swilling was the son of the plantation manager, while Margaret Farrar's father, Thomas Farrar, was the owner. (Incidentally, Thomas Farrar's mother was Judith Jefferson, the sister of Peter Jefferson and aunt to the third president, Thomas Jefferson.) Farrar's parents did not approve of the marriage, so the young couple eloped. It took three years for her parents to accept the match. In time, George Swilling became owner of the plantation. When Jack Swilling was 14, the family moved from South Carolina to Georgia. Three years later he and an older brother enlisted in a mounted battalion of Georgia volunteers for service during the Mexican–American War. After the war, the two young men returned to Georgia. Jack Swilling's whereabouts are unknown for a brief time afterwards, although he was reported in Georgia for the Christmas of 1849.