On November 18, 1836, at age 20, Cannon married James Swisshelm, from a nearby town. Jane was strong-willed making the circumstances of marriage difficult. They moved to Louisville, Kentucky, in 1838, where James intended to go into business with his brother, Samuel. This is where Jane first encountered slavery, which made a strong impression on her. Nearby was a man who had sold away his own mixed-race children. She wrote in her autobiography of some of the sights she saw and stories she heard.