Cartwright was born in 1820 to Alexander Cartwright Sr. (1784–1855), a merchant sea captain, and Esther Rebecca Burlock Cartwright (1792–1871). Alexander Jr. had six siblings. He first worked at the age of 16 in 1836 as a clerk for a Wall Street broker, later doing clerical work at the Union Bank of New York. After hours, he played bat-and-ball games in the streets of Manhattan with volunteer firefighters. Cartwright himself was a volunteer, first with Oceana Hose Company No. 36, and then Knickerbocker Engine Company No. 12.