Cullen was born at Prospect, Narraghmore, Athy, County Kildare, one of 16 children of Hugh and Judith Maher Cullen, six of whom were from Hugh's first marriage. His first school was the Quaker Shackleton School in nearby Ballitore. Following the relaxation of some of the Penal Laws, his father purchased some , giving him the status of a Catholic "strong farmer", a class that greatly influenced 19th-century Irish society. They were fervent in their Catholicism and fearful of the sort of social unrest that had led to the failed 1798 Rising. His great-nieces, Mary and Elizabeth Cullen became nuns, and two great-nephews entered the priesthood.