Charles' siblings included Father William Chandler Neale S.J. (1743-1799), who left America to enter the Jesuit order in Flanders and spent his life ministering to the Catholics of England, where he died. The next two brothers also took this step, but both died before they were able to complete their training. Charles was among the youngest three sons, all of whom also became Jesuit priests. They were Francis Neale S.J., the president of Georgetown College, and Leonard Neale S.J., who also served as president of Georgetown and then became the Archbishop of Baltimore.