Kilmer was born December 6, 1886, in New Brunswick, New Jersey, the fourth and youngest child, of Annie Ellen Kilburn (1849–1932), a minor writer and composer, and Dr. Frederick Barnett Kilmer (1851–1934), a physician and analytical chemist employed by the Johnson and Johnson Company and inventor of the company's baby powder. He was named Alfred Joyce Kilmer after two priests at Christ Church in New Brunswick: Alfred R. Taylor, the curate; and the Rev. Dr. Elisha Brooks Joyce (1857–1926), the rector. Christ Church is the oldest Episcopal parish in New Brunswick and the Kilmer family were parishioners. Rector Joyce, who served the parish from 1883 to 1916, baptised the young Kilmer, who remained an Episcopalian until his 1913 conversion to Catholicism. Kilmer's birthplace in New Brunswick, where the Kilmer family lived from 1886 to 1892, is still standing and houses a small museum to Kilmer, as well as a few Middlesex County government offices.