One writer asserts that Bayley's early school days were spent at Mendham Township, New Jersey. Another relates that he received a fair elementary education in the "public schools" of New York and displayed great studiousness and "an extraordinary love of miscellaneous reading." This may have referred to the schools in Mamaroneck or in New York City, where he may have lived with his grandfather. A later biographer infers he may have begun his earliest education in New Jersey and continued it in New York.