A precocious child, Peter started first grade at Manhattan's Hunter College Elementary School at the age of 6. He later attended Phillips Academy in Andover, Massachusetts, a prep school known for its art museum and, at the time, innovative art program. While at Andover, Halley took an interest in various forms of media and became the programming director for the school's low-wattage radio station. It was also during this time that he began painting, making his first works in his great uncle Aaron's art studio.