Phillips was born at Marden in Wiltshire. His father belonged to an old Welsh family, but settled in England as an officer of excise and married the sister of William Smith, a renowned English geologist. When both parents died when he was a child, Phillips's custody was assumed by Smith and Phillips was brought into Smith's London home during early 1815. During the next few years he attended various schools and helped his uncle with his geological research and writing; he also developed an interest in lithography (printing from prepared slabs of stone), and was among the earliest English practitioners of the process, experimenting with it between about 1816 and 1819. After ending school, Phillips accompanied Smith on his wanderings in connection with his preparation of geological maps. During the spring of 1824 Smith went to York to deliver a course of lectures on geology, and his nephew Phillips accompanied him. Phillips accepted engagements in the principal Yorkshire towns to arrange their museums and give courses of lectures on the collections contained therein. York became his residence, and he obtained during 1826 the situation of keeper of the Yorkshire museum and secretary of the Yorkshire Philosophical Society at the same time as Henry Robinson was Librarian of the YPS.