Phillips was drafted by the Milwaukee Brewers as a shortstop in the 41st round of the draft as a high school senior and again by the New York Yankees in the 7th round of the 1999 draft as a college graduate of the University of Alabama. He signed with the Yankees, but did not reach the Major Leagues until a call-up session towards the end of the 2004 season; he made his Major League debut on September 14, 2004. In his first Major League AB, he hit a home run over the Green Monster in Fenway Park off Terry Adams of the Boston Red Sox. Phillips is one of ten players to homer in his first at bat as a Yankee, along with Marcus Thames, Bubba Crosby, Todd Zeile, Wilson Betemit, Cody Ransom, Mark Reynolds, Curtis Granderson, Tyler Austin, and Aaron Judge. He became the 21st player in Major League history to hit a home run on the first pitch of his Major League career.