Victory over Curzon Ashton would have earned them a trip to Westfields, who play in English football's ninth tier.
But they were beaten 2-1 by their National League North opponents, after being held 1-1 at home on Saturday.
Mills was appointed on Sunday, with former Jackie McNamara switching to the role of chief executive.
Richard Brodie put the Minstermen ahead, but Alex Brown levelled before half-time and Niall Cummins scored Curzon's winner, netting the rebound after Luke Simpson saved his penalty.
Gary Mills suffered disappointment in the first game of his second spell as York City manager as they failed to reach the first round of the FA Cup.