The Welshman, 28, was made to work hard in his first defence against 36-year-old Montiel, who is a former multiple weight world champion.
Selby struggled to impose himself and found his shrewd opponent tough to hit.
His difficulties were not reflected on the judges' scorecards, though, winning 119-109, 118-110 and 116-112.
This was Selby's debut in America and the Barry boxer was hoping to put on a show to his enhance his profile.
He will walk away with valuable experience having faced an obdurate opponent with 20 world-title fights to his name and only one stoppage in a 61-bout career, but it was far from a slick display.
An obvious height advantage gave the champion the air of physical dominance as he faced a former flyweight world champion moving up through the weights in the twilight of his career.
But what Montiel lacked in stature, he made up for with compact power and ring craft.
He gave Selby early notice of his strength in the first round as he connected with two right hooks.
The Welshman attempted to impose himself with combinations behind his jab, but the pattern of the fight was soon set as he struggled to land any meaningful shots.
Selby finally found his target in the third round with a good right that seemed to rock Montiel but the challenger recovered and began to make inroads.
The sixth round was particularly uncomfortable for the champion, who seemed to be bullied by his more experienced opponent and a cut developed above Selby's right eye.
Selby's spark reappeared in the eighth with his best shots of the middle rounds, and he was given further encouragement in the ninth as Montiel's work-rate dropped.
The Welshman's combinations finally landed in the 10th, but Montiel continued to succeed with single shots to ensure Selby had no room to relax.
A second wind for Montiel in the final two rounds kept his opponent occupied, and there were plenty of anxious faces in Selby's corner as the bell rang for the final time.
Those concerns were unfounded, though, as the three judges gave Selby a generous points victory for a performance that often lacked accuracy and control.
Lee Selby retained his IBF featherweight title with a unanimous points victory over Mexican Fernando Montiel in Phoenix, Arizona.