The staff needs to make contact with skin (as vsz says, skin near the heart) to affect the person.
Tony's skin wasn't touched, it hit the reactor instead.
While the magic can penetrate a thin layer of clothing, apparently the metal of the reactor was sufficient shielding.
scifi.stackexchange.com/users/656/Jeff
The reason Loki's staff will not work on Tony has nothing to do with the ARC reactor technology per say, and everything to do with the core element of the 4th generation reactor, Vibranium .
One of the main abilities of the fictional element is absorbtion of energy directed at it.
This is also why Thor's mystical lightning bolt overcharges Tony's reactor to "Over 400%"
It is stated in the novelization of Iron Man II that the metal is indeed Vibranium.
scifi.stackexchange.com/users/8226/Monty129
From the comments Loki made when converting previous victims, it seems he has to poke very close to the heart with his staff .
Tony Stark just had his arc reactor in the way.
In all previous "conversions" Loki touched his victims on the chest, very close to the heart, so it seems that this was necessary, and it was not enough to touch the victim's skin on their hand, for example.
From the metallic clack, Loki could assume Tony has something there (e.g. a cuirass), so if it weren't necessary to "touch the heart" he could have instantly touched Tony elsewhere (on his arm, on his face, etc.), instead of trying once or twice again and then giving up.
scifi.stackexchange.com/users/4004/vsz
When he takes over Hawkeye at the beginning of the film, Loki says Barton "has heart".
When it doesn't work on Tony Stark the director is poking fun that Stark is 'heartless' in the emotional sense that he is extremely vain or self interested.
Although the movie shows the staff always touching close to the physical heart of the victim, the only time Loki explains its workings is in regard to Hawkeye, speaking of a metaphorical 'heart' vs a physical one.
We can confirm this through the reasoning that the arc reactor is in the center of Stark's chest where the physical heart lies to the left, Loki would have the sense to tap the second time on the left of Stark's chest if near physical contact with the heart is required.
scifi.stackexchange.com/users/9008/Gorden
I believe it was because of the energy field produced by the arc reactor in Tony's chest.
The metal had nothing to do with it.
It just interfered with the manipulation of energy that occurs when Loki uses the staff to augment his mind control.
scifi.stackexchange.com/users/9017/Terence
I have something to add, as I went back to watch all the other movies and something came to me.
If you look closely at the book Tony is reading in Iron Man 2 , you will see the Tesseract
and I thought the new element his father was talking about might have been the Tesseract.
I think that he was able to create his own Tesseract for his heart and suit.
That is why the staff didn't work on him.
scifi.stackexchange.com/users/10853/Smackey
In the Avengers movie, why didn't Loki's staff work on Tony Stark?  My theory is that it had something to do with the Arc Reactor, given the 'clink' sound it made when Loki tapped Tony on the chest.
Why didn't Loki's staff work on Tony Stark?
scifi.stackexchange.com
<marvel><marvel-cinematic-universe><avengers><the-avengers-2012>
scifi.stackexchange.com/questions/20276
scifi.stackexchange.com/users/4650/Wayne Werner
In all previous "conversions", Loki touched his victims on the chest, as it seems that he has to poke very close to the skin near the heart with his staff. From the metallic clack, Loki could assume Tony has something there (the arc reactor). However, some people feel that this may be due to the energy from the arc reactor in Tony's chest, or even due to the core element of the 4th generation reactor, Vibranium. Skin contact close to the heart is needed for the staff to work.
Skin contact close to the heart is needed for the staff to work.