When using a roll pin punch, make sure the axis of the shank of the roll pin punch is in line with the axis of the roll pin. Do not cant the roll pin punch off to one side.
To effectively use a roll pin punch, align its shank with the roll pin's axis. Avoid any deviation or tilting of the punch to one side.