Liverpudlian Smith, 30, grew into the fight after a shaky start but was knocked down in the ninth round.
The three judges scored the bout 117-110 116-111 116-111 in favour of the undefeated Puerto Rican in Connecticut.
It was the second defeat for Smith in his 25-fight career.
WBO light-middleweight champion Liam was in the United States to support his brother, who had been hoping to become the 13th current British world champion.
Stephen Smith failed to join brother Liam as a world champion as he lost a unanimous points decision to defending champion Jose Pedraza in their IBF super-featherweight title bout.