With the rapid growth in wireless communication and IoT technologies, Radio Frequency Identification (RFID) is applied to the Internet of Vehicles (IoV) to ensure the security of private data and the accuracy of identification and tracking. However, in traffic congestion scenarios, frequent mutual authentication increases the overall computing and communication overhead of the network. For this reason, in this work, we propose a lightweight RFID security fast authentication protocol for traffic congestion scenarios, designing an ownership transfer protocol to transfer access rights to vehicle tags in non-congestion scenarios. The edge server is used for authentication, and the elliptic curve cryptography (ECC) algorithm and the hash function are combined to ensure the security of vehicles’ private data. The Scyther tool is used for the formal analysis of the proposed scheme, and this analysis shows that the proposed scheme can resist typical attacks in mobile communication of the IoV. Experimental results show that, compared to other RFID authentication protocols, the calculation and communication overheads of the tags proposed in this work are reduced by 66.35% in congested scenarios and 66.67% in non-congested scenarios, while the lowest are reduced by 32.71% and 50%, respectively. The results of this study demonstrate a significant reduction in the computational and communication overhead of tags while ensuring security.