Through the surrounding town of Lyme, NY 12E reverts to a two-lane rural highway as it proceeds west into the hamlet of Herrick Grove, located from Chaumont. Here, the highway heads along a short commercial strip and intersects with the southern terminus of CR 5 (Depot and Church streets) off the shore of Chaumont Bay. After leaving Herrick Grove, NY 12E enters the nearby hamlet of Three Mile Bay, which comprises a few homes and a junction with CR 57 (Carrying Road). Past Three Mile Bay, the route makes a gradual bend to the northwest, leaving Lyme for the town of Cape Vincent, where NY 12E intersects with Bedford Corners Road (former CR 56) roughly from the town line. Heading northwest through the rural town, the route crosses over Kents Creek and intersects with CR 4 on its way into the village of Cape Vincent, located at the point where Lake Ontario meets the St. Lawrence River.