According to the United States Census Bureau, the town of Brighton has a total area of , of which is land and , or 2.76%, is water. Island Pond, originally called "Knowlton Pond" in the 19th century to honor Luke Knowlton, who surveyed and laid out the town, is Brighton's largest body of water. The Clyde River flows out of the north end of Island Pond and proceeds in a northwesterly direction towards Charleston. Spectacle Pond is a kettle lake just east (upstream) of Island Pond. It is only deep, with a thick bottom of muck. Most of the town, including Island and Spectacle ponds and the Clyde River, is part of the Lake Memphremagog watershed, and thus part of the Saint Lawrence River basin, via the Magog and Saint-François rivers of Quebec. The east side of the town drains to the Connecticut River, mostly to the east via the Nulhegan River, but a small portion to the south via the East Branch of the Passumpsic River. The highest point in Brighton is the southern summit of Bluff Mountain, in the northern part of the town.