The 28-year-old left Southend at the end of the season following 164 appearances across six years, having finished the campaign on loan at League Two champions Northampton Town.
He also won promotion out of League Two in 2014-15 with the Shrimpers.
Prosser is the second U's signing of the summer, after the acquisition of Maldon & Tiptree left-back Ben Wyatt.
The pair join a club that had the worst defence in the Football League, conceding 99 goals as they were relegated out of League One.
But Prosser told BBC Essex: "Colchester is a club in transition, I think. A few people have said that it might be the best thing that's happened to the club, to come down, rebuild and go again.
"Southend did it when I first went there. Colchester is a massive club, when you look at the facilities and stadium it's a League One, if not Championship club."
Find all the latest football transfers on our dedicated page.
Colchester United have signed defender Luke Prosser on a two-year deal after his release by Essex rivals Southend.