The 26-year-old made 133 appearances for the New York Stadium side, and helped them win promotion to the Championship in 2014.
He previously played under Rovers boss Tony Mowbray at Middlesbrough.
Smallwood is the second player to join Blackburn this summer after Cardiff midfielder Peter Whittingham.
Find all the latest football transfers on our dedicated page.
Blackburn Rovers have signed Rotherham midfielder Richie Smallwood on a two-year deal after he turned down a new contract with the Millers.