The 22-year-old Scotland Under-21 international was a free agent after leaving Scottish Championship side St Mirren earlier this summer.
Kelly, who can play at centre-back or left-back, becomes Wimbledon's sixth signing of the summer transfer window.
"We wanted someone young who we could develop," Dons boss Neal Ardley told the club website.
The south-west London club have not disclosed the length of Kelly's contract at Kingsmeadow.
Find all the latest football transfers on our dedicated page.
League One newcomers AFC Wimbledon have signed defender Sean Kelly following a successful trial spell.