John Rooney, 46, attacked the child, who cannot be named for legal reasons, at an address in the Borders between April 2009 and April 2010.
He was convicted of attempted rape following a trial last month at the High Court in Glasgow.
Judge Lord Stewart told Rooney, of Deanfield Place, that he had no option but to impose a prison term.
Rooney's advocate Victoria Dow told the High Court in Edinburgh that her client was a first offender who continued to deny attacking the girl.
She added: "He, however, accepts the verdict of the jury."
Lord Stewart told Rooney there was no alternative to a custodial sentence and jailed him for four years.
A Kelso man who sexually assaulted an 11-year-old girl has been jailed for four years.