It comes ahead of peace talks between the left-wing group and the government, which begin on Tuesday in Ecuador.
The soldier, Freddy Moreno, has been handed over to delegates from the International Committee of the Red Cross in Arauca province.
"The ELN has stood by its word," the rebel group tweeted.
The ELN talks follow successful negotiations between the government and Colombia's largest rebel group, the Farc, which took place in the Cuban capital, Havana, and lasted four years.
The head of the Red Cross in Colombia, Christoph Harnisch, said the gesture would "reinforce trust between the Colombian government and the ELN before the imminent opening of the public phase of peace negotiations".
The talks were due to begin at the end of October. But they were delayed as the government refused to go sit down for formal negotiations while the rebels still held Odin Sanchez, a former congressman.
Mr Sanchez was released last Thursday, after 10 months in captivity.
The rebels had demanded that in exchange the government pardoned two of its members serving time in Colombian jails.
The two sides struck a deal and the two ELN members were released on Saturday.
How significant is Colombia's ELN rebel group?
The two who have been released are expected to serve as rebel negotiators at the peace talks in the Ecuadorean capital, Quito.
The government reached a peace agreement with Colombia's largest rebel group, the Farc, last year.
Members of the Farc have been gathering in "transition zones", where they are to demobilise and lay down their weapons under the supervision of United Nations monitors.
The last of the Farc rebels are expected to reach the designated demobilisation areas by Wednesday, government officials said.
Colombia's second-largest rebel group, the National Liberation Army (ELN), has released a soldier it had been holding hostage for two weeks.