Woodfox was part of the group known as the "Angola Three", after the prison where they spent years in confinement.
He had been there since April 1972 for the murder of a prison guard.
Maintaining his innocence in the death of Brent Miller, Woodfox, 69, was finally freed after accepting the lesser charge of manslaughter.
The plea bargain was negotiated with state prosecutors.
Twice in decades of legal battles, his murder conviction was thrown out of court, yet Louisiana state prosecutors were preparing to try him a third time.
He finally agreed to plead no contest to lesser charges in exchange for freedom but insisted this was not an admission of guilt.
"Although I was looking forward to proving my innocence at a new trial, concerns about my health and my age have caused me to resolve this case now and obtain my release with this no contest plea to lesser charges," he said in a statement on Friday.
"I hope the events of today will bring closure to many."
How do you survive solitary confinement?
Before driving off from Feliciana Parish Detention Center, in St Francisville, with his brother, Woodfox told media he wanted to visit his mother's gravesite.
She died while he was in prison and Woodfox said he had not been allowed to go to the funeral.
At the time of Mr Miller's death, Woodfox was in Louisiana's notorious Angola prison for armed robbery and assault.
He was accused of grabbing the guard from behind while others stabbed him with a lawnmower blade and a hand-sharpened prison knife.
Woodfox was placed in solitary and ordered to be kept on "extended lockdown" every 90 days for decades.
His lawyers say he was confined to his cell for 23 hours a day.
The other members of the "Angola Three", Robert King and Herman Wallace, were released in 2001 and 2013 respectively.
Wallace, also convicted over Mr Miller's murder, died soon after his release pending a new trial. King's conviction was overturned.
Woodfox and Wallace were involved with the Black Panthers, a militant black rights movement formed in 1966 for self-defence against police brutality and racism.
The longest-serving prisoner to be held in solitary confinement in US history, Albert Woodfox, has walked free in Louisiana after 43 years.