The decision has been made by experts at Oxford City Council and the university after the large branches fell from the tree on Saturday.
Dr Alison Foster, of the Botanic Garden, said: "It's a tree like no other - it's just heart-breaking."
The tree was a favourite of JRR Tolkien during his time in Oxford.
Ms Foster said staff heard noises coming from the tree and moved visitors away from the area of the walled garden shortly before the branches fell.
She said: "A crack appeared and then in about five minutes the branches came down.
"It's really hard to say what the cause was - it's something that just happens in old trees - there are suggestions that prolonged hot, dry weather can lead to this kind of thing."
An area around the tree has been closed to the public while the tree is felled, a process which is expected to take several weeks.
The Botanic Garden said the tree was thought to have been planted in 1799 from a seed collected by the third Sherardian Professor of Botany, John Sibthorp, in Austria.
The garden said it intended to propagate from the black pine.
Tolkien wrote The Hobbit and The Lord Of The Rings while living and working in Oxford.
A 215-year-old black pine known as "Tolkien's tree" in Oxford University's Botanic Garden is to be felled after two of its branches came down.