The National Trust wants to introduce charges at Pont ar Daf car park near Storey Arms on the main A470.
The trust owns and manages the central Beacons, and says it wants to build a visitor centre and improve the car park's safety and security.
Ramblers Cymru criticised the move, saying walking should be free and accessible for everyone.
The trust, which spends more than Â£100,000 a year on the upkeep of Pen y Fan and other parts of the Brecon Beacons, is to submit a planning application later this year.
Pont ar Daf car park was created in the 1960s after road improvements left a section of the A470 isolated from the main carriageway.
'Brew and a view'
It was soon adopted by walkers because it offered easy access to Pen y Fan.
The mountain stands at 886m (2,907ft) and is the highest peak in southern Britain.
"If you arrive at the car park you have access to the three most important things, a loo, a brew and a view," said National Trust property manager Philip Park.
"If we are successful in gaining permission the offer will remain essentially the same, but it will be designed to meet the higher expectations of visitors to the area nowadays.
"We believe this development can really improve the enjoyment that visitors can experience in this wonderful part of Wales."
Recovering addicts
Mr Park said the trust's initial plans had gone out to consultation.
But director Angela Charlton said: "Ramblers Cymru believes walking should be free and accessible to all and we are disappointed that the National Trust feel they have to start charging for parking.
"People have been freely walking Pen y Fan for many years.
"Ramblers Cymru works with groups such as Turning Point who use Pen y Fan for inspirational training for recovering addicts.
"We hope this will not have a detrimental effect."
Walkers could be charged to visit Pen y Fan, the highest peak in the Brecon Beacons.