John O'Shea said it could help the Welsh Government's drive to have a million Welsh speakers by 2050 if more post-16 education opportunities in Welsh were available and adult courses.
Ministers announced their plans and a consultation at last week's National Eisteddfod in Abergavenny.
"We need to start thinking of different ideas," said Mr O'Shea.
"We have Welsh medium schools but we're not getting that breakthrough in take up of the language."
The 2011 census reported a drop in the number of Welsh speakers from 582,000 in 2001 to 562,000, about one in five of the population.
Mr O'Shea said: "If we don't have parity for the Welsh language then the chances of the Welsh language developing and growing are reduced."
He claims vocational courses or A levels for pupils aged 16 and over who want to study through the medium of Welsh are not as varied when compared to the range of subjects available to students who study through the medium of English.
He added: "In the first case, we would need to find a site in the middle of south east Wales where you could bring all of the Welsh-speaking staff together and all the Welsh speaking-students together in a tertiary college because, at the moment, there aren't enough numbers to set up a number of these colleges.
"As it succeeds then, of course, you could diversify and have more colleges in more local locations."
Iestyn Davies, the chief executive of Colleges Wales, which represents further education colleges, welcomed the idea.
He said reaching the Welsh-speakers target would be a challenge and different options should be considered.
In a statement, the Welsh Government said: "We want everyone to have the opportunity to learn through the medium of Welsh from early years to higher education and this should be an integral part of the general provision instead of something separate.
"The million speakers consultation is an opportunity for everyone to have their say on the future of what could be claimed the best asset of the country.
"We are confident that this will enable the language to develop in a proactive and balanced way."
The principal of Merthyr Tydfil College has called for a Welsh language college to be established in the region.