Many of the delays that passengers on Virgin Rail suffer are down to Network Rail, he said.
"Network Rail is far too big a company," he said. "I think that companies that kind of size should be broken up into small units."
Network Rail said decisions on its future would be taken by the government.
Sir Richard added that "ideally" the train operators should manage the track they use.
"We get enormously frustrated that people say will say that Virgin Rail has delays, but 90% of those delays are down to Network Rail," he told the BBC.
"If we were running the track underneath, because we've got our trains running on that track we'd make absolutely certain that track was fixed and running well, because we value our reputation."
When asked if a break-up of Network Rail was likely, he said: "I don't know; I've thrown that idea out before. Maybe right now it's an option that might be considered."
A spokesman for Network Rail described Sir Richard's figures as "incorrect", adding: "Over the last year (to 30 May), infrastructure faults were responsible for around a third (36%) of delays to Virgin Trains services."
On Thursday, the government said it would delay or cut back a number of modernisation projects planned for Network Rail.
Transport Secretary Patrick McLoughlin said rising costs and missed targets made an existing Â£38.5bn investment plan untenable,
Mr McLoughlin said Network Rail should have foreseen the improvements would cost more and take longer.
Network Rail said the plan, which was launched last year as the "largest modernisation of the railways since Victorian times", was too ambitious.
Under the changes, the government said electrification work on the Midland main line and on the Trans-Pennine route between Leeds and Manchester would be "paused".
Network Rail is too big and should be broken up into regional units, Sir Richard Branson has told the BBC.