Writing in the Sunday Times, Mr Javid said people could not play a "positive role" in public life unless they accepted basic values.
These included democracy, equality and freedom of speech, he said.
Mr Javid's intervention comes after a report by Dame Louise Casey warned of increasing ethnic segregation.
Mr Javid's proposals would mean every new recruit in the public sector, including councillors, school governors and civil servants would be expected to commit to the oath, which may have to be read out loud before starting the role.
This could extend to those working in the NHS and the BBC.
Dame Louise said some sections of society did not accept British values such as tolerance.
Mr Javid said he was "drawn" to Dame Louise's recommendation to bring in an oath of allegiance.
"If we are going to challenge such attitudes, civic and political leaders have to lead by example," he said.
"We can't expect new arrivals to embrace British values if those of us who are already here don't do so ourselves, and such an oath would go a long way to making that happen."
Mr Javid said he did not want to see a "government-approved, one-size-fits-all identity" where everyone "drinks tea, watches cricket and bobs up and down at the Last Night of the Proms".
But, he added, people would struggle to play a positive role in British life if they did not accept the "building blocks of our society".
Mr Javid recalled how, aged about eight, he went to see a GP with his mother to act as her interpreter.
He said while today she is completely fluent in English, a minority of immigrants have failed to make such an "effort".
The new oath could include "tolerating the views of others even if you disagree with them", "believing in freedom of speech, freedom of religion, freedom from abuse", "a belief in equality, democracy and the democratic process" and "respect for the law, even if you think the law is an ass," Mr Javid writes.
A government source acknowledged that the oath on its own would be insufficient to weed out extremism or promote integration.
Mr Javid will set out his full response to Dame Louise's report on social cohesion in the spring.
Civil servants and other holders of public office should swear an oath to British values, Communities Secretary Sajid Javid has said.