Brown was born in San Francisco, California, one of four children of Ida (née Schuckman) and Edmund Joseph Brown. His father came from an Irish Catholic family, with his grandfather Joseph immigrating from County Tipperary, Ireland. His mother was from a German Protestant family. He acquired the nickname "Pat" during his school years; the nickname was a reference to his Patrick Henry–like oratory. When he was 12 and selling Liberty Bonds on street corners, he would end his spiel with, "Give me liberty, or give me death."