Brown was ordained an apostle and became a member of the Quorum of the Twelve Apostles on April 10, 1958, replacing Adam S. Bennion, who had died the previous February. He was called to the First Presidency as a third counselor to church president David O. McKay on June 22, 1961. He was called as Second Counselor in the First Presidency on October 12, 1961, upon the death of First Counselor J. Reuben Clark. He was later called as First Counselor in 1963 when the First Counselor, Henry D. Moyle, died.