Broyhill also became known as a strident opponent of integration. In 1955, he was one 81 US Representatives who vowed to oppose by "every lawful means", the U.S. Supreme Court holding in "Brown v. Board of Education" which outlawed segregation. He and Richard Harding Poff of Virginia were the only two Republicans to sign the Southern Manifesto. Broyhill voted against the Civil Rights Acts of 1957, 1960, 1964, and 1968, and the Voting Rights Act of 1965, but voted in favor of the 24th Amendment to the U.S. Constitution. As a longtime member of the committee overseeing the District of Columbia he, along with three other members of Congress, recommended that schools in the District reinstitute segregation.