His belief in limited government conflicted with many of his colleagues, who expected to trade for votes on various issues and projects, something he refused to do. In the era of civil rights, he believed that solutions lay with local, not national government. He maintained that the national government should concentrate on defense and foreign affairs. He believed that the responsibility for social programs belonged at the local level. He was the only member of the House, for example, to oppose the popular school lunch program. Despite being in the majority of the Texan delegation to decline to sign the 1956 Southern Manifesto opposing the desegregation of public schools ordered by the Supreme Court in "Brown v. Board of Education", Alger voted against the Civil Rights Acts of 1957, 1960, and 1964, as well as the 24th Amendment to the U.S. Constitution.