Brown's most notable action in the House was a speech he made on February 4, 1875, in response to Massachusetts Representative Benjamin F. Butler's call to pass the Civil Rights Act of 1875. Referring to comments Butler had made the previous day about lawlessness against African-Americans in the South, Brown claimed that unjust charges had been made against Southerners by an individual "who is outlawed in his own home by respectable society, whose name is synonymous with falsehood, who is the champion, and has been on all occasions, of fraud; who is the apologist of thieves, who is such a prodigy of vice and meanness that to describe him would sicken the imagination and exhaust invective." Brown continued by referencing notorious Scottish murderer William Burke, whose method of murdering his victims became known as "Burking." At this point in the speech, Speaker of the House James G. Blaine interrupted Brown, asking if he was referring to a member of the House; Brown gave an ambiguous response before continuing: "If I wished to describe all that was pusillanimous in war, inhuman in peace, forbidden in morals, and infamous in politics, I should call it 'Butlerizing'." The House gallery exploded in protest at Brown's remark, and incensed Republican legislators called for Brown's immediate expulsion. Though not expelled, he was officially censured by the House for the use of unparliamentary language. The censure was expunged from the record by a subsequent Congress.