McCay became the sixth Minister for Defence in four years. His predecessor, Senator Anderson Dawson, had chaired a committee that had produced a detailed report recommending the abolition of the post of General Officer Commanding Australian Military Forces and the creation of a Council of Defence, a Naval Board and a Military Board. It fell to McCay to implement the report's recommendations and create a five-man Military Board consisting of himself, a finance member and three military officers. McCay preferred the senior member not be styled the Chief of the General Staff. This change would not be made until 1909. At the first meeting of the Council of Defence, McCay rejected the arguments of Captain William Rooke Creswell for the majority of the defence budget to be spent on supporting the British fleet. In 1905 the Reid government collapsed and McCay became a backbencher once more. Since the Federal parliament sat in Parliament House, Melbourne, McCay lived at the Stock Exchange Club in Collins Street, Melbourne while his family remained in Castlemaine. He maintained a liaison with a married woman, Ella Gavan Duffy.