Fenech was born in St Peters, Sydney. After playing junior rugby league and getting into trouble with the police as a child, Fenech was turned onto boxing when he attended the Newtown Police Boys Club in Sydney where he met Johnny Lewis. From there Fenech had a stellar amateur career, which led to him representing his country at the 1984 Summer Olympics in Los Angeles where he was selected as the boxing team captain. At the Olympics, Fenech lost a controversial quarter-final bout to Yugoslavian Redžep Redžepovski. Fenech was initially given the decision, but after intervention by the Olympic Boxing Committee and a total recount, the decision was reversed with Redžepovski being awarded the win. Many of the other boxers and those in the press felt that Fenech had been robbed of a chance to win an Olympic medal and most boxing writers noted how political amateur boxing was, especially at the Olympic Games.