Ward was born Windsor, New South Wales in 1835. His father, Michael Ward, had been sent to Australia as a convict in 1815. Fred was one of eleven children. He was arrested in 1856 for selling 50 stolen horses, and sent to prison on Cockatoo Island for ten years. After four years he was given a ticket of leave which meant he was able to leave the gaol as long as he reported to the police every three months.