Loxton was born on 29 March 1921 at Albert Park, Victoria, the son of Sam Sr. and Annie. The elder Sam Loxton was an electrician who played second grade cricket for Collingwood. The younger Sam started his education at Yarra Park State School, where he learned to bat, using a pine tree in the schoolyard as the stumps; the same tree was used for the same purpose years earlier by Test players Vernon Ransford and Ernie McCormick, and long-serving Victorian batsman Jack Ledward. The family moved to Armadale, and young Loxton attended Armadale Public School before completing his secondary education at Wesley College, Melbourne, an elite private boys’ school. One of his colleagues at Wesley College was Ian Johnson, a future teammate for Victoria and Australia. The boys’ school coach was P. L. Williams, a renowned mentor of teenagers who had earlier coached Ross Gregory and future Test captain Lindsay Hassett. Away from his sporting commitments at school, Loxton played district cricket for Prahran's third grade team when he was just 12. The young cricketer's parents were stalwarts of the club; Sam Sr. was the scorer and served as a transport man, driving the matting and equipment to matches, while Annie made cucumber sandwiches for 25 years—due to the economic difficulties caused by the Great Depression and World War II, meat catering for players was a luxury even at first-class and international matches. The elder Sam was a member of the club committee from 1941 until his death in 1974, and was a vice-president for the last 17 years of his life. At the age of 16, the younger Loxton was selected in the Victorian Cricket Association Colts team that played in the first grade competition in 1937–38; he played three seasons with the outfit, which was effectively a state youth team. The squad was coached by Bert Cohen and former Test batsman and captain Jack Ryder, and Loxton credited the latter as the biggest influence on his career, saying