Penny was one of the founding members of the local division of the Sons of Temperance group (Star of Moorabbin Division No29.), a friendly society aimed to assist members and their families at times of unemployment, illness and death. He served in many roles in the group including starting off as Assistant Conductor but also taking on the roles of Worthy Patriarch, Recording Scribe and Conductor. The Sons of Temperance was absorbed into the Independent Order of Oddfellows (IOOF) which subsequently became part of NIB Health Funds. Whilst unable to drink alcohol as a member of the Temperance Society, Penny did actually get in trouble once for doing so. Whilst not a drinker, Penny and his family regularly attended the Exchange Hotel (now the Tudor inn) on the Point Nepean Road for meals and meetings, creating a pathway through the area. When the railway line was built in 1881, the Penny's kept using the same pathways, creating a crossing over the line at what is now Heather Grove. When Vic Rail formalised all crossings in the 1950s, this crossing point was retained as it had grown in usage thanks largely to the walking habits of the Penny family "(Graham Whitehead. Article 647 City of Kingston Historical Website) (Penny family history)"