When Victor was three years old, his father taught him to make pro-union speeches and would take his son to rallies and union meetings and have the boy recite the speeches for onlookers. Attending union meetings, indoor and outdoor rallies, and standing on street corners promoting the union formed many of Victor Riesel's childhood and teenage memories. In the 1920s and 1930s, Nathan Riesel successfully opposed Communist Party USA attempts to infiltrate activists into the local union and turning its purpose to promotion of the party (a strategy known as "boring from within"). Throughout his childhood and teenage years, he saw his father come home bleeding many times after fistfights with communist activists or gangsters. This conflict left a deep impression on Victor.