In 1912, at the age of 42, Downs decided to retire in Iowa. After moving back to Marshalltown, he rarely left the area. Not quite comfortable with idle retirement, Downs kept busy by opening a vaudeville house on Main Street, a few blocks from his own home at 7 South 3rd Avenue. He also sold a line of magic equipment. Many visitors, other magicians, streamed to his home to share gossip and the latest tricks as well as to brush elbows with the famous sleight of hand artist. The most famous of these visitors included Chung Ling Soo (William Robinson) and Dai Vernon.