Two days after his birth, Palmer was adopted by Moe Wiesen and his wife Polly, a wealthy Manhattan dress designer and a boutique owner respectively, who lived on Park Avenue. His sister Bonnie was also adopted by the Wiesens. The family's butler taught the young Jim to throw a baseball at Central Park. After his adoptive father died of a heart attack in 1955, the nine-year-old Jim, his mother and his sister moved to Beverly Hills, California where he began playing in youth-league baseball. In 1956, his mother married actor Max Palmer, but Jim continued to go under the name Jim Wiesen until a year later. At a Little League banquet, just before being presented with an award, he asked the coaches to identify him as "James Alvin Palmer." "Through all these years, that night was the highlight of my entire life," Max recalled. Max was a character actor and there were two men who shared that name who worked in show business during similar time periods. The Max who was Jim's second dad worked mostly on TV on such programs as "Dragnet", "Bat Masterson" and "The Colgate Comedy Hour". He was Jewish, and he also earned a living by selling shoes. The other Max Palmer, often erroneously credited as Jim's father, worked in several movies as a monster. He was 8'2" tall and later became a professional wrestler and eventually a Christian evangelist.