Jim's mother died when he was quite young, leaving Jim's father (later played by Victor Buono) to raise the family. (Jim states while applying for his taxi driver's license that his father's name was Ignatowski—another example of his faulty memory.) Being busy with his thriving urology practices, Dr. Caldwell left much of this task to various family servants. The father's absence seemed to affect all the Caldwell children in some way: Jim was an extremely uptight and humorless person before he became a drop-out; his brother Tom remained humorless and somewhat mean-spirited throughout his adulthood; and his sister Lila exhibited marked nymphomania.