Padamsee was born into a traditional Khoja Muslim family hailing from the Kutch region of Gujarat. Their ancestors had belonged to the "charan" caste of bards and court musicians. The family had been settled in the nearby Kathiawar region for some generations; Padamsee's grandfather, who had been the "sarpanch" (headman) of Vāghnagar, a village in Bhavnagar district, had earned the honorific name "Padamsee" (a corruption of "Padmashree") after he distributed his entire granary to the village during a famine. His original family name was "Charanyas", due to their ancestors being court poets.