Padamsee's father, Hassan Padamsee, was an affluent businessman who owned 10 buildings and also ran a glassware and furniture business. His mother, Jenabhai Padamsee, was a home-maker. Akbar Padamsee was one of their eight children; one of his brothers is the actor Alyque Padamsee. Although rich, the family was not well-educated, and neither of his parents had received much education. Alyque and his brothers (but not his sisters) were the first to attend school and learn English there; the parents later picked up a smattering of the language from their sons.