Roy Choudhury was born in Patna, Bihar, India, into the Sabarna Roy Choudhury clan, which owned the villages that became Kolkata. He grew up in Patna's Imlitala ghetto, which was mainly inhabited by Dalit Hindus and Shia Muslims. His was the only Bengali family. His father, Ranjit (1909–1991) was a photographer in Patna; his mother, Amita (1916–1982), was from a progressive family of the 19th-century Bengali Renaissance. His grandfather, Laksmikanta Roy Choudhury, was a photographer in Kolkata who had been trained by Rudyard Kipling's father, the curator of the Lahore Museum.