The tribe trace their descent from a Sardar Kallar Shah, son of Sultan Sujann Shah, who is believed to have come from southern Afghanistan. It is believed that he was a descendant of Alexander the Great. In Haripur and Abbottabad, they are known as Sultan Chief Sardars, a name they acquired during the time of the Delhi Sultanate (1206–1296), when the tribal members were made rulers of the galliyat region of Hazara. There is also some argument as to the religion of the Karlal, as whether they were Hindu or not. As some of the oldest inhabitants of Hazara, a region that was largely Hindu till the early 13th Century, it is likely that they were either Hindus, or strongly influenced by Hinduism.