King John of Bohemia pledged the castle to nobleman Jan of Vartemberk in 1311, but his son, Margrave of Moravia Charles (later Holy Roman Emperor Charles IV), received the property as a debt settlement in 1335. Charles´ younger brother, Margrave Jan Jindřich (John Henry), then took a fancy to Veveří Castle. He was responsible for the building of its rear part with two towers and an outer ward. In the central area around the keep, he developed the main palace, which included a large hall and the Chapel of St. Procopius (later of St. Wenceslaus). The present appearance of the compound is the result of these building activities, giving the castle its basic silhouette of a medieval fortress.