The Norman conquest of England brought a major shift in land ownership. The Penn area, along with a wide tract of Mercia, was assigned by William the Conqueror to Ansculf de Picquigny, who built a motte and bailey fortress at Dudley. By 1087, the time of Domesday Book, the Penn area belonged to Ansculf's son, William Fitz-Ansculf. He had installed in Upper Penn a tenant called Robert, who also held lands from him in Bushbury, Ettingshall, Moseley and Oxley. The settlement had eight villagers and two smallholders. In Lower Penn, which had only 6 villagers with one freeman, he had another tenant, called Gilbert. Clearly both places were still mere hamlets.