By the 1790s, the settlement at St. Asaph's developed into the town of Stanford. Whitley and his family built a large brick house outside town, near what would later become Crab Orchard, Kentucky. The plantation was named Sportsman's Hill. It was the first brick house built in Kentucky and still stands, preserved as the William Whitley House State Historic Site. The house includes a secret passage for escape and survival during raids by Native Americans. The plantation originally included a racetrack. This racetrack set several traditions for horse racing in the United States. It had the first clay (instead of turf) track in the United States and here horses were raced counterclockwise (instead of clockwise, as was the British tradition).