Neighboring Loudon County was home to several small pockets of Union supporters, including Quakers who lived in villages like Waterford and Lincoln who did not support secession or the Confederate cause for defending the institution of slavery. Point of Rocks became a haven for those families who were forced to flee Virginia. In 1862, Captain Samuel C. Means, a native of Waterford, Virginia, but then living in Point of Rocks where he was a merchant and the B&O Railroad station manager, raised a cavalry unit called the Loudoun Rangers, the only organized unit from Virginia to fight for the Union. The Loudoun Rangers spent most of 1862 and 1863 fighting alongside Cole's Maryland Cavalry (the First Potomac Home Brigade) to protect the C&O Canal and the B&O Railroad from frequent Confederate raids. Cole's Maryland Cavalry encamped at Point of Rocks, occupying St. Paul's Episcopal Church where they burned the interior furnishings.