Allen spent the next several years involved in Vermont's political and military matters. While his family remained in Arlington, he spent most of his time either in Bennington or on the road, where he could avoid his wife's nagging. Shortly after his arrival, Vermont's Assembly passed the Banishment Act, a sweeping measure allowing for the confiscation and auction by the republic of property owned by known Tories. Allen was appointed to be one of the judges responsible for deciding whose property was subject to seizure under the law. (This law was so successful at collecting revenue that Vermont did not impose any taxes until 1781.) Allen personally escorted some of those convicted under the law to Albany, where he turned them over to General John Stark for transportation to the British lines. Some of these supposed Tories protested to New York Governor George Clinton that they were actually dispossessed Yorkers. Clinton, who considered Vermont to still be a part of New York, did not want to honor the actions of the Vermont tribunals; Stark, who had custody of the men, disagreed with Clinton. Eventually the dispute made its way to George Washington, who essentially agreed with Stark since he desperately needed the general's services. The prisoners were eventually transported to West Point, where they remained in "easy imprisonment".