Baillie went to London at age of 18 to study law, but liked art more. Shortly after getting to London, he joined the army. He served at the Battle of Culloden the Battle of Minden and retired with the rank of Captain. Baillie was later made the Commissioner of Stamps, which he was until 1795.