He was baptized as "Phillipus Ludovicus Brion", son of the merchant Pierre Louis Brion and Marie Detrox, both from what is now Belgium. They arrived in Curaçao in 1777. In 1794 they sent their son to the Netherlands to complete his education. While he was there, he enlisted in the forces of the Batavian Republic to fight the British invasion of the northern Netherlands. He participated in the battles of Bergen (September 19, 1799) and Castricum (October 16, 1799). He was taken prisoner by the British but freed after a short time in the prisoner exchange under the Convention of Alkmaar.