The son of Hugh Pollen, a dock master, Pollen was born in Ringsend, Dublin. Little is known about the early part of his life, but it is supposed that he grew up in Ireland and in the United States of America. However, his father was dock master of the Grand Canal Company at Ringsend in 1812, still held that office in 1832, and died in 1837 to be succeeded as dock master by Thomas Pollen. On some accounts, Pollen's father helped to build the United States Capitol.