Weir was born on August 30, 1852, the second to last of sixteen children, and raised in West Point, New York. His father was painter Robert Walter Weir, a professor of drawing at the Military Academy at West Point who taught such artists as James Abbott McNeill Whistler. His older brother, John Ferguson Weir, also became a well-known landscape artist who painted in the styles of the Hudson River and Barbizon schools. He was professor of painting and design at Yale University from 1869, starting the first academic art program on an American campus.