Morley was born in Chester, Pennsylvania, the eldest of six children. His father, Colonel Benjamin F. Morley, was at the time vice-president and professor of chemistry, mathematics and tactics at Pennsylvania Military College (PMC). His mother Sarah also had a connection with the college, where her father Felix de Lannoy had been a professor of Modern Languages. Felix (Sylvanus' maternal grandfather) was an immigrant to the United States from newly independent Belgium, where his father had been a judge in the Belgian Supreme Court.