Ellis played for the school's basketball team, recording 21 assists in one game. He played baseball as an infielder for a local semi-professional team called the Pittsburgh Pirates Rookies, along with future major leaguers Willie Crawford, Bill Rohr, Tom Harrison, Bobby Tolan, Roy White, Ron Woods, Reggie Smith, Don Wilson, Bob Watson and Dave Nelson. The team was managed by Chet Brewer. However, Ellis refused to play for the Gardena High School baseball team because a baseball player referred to him as a "spearchucker".