The difficulty arises mostly from the assumption that the ladder is rigid (i.e., maintains the same shape). Ladders seem rigid in everyday life. But being completely rigid requires that it can transfer force at infinite speed (i.e., when you push one end the other end must react immediately, otherwise the ladder will deform). This contradicts special relativity, which states that information can travel no faster than the speed of light (which is too fast for us to notice in real life, but is significant in the ladder scenario). So objects cannot be perfectly rigid under special relativity.