Every cell biological textbook teaches us that the main role of the plasma membrane is to separate cells from their neighborhood to allow for a controlled composition of the intracellular space. The mostly hydrophobic nature of the cell membrane presents an impenetrable barrier for most hydrophilic molecules larger than 1 kDa. On the other hand, cell-penetrating peptides (CPPs) are capable of traversing this barrier without compromising membrane integrity, and they can do so on their own or coupled to cargos. Coupling biologically and medically relevant cargos to CPPs holds great promise of delivering membrane-impermeable drugs into cells. If the cargo is able to interact with certain cell types, uptake of the CPP–drug complex can be tailored to be cell-type-specific. Besides outlining the major membrane penetration pathways of CPPs, this review is aimed at deciphering how properties of the membrane influence the uptake mechanisms of CPPs. By summarizing an extensive body of experimental evidence, we argue that a more ordered, less flexible membrane structure, often present in the very diseases planned to be treated with CPPs, decreases their cellular uptake. These correlations are not only relevant for understanding the cellular biology of CPPs, but also for rationally improving their value in translational or clinical applications.