Antibody affinity maturation enables adaptive immune responses to a wide range of pathogens. In some individuals broadly neutralizing antibodies develop to recognize rapidly mutating pathogens with extensive sequence diversity. Vaccine design for pathogens such as HIV-1 and influenza has therefore focused on recapitulating the natural affinity maturation process. Here, we determine structures of antibodies in complex with HIV-1 Envelope for all observed members and ancestral states of the broadly neutralizing HIV-1 V3-glycan targeting DH270 antibody clonal B cell lineage. These structures track the development of neutralization breadth from the unmutated common ancestor and define affinity maturation at high spatial resolution. By elucidating contacts mediated by key mutations at different stages of antibody development we identified sites on the epitope-paratope interface that are the focus of affinity optimization. Thus, our results identify bottlenecks on the path to natural affinity maturation and reveal solutions for these that will inform immunogen design aimed at eliciting a broadly neutralizing immune response by vaccination.