The ability of human immune system to generate antibodies to any given antigen can be strongly influenced by immunoglobulin V gene (IGV) allelic polymorphisms. However, previous studies have provided only a limited number of examples. Therefore, the prevalence of this phenomenon has been unclear. By analyzing >1,000 publicly available antibody-antigen structures, we show that many IGV allelic polymorphisms in antibody paratopes are determinants for antibody binding activity. Biolayer interferometry experiment further demonstrates that paratope allelic mutations on both heavy and light chain often abolish antibody binding. We also illustrate the importance of minor IGV allelic variants with low frequency in several broadly neutralizing antibodies to SARSCoV-2 and influenza virus. Overall, this study not only highlights the pervasive impact of IGV allelic polymorphisms on antibody binding, but also provides mechanistic insights into the variability of antibody repertoires across individuals, which in turn have important implications for vaccine development and antibody discovery.