The ongoing COVID-19 pandemic has killed at least 1.1 million people in the United States and over 6.7 million globally. Accurately estimating the age-specific infection fatality rate (IFR) of SARS-CoV-2 for different populations is crucial for assessing and understanding the impact of COVID-19 and for appropriately allocating vaccines and treatments to at-risk groups. We estimated age-specific IFRs of wild-type SARS-CoV-2 using published seroprevalence, case, and death data from New York City (NYC) from March to May 2020, using a Bayesian framework that accounted for delays between key epidemiological events. IFRs increased 3-4-fold with every 20 years of age, from 0.06% in individuals between 18–45 years old to 4.7% in individuals over 75. We then compared IFRs in NYC to several city- and country-wide estimates including England, Switzerland (Geneva), Sweden (Stockholm), Belgium, Mexico, and Brazil, as well as a global estimate. IFRs in NYC were higher for individuals younger than 65 years old than most other populations, but similar for older individuals. IFRs for age groups less than 65 decreased with income and increased with income inequality measured using the Gini index. These results demonstrate that the age-specific fatality of COVID-19 differs among developed countries and raises questions about factors underlying these differences, including underlying health conditions and healthcare access.