Refugee communities are vulnerable to housing insecurity, which drives numerous health disparity outcomes in a historically marginalized population. The COVID-19 pandemic has only worsened the ongoing affordable housing crisis in the United States while continuing to highlight disparities in health outcomes across populations. We conducted interviewer-administered surveys with refugee and asylum seekers in San Diego County at the height of the COVID-19 pandemic to understand the social effects and drivers of COVID-19 in one of the largest refugee communities in the United States. Staff from a community-based refugee advocacy and research organization administered the surveys from September—November 2020. 544 respondents participated in the survey, which captured the diversity of the San Diego refugee community including East African (38%), Middle Eastern (35%), Afghan (17%), and Southeast Asian (11%) participants. Nearly two-thirds of respondents (65%) reported living in overcrowded conditions (>1 individual per room)and 30% in severely crowded conditions (>1.5 individuals per room). For each additional person per room, self-reported poor emotional health increased. Conversely, family size was associated with a lower likelihood of reporting poor emotional health. Crowded housing was significantly associated with a lower probability of accessing a COVID-19 diagnostic test, with every additional reported person per room there was approximately an 11% increase in the probability of having never accessed a COVID-19 testing. Access to affordable housing had the largest effect size and was associated with fewer people per room. Overcrowding housing is a structural burden that reduces COVID-19 risk mitigation behaviors. Improved access to affordable housing units or receiving vouchers could reduce overcrowded housing in vulnerable refugee communities.