Glycemic control is of paramount importance in care and management for patients with type 2 diabetes mellitus (T2DM). Poor glycemic control is a major health problem that greatly contributes to the development of diabetes related complications. This study aims to assess the prevalence of poor glycemic control and associated factors among outpatients with T2DM attending diabetes clinic at Amana Regional Referral Hospital in Dar-es-salaam, Tanzania from December 2021 to September 2022. A face to face interviewer semi-structured questionnaire was administered during data collection. Binary logistic regression under multivariable analysis was used to determine the independent predictors of poor glycemic control. A total of 248 patients with T2DM were included in the analysis with mean age of 59.8 ± 12.1 years. The mean fasting blood glucose was 166.9 ± 60.8 mg/dL. The prevalence of poor glycemic control was 66.1% (fasting blood glucose > 130 mg/dL or < 70 mg/dL). Failure to adhere to regular follow-up (AOR = 7.53, 95% CI = 2.34–19.73, p < 0.001) and alcoholism (AOR = 4.71, 95% CI = 1.08–20.59, p = 0.040) were the independent predictors of poor glycemic control. The prevalence of poor glycemic control observed in this study was significantly high. Emphasis should be placed on ensuring that patients have regular follow-up for their diabetes clinics and they should also continue modifying some of lifestyle behaviors including refraining from alcoholism, this can help them to have good glycemic control.