During the processes of rice domestication and improvement, a trade-off effect between grain number and grain weight was a major obstacle for increasing yield. Here, we identify a critical geneCOG1, encoding the transcription factor OsMADS17, with a 65-bp deletion in the 5′ untranslated region (5′ UTR) presented in cultivated rice increasing grain number and grain weight simultaneously through decreasing mRNA translation efficiency. OsMADS17 controls grain yield by regulating multiple genes and that the interaction with one of them,OsAP2-39, has been characterized. Besides, the expression ofOsMADS17is regulated by OsMADS1 directly. It indicates thatOsMADS1-OsMADS17-OsAP2-39participates in the regulatory network controlling grain yield, and downregulation ofOsMADS17orOsAP2-39expression can further improve grain yield by simultaneously increasing grain number and grain weight. Our findings provide insights into understanding the molecular basis co-regulating rice yield-related traits, and offer a strategy for breeding higher-yielding rice varieties.