Arginine, glutamine, and the branched chain amino acids (BCAAs) are a focus of increased interest in the field of oncology due to their importance in the metabolic reprogramming of cancer cells. In the tumor microenvironment (TME), these amino acids serve to support the elevated biosynthetic and energy demands of cancer cells, while simultaneously maintaining the growth, homeostasis, and effector function of tumor-infiltrating immune cells. To escape immune destruction, cancer cells utilize a variety of mechanisms to suppress the cytotoxic activity of effector T cells, facilitating T cell exhaustion. One such mechanism is the ability of cancer cells to overexpress metabolic enzymes specializing in the catabolism of arginine, glutamine, and the BCAAs in the TME. The action of such enzymes supplies cancer cells with metabolic intermediates that feed into the TCA cycle, supporting energy generation, or providing precursors for purine, pyrimidine, and polyamine biosynthesis. Armed with substantial metabolic flexibility, cancer cells redirect amino acids from the TME for their own advantage and growth, while leaving the local infiltrating effector T cells deprived of essential nutrients. This review addresses the metabolic pressure that cancer cells exert over immune cells in the TME by up-regulating amino acid metabolism, while discussing opportunities for targeting amino acid metabolism for therapeutic intervention. Special emphasis is given to the crosstalk between arginine, glutamine, and BCAA metabolism in affording cancer cells with metabolic dominance in the TME.