Olive tree cultivation in new warmer areas and climate change have increased the global interest in understanding how air temperature affects both fruit growth and oil accumulation. The aims of this study were to evaluate the rate and duration of fruit growth and oil accumulation in response to experimental warming (+3) in a semiarid region of Argentina; and assess how warming affected fatty acid composition. Young, potted olive trees (cvs. Arbequina, Coratina) were warmed (T+) or maintained near ambient temperature (T0) inside open top chambers in the field during oil accumulation in 2014–2015 or 2015–2016 using different trees in each season. Warming reduced the rate of both fruit growth and oil accumulation in T+ compared to T0 in both cultivars. These rate reductions under T+ led to large decreases in final fruit dry weight and oil concentration. In contrast, the durations (i.e., days) of fruit growth and oil accumulation were most often not affected. Cultivar x temperature interactions were observed in 2014–2015 with warming decreasing oleic acid and increasing linoleic acid in cv. Arbequina, while cv. Coratina showed no response to warming. However, no interactions were found in 2015–2016. Studying how fruit growth and oil accumulation respond to adaptation strategies against increasing air temperatures should be a priority in both young and mature olive trees of numerous cultivars given crop expansion to new regions and future climate scenarios.