Troicki started 2011 in ATP Doha, where he lost to Roger Federer 2–6, 2–6, in the quarterfinals. Troicki advanced to the finals at ATP Sydney, where he beat Juan Ignacio Chela, Richard Gasquet, Florian Mayer, and in the final, lost to Gilles Simon 5–7, 6–7. Then at the Australian Open, he reached the third round for the first time, but retired against compatriot and eventual champion Novak Djokovic due to a stomach pain, after losing the first set 2–6. He next played at the ABN AMRO World Tennis Tournament and reached the semifinals without dropping a set, but fell to Robin Söderling 5–7, 4–6. He then fell to Philipp Kohlschreiber 1–6, 6–7, in the first round of the Dubai Tennis Championships. He represented Serbia in the first round of 2011 Davis Cup and won both his matches. He then played the Masters 1000 events, the BNP Paribas Open and the Sony Ericsson Open, falling to eventual champion Novak Djokovic in the fourth round in both tournaments. He then reached his first Masters 1000 quarterfinals at the Monte-Carlo Masters, losing to David Ferrer 3–6, 3–6, after his opponent in the previous round Tommy Robredo retired while leading the match 6–3, 1–2. He then suffered early losses in the Serbia Open, the Madrid Open, and the Italian Open. Troicki then represented Serbia in the World Team Cup, winning his matches against Mikhail Youzhny and Marcel Granollers, but losing his match to Florian Mayer. At the French Open, Troicki reached his first Grand Slam fourth round with wins over Julian Reister, Tobias Kamke, and Alexandr Dolgopolov. In the fourth round, he faced fourth seed Andy Murray, where he lost 6–4, 6–4, 3–6, 2–6, 5–7, despite serving for the match at 5–3 and 30–0 in the final set in a match that was played over two days. He then reached a new career high of no. 12.