Campaigners said the seven-storey block would have restricted daylight, and would have obscured much of the A-listed building's south facade.
A spokesman for Haus architects said the design aimed to enhance a negative view of the building's gable end.
Glasgow City Council planning committee ruled 12 to six against the plans.
Jane Sutherland, chairwoman of the Garnethill Community Council, had made a plea to the local authority to reject the proposed building.
She said it would be like parking "an ice-cream cart in front of the Mona Lisa".
The plot on the corner of Dalhousie Street and Sauchiehall Street is currently occupied by a brown brick building topped with a flat green metal roof structure, and most recently operated as a nightclub.
The planning application noted the current building's "negative impact on the character of the Conservation Area".
GSA had raised an objection to the proposed development behind the Mackintosh building, which was damaged in a fire in 2014.
A spokeswoman said that restricting light into the building would undermine students' experience of the interior.
She added that Mackintosh-designed buildings as a whole, and the view of the building's south facade, was just as important as any other part of the building.
A proposed block of student flats next to Glasgow School of Art's (GSA) Mackintosh building has been rejected by planners.