Those with higher levels of emotional investment in social media, and who use it at night, were more likely to feel depressed and anxious, they say.
Glasgow University researchers questioned 467 teenagers about their use of social media and state of mind.
They found many felt a pressure to respond immediately to texts or posts.
Lead researcher Dr Heather Cleland Woods and Holly Scott asked the teenagers about how and when they used social media.
Sleep quality, self-esteem, anxiety, depression and the subjects' emotional investment in social media were also measured.
This related to the pressure to be available 24/7 and any anxiety around, for example, not responding immediately to texts or posts.
Dr Cleland Woods said: "Adolescence can be a period of increased vulnerability for the onset of depression and anxiety, and poor sleep quality may contribute to this.
"It is important that we understand how social media use relates to these. Evidence is increasingly supporting a link between social-media use and wellbeing, particularly during adolescence, but the causes of this are unclear."
General and night-time specific social-media use along with emotional investment in social media were related to poorer sleep quality, lower self-esteem as well as higher anxiety and depression levels.
Dr Cleland Woods said: "While overall social-media use impacts on sleep quality, those who log on at night appear to be particularly affected.
"This may be mostly true of individuals who are highly emotionally invested. This means we have to think about how our kids use social media, in relation to time for switching off."
The study was presented at the British Psychological Society annual conference in Manchester.
Teenagers are getting more anxious and depressed because of the 24-hour demands of their social media accounts, researchers say.