Monday, July 25, 2011

Who's your SSH buddy?

I popped out of the office to cycle to meet some folks arriving at a London railway station on Friday when my phone started going wild with text messages from my custom intrusion detection system. Stopping by the side of the road my heart leaped to see text messages informing me that someone had logged in as root on one of my servers:

This was particularly worrying because I'm pretty careful with machine security. I have a complex password scheme and trip wires to catch intruders. I like to keep software up to date and restrict access to only the ports necessary, etc. etc.

But intrusion detection helps to answer the "what if someone gets in?" question.

On Friday, a total 65 messages were sent over a space of a few minutes. My intrusion detection system uses a combination of iptables rules and constant inspection of log files to signal odd behaviour at the packet level, Apache and other service level.

And there I was in the middle of the street without access to the server while it appeared that someone was inside the machine as root. What I needed right then was an SSH buddy: someone I could call and give credentials to so they could log in and shutdown the machine.

There was one family member who I knew would be capable of this, but when I called I found that they were in a car driving somewhere. Finally, I went with a colleague and the machine was shutdown 8 minutes after the first text message was sent.

Ultimately this turned out to be a false alarm. Although the machine was under attack (on many levels: there was activity hitting the packet filter, trying all sorts of injection at the Apache level and having a go at SSH) the actual alert (based on looking in auth.log) was a false alarm based on a bad regexp.

Despite the heart attack I'm still glad I had my out-of-band mechanism (in this case, SMS) for getting machine alerts. But it made me realize that I need to tighten up my SSH buddy plans for the next time.

Who's your SSH buddy?

PS A number of people have mentioned having an SSH client on my iPhone. I do now. But this still doesn't mitigate the need for an SSH buddy: if I'm abroad or in an area without data access I still need someone I can call upon.

PPS Other people have suggested that I make the SMS system two-way so I can SMS in some standard commands (such as shutting the machine down). This is a good idea.


Maht said...

Why do you even let root log in remotely ?

echo PermitRootLogin no > /etc/ssh/cchd_config

And shame your fancy iPhone doesn't have a free SSH client, perhaps you should spring the $5 and buy one. Every system administrator worth his salt should have ssh on her smartphone, anything else is unprofessional !

NilsR said...

I highly recommend (since you obviously have an iphone) to buy prompt:

Tried several ssh clients and this is the best around (at least for me). Saved me once already.

ichi said...

Use ssh key based authentication and don't allow remote root login?

ichi said...

use ssh key based authentication and don't allow remote root login?

Anonymous said...

I don't have an iPhone client, but I do like Terminus for the iPad. A suggestion for the SSH buddy implementation: generate a unique SSH key, enable key-only access to the box and set them up with a special account with limited access to commands (like possibly only the ability to shut down the system).

I've been in situations like this before where I only had one set of credentials, and it's no fun. While I do have key-only access across a number of different machines, I don't have anything like your 'ssh buddy' concept. Depending on the system (and your trusted friends), it might also make sense to have multiple ones across time zones--just in case. ;)

Thanks for the post. HT @newsycombinator.

Barry said...

There are many things you can do to improve the situation.

Key based authentication makes it a LOT harder to break into your server.
Also, I use Config Server Firewall with bruteforce detection to shut people out after 3 attempts.

My SSH buddy is support from my host, I can call them in case of a problem and they are monitoring my servers as well

Joey Robert said...

I have an SSH buddy just to prevent me from fucking up. Too many times have I removed myself /etc/group only to have my SSH buddy, also with root privileges, save the day.

Mike Q. Bailey Esquire said...

All of my machines are in a datacenter that has 24/7/365 onsit engineers who will shut down or reboot a machine without any sort of charge. If I get a weird notice, I can dial them up, tell them the server ip, and have them shut the box down by holding down the power button.

cpmaynard said...

Buddy? You have no buddies in this world, get an iphone and a ssh client(prompt) and carry it with you at all times. Welcome to the future.

Greg said...

Wow. Some people have a monitored server. That's cool, and we don't care. This article is meaningful to remind us, people who manage their own machines, to get an SSH buddy in case something bad happens when we're not in front of a PC. Thanks for the tip.

Daniel Einspanjer said...

You mention that some people suggested having SMS commands you could send to your server. That sounds nice, but I'd say there is still a potential problem where you could be somewhere without your phone or coverage and you wouldn't even know there was a problem until it was too late.

I'd say since you have this nice IDS, it probably wouldn't hurt to put a dead-man's switch on it such that if you don't log in and give an all clear after a series of alerts like this, the machine will automatically halt.

asdf said...

As others have said, SSH public key authentication is a very good idea on the public internet. Failing that, a seriously long password is the next best thing... "pwgen -y 64" will generate a 64 character, alpha-numeric, mixed-case, special character, password.

mike said...


You talk about account security and password complexity, then allow root logins (what?!!) and think that turning on the ability to SMS commands to your actual server is a "good idea"

Anonymous said...

SMS is spoofable! Please don't use it for sending the machine commands.

Francis Turner said...

If you ran Deny hosts with shared data or used the Dshield block list you'd never have had this problem. That address was detected by both on July 20/21.

The easiest way (blatent commercial plug) to get those sorts of lists into your server or firewall is to use ThreatSTOP which is the company I work for but you can do it manually if you want to.

I have to say I'm not exactly thrilled with wonder at the SMS command approach. At least not without some kind of (one time?) authentication.