« Presentation: Stanford iPhone - iStanford | Main | Scary Excel "Share Workbook" feature behavior »

ssh keys on Mac OS X Server: the fuller, longer, better instructions

Establishing password-less logins using public/private keys turned out to be a bigger pain in the butt than I had expected. There are a lot of resources on the web on how to do this, typically for BSD or Linux; and there are even scripts that purport to do this with assistance, but only the simple part. Whenever I do this, I have to pause and give this more thought that should be necessary, googlin' around for an answer to solve what I screwed up.

The problems are, there are always other components at play which are Mac-specific, and the guides are usually generic or aimed at Linux, BSD or some other sort of unix-like operating system. Even Apple's own documentation (from the Developer Connection, no less) is a little scant or can be intimidating for green sysadmins. This blog post is very long because it's very detailed. It's not quite step-by-step, but it's close.

A note: if there's an omission or an error in the instructions, and thousands of credit card numbers, Social Security numbers, cancer test results or SAT scores escape into the open, don't come knocking on my door.

Using password-less ssh sessions with public/private keys isn't recommended for daily use from user consoles (ie, a typical desktop computer). If your workstation is accessed without authorization, the keys to the kingdom belong to the crook. If you want the convenience of password-less ssh sessions, you should use Kerberos instead. At least with Kerberos authentication, someone, at some point, needs to provide Kerberos credentials prior the expiration period. Moreover, there are no files on the workstation to steal with Kerberos. It's a good thing.

Most admins use pub/pri keys for mundane things, like automated rsyncing between two servers. There are, like everything else unix, many ways to skin a cat; but if done properly, you can do this in a reasonably secure way, with layers of security corresponding to your level of paranoia. In my case, to make things even more nerve-wracking, I'm using this for root-to-root ssh sessions. Call me crazy.

Sidebar: An alternative method to all this is to use remctl . It's a Kerb5 alternative to pub/pri keys useful for issuing commands between two machines. It compiles cleanly on Mac OS X 10.5 with Xcode 3.1 — at least with version 2.12; however the current version at the time of this writing, 2.13, errors out on the make. My Mac servers use it for the sake of wallet but I should also take the time to learn about its magic first.

Also, be sure you have an out to get back in if you're working remotely, especially if the servers are headless and in a data center across campus. If you make some stupid typo, you won't be able to correct the situation — because you won't be able to shell in. You can use Apple Remote Desktop if you like, but make sure you've already connected and you don't have the Encrypt All Network Data option enabled for that host. If I remember correctly, if you're connected via ssh and you make a change to its configuration file, your older settings apply until you reconnect. Or something like that.

Another thing to keep in mind is that Mac OS X doesn't actually have an always-running ssh daemon. It's an on-demand thing, kicked off by launchd (not xinitd or /etc/rc files).

The Gist:

This works by creating a set of keys on two machines (we'll call them "foo.stanford.edu" and "bar.stanford.edu" ) and swapping them so each server has set of private (its own) and public (the other) keys. You move the public key from one to the other, and vice-versa (because for our purposes, we want mutual, keyless ssh sessions from one to the other, and be able to reverse the process). Many instructions just end here, and when things just don't work, it's annoying. So let's take it from the top. Again, we're doing this to allow root-to-root ssh sessions. I told you this was scary.

Step 1:

Ssh into foo using your typical admin account, in a typical fashion, then follow the example commands. The process will ask a series of questions and even make a pretty picture for you. Again, I'm logged into the server "foo" for the sake of demonstration.

[nbfa@foo 11:17:44 ~]$ sudo ssh-keygen -t dsa    

Remember, my tutorial is for generating pub/pri keys for the root account, which is why I preface the do-ing command (ssh-keygen) with sudo. If you choose to run ssh-keygen as you, dear reader, you'll end up putting your keys in your own home directory. That's useful for other purposes, I'm sure. Also, the -t flag indicates "type" and we're using DSA encryption "just because". The alternative is RSA, and you're welcome, even encouraged to learn more by google-ing around. Using DSA vs RSA can be a contentious topic anywhere nerds congregate.

Generating public/private dsa key pair.
Enter file in which to save the key (/var/root/.ssh/id_dsa): /var/root/.ssh/foo_dsa   

Here, I'm going to mitigate future confusion by not accepting the default name of id_dsa. I'm going to give it the name foo_dsa instead. This will change down the line, but it's helpful for moving across machines when the time comes.

Enter passphrase (empty for no passphrase):    
Enter same passphrase again:
Your identification has been saved in /var/root/.ssh/foo_dsa.
Your public key has been saved in /var/root/.ssh/foo_dsa.pub.
The key fingerprint is:
88:c0:29:dc:4f:68:a0:1b:b7:fe:0d:37:63:00:86:85 root@foo.local
The key's randomart image is:
+--[ DSA 1024]----+
| o. |
|E+o.. |
|=.B+ . |
| *.+o. . |
|. . o.. S |
| . . |
| . . = |
| . = o |
| . . |
[nbfa@foo 12:11:50 ~]$

All the rest is just confirmation and some random art thing someone thought was clever.

Step 2:

Do the same thing on your other server. Rather than repeat myself, replace "foo" with "bar" from above.

Step 3: (in easyland)

Since you're already on bar, we'll need to get the public key over to foo. That sounds easy enough, but by default, you can's just scp it over as root, to root's home directory. We can either make the adjustments in the other configuration files right now, or you can do something like put it on a thumb drive. For the sake of simplicity, let's pretend you have a thumb drive, that foo and bar are ten feet away from your cubicle, attached to a KVM, and you have the key to open the server room. That's highly unlikely, I know. We'll get to alternatives later; this part is just for concept.

Let's head over to foo first. Go ahead ahead and log in with your credentials and fire up Terminal and copy foo_dsa.pub over to your thumb drive. Don't try to use the Finder; beside, you'll get confused when you can't see the dot-ssh directory in root's home.

# sudo cp /var/root/.ssh/foo_dsa.pub /Volumes/mycheapthumbs    

Switch over to bar and do the same, transferring its .pub key to the thumb drive. You now need to reverse the process by copying foo_dsa.pub over to bar, and then vice versa. So in root's home directory, it should look like this:

[nbfa@bar 14:15:14 ~]# sudo ls -la /var/root/.ssh/   
total 32
drwx------ 6 root wheel 204 Mar 8 11:32 .
drwxr-x--- 10 root wheel 340 Nov 24 19:18 ..
-rw-r--r-- 1 root wheel 668 Mar 8 10:40 foo_dsa.pub
-rw-r--r-- 1 root wheel 622 Mar 8 10:40 bar_dsa.pub
-rw------- 1 root wheel 622 Mar 8 10:40 bar_dsa
-rw-r--r-- 1 root wheel 3319 Mar 8 11:01 known_hosts

Looking at the above, we see bar's private key. This is really, really important and should be guarded with a rout of angry wolverines. Make sure the permissions are 600 (eg -rw------- ). The permissions are typically the default, but it's good to check it out. Follow these example commands for the server named bar, then do that for foo, too.

[nbfa@bar 14:15:14 ~]# sudo chmod 600 /var/root/.ssh/bar_dsa

The public keys are nothing special; they're public after all.

Now jump to Step 4 below.

Step 3:

It's a good chance that you're no where near your server, or it's headless or locked away far from USB flash drives. In this case, you'll have to scp (that's secure copy) the public keys from one server to the other. Most likely, however, you can't scp as root. (Don't fret; hold on a little bit longer.) In fact, you can't even get to the /var/root/.ssh directory as yourself. To get around all this, you have to do the equivalent of three-point parallel park. First, sudo to root to copy your server's .pub file (just that server's) to your home directory.

[nbfa@foo 11:42:02 ~]# sudo cp /var/root/.ssh/foo_dsa.pub ~/Desktop/    

Now, let's scp that key to your home directory on bar. You don't want to use sudo to scp it over, because that would really be root trying to connect to the other server (which is currently disallowed).

[nbfa@foo 11:45:16 ~/Desktop]$ scp foo_dsa.pub bar.stanford.edu:~/Desktop/    

Ssh to bar to make sure you copied it on your other desktop. Finally, we'll copy the public key on our server's desktop over to its proper place, then change ownership and permission.

[nbfa@bar 11:48:18 ~/Desktop]$ sudo cp foo_dsa.pub /var/root/.ssh/    
[nbfa@bar 11:48:18 ~/Desktop]$ sudo chown root:wheel /var/root/.ssh/foo_dsa.pub
[nbfa@bar 11:48:18 ~/Desktop]$ sudo chmod 644 /var/root/.ssh/foo_dsa.pub

Step 4:

When we did all this moving around of public keys, we used names like foo_dsa.pub and bar_dsa.pub to keep things straight. But these aren't really the normal names, so we'll have to change them to something expected. Technically you can use this custom name (foo or bar) but you'd have to edit a few other configuration files, and besides, you only get one public key file (though that file can have multiple server's keys concatenated). You'd think that you would just change the name back to something.pub, but that would change that server's public key. Instead, we need to name it (or add to) authorized_keys2. Why the 2 in the name? Because we chose DSA as the encryption method, and that's what it likes.

[nbfa@foo 11:42:02 ~]# sudo mv /var/root/.ssh/bar_dsa.pub /var/root/.ssh/authorized_keys2    

If you've inherited a machine that already has one of these files, we'll just concatenate our other server's public key.

[nbfa@foo 11:42:02 ~]# sudo cat /var/root/.ssh/bar_dsa.pub >> /var/root/.ssh/authorized_keys2    

Clean up after yourself and delete the public key sitting on your desktop (if only to reduce clutter and prevent future confusion). Wash, rinse and repeat the other way around.

Step 5:

Most of the work has been done, which is pretty awesome. At this point, you could try opening a sudo root shell and ssh-ing into the other server.

[nbfa@bar 11:45:22 ~/Desktop]$ sudo -s    
[root@bar 11:54:07 ~/Desktop]#

Then give it a whirl:

[root@bar 11:54:07 ~/Desktop]# ssh foo.stanford.edu    

If you are successful, hats off to you; go get a latte. For the rest of us, it may just hang there, cursor blinking, mocking you until it eventually times out (or you type control-c) or it will prompt you for a password, which is what we were trying to avoid all along. The fun continues.

Step 6:

There are multiple points where this can fail; we'll enumerate a few.

A) Check the firewall settings. You might have a local software or network-based firewall that prevents ssh connections. This is port 22 using TCP.

B) You might not have the root account active. On Mac OS X Server, it's on by default (which is worrisome but convenient); the initial password is the same as the initial account when the server was configured. You can (and should) change the root account's password to something unique and super long and complicated, since you likely won't need to enter it very much (we use sudo and our own admin password to get our work done anyway). You can use Directory Access.app in the Utilities folder (or Workgroup Manager) to make the change to enable/disable/modify the root account. Make sure it's alive, otherwise you can't ssh in.

C) Your sshd_config file hasn't been modified. You need this edited from the default to make password-less root ssh-ing to happen. Get out your Terminal and crack your finger knuckles, because we'll do some command line typing. You can use your favorite editor to make the changes. What? You don't have a favorite editor? Use nano then. (As you would expect, this file is protected, so we need to use sudo again. If not, you'll probably do all these edits and then be told you can't save it. Rats.) This is where screwing up with typos can lock yourself out, so let's use caution and make a backup, just in case.

[nbfa@bar 11:48:18 ~/Desktop]$ sudo cp /etc/sshd_config /etc/sshd_config.org    
[nbfa@bar 11:48:18 ~/Desktop]$ sudo nano /etc/sshd_config

This will open the ssh daemon configuration file. This is the config for the receiving ssh daemon, not the ssh command you initiate when you connect to a server. It should look a little like this:

GNU nano 2.0.1 File: /etc/sshd_config    
# $OpenBSD: sshd_config,v 1.74 2006/07/19 13:07:10 dtucker Exp $
# sshd_config(5) for more information.
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options change a
# default value.
#Port 22
Protocol 2
#AddressFamily any
#ListenAddress ::

At the top of the text, where it says "See sshd_config for more information" means you can learn all about the different and fascinating variables and options you want ssh to assume when it responds to an incoming connection. It's rather confusion, because you think you're editing sshd_config, so what could you learn from by reading the file you're working on! What is not said is that you can read the manual for sshd_config and get the options and instructions there. Or you can google it. But let's open another terminal window and look at the man page.

[nbfa@bar 11:48:18 ~/Desktop]$ man sshd_config    

SSHD_CONFIG(5)     BSD File Formats Manual     SSHD_CONFIG(5)

sshd_config -- OpenSSH SSH daemon configuration file   

sshd(8) reads configuration data from /etc/sshd_config (or the file specified with -f on the command line).
The file contains keyword-argument pairs, one per line. Lines starting with # and empty lines are interpreted as comments.   
Arguments may optionally be enclosed in double quotes (") in order to represent arguments containing spaces.   
The possible keywords and their meanings are as follows (note that keywords are case-insensitive and arguments are case-sensitive):   

Scrolling down, we have an idea of what some defaults are and what some options are appropriate. People can go nuts editing this file, and by default it's pretty secure; but let's stick to only a few changes for our purposes. Going back to our terminal window with our text editor, lets scroll on down to make the changes. We'll want these lines to look like this:

PermitRootLogin without-password (This limits the root
account to using public/private keys or Kerberized ssh sessions.)    
RSAAuthentication no
DSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys2

D) Your server uses service access control lists (SACLs) and root isn't allowed to ssh in. See this Apple technote on the topic: . Be sure to add root (or the long name, System Administrator).

Now let's try again. Ssh as root to the other server. If any of the letters above have been resolved, hip hip hooray. Oh, and another trick to debugging this is to use the -v flag so that your ssh command verbosely demonstrates what it's attempting to do. There's even -vvv for super-verbose.

Security Considerations:

The scope of this document doesn't really cover ssh best practices and doesn't include certain security measures that are important. Here are quick and cheap suggestions, though.

Use the built-in firewall and/or a network-based firewall to exclude all the bad guys, especially the internet. Which is to say, allow only the hosts or a tiny subnet from which you will allow connections. Consider only allowing connections from the VPN pool or yours and the other server's IP address.

Modify the SACLs on Server Admin.app to only allow certain, trusted entities to ssh in.

Use an assortment of allow and deny parameters in the sshd_config file. For example, AllowUsers or AllowGroups effectively denies those not deliberately added. You can even eliminate challenge/response by enabling the PasswordAuthentication to "no". That will allow only Kerberized and public/private keys. This was enabled for the root account only specifying "without-password" for the PermitRootLogin parameter above.

In Workgroup Manager, eliminate the login shell variable for the user accounts you manage. In the account's "Advanced" tab, select "none" for the login shell.

Hopefully you'll be able to get your work done now, perhaps using the root pub/pri keys to do things like rsync-ing files from a primary to backup server.


TrackBack URL for this entry:

Post a comment



This page contains a single entry from the blog posted on March 10, 2009 1:13 PM.

The previous post in this blog was Presentation: Stanford iPhone - iStanford.

The next post in this blog is Scary Excel "Share Workbook" feature behavior.


Creative Commons License
This weblog is licensed under a Creative Commons License.