Understanding IPSEC

IPSec is a set of protocols and standards developed by the Internet Engineering Task Force (IETF) to support secure communication at the IP layer. IPSEC allows for two types of VPN configuration.

Remote-Access VPN: These let individual users, such as telecommuters, connect to secured networks using VPN client connection software.

LAN to LAN VPN: This VPN connects secured sites together via gateway tunnels.


Cryptography is the set of mathematical functions that form the foundation of all IPSEC security protocols.

Encryption and decryption functions ensure confidentiality. Encryption is the process of taking plain text and converting it to an unintelligible “cipher text” of equal length using a unique encryption key. The “cipher text” is transmitted across the insecure network and on the receiving end; decryption is performed using the same key that converts the “cipher text” back to plain text. Decryption is only possible with the key. The more complex the cryptography the harder it is to try and guess the key and read the “cipher text”

Two types of key cryptography are supported.

Symmetric-Key Cryptography:  A shared key is known to both peers, but it is not known to any other party. The shared key is then used by both ends to encrypt and decrypt the data using different mathematical algorithms. These algorithms include DES, 3DES, AES and process one block of data at a time and transform it into cipher text. Additional security can be obtained by using increasing cipher strength with longer keys.

  • DES (56 bit Key)

This is a low strength cipher that can provide basic protection from eavesdropping. Recent improvements in hacking software have resulted in DES ciphers being breakable within a fairly short time period so its use is not recommended.

  • 3DES (168 bit Key) / AES (128 bit)

These are medium strength ciphers that can provide good protection from eavesdropping. Although breakable the cipher does offer a good challenge to any hacking software in that the time taken to decode a packet would be long enough to not allow for recent data exposure. These ciphers should be considered safe to use for temporary or non-critical data transmission.

  • AES256  (256 bit)

This is the highest strength cipher available and provides excellent protection from eavesdropping. When the data content is considered high risk or ultra confidential then the usage of AES256 cipher strength is highly recommended.

Asymmetric-Key Cryptography: Also known as “public-key” cryptography. A pair of keys is generated by each peer. This consists of a public and a private key. The private key is always kept safe on the peer gateway while the public key can be made known to the other peer across an insecure network.  The public key can only be used to encrypt the data and the private key to decrypt it. That way a trust relationship can be set up by the two peers knowing that only they can decrypt each others packets.

Symmetric key algorithms are computationally much faster than public key algorithms. For efficiency, symmetric-key algorithms like DES or 3DES are used for preserving confidentiality. Public-key algorithms are used in a “key-management” mode to achieve authentication and non-repudiation.

One thing that encryption/decryption does not ensure is data integrity. This is achieved by using integrity checksum algorithms to generate a “fingerprint” that uniquely describes a message. Hashing functions result in a “one-way” result that ensures that a message has not been tampered with. The result cannot be used to generate the original message but each peer can hash the same packet and if the fingerprints agree they know that the data has not been altered. Popular hash functions are MD5 and SHA1.

By combining a public-key algorithm with a one-way hash function the peers can digitally sign each packet to ensure that the confidentiality and integrity has been assured.

Security Associations

The security association database (SAD) hold unique 32 bit wide records for each unidirectional logical connection between two IPSec systems. Each of these security association (SA) entries is uniquely identifiable by the Security Parameter Index number (SPI).


An SA indicates all the security services to the traffic carried by it (pointed to by the authentication transform, encryption transform, and the replay protection—a replay attack is a denial of service attack where an eavesdropper saves already traversed packets and sends them at a later point of time).

It also has a lifetime associated with it, by the end of which the SA state becomes expired and cannot be used for further communication. Security associations are created either manually or through an automatic key-exchange protocol called Internet Key Exchange (IKE) which is managed in the IPSEC system by the IKE daemon.

Security Policy

The Security Policy Database (SPD) contains what actions are to be applied to IP traffic, classified by a set of fields that indicate source and destination IP addresses and IP protocol. These identifying “encryption domains” basically allow the IPSEC engine to classify all packets and either

  • Process the packet by the IPSec module, in which case the SPD entry points to an SA
  • Pass through (pass the packet to the IP stack for normal forwarding).

The policy manager acts as the interface between the user-configured security policies and the SPD.


A packet is received through the receive queue and passed to the IPSec packet processing module. The IPSec packet processing module extracts information from the packet header and looks up the SPD for a matching policy. If a matching policy is not found then the forwarding engine forwards the packet normally.

If the policy is found then the SPD entry should point to an SA in the SAD. The module then fetches the corresponding SAD entry and checks for validity. If the SA state is expired, the module requests the IKE Daemon for another SA negotiation. All the encryption/decryption transforms depicted in the SA are performed on the packet with the help of the “cryptography” module which performs all the mathematics required. The transformed packet is sent to the “transmit queue” for transmission.

IPSec Protocols

IPSec standards are defined by three main protocols: AH, ESP, IKE.

Authentication Header (AH)

AH provides data integrity and replay protection for the whole IP datagram and is an effective measure against IP-spoofing and session-hijacking attacks. AH uses an established authentication key as input to the standard hashing algorithm. The result is included as part of the AH header. The receiver re-computes the hash result on the received packet and checks for equality. AH does not encrypt data and therefore does not provide data confidentiality.

Encapsulating Security Payload (ESP)

ESP provides data confidentiality, data integrity, and replay protection for the whole IP datagram. It uses a symmetric key algorithm (like 3DES or AES) to encrypt the data and, like AH, uses a secure hash algorithm to compute a unique checksum. In this case the checksum (ICV) is stored at the end of the packer and not in the ESP header


AH and ESP examples

Internet Key Exchange (IKE)

IKE is the secure mechanism used to automatically establish SA credentials needed to secure the packets between two IPSec peers. The IKE daemon runs as a IP service on UDP port number 500. The protocol operates in two phases:

Phase 1 (Initial Authentication Phase)

The objective of phase 1 is to establish a secure channel, authenticate the negotiating parties, and generate shared keys.  This is achieved using 6 messages.

The first two messages are used to negotiate the security policy that will be used to protect the phase 2 messages. Both sides will have configured a list of proposed security polices and each side will check that list to see if a match can be found. If a match is chosen then the next two messages perform a Diffie-Hellman (DH) asymmetric public-private key exchange.

The last two messages are used to authenticate the peers using one of the methods below.

  • Pre-Shared Keys (PSK):  A shared secret is distributed out-of-band to the peers. The peers use this information to create a hash that is used to authenticate messages.
  • Digital Signatures (RSA or DSS): Certificates of the peers are exchanged in the last two messages and hashes are calculated from these certificates to authenticate each other.

An alternative to the 6 message mode is to use only 3 messages in “Aggressive” mode. This mode is quicker to establish but does expose some Phase 2 information during setup.

Phase 2 (Key Exchange)

Phase 2 is used to establish the IPSec SA and to generate new keying material. A full Diffie-Hellman key exchange may be done to provide perfect forward secrecy (PFS), otherwise the keys are derived from the initial phase 1 keying material.

Once IKE is fully established then all data packets are encrypted using the chosen encryption scheme and keys derived from phase 2. To ensure security within an established VPN the keys to phase 2 need to be refreshed often. Phase 1 keys are only renegotiated after numerous phase 2 cycles are complete.

Phase 1 renegotiation will also tear down any phase 2 SA data at the same time since phase 2 is derived from phase 1.