IP Security (IPSec)

Last Updated : 29 Apr, 2026

IP Security (IPsec) is a set of network security protocols used to protect data transmitted over an IP network, such as the Internet. It provides security by authenticating, encrypting, and ensuring the integrity of IP packets during communication between devices.

  • It works at the Network Layer of the OSI model
  • Encrypts data to prevent unauthorized access.
  • Ensures that the data is not modified during transmission.
  • Helps to create a secure communication channel between the sender and receiver.

Features

  • Confidentiality: Encrypts the data so that unauthorized users cannot read the information.
  • Key Management: Manages the creation, exchange, and updating of cryptographic keys securely.
  • Tunneling: Allows IP packets to be encapsulated inside another packet to create a secure communication path.
  • Flexibility: Can be used in different network types such as point-to-point, site-to-site, and remote access connections.
  • Interoperability: Works across different devices and vendors because it is an open standard protocol.

Working

IPsec is used to secure data when it travels over the Internet by creating a protected connection between communicating devices. It ensures that the transmitted information remains confidential, authentic, and unchanged during transmission.

ipsec_tunnel
  • Secure Connection Establishment: Two devices first initiate communication and agree on security methods to protect data.
  • Key Exchange using IKE: Cryptographic keys are created and shared using Internet Key Exchange so that both devices can communicate securely.
  • Use of Security Protocols: IPsec uses AH for authentication and integrity, and ESP for encryption along with authentication.
  • Modes of Operation: IPsec operates in two modes, Transport Mode and Tunnel Mode, depending on how the data needs to be protected.
  • Authentication: IPsec verifies that the data is coming from a trusted sender.
  • Encryption: IPsec encrypts the data so that unauthorized users cannot read it.
  • Integrity Check: IPsec ensures that the data is not modified during transmission.
  • Secure Data Transmission: After establishing a secure tunnel, data is safely transmitted between devices.
  • Connection Termination: After communication is complete, the secure connection is closed.

Connection Establishment Process

IPsec establishes a secure communication channel by authenticating devices and encrypting the data transmitted over the network. The connection establishment process takes place in two main phases.

Phase 1: Establishing the IKE Tunnel

In Phase 1, a secure communication channel is created using Internet Key Exchange. This secure channel is used for further negotiation of security parameters and key exchange.

  • Main Mode: Uses six message exchanges and provides higher security because identity information is protected during negotiation.
  • Aggressive Mode: Uses three message exchanges and establishes the connection faster, but is less secure because more identity information is exposed.

Phase 2: Establishing the IPsec Tunnel

In Phase 2, also known as Quick Mode, the devices negotiate the IPsec Security Associations (SA) and decide how data will be protected during communication.

  • Tunnel Mode: Encrypts the entire original IP packet, including header and data, commonly used in site-to-site VPN connections.
  • Transport Mode: Encrypts only the data part of the IP packet, while the IP header remains unchanged, commonly used in end-to-end communication between hosts.
  • Phase 1 creates a secure channel for key exchange, and Phase 2 establishes the actual secure IPsec tunnel for protected data transmission.

IPsec Tunnel Mode vs Transport Mode

Tunnel ModeTransport Mode
Protects the entire original IP packet (header + data)Protects only the payload (data) part
Adds a new outer IP headerUses the original IP header
Provides higher securityProvides comparatively lower security
Encapsulates the whole original packetDoes not encapsulate the full packet
Used in site-to-site VPN connectionsUsed in host-to-host communication
Hides source and destination IP addressesSource and destination IP addresses remain visible
More overhead due to the extra IP headerLess overhead
Commonly used in VPN gatewaysCommonly used for end-to-end communication

Protocols Used in IPSec

  • Encapsulating Security Payload (ESP): It provides data integrity, encryption, authentication, and anti-replay. It also provides authentication for the payload.
  • Authentication Header (AH): Provides authentication and data integrity for transmitted data, but does not provide encryption or confidentiality.
  • Internet Key Exchange (IKE): Establishes Security Association (SA) and exchanges cryptographic keys required for secure communication. 

IP Security Architecture

IPSec (IP Security) architecture uses two protocols to secure the traffic or data flow. These protocols are ESP (Encapsulation Security Payload) and AH (Authentication Header).

  • Architecture includes protocols, algorithms, DOI, and Key Management.
  • All these components are very important in order to provide the three main services such as Confidentiality, Authenticity and Integrity.
architecture

IPsec VPN

This is a secure networking method that allows users to communicate over the internet safely and privately. IPsec VPN creates an encrypted tunnel between sender and receiver, ensuring that transmitted data remains confidential and protected from unauthorized access.

  • Provides secure communication between two endpoints.
  • Ensures confidentiality and privacy of transmitted data.
  • Offers end-to-end encryption, where data is encrypted at sender side and decrypted at receiver side.
  • Commonly used for remote access and site-to-site secure connections.

Uses

  • Encrypts application layer data to keep information secure during transmission.
  • Provides security for routers while sending routing data over the public internet.
  • Provides authentication to verify that data comes from a trusted sender, even without encryption.
  • Creates secure communication tunnels between networks, such as in a Virtual Private Network (VPN).

Advantages

  • Provides strong cryptographic security to protect sensitive data and maintain privacy and integrity.
  • Open standard protocol supported by many vendors, making it suitable for different network environments.
  • Can secure various network types such as point-to-point, site-to-site, and remote access connections.
  • Suitable for both small and large networks, as it can be scaled according to requirements.
  • Helps improve secure communication efficiency across networks.

Disadvantages

  • Complex to configure and requires technical knowledge for proper implementation.
  • May face compatibility issues with some devices or applications.
  • Encryption and decryption process may reduce network performance due to extra processing.
  • Requires proper key management to maintain security of encryption keys.
  • IPsec protects all IP-based protocols, including ICMP and DNS, but only when traffic is configured to use IPsec; otherwise, it remains unprotected.
Comment

Explore