Cryptography 18CS744 Module 5: IP Security

MODULE V

IP Security
IP Security: IP Security overview, applications of IPsec, benefits of IPsec,
Routing applications, IPsec documents, IPsec services, transport and tunnel
modes, IP Security policy, Security associations, Security associations
database, Security policy database, IP traffic processing, Encapsulating
Security payload, ESP format, encryption and authentication algorithms,
Padding, Anti replay service
Transport and tunnel modes: combining security associations,
authentication plus confidentiality, basic combinations of security associations,
internet key exchange, key determinations protocol, header and payload
formats, cryptographic suits.
Textbook 1: Ch. 20.1 to 20.3 RBT: L1, L2
Cryptography 18CS744 Module 5: IP Security

IP level security
If an organization requires a secure network by disallowing links to
untrusted sites, encrypting packets that leave the premises, and
authenticating packets that enter the premises then security at IP level
needs to be implemented.
IP-level security provides: authentication, confidentiality, and key
management.
The authentication mechanism assures that a received packet was, in fact,
transmitted by the party identified as the source in the packet header.
The confidentiality facility enables communicating nodes to encrypt
messages to prevent eavesdropping by third parties.
The key management facility is concerned with the secure exchange of keys
Applications of Ipsec
Secure branch office connectivity over the Internet: A company can
build a secure virtual private network over the Internet or over a public
WAN. This enables a business to rely heavily on the Internet and reduce its
need for private networks, saving costs and network management
overhead.
Secure remote access over the Internet: An end user whose system is
equipped with IP security protocols can make a local call to an Internet
Service Provider (ISP) and gain secure access to a company network. This
reduces the cost of toll charges for traveling employees and telecommuters.
Establishing extranet and intranet connectivity with partners: IPsec
can be used to secure communication with other organizations, ensuring
authentication and confidentiality and providing a key exchange
mechanism.

Enhancing electronic commerce security: Even though some Web and

electronic commerce applications have built-in security protocols, the use
of Ipsec enhances that security. IPsec guarantees that all traffic designated
by the network administrator is both encrypted and authenticated, adding
Cryptography 18CS744 Module 5: IP Security

an additional layer of security to whatever is provided at the application

layer.

IP security scenario

As shown in the above figure IPSec headers are introduced between IP

header and IP payload.

IPsec Documents

IPSec documents can be categorized into the following groups.

Architecture: Covers the general concepts, security requirements,

definitions, and mechanisms defining IPsec technology.

Authentication Header (AH): AH is an extension header to provide

message authentication. Because message authentication is provided by
ESP, the use of AH is deprecated. It is included in IPsecv3 for backward
compatibility but should not be used in new applications.
Cryptography 18CS744 Module 5: IP Security

Encapsulating Security Payload (ESP): ESP consists of an

encapsulating header and trailer used to provide encryption or combined
encryption/authentication.

Internet Key Exchange (IKE): This is a collection of documents

describing the key management schemes for use with IPsec.

Cryptographic algorithms: This category encompasses a large set of

documents that define and describe cryptographic algorithms for
encryption, message authentication, pseudorandom functions (PRFs), and
cryptographic key exchange.

IPsec Services

• Access control

• Connectionless integrity

• Data origin authentication

• Rejection of replayed packets (a form of partial sequence integrity)

• Confidentiality (encryption)

• Limited traffic flow confidentiality

Transport and Tunnel Modes

Both AH (Authenticating Header) and ESP (Encapsulating Security Payload)

support two modes of use: transport and tunnel mode

Transport Mode: Transport mode provides protection primarily for upper-

layer protocols. Examples include a TCP or UDP segment or an ICMP packet,
all of which operate directly above IP in a host protocol stack.

Transport mode is used for end to-end communication between two hosts
(e.g., a client and a server, or two workstations).

ESP in transport mode encrypts and optionally authenticates the IP payload

but not the IP header. AH in transport mode authenticates the IP payload
and selected portions of the IP header.
Cryptography 18CS744 Module 5: IP Security

Tunnel Mode: Tunnel mode provides protection to the entire IP packet. To

achieve this, after the AH or ESP fields are added to the IP packet, the
entire packet plus security fields is treated as the payload of new outer IP
packet with a new outer IP.

IP Security Policy

IPsec policy is determined primarily by the interaction of two databases,

the Security Association Database (SAD) and the Security Policy
Database (SPD).

IPsec Architecture

Security Associations

An association is a one-way logical connection between a sender and a

receiver that affords security services to the traffic carried on it.

A security association is uniquely identified by three parameters.

Security Parameters Index (SPI): A 32-bit unsigned integer assigned

to this SA. The SPI is carried in AH and ESP headers to enable the receiving
system to select the SA under which a received packet will be processed.

IP Destination Address: This is the address of the destination endpoint

of the SA, which may be an end-user system or a network system such as
a firewall or router.
Cryptography 18CS744 Module 5: IP Security

Security Protocol Identifier: This field from the outer IP header indicates
whether the association is an AH or ESP security association.

Security Association Database

Security Association Database normally defined by the following

parameters in an SAD entry.

Security Parameters Index (SPI):

Sequence Number Counter: A 32-bit value used to generate the

Sequence Number field in AH or ESP headers

Sequence Counter Overflow: A flag indicating whether overflow of the

Sequence Number Counter

Anti-Replay Window: Used to determine whether an inbound AH or ESP

packet is a replay

AH Information: Authentication algorithm, keys, key lifetimes, and

related parameters being used with AH

ESP Information: Encryption and authentication algorithm, keys,

initialization values, key lifetimes, and related parameters being used with
ESP

Lifetime of this Security Association: A time interval or byte count after

which an SA must be replaced with a new SA

IPsec Protocol Mode: Tunnel, transport, or wildcard.

Security Policy Database

Each SPD entry is defined by a set of IP and upper-layer protocol field

values, called selectors. These selectors are used to filter outgoing traffic
in order to map it into a particular SA. The following selectors determine an
SPD entry:

Remote IP Address:

Local IP Address:
Cryptography 18CS744 Module 5: IP Security

Next Layer Protocol:

Name: A user identifier from the operating system.

Local and Remote Ports: These may be individual TCP or UDP port values.

IP Traffic Processing (Outbound Packets)

1. IPsec searches the SPD for a match to this packet.

2. If no match is found, then the packet is discarded and an error message

is generated.

3. If a match is found, further processing is determined by the first

matching entry in the SPD. If the policy for this packet is DISCARD, then
the packet is discarded. If the policy is BYPASS, then there is no further
IPsec processing; the packet is forwarded to the network for transmission.
Cryptography 18CS744 Module 5: IP Security

4. If the policy is PROTECT, then a search is made of the SAD for a matching
entry. If no entry is found, then IKE is invoked to create an SA with the
appropriate keys and an entry is made in the SA.

5. The matching entry in the SAD determines the processing for this packet.
Either encryption, authentication, or both can be performed, and either
transport or tunnel mode can be used. The packet is then forwarded to the
network for transmission.

IP Traffic Processing (Inbound Packets)

1. IPsec determines whether this is an unsecured IP packet or one that

has ESP or AH headers/trailers, by examining the IP Protocol field (IPv4)
or Next Header field (IPv6).
Cryptography 18CS744 Module 5: IP Security

2. If the packet is unsecured, IPsec searches the SPD for a match to this
packet. If the first matching entry has a policy of BYPASS, the IP header is
processed and stripped off and the packet body is delivered to the next
higher layer, such as TCP. If the first matching entry has a policy of
PROTECT or DISCARD, or if there is no matching entry, the packet is
discarded.

3. For a secured packet, IPsec searches the SAD. If no match is found, the
packet is discarded. Otherwise, IPsec applies the appropriate ESP or AH
processing. Then, the IP header is processed and stripped off and the
packet body is delivered to the next higher layer, such as TCP.

Encapsulating Security Payload (ESP)

ESP can be used to provide confidentiality, data origin authentication,

connectionless integrity, an anti-replay service (a form of partial sequence
integrity), and (limited) traffic flow confidentiality. The set of services
provided depends on options selected at the time of Security Association
(SA) establishment.
Cryptography 18CS744 Module 5: IP Security

Security Parameters Index (32 bits): Identifies a security association.

Sequence Number (32 bits): A monotonically increasing counter value;

this provides an anti-replay function

Payload Data (variable): This is a transport-level segment (transport

mode) or IP packet (tunnel mode) that is protected by encryption.

Padding (0–255 bytes)

Pad Length (8 bits): Indicates the number of pad bytes immediately

preceding this field.

Next Header (8 bits): Identifies the type of data contained in the payload
data

Integrity Check Value (variable): A variable-length field (must be an

integral number of 32-bit words) that contains the Integrity Check Value
computed over the ESP packet minus the Authentication Data field.

An initialization value (IV), is present if this is required by the encryption

or authenticated encryption algorithm used for ESP.

If tunnel mode is being used, then the IPsec implementation may add
Traffic Flow Confidentiality (TFC)
Cryptography 18CS744 Module 5: IP Security

Anti-Replay Service

A replay attack is one in which an attacker obtains a copy of authenticated

packet and later transmits to the intended destination.

When a new SA is established the sender initializes a sequence number

counter to 0, each time the packet is sent on this SA, the sender increments
the counter and places the value in sequence number field.

Because IP is a connectionless, unreliable service, the protocol does not

guarantee that packets will be delivered in order and does not guarantee
that all packets will be delivered.

The receiver will implement a window size W, with a default of W = 64.

The right edge of the window represents the highest sequence number, N,
so far received for a valid packet for any packet with a sequence number
in the range from N - W + 1 to N that has been correctly received (i.e.,
properly authenticated), the corresponding slot in the window is marked.

1. If the received packet falls within the window and is new, the MAC is
checked. If the packet is authenticated, the corresponding slot in the
window is marked.

2. If the received packet is to the right of the window and is new, the MAC
is checked. If the packet is authenticated, the window is advanced so that
this sequence number is the right edge of the window, and the
corresponding slot in the window is marked.
Cryptography 18CS744 Module 5: IP Security

3. If the received packet is to the left of the window or if authentication

fails, the packet is discarded; this is an auditable event.

Transport and Tunnel Modes

In tunnel mode, the entire original IP packet is encapsulated to become the

payload of a new IP packet. Additionally, a new IP header is added on top
of the original IP packet. Since a new packet is created using the original
information, tunnel mode is useful for protecting traffic between different
networks.

Transport mode retains the original IP header. In other words, payload data
transmitted within the original IP packet is protected, but not the IP header.

Transport Mode ESP

1. At the source, the block of data consisting of the ESP trailer plus the
entire transport-layer segment is encrypted and the plaintext of this
block is replaced with its ciphertext to form the IP packet for
transmission. Authentication is added if this option is selected.

2. The packet is then routed to the destination. Each intermediate router

needs to examine and process the IP header plus any plaintext IP
extension headers but does not need to examine the ciphertext.
Cryptography 18CS744 Module 5: IP Security

3. The destination node examines and processes the IP header plus any
plaintext IP extension headers. Then, on the basis of the SPI in the ESP
header, the destination node decrypts the remainder of the packet to
recover the plaintext transport-layer segment.

Tunnel Mode ESP

1. The source prepares an inner IP packet with a destination address of the

target internal host. This packet is prefixed by an ESP header; then the
packet and ESP trailer are encrypted and Authentication Data may be
added. The resulting block is encapsulated with a new IP header (base
header plus optional extensions such as routing and hop-by-hop options for
IPv6) whose destination address is the firewall; this forms the outer IP
packet.
Cryptography 18CS744 Module 5: IP Security

2. The outer packet is routed to the destination firewall. Each intermediate

router needs to examine and process the outer IP header plus any outer IP
extension headers but does not need to examine the ciphertext.

3. The destination firewall examines and processes the outer IP header plus
any outer IP extension headers. Then, on the basis of the SPI in the ESP
header, the destination node decrypts the remainder of the packet to
recover the plaintext inner IP packet. This packet is then transmitted in the
internal network.

4. The inner packet is routed through zero or more routers in the internal
network to the destination host.
Cryptography 18CS744 Module 5: IP Security

Combining Security Associations

An individual SA can implement either the AH or ESP protocol but not both.

Sometimes a particular traffic flow will call for the services provided by both
AH and ESP. Multiple SAs must be employed for the same traffic flow to
achieve the desired IPsec services. The security association bundle refers
to a sequence of SAs through which traffic must be processed to provide a
desired set of IPsec services.
Cryptography 18CS744 Module 5: IP Security

Security associations may be combined into bundles in two ways:

• Transport adjacency: Refers to applying more than one security

protocol to the same IP packet without invoking tunneling. This approach
to combining AH and ESP allows for only one level of combination.

• Iterated tunneling: Refers to the application of multiple layers of

security protocols effected through IP tunneling. This approach allows for
multiple levels of nesting, since each tunnel can originate or terminate at a
different IPsec site along the path.

Authentication Plus Confidentiality

Encryption and authentication can be combined in order to transmit an IP

packet that has both confidentiality and authentication between hosts.

ESP with Authentication Option :

• Transport mode ESP: Authentication and encryption apply to the IP

payload delivered to the host, but the IP header is not protected

• Tunnel mode ESP: Authentication applies to the entire IP packet

delivered to the outer IP destination address (e.g., a firewall), and
authentication is performed at that destination. The entire inner IP
packet is protected by the privacy mechanism for delivery to the inner
IP destination.

Transport Adjacency

Use two bundled transport SAs, with the inner being an ESP SA and the
outer being an AH SA. In this case, ESP is used without its authentication
option. Because the inner SA is a transport SA, encryption is applied to the
IP payload.

Transport-Tunnel Bundle
Cryptography 18CS744 Module 5: IP Security

One approach to applying authentication before encryption between two

hosts is to use a bundle consisting of an inner AH transport SA and an outer
ESP tunnel SA. In this case, authentication is applied to the IP payload plus
the IP header.

Basic Combinations of Security Associations

Case 1. All security is provided between end systems that implement IPsec.
For any two end systems to communicate via an SA, they must share the
appropriate secret keys. Among the possible combinations are

a. AH in transport mode

b. ESP in transport mode

c. ESP followed by AH in transport mode (an ESP SA inside an AH SA)

d. Any one of a, b, or c inside an AH or ESP in tunnel mode

Case 2. Security is provided only between gateways (routers, firewalls,

etc.) and no hosts implement IPsec. This case illustrates simple virtual
private network support. The security architecture document specifies that
only a single tunnel SA is needed for this case. The tunnel could support
AH, ESP, or ESP with the authentication option. Nested tunnels are not
required, because the IPsec services apply to the entire inner packet.

Case 3. This builds on case 2 by adding end-to-end security. The same

combinations discussed for cases 1 and 2 are allowed here. The gateway-
to-gateway tunnel provides either authentication, confidentiality, or both
for all traffic between end systems. When the gateway-to-gateway tunnel
is ESP, it also provides a limited form of traffic confidentiality. Individual
hosts can implement any additional IPsec services required for given
applications or given users by means of end-to-end SAs.

Case 4. This provides support for a remote host that uses the Internet to
reach an organization’s firewall and then to gain access to some server or
Cryptography 18CS744 Module 5: IP Security

workstation behind the firewall. Only tunnel mode is required between the
remote host and the firewall. As in case 1, one or two SAs may be used
between the remote host and the local host.

Internet Key Exchange

The key management portion of IPsec involves the determination and

distribution of secret keys.

The IPsec Architecture document mandates support for two types of key
management:

Manual: A system administrator manually configures each system with its

own keys and with the keys of other communicating systems. This is
practical for small, relatively static environments.

Automated: An automated system enables the on-demand creation of

keys for SAs and facilitates the use of keys in a large distributed system
with an evolving configuration.

The default automated key management protocol for IPsec is referred to as

ISAKMP/Oakley and consists of the following elements:

• Oakley Key Determination Protocol: Oakley is a key exchange

protocol based on the Diffie-Hellman algorithm but providing added
security. Oakley is generic in that it does not dictate specific formats.

• Internet Security Association and Key Management Protocol

(ISAKMP): ISAKMP provides a framework for Internet key management
and provides the specific protocol support, including formats, for
negotiation of security attributes.

Key Determination Protocol

IKE key determination is a refinement of the Diffie-Hellman key exchange

algorithm.

The Diffie-Hellman algorithm has two attractive features:

Cryptography 18CS744 Module 5: IP Security

• Secret keys are created only when needed. There is no need to store
secret keys for a long period of time, exposing them to increased
vulnerability.

• The exchange requires no pre-existing infrastructure other than an

agreement on the global parameters.

There are a number of weaknesses to Diffie-Hellman,

• It does not provide any information about the identities of the parties.

• It is subject to a man-in-the-middle attack, IKE key determination is

designed to retain the advantages of Diffie-Hellman, while countering
its weaknesses.

Internet Key Exchange

Features of IKE key determination The IKE key determination algorithm is

characterized by five important features:

1. It employs a mechanism known as cookies to prevent clogging attacks.

2. It enables the two parties to negotiate a group; this, in essence, specifies

the global parameters of the Diffie-Hellman key exchange.

3. It uses nonces to ensure against replay attacks.

4. It enables the exchange of Diffie-Hellman public key values.

5. It authenticates the Diffie-Hellman exchange to thwart man-in-the-

middle attacks.

IKE mandates that cookie generation satisfy three basic requirements:

1. The cookie must depend on the specific parties. This prevents an attacker
from obtaining a cookie using a real IP address and UDP port and then using
it to swamp the victim with requests from randomly chosen IP addresses
or ports.

2. It must not be possible for anyone other than the issuing entity to
generate cookies that will be accepted by that entity. This implies that the
Cryptography 18CS744 Module 5: IP Security

issuing entity will use local secret information in the generation and
subsequent verification of a cookie. It must not be possible to deduce this
secret information from any particular cookie. The point of this requirement
is that the issuing entity need not save copies of its cookies, which are then
more vulnerable to discovery, but can verify an incoming cookie
acknowledgment when it needs to.

3. The cookie generation and verification methods must be fast to thwart

attacks intended to sabotage processor resources.

IKE key determination employs nonces to ensure against replay attacks.

Each nonce is a locally generated pseudorandom number. Nonces appear
in responses and are encrypted during certain portions of the exchange to
secure their use.

Three different authentication methods can be used with IKE key

determination:

• Digital signatures: The exchange is authenticated by signing a

mutually obtainable hash; each party encrypts the hash with its private
key. The hash is generated over important parameters, such as user IDs
and nonces.

• Public-key encryption: The exchange is authenticated by encrypting

parameters such as IDs and nonces with the sender’s private key.

• Symmetric-key encryption: A key derived by some out-of-band

mechanism can be used to authenticate the exchange by symmetric
encryption of exchange parameters.

Header and Payload Formats

An IKE message consists of an IKE header followed by one or more

payloads. All of this is carried in a transport protocol. The specification
dictates that implementations must support the use of UDP for the
transport protocol.
Cryptography 18CS744 Module 5: IP Security

Figure 20.12a shows the header format for an IKE message. It consists of
the following fields:

• Initiator SPI (64 bits): A value chosen by the initiator to identify a unique
IKE security association (SA).

• Responder SPI (64 bits): A value chosen by the responder to identify a

unique IKE SA.

• Next Payload (8 bits): Indicates the type of the first payload in the
message; payloads are discussed in the next subsection.

• Major Version (4 bits): Indicates major version of IKE in use.

• Minor Version (4 bits): Indicates minor version in use.

• Exchange Type (8 bits): Indicates the type of exchange; these are

discussed later in this section.

• Flags (8 bits): Indicates specific options set for this IKE exchange. Three
bits are defined so far. The initiator bit indicates whether this packet is sent
by the SA initiator. The version bit indicates whether the transmitter is
capable of using a higher major version number than the one currently
indicated. The response bit indicates whether this is a response to a
message containing the same message ID. • Message ID (32 bits): Used
Cryptography 18CS744 Module 5: IP Security

to control retransmission of lost packets and matching of requests and

responses.

• Length (32 bits): Length of total message (header plus all payloads) in
octets

IKE Payload Types

All IKE payloads begin with the same generic payload header shown in
Figure 20.12b. The Next Payload field has a value of 0 if this is the last
payload in the message; otherwise its value is the type of the next payload.
The Payload Length field indicates the length in octets of this payload,
including the generic payload header

Table 20.3 summarizes the payload types defined for IKE and lists the
fields, or parameters, that are part of each payload. The SA payload is used
to begin the establishment of an SA. The payload has a complex,
hierarchical structure. The payload may contain multiple proposals. Each
proposal may contain multiple protocols. Each protocol may contain
multiple transforms. And each transform may contain multiple attributes.
These elements are formatted as substructures within the payload as
follows.

• Proposal: This substructure includes a proposal number, a protocol ID (AH, ESP,

or IKE), an indicator of the number of transforms, and then a transform
substructure. If more than one protocol is to be included in a proposal, then there
is a subsequent proposal substructure with the same proposal number.

• Transform: Different protocols support different transform types. The

transforms are used primarily to define cryptographic algorithms to be used with
a particular protocol.

• Attribute: Each transform may include attributes that modify or complete the
specification of the transform. An example is key length.
Cryptography 18CS744 Module 5: IP Security