Advanced DMVPN Deployments

BRKSEC-3006 Frederic Detienne, Distinguished Services Engineer

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Housekeeping
We value your feedback- don't forget to complete your online session evaluations after each session & complete the Overall Conference Evaluation which will be available online from Thursday Visit the World of Solutions Please remember this is a 'non-smoking' venue! Please switch off your mobile phones Please make use of the recycling bins provided Please remember to wear your badge at all times including the Party
Session ID Presentation_ID

2007 Cisco Systems, Inc. All rights reserved.

Cisco Public

Agenda
DMVPN phases
Phase 2 and phase 3 comparison Shortcut Switching NHRP forwarding

Designing with DMVPN phase 3

Basic Scalable Design passing the 1,000 nodes barrier with a single hub Dual Homed Scalable Design hub resilience, beyond 1,000 nodes using IP SLA Very large scale DMVPN design limitless aggregation

Deployment tips and tricks

Using ISAKMP profiles to map users to tunnels VRF and DMVPN

Recent DMVPN enhancements

DMVPN and IPv6 Per Tunnel QoS

GET vs DMVPN

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Sessions objectives
DMVPN phase 2 and 3 comparison Large IPsec VPN meshes designs Integrating DMVPN with other features
VRF, PKI, IPv6, QoS

In-depth knowledge of DMVPN is assumed

This includes IKE and IPsec

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

DMVPN phase 2-3 comparison

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Nomenclature Transport
Transport Network
Hub 192.168.254.0/24

NBMA Address

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

DMVPN Tunnels
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Nomenclature Overlay
Overlay network
Hub 192.168.254.0/24

Tunnel Address

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Overlay/Private Addresses
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Spoke Registration
ip address 10.0.0.254 255.255.255.0 ip nhrp network-id 1 Hub
192.168.254.0/24
NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1
t es u eq R

Physical: 172.16.254.1 Tunnel: 10.0.0.254

n tio tra is Physical: 172.16.1.1 Reg

Tunnel:

10.0.0.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

ip ip ip ip
BRKSEC-3006

address 10.0.0.1 255.255.255.0 nhrp network-id 1 nhrp map 10.0.0.254 172.16.254.1 nhrp nhs 10.0.0.254
2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Route exchange
Routing table C 10.0.0.0 Tunnel0 D 192.168.0.0/29 10.0.0.1 D 192.168.0.8/29 10.0.0.2

ip nhrp map multicast dynamic Hub

192.168.254.0/24

NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Physical: 172.16.254.1 Tunnel: 10.0.0.254

g ng ttiin u u Ro Ro
e e att da d p Up U

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0

ip nhrp map multicast 172.16.254.1

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0

IT DEPENDS !!

IT DEPENDS !!
9

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Hub & Spoke design

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0 Tunnel0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/16 10.0.0.254

Hub via transport network 192.168.0.0/16 encrypted & tunneled to hub

Cisco Public

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/16 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

10

DMVPN phase 2 design style

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Hub 192.168.254.0/24

Hub advertises back individual prefixes pointing to corresponding spoke.

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.8/29 10.0.0.2

Tunnels via transport network Lots of individual prefixes

Cisco Public

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/29 10.0.0.1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

11

DMVPN phase 2 shortcuts (1)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1
st ue q Re

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

n tio lu so Physical: 172.16.1.1 Re

Tunnel:

10.0.0.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

NHRP table 10.0.0.254 172.16.254.1 10.0.0.2

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.8/29 10.0.0.2

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

12

DMVPN phase 2 shortcuts (2)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1
ly ep R

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1 10.0.0.2 172.16.2.1

n io ut l so Re

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.8/29 10.0.0.2

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

DMVPN phase 2 shortcuts (3)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1 10.0.0.2 172.16.2.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.8/29 10.0.0.2

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

14

DMVPN phase 3 design style

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Hub 192.168.254.0/24

Hub advertises back summary prefix pointing to hub.

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/16 10.0.0.254

Tunnels via transport network 192.168.0.0/16 summary tunneled to hub

Cisco Public

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0 Dialer0 D 192.168.0.0/29 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

15

DMVPN phase 3 design style

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29

ip nhrp redirect Hub

192.168.254.0/24

NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/16 10.0.0.254

ip nhrp shortcut

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

DMVPN phase 3 shortcuts (1)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1
) .9 .0 68 .1 92 (1

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

n io ct e Physical: 172.16.1.1 ndir I

Tunnel:

10.0.0.1

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1

NHRP table 10.0.0.254 172.16.254.1

Spoke 1 192.168.0.0/29

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/16 10.0.0.254

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

17

DMVPN phase 3 shortcuts (2)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1
) .9 .0 68 .1 92 (1
Re so Tunnel: 10.0.0.254 lu tio n (1 92 .16 8. 0.9 ) Physical: 172.16.2.1 Tunnel: 10.0.0.2 Resolution Reply

Hub 192.168.254.0/24

Physical: 172.16.254.1

n tio lu Physical: 172.16.1.1 eso Tunnel: 10.0.0.1 R

192.168.0.8/29 10.0.0.2 172.16.2.1 192.168.0.8/29

NHRP table 10.0.0.254 172.16.254.1 Spoke 1 10.0.0.2 172.16.2.1 192.168.0.8/29 172.16.2.1 192.168.0.0/29 Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/16 10.0.0.254

Spoke 2 192.168.0.8/29

NHRP table 10.0.0.254 172.16.254.1 10.0.0.1 172.16.1.1

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

18

DMVPN phase 3 shortcuts (3)

Routing table C 10.0.0.0 Tunnel0 C 192.168.254.0/24 Eth0 D 192.168.0.0/29 10.0.0.1 10.0.0.2 D 192.168.0.8/29 NHRP table 10.0.0.1 172.16.1.1 10.0.0.2 172.16.2.1

Hub 192.168.254.0/24

Physical: 172.16.254.1 Tunnel: 10.0.0.254

Physical: 172.16.1.1 Tunnel: 10.0.0.1

NHRP table 10.0.0.254 172.16.254.1 Spoke 1 10.0.0.2 172.16.2.1 192.168.0.8/29 172.16.2.1 192.168.0.0/29 Routing table C 192.168.0.0/29 Eth0 C 10.0.0.0 Tunnel0 S 0.0.0.0/0 Dialer0 D 192.168.0.0/16 10.0.0.254

Physical: 172.16.2.1 Tunnel: 10.0.0.2

NHRP table 10.0.0.254 172.16.254.1 10.0.0.1 172.16.1.1

Spoke 2 192.168.0.8/29

Routing table C 192.168.0.8/29 Eth0 C 10.0.0.0 Tunnel0 S 172.16.254.1 Dialer0 D 192.168.0.0/29 10.0.0.254

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

19

DMVPN phase 3 data packet forwarding

The packet and next-hop are passed to the interface Assuming the interface is NHRP enabled

For your reference

Route lookup determines output interface and next-hop

Destination address is looked up in the NHRP cache

If success, use entry to encapsulate

Next-hop address is looked up in the NHRP cache

Is success, use entry to encapsulate

Fallback: send packet to configured NHS

Use NHS NHRP entry Resolve next-hop address via resolution-request

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

DMVPN phase 3 resolution triggers

If packet forwarding falls back to NHS
Issue resolution-request for next-hop address (/32)

For your reference

If router receives indirection-notification

Aka NHRP Redirect Issue resolution-request for address in notification A /32 address is looked-up

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

21

DMVPN phase 3 resolution forwarding

Address look up in NHRP cache
If authoritative entry present, answer w/ entry

For your reference

Otherwise lookup address in routing table (RIB) If next-hop belongs to same DMVPN
i.e. nhrp network-id of next-hop same as incoming request Treat found next-hop as NHS Forward resolution-request to next-hop

If next-hop does not belong to DMVPN

i.e. Network-id is different or interface not NHRP-enabled Respond with full prefix found in routing table maybe < /32

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

22

Phase 3: Platform Support Summary

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

23

Cisco IOS Code and Platform Support

IOS Code
Phase 1 & 2 12.3(17), 12.3(14)T6, 12.4(7), 12.4(4)T Phase 1, 2 & 3 12.4(6)T

Platforms
6500/7600 (12.2(18)SXF4) with VPN-SPA + sup720 No Phase 3 capability yet 7301, 7204/6, 38xx, 37xx, 36xx, 28xx, 26xx, 18xx, 17xx, 87x, 83x Phase 1, 2 & 3

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

24

Dual Homed Spokes Scalable Design Using IP SLA

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

IP SLA and Reliable Static Routing

IP SLA is an IOS feature to monitor an Service Levels Probes are sent to measure network performances
Availability, delay, jitter, Probes can be ICMP, UDP,

Tracking Objects report the status of SLA probes

Object status goes up or down as the SLA monitor hits triggers

Routes can be injected based on the Tracking Object

Routes injected when the tracked object is up

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

26

Dual homed DMVPN spokes

Single DMVPN Dual Hub Single mGRE tunnel on all nodes
Physical: 172.17.0.5 Tunnel0: 10.0.0.2 192.168.0.0/24 .2 .1

Physical: 172.17.0.1 Tunnel0: 10.0.0.1

Physical: (dynamic) Tunnel0: 10.0.0.12

Spoke B Physical: (dynamic) Tunnel0: 10.0.0.11

.1

.37 Web

192.168.2.0/24

.1 192.168.1.0/24

Spoke A .25

...
= Dynamic&Temporary Spoke-to-spoke IPsec tunnels
27

PC

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

..

Dual homed DMVPN spokes Hub1

Common Subnet Activate redirection interface Tunnel0 bandwidth 1000 ip address 10.0.0.1 255.255.255.0 ip mtu 1400 ip nhrp map multicast dynamic ip nhrp redirect ip nhrp network-id 1 tunnel source Serial1/0 tunnel mode gre multipoint tunnel protection ipsec profile vpnprof ! router rip network 10.0.0.0 passive-interface default ! ip sla responder ip sla responder udp-echo ipaddress 10.0.0.1 port 2000

Make RIP Passive Make hub SLA responder

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

28

Dual homed DMVPN spokes Hub2

Common Subnet Activate redirection interface Tunnel0 bandwidth 1000 ip address 10.0.0.2 255.255.255.0 ip mtu 1400 ip nhrp map multicast dynamic ip nhrp redirect ip nhrp network-id 1 tunnel source Serial1/0 tunnel mode gre multipoint tunnel protection ipsec profile vpnprof ! router rip network 10.0.0.0 passive-interface default

Make RIP Passive

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Dual homed DMVPN spokes Spokes part 1

Hub1 NHRP mappings Hub2 NHRP mappings
interface Tunnel0 bandwidth 1000 ip address 10.0.0.<x> 255.255.255.0 ! <x> = 11,12, ip mtu 1400 ip nhrp map multicast 172.17.0.1 ip nhrp map 10.0.0.1 172.17.0.1 ip nhrp map multicast 172.17.0.5 ip nhrp map 10.0.0.2 172.17.0.5 ip nhrp network-id 1 ip nhrp holdtime 360 ip nhrp nhs 10.0.0.1 ip nhrp nhs 10.0.0.2 tunnel source Serial1/0 tunnel mode gre multipoint tunnel protection ipsec profile vpnprof router rip network 10.0.0.0 network 192.168.<x>.0

Activate RIP

!<x> = 1,2,

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

30

Dual homed DMVPN spokes Spokes part 2

Poll every second Timeout: 1 second Fail after 21 seconds

Poll 10.0.0.1 UDP Port 2000

ip sla 1 udp-echo 10.0.0.1 2000 control disable timeout 1000 frequency 1 threshold 21000 ip sla schedule 1 life forever start-time now track 1 rtr 1 reachability ip route 192.168.0.0 255.255.255.0 10.0.0.1 track 1 ip route 10.0.0.0 255.0.0.0 10.0.0.1 track 1 ip route 192.168.0.0 255.255.255.0 10.0.0.2 254 ip route 10.0.0.0 255.0.0.0 10.0.0.2 254 ip route 0.0.0.0 0.0.0.0 Serial 1/0

Monitor SLA probes Primary routes When track 1 is up Floating routes Kick-in if probes fail (floating statics)

Model shown here makes hub1 primary, hub2 backup Track both hubs to make active-active if desired

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Large Scale DMVPN Hub & Spoke

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

32

Overall solution
HQ network Aggregation router Cluster of DMVPN hubs Aggregates user tunnels SLB balances connections Owns virtual IP address

Hubs

Server Load Balancer

GRE/IPsec tunnels IGP + NHRP

Spokes

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

33

High level description

Spokes believe there is a single hub NHRP map points to the Load Balancers Virtual IP Address The Load Balancer is configured in forwarding mode (no NAT) All the hubs have the same DMVPN configuration
Same Tunnel interface address Same Loopback address (equal to the VIP)

All the spokes have the same DMVPN configuration

Same hub NBMA address Same NHS

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

34

The Load Balancer in general

The Load Balancer owns a Virtual IP Address (VIP) When IKE or ESP packets are targeted at the VIP, the LB chooses a hub The hub choice is policy (predictor) based:
weighted round-robin least-connections

When hub chosen for a tunnel, all packets go to the same hub
stickyness

Once a decision is made for IKE, the same is made for ESP
buddying

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

35

Topology and addresses

10.1.2.0/24

.2
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

.1 10.1.1.0/24 10.1.0.0/24 .1

.3
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

.2

.3

Load Balancer VIP: 172.17.0.1 (no tunnel) Physical: (dynamic)172.16.2.1 Tunnel0: 10.0.0.2

Physical: (dynamic)172.16.1.1 Tunnel0: 10.0.0.1

192.168.1.1/29

Spoke A

Spoke B

192.168.2.1/29

Supernet: 192.168.0.0/16
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

36

Load Balancer
We will use an IOS-SLB
IOS SLB runs on top of c7200 or Catalyst6500 As of today, opt for 12.2S or 12.1E releases

The LB must be able to do layer 3 and 4 load balancing. Upper layers are useless (encrypted) Content Switching Module 3.1 or above will work too but we do not need most of its features (layer 5+) ACE is ok but need to disable NAT-T Any SLB will do

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

37

IOS SLB performances

IOS SLB on a Cat6500 (MSFC-2)
Can manage 1M connections w/ 128MB RAM Can create 20,000 connections per second Switches packets at 10Gbps (64 bytes)

IOS SLB on a c7200 (NPE-400)

Can create 5,000 connections per second Switches packets at the CEF rate (depending on other features)

Typically not a bottleneck

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

38

IOS SLB cluster definition

ip slb probe PINGREAL ping faildetect 2 ip slb serverfarm HUBS failaction purge probe PINGREAL predictor leastconn real 10.1.0.2 weight 4 inservice real 10.1.0.3 weight 4 inservice

For your reference

Least connections (default is round-robin)

If all the hubs are equivalent, the weight is the same for all

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

39

IOS SLB VIP definition

ip slb vserver ESPSLB virtual 172.17.0.1 esp serverfarm HUBS sticky 60 group 1 idle 30 inservice ip slb vserver IKESLB virtual 172.17.0.1 udp isakmp serverfarm HUBS sticky 60 group 1 idle 30 inservice

For your reference

Same farm

Buddying

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

40

Monitoring and managing

SLB-7200#sh ip slb connections

For your reference

vserver prot client real state nat ------------------------------------------------------------------------------IKESLB UDP 64.103.8.8:500 10.1.0.2 ESTAB none ESPSLB ESP 217.136.116.189:0 10.1.0.2 ESTAB none IKESLB UDP 213.224.65.3:500 10.1.0.2 ESTAB none ESPSLB ESP 80.200.49.217:0 10.1.0.2 ESTAB none ESPSLB ESP 217.136.132.202:0 10.1.0.3 ESTAB none SLB-7200#clear ip slb firewallfarm Clear serverfarm Clear vserver Clear <cr> connections connections connections connections ? for a firewallfarm for a specific serverfarm for a specific virtual server

SLB-7200#sh ip slb reals real farm name weight state conns ------------------------------------------------------------------10.1.0.2 HUBS 4 OPERATIONAL 4 10.1.0.3 HUBS 4 OPERATIONAL 1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

41

Hub Tunnel configuration

interface Loopback0 ip address 172.17.0.1 255.255.255.255 end

interface Tunnel0 bandwidth 10000 ip address 10.0.255.254 255.255.0.0 Must be same on all hubs no ip redirects Mask is /32 ip mtu 1400 ip nhrp map multicast dynamic ip nhrp network-id 1 Must be same on all hubs ip nhrp holdtime 3600 Mask allows 216-2 nodes tunnel source Loopback0 tunnel mode gre multipoint tunnel protection ipsec profile tp cdp enable end interface FastEthernet0/0 ip address 10.1.0.{2,3} 255.255.255.0 interface FastEthernet0/1 ip address 10.1.1.{2,3} 255.255.255.0

Physical interface ip addresses unique on each hub

BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

42

Spoke tunnel configuration

Basic DMVPN / ODR configuration
interface Tunnel0 ip address 10.0.0.1 255.255.255.0 ip nhrp map 10.0.255.254 172.17.0.1 ip nhrp nhs 10.0.255.254

Remember
All the spokes have the same configuration

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

Current status Tunnel setup

We now allow spokes to
build a DMVPN tunnel to a virtual hub NHRP-register to their assigned hub
Hubs
NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

Server Load Balancer

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.2.1 10.0.0.2

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spokes
Spoke 1 Spoke 2 Spoke 3 Spoke 4
44

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

Spoke routing configuration

interface Tunnel0 Activate ODR over tunnel Tunnel packet physical cdp enable ip route 0.0.0.0 0.0.0.0 Dialer0 ip route 192.168.0.0 255.255.0.0 10.0.0.1 ip route 10.0.0.0 255.0.0.0 10.0.0.1

Private traffic (summary) Tunnel 0

Physical: (dynamic)172.16.1.1 Tunnel0: 10.0.0.11

Spoke A

192.168.1.1/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

45

Hub Routing Protocol configuration

Only allow private networks in the routing table Prevents recursive routing

interface Tunnel0 Activate ODR over tunnel Tunnel packet physical cdp enable router odr distribute-list 1 in access-list 1 permit 192.168.0.0 0.0.255.255

Redistribute ODR BGP Send information to aggregation router

router bgp 1 redistribute odr neighbor 10.1.1.1 remote-as 1 neighbor 10.1.1.1 next-hop-self
46

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

HQ Edge BGP configuration

For your reference

router bgp 1 no synchronization bgp log-neighbor-changes aggregate-address 10.0.0.0 255.0.0.0 summary-only aggregate-address 192.168.0.0 255.0.0.0 summary-only neighbor HUB peer-group neighbor HUB remote-as 1 neighbor 10.1.1.2 peer-group HUB neighbor 10.1.1.3 peer-group HUB neighbor <other hubs> peer-group HUB no auto-summary

HQ network Aggregation router Cluster of DMVPN hubs Aggregates user tunnels

Hubs

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

Edge router OSPF configuration

OSPF attracts traffic from the HQ DMVPN Floating static route to Null0 discards packets to unconnected spokes
ip route 192.168.0.0 255.255.0.0 Null0 254 router ospf 1 redistribute static network 10.1.2.0 0.0.0.255 area 1 HQ network (10.0.0.0/8) Runs OSPF segment in area 1
10.1.2.0/24

For your reference

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

48

Routing protocols Route Propagation spoke

HQ

aggregation
10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table B 192.168.0.0/29 network 192.168.0.8/29 B B 192.168.0.16/29 B 192.168.0.24/29

NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1 Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.0.16/29 10.0.0.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1

NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1 Routing table o 192.168.0.8/29 10.0.0.2 o 192.168.0.24/29 10.0.0.4 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.2.1 10.0.0.2

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.0.8/29 192.168.0.16/29 192.168.0.24/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

Hub&Spoke packet flow

HQ
Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.2.1 10.0.0.2

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

50

Large Scale DMVPN Spoke Spoke

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

51

Shortcut switching
Spoke configurations get a single extra line:
interface Tunnel0 ip nhrp shortcut ! thats it!!

Hub get an extra line:

interface Tunnel0 ip nhrp redirect ! thats it!!

Spokes on a given hub will create direct tunnels Spokes on different hubs will NOT create tunnels

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

52

Basic spoke-spoke packet flow

HQ
Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

red ire c

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.2.1 10.0.0.2

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

53

Basic spoke-spoke packet flow

HQ
Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

re s rreolu eq dit io ue r ec st n t

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

NHRP table 10.0.255.254 Physical: 172.16.1.1 172.16.0.1

Tunnel:

10.0.0.1

172.16.2.1 10.0.0.2

NHRP table 10.0.255.254 172.16.4.1 172.16.0.1 172.16.3.1

10.0.0.3

10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

54

Basic spoke-spoke packet flow

HQ
Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

re s r e o lu qu tio es n t

NHRP table 10.0.255.254 Physical: 172.16.1.1 172.16.0.1 192.168.16.0/29 172.16.3.1 Tunnel: 10.0.0.1 10.0.0.3 172.16.3.1

172.16.2.1 10.0.0.2

NHRP table 10.0.255.254 172.16.4.1 172.16.0.1 172.16.3.1 10.0.0.1 172.16.1.1 10.0.0.4 10.0.0.3 192.168.1.0/29 172.16.1.1

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

55

Cross-hubs spoke-spoke tunnels

We want spokes to create direct tunnels even if they are on different hubs For this, we link the hubs via a DMVPN NOT a daisy chain!!!

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

56

Linking the hubs

interface Tunnel1 ip address 10.1.3.2 255.255.255.0 no ip redirects ip mtu 1400 ip nhrp network-id 1 ip nhrp redirect ip nhrp map 10.1.3.3 10.1.0.3 tunnel source FastEthernet0/1 end
10.1.2.0/24

Same network ID as Tunnel0 !! Send indirection notifications

.2
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

.1 10.1.1.0/24

.3
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

Tunnel1: 10.1.3.2/24

Tunnel1: 10.1.3.3/24

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Routing across hubs

10.1.2.0/24

.2
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

.1 10.1.1.0/24

.3
Loopback: 172.17.0.1 Tunnel0: 10.0.255.254/16

router bgp 1 neighbor 10.1.3.3 remote-as 1 neighbor 10.1.3.3 next-hop-self

router bgp 1 neighbor 10.1.3.2 remote-as 1 neighbor 10.1.3.2 next-hop-self

Hubs exchange their ODR information directly via BGP The exchange occurs over the inter-hub DMVPN

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

Hub&Spoke packet flow

HQ
Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.8.0/29 10.1.3.3 B 192.168.24.0/29 10.1.3.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/29 10.1.3.2 B 192.168.16.0/29 10.1.3.2 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

red ire c
172.16.2.1 10.0.0.2

NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

59

Hub&Spoke packet flow

HQ
Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.8.0/29 10.1.3.3 B 192.168.24.0/29 10.1.3.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/29 10.1.3.2 B 192.168.16.0/29 10.1.3.2 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1

re s rreolu eq d t io ue ir n st ec
172.16.2.1 10.0.0.2

NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1

Physical: 172.16.1.1 Tunnel: 10.0.0.1

172.16.3.1 10.0.0.3

172.16.4.1 10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

Hub&Spoke packet flow

HQ
Routing table o 192.168.0.0/29 10.0.0.1 o 192.168.16.0/29 10.0.0.3 B 192.168.8.0/29 10.1.3.3 B 192.168.24.0/29 10.1.3.3 B 192.168.0.0 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.1 172.16.1.1 10.0.0.3 172.16.3.1 Routing table B 192.168.0.0/29 network 192.168.8.0/29 B B 192.168.16.0/29 B 192.168.24.0/29 10.1.1.2 10.1.1.3 10.1.1.2 10.1.1.3

Routing table o 192.168.8.0/29 10.0.0.2 o 192.168.24.0/29 10.0.0.4 B 192.168.0.0/29 10.1.3.2 B 192.168.16.0/29 10.1.3.2 B 192.168.0.0/16 10.1.1.1 B 10.0.0.0 10.1.1.1 NHRP table 10.0.0.2 172.16.2.1 10.0.0.4 172.16.4.1 NHRP table 10.0.255.254 172.16.0.1 10.0.0.1 172.16.1.1 172.16.4.1 192.168.1.0/29 172.16.1.1

NHRP table 10.0.255.254 Physical: 172.16.1.1 172.16.0.1 192.168.24.0/29 172.16.4.1 Tunnel: 10.0.0.1 10.0.0.4 172.16.4.1

re s r e o lu qu tio es n t
172.16.2.1 10.0.0.2

172.16.3.1 10.0.0.3

10.0.0.4

Spoke 2 Spoke 4 Spoke 3 Spoke 1 192.168.0.0/29 192.168.8.0/29 192.168.16.0/29 192.168.24.0/29

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

Adding hubs

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

62

Linking the hubs option 1

interface Tunnel1 ip address 10.1.3.2 255.255.255.0 . . . ip nhrp map 10.1.3.3 10.1.0.3 ip nhrp map 10.1.3.4 10.1.0.4 ip nhrp map 10.1.3.5 10.1.0.5 . . . end

Create a manual full mesh Do the same with BGP

10.1.2.0/24

.2

.3

.1 10.1.1.0/24

.4

.5

Tunnel1: 10.1.3.2/24

Tunnel1: 10.1.3.3/24

Tunnel1: 10.1.3.4/24

Tunnel1: 10.1.3.5/24

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

63

Linking the hubs option 2

interface Tunnel1 ip address 10.1.3.2 255.255.255.0 ip nhrp network-id 1 ip nhrp redirect ip nhrp map 10.1.3.1 10.1.0.1 ip nhrp nhs 10.1.3.1 end
10.1.2.0/24

Use the edge router as NHRP hub Use the edge as a RR

Tunnel1: 10.1.3.1/24

.2

.3

.1 10.1.1.0/24

.4

.5

Tunnel1: 10.1.3.2/24

Tunnel1: 10.1.3.3/24

Tunnel1: 10.1.3.4/24

Tunnel1: 10.1.3.5/24

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

64

Large Scale Design Summary

Virtually limitless scaling w/ automatic load management Load balancing AND resilience Multiply performances by number of hubs
Tunnel creation rate, speed, max SAs

Resilience in N+1 No need to touch the hubs while adding a spoke All spokes have the same configuration New hubs can be added/removed on the fly
BGP needs to be told about the new hub EIGRP may be used instead of BGP full automatic

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

65

Virtual Routing & Forwarding (VRF)

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

66

VRFs very short rehearsal

VRFs are virtual routers inside a router Each VRF has its own routing table that it does not share with other VRFs An interface can belong to a single VRF at a time
! define VRF red ip vrf red ! give interfaces to VRF red interface FastEthernet 0/0 ip vrf forwarding red

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

Router without VRF

Layer 5+ IKE AAA

Layer 4 Layer 3 helpers Layer 3 Layer 2 Loopback Tunnel

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

68

Forwarding without encapsulation

Layer 5+ IKE AAA

Layer 4 Layer 3 helpers Layer 3 Layer 2

Routing

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

69

Forwarding with encapsulation

Layer 5+ IKE AAA

Layer 4 Layer 3 helpers Layer 3 Layer 2

Routing

Routing

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

70

Add VRFs to the router

ip vrf red ip vrf blue ip vrf green interface FastEthernet 0/0 ip vrf forwarding red interface FastEthernet 0/1 ip vrf forwarding red interface Tunnel 0 ip vrf forwarding red

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

71

Router with VRFs

Layer 5+ IKE AAA

Layer 4 Layer 3 helpers Layer 3 Layer 2 VRF Global VRF Red VRF Blue VRF Green

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

72

Source the tunnel from a VRF

interface FastEthernet 1/0 ip vrf forwarding blue interface Tunnel 0 ip vrf forwarding red tunnel source FastEthernet 1/0 tunnel destination tunnel vrf blue Determines how GRE packets are routed out
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

73

VRF tunneling
Layer 5+ IKE AAA

Layer 4 Layer 3 helpers Layer 3 Layer 2 VRF Global VRF Red VRF Blue VRF Green

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

74

Watch out the network ID

interface Tunnel 0 ip vrf forwarding red tunnel source FastEthernet 1/0 tunnel destination ip nhrp network-id 1 tunnel vrf blue Several tunnels can share the same nhrp network-id BUT Any given network-id can only appear in a single VRF
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

75

ISAKMP profiles in DMVPN

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

76

Purpose of the exercise

Assume two groups of users
Finance and Engineering

The hub hosts two DMVPNs,

On the same the tunnel-source

Each group of user should access its own DMVPN

And not the other

Each DMVPN sits in its own VRF

To fully separate the traffic from each group

We will use ISAKMP profiles to solve the exercise

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

Multi-DMVPN on a single hub

Single HUB terminating Two distinct DMVPNs
192.168.0.0/24 .1 Physical: 172.17.0.1 Tunnel2: 10.0.1.1 Physical: 172.17.0.1 Tunnel1: 10.0.0.1

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

Assume two groups of users

Group 1 Engineering
Certificate Status: Available Certificate Serial Number: 100 Certificate Usage: General Purpose Issuer: cn=blue-lab CA o=CISCO Subject: Name: Router100.cisco.com o=CISCO ou=Engineering ou=Engineering Validity Date: start date: 14:34:30 UTC Mar 31 2004 end date: 14:34:30 UTC Apr 1 2009 Associated Trustpoints: LaBcA

Group 2 Finance
Certificate Status: Available Certificate Serial Number: 300 Certificate Usage: General Purpose Issuer: cn=blue-lab CA o=CISCO Subject: Name: Router300.cisco.com o=CISCO ou=Finance ou=Finance Validity Date: start date: 14:34:30 UTC Mar 31 2004 end date: 14:34:30 UTC Apr 1 2009 Associated Trustpoints: LaBcA

There is a single CA Each user either belongs to ou=Engineering or ou=Finance

79

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

What are ISAKMP profiles ?

ISAKMP profiles map an IKE session to an IPsec SA IKE sessions are identified by
Peer identity VRF Local-address

IPsec SAs can be derived from

a crypto map an IPsec profile

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

80

Certificate maps
We need to map users to their respective tunnels The only useful attribute is the Organization Unit (ou)

crypto pki certificate map engineering_map 10 subject-name co ou = Engineering crypto pki certificate map finance_map 10 subject-name co ou = Finance

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

81

Defining the ISAKMP profiles

We now define one ISAKMP profile per group Each ISAKMP profile will match users of a given group
crypto isakmp profile eng-ikmp-prof eng- ikmppki trustpoint LaBcA match certificate engineering_map

crypto isakmp profile fin-ikmp-prof fin- ikmpset trustpoint LaBcA -ikmp-prof isakmpeng pki isakmp-profile eng-ikmpmatch certificate finance_map

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

82

The IPsec profiles

Two IPsec profiles are necessary
Each profile maps to a distinct ISAKMP profile
crypto ipsec profile eng-ipsec-prof eng- ipseccrypto ipsec transform-set high-security set isakmp-profile eng-ikmp-prof isakmpeng- ikmp-

crypto ipsec profile fin-ipsec-prof fin- ipseccrypto ipsec transform-set high-security set isakmp-profile fin-ikmp-prof isakmpfin- ikmp-

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

83

Defining the tunnels

interface tunnel1 ip vrf forwarding Engineering ip address 10.0.0.1 255.255.255.0 tunnel key 1 ip nhrp network-id 1 ip nhrp tunnel source loopback0 tunnel protection ipsec profile eng-ipsec-prof eng- ipsecinterface tunnel2 ip vrf forwarding Finance ip address 10.0.1.1 255.255.255.0 tunnel key 2 ip nhrp network-id 2 ip nhrp tunnel source loopback0 tunnel protection ipsec profile fin-ipsec-prof fin- ipsecBRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Each tunnel links To a specific ISAKMP Profile

84

Session mapping example

Incoming session Cert. authentication Engineering ISAKMP Profile Certificate inspected

IKE
Finance ISAKMP Profile

Turns out ou=Engineering

Engineering Tunnel

Finance Tunnel

IPsec SAs are linked to the Engineering tunnel

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

85

DMVPN IPv6

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

86

DMVPN IPv6
NHRP supports IPv6 since 12.4(20)T Feature is very similar to v4 support
Registrations, resolutions, Only DMVPN phase 3 is supported no phase 2 design!! No VRF support yet due to routing protocols limitations

Only support for IPv6 over IPv4

All NBMA addresses must be IPv4

V4 and V6 overlays supported simultaneously

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

87

Spoke configuration
Spoke
interface Tunnel0 ipv6 address fe80::2002 link-local ipv6 address 2001::2/64 ipv6 nhrp map 2001::1 172.17.0.1 ipv6 nhrp map multicast 172.17.0.1 ipv6 nhrp nhs 2001::1 ipv6 nhrp network-id 1 tunnel mode gre multipoint tunnel source tunnel protection ipsec profile
IPv4 address or interface Business almost as usual Unique Link-Local address Global or Locally Reachable addr.

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

88

Hub configuration
Hub
interface Tunnel0 ipv6 address fe80::2001 link-local ipv6 address 2001::1/64 ipv6 nhrp network-id 1 ipv6 nhrp map multicast dynamic tunnel mode gre multipoint tunnel protection ipsec profile tunnel source
IPv4 address or interface Business almost as usual Unique Link-Local address Global or Locally Reachable addr.

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

89

Subtle differences
Hub#show ipv6 nhrp 2001::2/128 via 2001::2 Tunnel0 created 00:04:47, expire 01:59:49 Type: dynamic, Flags: unique registered used NBMA address: 1.0.0.2 2001::3/128 via 2001::3 Tunnel0 created 00:04:03, expire 01:59:49 Type: dynamic, Flags: unique registered used NBMA address: 1.0.0.3 FE80::2/128 via 2001::2 Tunnel0 created 00:04:47, expire 01:59:49 Type: dynamic, Flags: unique registered NBMA address: 1.0.0.2 FE80::3/128 via 2001::3 Tunnel0 created 00:04:43, expire 01:59:49 Type: dynamic, Flags: unique registered NBMA address: 1.0.0.3

Global and Link-Local registered!!

Global and Link-Local registered!!

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

90

Per-tunnel QoS

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

91

The need for QoS the obvious

QoS is needed for
Sharing network bandwidth Marshaling applications bandwidth usage Meeting applications latency and speed requirements

MQC is a CLI allowing the configuration of

Bandwidth upper limits (policing, shaping) Bandwidth lower limits (cbwfq) Low Latency Queuing (priority queuing)

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

92

Need for QoS the greedy spoke

Interface w/ limited downstream rate

Hub

ISP router

Greedy Spoke 3

Crypto engine or Wan link

Spoke 1 Spoke 2

The greedy spoke calls for a lot of traffic (VoIP calls, DB x-fer,...) It overruns the hub CE or the WAN link
Packets are dropped Starves other spokes

Greedy spoke downlink gets overloaded and packets are dropped

damages data throughput, impacts phone conversations

We want to limit the amount of traffic sent to each spoke

BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

93

QoS and (DM)VPN problem statement

QoS with MQC is complex to deploy with DMVPN Static MQC configuration
Long configurations on hubs Only works with static spoke addresses

Performances of QoS/MQC is weak with lots of shapers Pre-Crypto-Engine QoS is limited

Only priority queuing

Serious QoS can only be applied after the crypto engine

Classification uneasy after packet encapsulation (DSCP) Pre-classification not always useful (e.g. NBAR) Shaping, multiple classes, etc only in MQC

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

94

Horror MQC policy DMVPN

Problem: static and slow
policy-map child class routing-protocol bandwidth 100 kbps class voice priority 200 kbps class data police 500 kbps class class-default ! policy-map parent class tunnel1 bandwidth 400 kbps shape average 1mbps service policy child class tunnel2 bandwidth 400 kbps shape average 1mbps service policy child class tunnel3 bandwidth 400 kbps shape average 1mbps service policy child class class-default shape average 2mbps Interface Tunnel0 (qos pre-classify optional) Interface GigabitEthernet0/1 service-policy out parent

Access-list 101 permit esp hub class-map tunnel1 match access-group 101 Access-list 102 permit esp hub class-map tunnel2 match access-group 102 Access-list 103 permit esp hub class-map tunnel3 match access-group 103

spoke1

spoke2

spoke3

The configuration goes on and on

BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

95

Changes to the QoS infrastructure

MQC stands for Modular QoS CLI
MQC was also the name of the queuing and scheduling infrastructure

The situation has changed

12.4(15)T introduced CCE 12.4(20)T introduced HQF

Mostly internal changes but there is an impact

MQC CCE HQF

CLI Common Classification Engine Hierarchical Queuing Framework

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

96

Per-tunnel QoS
Per Tunnel QoS will apply dynamic per spoke QoS policy on hub
Spokes are be split into groups Groups are mapped to a QoS template

HQF / CCE framework will be used

Performances improve over current MQC framework

The feature will apply to DMVPN and EzVPN dVTI

Not supported for crypto map based designs

Hub CE and WAN link overruns are rare

WAN link overrun could be addressed with aggregate QoS

Spoke downlinks overruns are more frequent

Nothing could be done This is the primary goal of per-tunnel QoS

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

97

Per-tunnel QoS high level view

Classification happens at the tunnel level
Before encapsulation and before the crypto engine

Policing (dropping) and marking also applied at tunnel Queuing and scheduling happen at the physical interface
QoS policy Classification QoS Policy policing, marking Tunnel 1 - data SA classification Tunnel 1 - voice Tunnel 2 - data Tunnel 2 - voice Tunnel 3 - data Tunnel 3 - voice Crypto Engine Hierachical queueing per Tunnel Data Voice Data Voice Data Voice Derived Interface QoS policy Tunnel 1 policy Tunnel 2 policy Tunnel 2 policy

Physical Interface

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

98

More per-tunnel QoS information

Performances depend on
The number of tunnels The number of active shapers

Policy Provisioning via CLI and AAA Available on 7200 and 3800
Catalysts will require next-generation VPN hardware (5g) ASR agenda still TBD

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

99

Provisioning DMVPN QoS

Spokes group 1
interface Tunnel0 ip nhrp group <name1> policy-map PM1 class class-default shape average 1000000 Offer 1 Mbps to each tunnel

Group 2
interface Tunnel0 ip nhrp group <name2>

HUB

Policy-map PM2 class class-default shape average 500000 interface Tunnel0 ip nhrp map group <name1> service policy output PM1 ip nhrp map group <name2> service policy output PM2 Offer 500 kbps to each tunnel

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

100

QoS policy limiting tunnel bandwidth

Hub
class-map Control match ip precedence class-map Voice match ip precedence policy-map PM1 class class-default shape average 1000000 interface Tunnel0 ip nhrp map group G1 service-policy output PM1
Offer 1Mbps to each tunnel

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

101

Hierarchical shaper
Tunnel bandwidth parent policy
Each tunnel is given a maximum bandwidth A shaper provides the backpressure mechanism

Protected packets are processed by the client policy

There would be several policies: bandwidth, llq, etc.

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

102

QoS policy limiting tunnel bandwidth

Hub
class-map Control match ip precedence class-map Voice match ip precedence policy-map PM1 class class-default shape average 1000000 service-policy SubPolicy policy-map SubPolicy class Control bandwidth 20 class Voice priority percent 60
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

Offer 1Mbps to each tunnel

20Kbps guaranteed to Control

LLQ for voice

103

DMVPN vs. GET VPN

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

104

GET VPN in a nutshell

GET VPN introduces two entities
Group Members (GM) Key Servers (KS)

GMs register to KS using IKE and GDOI

GDOI is Group-IKE or IKE for multicast

KS send the same Traffic Encryption Key to the GMs

The GMs use that TEK to encrypt/decrypt data packets

Data packets are encapsulated in ESP but the IP header is preserved

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

105

10,000 feet over GET VPN

TEK
192.168.0.0/24 .1 Key server

Group Member

TEK

TEK
IP(s=PC,d=Web) TCP .37 IP(s= PC,d=Web) .1 192.168.3.0/24 Web

IP(s=PC,d=Web) ESP IP(s= PC,d=Web)

TEK TEK
.1 192.168.1.0/24 .25 IP(s=PC,d=Web) TCP IP(s= PC,d=Web) PC 192.168.2.0/24 .1

.37 Web

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

106

Scopes of DMVPN and GET VPN

DMVPN is an overlay VPN Creates tunnels over the transport network
Isolates protected networks from transport network Allows private protected addresses over a public transport network

Hubs concentrate connections all spokes must connect

Hubs concentrate part of the spoke-spoke traffic Hubs need to know about all the private networks RP scale

Multicast requires replication before encryption usually on hubs GET VPN is a proxy VPN Encrypted packets have the same addresses as the protected packets
Does not isolate address spaces requires end-to-end routing

KS concentrate connections all GM must connect

KS do not concentrate any traffic

Transport network takes care of routing packets Multicast can happen in the core if core supports it
BRKSEC-3006 2009 Cisco Systems, Inc. All rights reserved. Cisco Public

107

GET and DMVPN not enemies

GET only works if protected addresses are routable
Usually recommended over an other (Virtual) Network (MPLS) Core needs to be multicast aware for mcast to work at all

When the transport network is optimized When the transport network is dumb Some designs link GET and DMVPN
Making DMVPN hubs also Group Members

GET has a lead DMVPN just works

DMVPN over Internet links to GET over MPLS Takes the best of both worlds

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

108

12.4 T DMVPN New Features Summary

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

109

DMVPN Enhancements
Previous Limitation
Large routing tables at spokes sometimes caused network instability.

For your reference

New Feature & Associated Benefits

Shortcut switching introduced
Route summarization now possible Higher scalability 12.4(6)T

Packets CEF switched via hub

Delays in setting up voice calls between spokes. Reduced latency during call setup 12.4(6)T Complex interconnection of Hubs to expand DMVPN Spoke-toSpoke Networks.

NHRP resolution requests forwarding

Simplified hub network design Improved resiliency. 12.4(6)T

NAT/PAT not possible in spokespoke designs

NAT and static PAT now supported

12.4(9)T

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

110

DMVPN Enhancements
Previous Limitation New Feature & Associated Benefits
DMVPN debug enhancements
Complex troubleshooting All tables with a single show command Per-peer debugging also possible 12.4(9)T

For your reference

NHRP MIB
Network monitoring difficult or impossible Monitoring of NHRP tables via SNMP 12.4(20)T

DMVPN IPv6
Limited to IPv4 Allows IPv6 in the overlay network 12.4(20)T

Complex QoS configuration. Not working well with dynamic spoke NBMAs.

Per-tunnel QoS introduced

12.4(22)T

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

111

Session Summary

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

112

Shortcut switching Routing protocols revisited

OSPF does not bring anything new
Same requirements as in phase 2

EIGRP can be tuned to summarize routes to spokes

Number of neighbors increases still requires attention

ODR can now be used for spoke-to-spoke configs

1200 neighbors possible

RIP passive can now be used for spoke-to-spoke

1500 neighbors possible

Different protocols can be used between hubs and between hub-spoke

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

113

Summary
Phase 3 subtly different from phase 2
Most visible on the routing topology

Shortcut switching helps picking the best protocol

Usually, the choice relates to scalability

DMVPNv6 is now a reality

one more step in the right direction

Per-SA QoS finally made it ISAKMP profiles enhance security of multi-DMVPN

Very useful for VRF separation

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

114

Recommended Sessions

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

115

Recommended sessions
Server Load Balancing Design
BRKAPP-2002 by Floris Gransvarle

Advanced IPsec with GET VPN

BRKSEC-3011 by Frederic Detienne

Advanced Topics in Encryption Standards and Protocols

BRKSEC-3014 by Frederic Detienne

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

116

Q and A

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

117

Meet The Expert

To make the most of your time at Cisco Networkers 2009, schedule a Face-to-Face Meeting with a top Cisco Expert. Designed to provide a "big picture" perspective as well as "in-depth" technology discussions, these face-to-face meetings will provide fascinating dialogue and a wealth of valuable insights and ideas. Visit the Meeting Centre reception desk located in the Meeting Centre in World of Solutions

Session ID Presentation_ID

2007 Cisco Systems, Inc. All rights reserved.

Cisco Public

118

BRKSEC-3006

2009 Cisco Systems, Inc. All rights reserved.

Cisco Public

119