Network Automation

Master Class

Network Automation

[20110324 - Copenhagen]

Bruno Klauser
Consulting Engineer NMS/OSS
European Markets
[email protected]
wwwin-people.cisco.com/bklauser
Schedule
Day 1
8:30-09:00 Welcome & Coffee
9:00-12:30 Theory Block I
0. Introduction
1. Service planning
2. Deployment and activation
13:30-17:00 Hands-on Lab

Day 2
8:30-09:00 Welcome & Coffee
9:00-12:30 Theory Block II
3. Testing and verification
4. Ongoing service assurance
5. Troubleshooting and optimization
13:30-17:00 Hands-on Lab
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 3
Agenda

 Introduction & Overview

Service Planning
Service Deployment & Activation
Service Testing, Verification & Assurance
Troubleshooting & Optimization
Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 4
Why Network
Automation?
 Availability
Excessive OSPF messages
force US Telco to bring down
parts of ATM network:
 26 hrs Outage
 several Million US$ Impact

Bad redundancy implementation

forces traffic through a 64kbit
Resolution Time

undersea cable:
 4 hrs Outage
 several Million £ Impact

LSP black hole issue forces

Airline to ground all planes:
 20 minutes Outage Lack of memory in a switch
 several Million US$ Impact causes Intermitted outages
on trading floor – Impact:
 1 Million € per 1 minute
Inadequate QoS on GigE link
of bookstore impacts 10‘000
transactions per second:
 Millions of US$ in seconds

1995 2000 2005 2010 2015

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 6
Design Variability and Complexity – 1/2
What do these have in common ?

 IP Connectivity
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 7
Design Variability & Complexity
Design Variability and Complexity – 2/2

Source: http://networkcomplexity.org/wiki/index.php?title=Definition

1995 2000 2005 2010 2015

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 8
 Automation and Differentiation
Program
Embedded Automation
Systems (EASy)
Business Value / Revenue Potential

Compute
Customize Cloud, XaaS,
Computing
Device Manageability
Instrumentation (DMI)
Transaction Experience SLA
Collaborate
Unified Comms
Configure Security
Basic Instrumentation
GET / SET
Quality of Service SLA
Increase in
Connect
- Autonomicy
Managed Network
- Application awareness
Services
- Real-time management
- Custom requirements
Basic SLA - Programmability

1995 2000 2005 2010 2015

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 9
An Analogy

Airplane Router

Embedded
Instruments
Automations

21‘000 sensors OIDs in MIBs

With increasing scale, complexity, differentiation and availability

requirements, operators rely on Embedded Automations

From: Full control by a single central authority

To: Operating a system of self-managing components
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 10
What is Network
Automation?
Network Automation
Taxonomy of Network Automations

By Infrastructure Span
 Device level
 Domain wide
 Service end-to-end

By Automation Function
 Task Execution
 Workflow Orchestration
 Decision Triggers

By Adoption Type and Benefit …

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 12
 Network Automation
Network Automation Adoptions
Automation
as Integral Part of
Network Design
Value to the Business

Benefits:
Automate - Revenue Enabler
New Task - OPEX 
- CAPEX 
Benefits: - Quality 
Automate - OPEX  - Reactive  Proactive
Existing Task - CAPEX  - Corporate Learning
- Quality  Enabler
- Reactive  Proactive
Benefits:
- OPEX 
- Quality 

Level of Experience and Sophistication

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 13
 Example: Sharing Information 1/2
Problem: Sometimes we need to quickly get some parameters from a
website and share information from the router (or a neighboring device)
across organizational and technical borders …
Solution I: Initiate a Project to make use of SNMP, Syslog, Event
Management Software, Reporting, Provisioning and CRM Systems ...
Solution II: Use Cisco IOS DMI to gather the information and EEM/Tcl to
post it via http to a shared location
1. Import the http package into your EEM TCL Policy
namespace import ::http::*

2. Gather and format whatever information you need

3. Build your query for the HTTP POST operation

set my_query [::http::formatQuery "status" $my_info]
4. Use the HTTP reply to get and HTTP POST to share your information
set my_reply [::http::geturl $my_server_url -query $my_query]

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 14
 Example: Sharing Information 2/2

> 900 d
ownloa
ds from
ciscobe
yo n d

See: http://twitter.com/EASyDMI
Note: it is NOT recommended to use a public site or feed other than for demo purpose
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 15
 Network Automation
Example: Integrating CleanAir and Security
Problem: A new rogue WLAN device in sensitive areas should be detected
by Cisco CleanAir and automatically focus/pan/zoom a security camera.

Solution: Use Network Automation based on Cisco IOS Embedded Event

Manager to receive an SNMP Notification from WLC and trigger the Video
Operations Manager via HTTP

1. Rogue WLAN Device added

2. Rogue Device detected by CleanAir AP

1
2 6
3. WLC sends SNMP Notification

4. EEM triggers upon SNMP Notification

5. EEM notifies VSOM via HTTP

ATM
6. Security Camera Focus/Pan/Zoom
EEM
4
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 16
Example 1: NBAR Effectiveness Monitoring
Problem: Application protocols as well as user behavior are changing, hence
the traffic mix changes too. We need to permanently assess how effective the
NBAR deployment is – especially when using CBQoS with match protocol.

Solution: Automate the comparison between ‘unknown’ versus ‘total’ traffic

Router# show ip nbar protocol-discovery top-n 5 Serial0/0

Input Output
Protocol Packet Count Packet Count
Byte Count Byte Count
5 minute bit rate (bps) 5 minute bit rate (bps)
---------- ------------------------ ------------------------
: : :
unknown 205 204
14976 10404
0 0
Total 41304 40944
2649809 2619839
3000 3000

Upon low % of traffic recognized by NBAR, it’s time to check for new PDLMs …

See: Available as an EASy Package:

http://www.cisco.com/go/easy
See: Scripts available from CiscoBeyond:
http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=2101
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 17
Example 2: Connectivity Verification
Problem: We need a failover from primary to secondary link – but with
flexibility and custom notification beyond what a simple routing protocol based
solution provides

Solution: Automate based on IP SLA, EOT and Embedded Event Manager

Upon State Change

Did
IP SLA
succeed Operation timeout

Tracked object is up, Tracked object is down,

Execute up commands Execute down commands

Is Is
No No
up-syslog down-syslog
set? set?
Yes Yes

Send up syslog Send down syslog

done

See: Available as an EASy Package:

http://www.cisco.com/go/easy
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 18
What are the key IOS
Technologies for EASy?
 That is exactly what we will
discuss during the next
537 slides …

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 20
 Operating Models – 2/2

Is there room for yet How to configure? Is it working as specified?

another service?

What if something goes wrong? Are we meeting SLA?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 21
 Operating Models – 2/2

Is it built to
Specification? How to take
Does it meet out of service?
Requirements?
…

Network Automation goes beyond ‚just‘ the Operational Life Cycle

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 22
 Introduction & Overview
Feature Availability
 Main focus on what is available in IOS 15.1T on ISR platforms
 Most Features have been around for some time already
 More Details in Appendix II
 Feature Navigator: www.cisco.com/go/fn
12.4(4)T 12.4(2)T 12.3(14)T 12.3(4)T 12.3(2)T 12.2(12)T
Cisco
Cisco 7301 Cisco
Cisco 7304
and 7200 Catalyst X
Cisco Catalyst 3750X& X X X X
Router 4500 Series 2900
Routers 6500 Series
X X
Series X X X X
12.2SB 12.2SB/SR 12.2SX/ SR 12.2SG 12.2SE T
X X X X X
12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXH 12.2(12th)SG 12.2(6th)SE 12.3(2)T
X X X X
12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXH 12.2(12th)SG 12.2(6th)SE 12.3(4)T

12.2(25)S 12.2(31)SB 12.2(1st)SXH

X
12.2(12th)SG
X
12.2(6th)SE
X
12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXI 12.2(11th)SG 12.2(44)SE 12.3(14)T

12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXI 12.2(12th)SG 12.2(6th)SE 12.4(2)T

12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXI 12.2(12th)SG 12.2(6th)SE 12.4(4)T

12.2(1st)SB5 12.2(1st)SRC 12.2(1st)SXI 12.2(31)SGA NA NA

12.2(31)SB 12.2(31)SB 12.2(1st)SXH 12.2(12th)SG 12.2(6th)SE

12.2(31)SB 12.2(31)SB HD 12.2(13th)SG 12.2(7th)SE 12.5(2nd)T

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 23
 Device Manageability Instrumentation

DC Headquarters

Cisco IOS® Device Manageability Instrumentation (DMI)

Fault Configuration Performance Accounting
 IPIPOAM—Ping,
OAM—Ping,Trace,
Trace, BFD,  Config
ConfigCLI
CLI—diff, logging, AutoIPIPSLA—delay,
 Auto SLA—delay,jitter,
jitter, FlexibleNetFlow
 Flexible NetFlow—
ISG per session lock, replace, rollback loss probability
packet loss, IETF IPFIX
 802.3ah—Link monitoring  E-LMI—parameter and  CBQoS MIB—class-based  BGP policy accounting –
and remote fault indication status signaling QoS includes AS information
 802.1 ag—Continuity  E-DI—Enhanced Device  NBAR  Periodic MIB bulk data
check, L2 ping, trace, AIS Interface, CLI, Perl, IETF  RMON collection and transfer
 MPLS OAM—LSP ping, Netconf  EPC – Embedded Packet  …
LSP trace, VCCV  EMM — Embedded Menu Capture
 EEM—Embedded Event Manager  ERM—Embedded
Manager  NETCONF—IETF Resource Manager
 EVENT-MIB—OID-based NETCONF XML PI  GOLD—Generic Online Security
triggers, events, or SNMP  CNS and WSMA Diagnosis
Set, IETF DISMON  TR-069  Smart Call Home—  Auto Secure—one-touch
 EXPRESSION-MIB—OID  KRON—command preventive maintenance device hardening
expression-based triggers, scheduler  VidMon—Video Monitoring  LDP Auth—message
IETF DISMON  AutoInstall—bootstrapping  … authentication
 …  IOS.sh —IOS Shell  Routing Auth—MD5
 SmartInstall authentication, BGP, OSPF
 Auto SmartPorts …
 …

Device Manageability Instrumentation Has Evolved

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 24
 Packaging Embedded Automations
Problem: Automations may consist of multiple elements – how to deploy
them in a professional and efficient manner ?
Solution I: Write detailed requirements and step-by-step instructions
Solution II: Create an installable EASy package
 Package Description
 Pre-Requisite Verification
EASy Installer = Menu Guided Installation

 Pre-Installation Config +
 Pre-Installation Exec MyPackage.tar
 Environment Variables
 Configuration Router# easy-installer tftp://10.1.1.1/mypackage.tar flash:/easy
-----------------------------------------------------------------
 Files Configure and Install EASy Package ‘mypackage-1.03'
 Post-Requisite Verification -----------------------------------------------------------------
1. Display Package Description
 Post-Installation Config 2. Configure Package Parameters
 Post-Installation Exec 3. Deploy Package Policies
4. Exit
 Uninstall
Enter option: 2
See: http://www.cisco.com/go/easy
See: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps10777/application_note_c27-574650.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 25
 For Your
Reference
Embedded Automation Systems
Embedded Automation Systems (EASy)
1. Browse and Download EASy Packages
www.cisco.com/go/easy

2. Make Sure to also download EASy Installer

3. Browse Other Embedded Automations

www.cisco.com/go/ciscobeyond

4. Learn About The Technology Under The Hood

www.cisco.com/go/instrumentation
www.cisco.com/go/eem
www.cisco.com/go/pec

5. Discuss, Ask Questions, Suggest Answers

supportforums.cisco.com

6. Upload your own Examples to CiscoBeyond

www.cisco.com/go/ciscobeyond

7. Engage via [email protected]

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 26
Agenda

Introduction & Overview

 Service Planning
Service Deployment & Activation
Service Testing, Verification & Assurance
Troubleshooting & Optimization
Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 27
How is my current Use
of Resources ?
 Service Planning
Embedded Resource Manager (ERM)
 The ERM framework tracks resource depletion and resource
dependencies across processes and within a system
 Monitor thresholds for CPU, buffer, and/or memory
 For system or line card
 ERM can define “group”, i.e.
group of different CPU processes
 CISCO-ERM-MIB
 Interface into EEM

Available from: IOS 12.2(33)SRB, 12.4(15)T

Platforms: UC520, 800, x8xx ISR,x900x ISR, 65xx, 72xx, 73xx, 75xx, 76xx, 10xxx
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 29
 Service Planning
Example – Monitoring Resources

Problem: During the planning cycle, we would like to understand if total

CPU usage reaches critical levels
Solution: Define an ERM policy to notify upon resource depletion
resource policy
policy my-erm-policy-1 type iosprocess
system
cpu total
critical rising 90 interval 15 falling 20 interval 10 global
major rising 70 interval 15 falling 15 interval 10 global
minor rising 60 interval 15 falling 10 interval 10 global
!

 If Total CPU usage count rises above 90% at an interval of 15s, a

Critical Up notification is sent

Feb 17 13:32:18.283: %SYS-4-CPURESRISING: System is seeing global

cpu util 62% at total level more than the configured minor limit 60%

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 30
 Service Planning
Example – Monitoring Multiple Processes
Problem: In order to detect resource consumption caused by brute force login
attempts, we want to keep an eye on CPU utilization by the login processes
Solution: Define an ERM policy to notify upon critical / suspicious levels
resource policy
policy my-login-policy type iosprocess
system
cpu process
critical rising 30 interval 10 falling 20 interval 10
major rising 20 interval 10 falling 10 interval 10
minor rising 10 interval 10 falling 5 interval 10
user group my-login-group type iosprocess
instance "SSH Process"
instance "SSH Event handler“
:
policy my-login-policy
 Syslog if Group CPU Usage Count Rises Above 10% at an Interval of 10s
*Aug 25 12:56:26.089: %SYS-4-CPURESRISING: Resource group my-login-group is
seeing local cpu util 16% at process level more than the configured minor limit
10%
*Aug 25 12:56:41.089: %SYS-6-CPURESFALLING: Resource group my-login-group is no
longer seeing local high cpu at process level for the configured minor limit
10%, current value 0%
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 31
Exporting MIB
Statistics?
 Service Planning
Quickly export SNMP Statistics?
Problem: Sometimes we need data from one or multiple MIBs, but
- we may not want to (re-)configure an NMS
- don’t want to constantly poll
- need to gather data during temporary loss of connectivity

Solution: Use Bulk File MIB to define the data we need and
periodically transfer it to a convenient location
- group data from multiple MIBs
- single, common polling interval
- buffer data
- transfer using RCP, FTP, TFTP
- format ASCII or Binary

Feature Name: Periodic MIB Data Collection and Transfer Mechanism

Available from: IOS 12.0(24)S, 12.2(25)S, 12.3(2)T, IOS XE 2.1, IOS XR 3.2
Platforms: ASR1k, x8xx ISR, x900x ISR, 72xx, 73xx, 76xx, 10xxx, ME3400, C4k, C6k, …
See: http://tools.cisco.com/Support/SNMP/do/BrowseOID.do?local=en&translate=Translate&objectInput=1.3.6.1.2.1.2
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 33
 Service Planning
Configuration – Example
1. Define Lists of relevant OIDs (Names for IF-MIB, ASN.1 for all others)
Router(config)# snmp mib bulkstat object-list my-if-data
Router(config-bulk-objects)# add ifIndex
Router(config-bulk-objects)# add ifDescr
What Data am I add
Router(config-bulk-objects)# interested in?
ifAdminStatus
Router(config-bulk-objects)# add ifOperStatus
Router(config-bulk-objects)# exit

2. Specify Polling Schema

Router(config)# snmp mib bulkstat schema my-if-schema
Router(config-bulk-sc)# object-list my-if-data
Where and when
Router(config-bulk-sc)# do I want1to poll Data?
poll-interval
Router(config-bulk-sc)# instance exact interface FastEthernet0
Router(config-bulk-sc)# exit

3. Configure the Transfer Mechanism – and enable it !

Router(config)# snmp mib bulkstat transfer my-fa0-transfer
Router(config-bulk-tr)# schema my-if-schema
Router(config-bulk-tr)# transfer-interval 5
How do I want
Router(config-bulk-tr)# to export
url primary Data?
tftp://10.10.10.10/folder/
Router(config-bulk-tr)# retain 30
Router(config-bulk-tr)# buffer-size 4096
Router(config-bulk-tr)# enable
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 34
What if it’s neither in
ERM nor a MIB?
 Service Planning
Expression MIB
 Allows you to create new SNMP objects based upon existing MIB
variables and formulas
 Interesting when combined with the EVENT-MIB
 EXPRESSION MIB proposed by Cisco to IETF DISMON Working
Group, accepted standard track RFC-2982
Based on IETF draft, again in the DISMON Working Group, and numbered in
Cisco’s namespace

 3 Phases:
MIB Introduction, SNMP Only - 12.0(5)T
However “show command” exists
However “debug command” exists
Introduction of Scriptable Interface
Introduction of CLI Support - 12.4(20)T

See: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_cfg_snmp_sup.html
Available from: IOS 12.0(5)T (EXPRESSION-MIB), 12.3(7)T (SNMPset in TCL script), 12.4(20)T (CLI)
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 36
 Service Planning
Event-MIB

 The EVENT MIB provides a superset of the capabilities of the

RMON alarm and event
 EVENT MIB can monitor
- any MIB object (existence)
- any integer/counter (boolean, threshold)
 EVENT-MIB sends an SNMP notification in response to a
trigger (like RMON) but add the concept of setting a MIB
object (integers)
 EVENT-MIB can specify which variables to add to the
notification
 RFC 2981-compliant introduced in 12.2(4)T
 Configuration support via CLI added in 12.4(20)T

See: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_cfg_snmp_sup_ps6441_TSD_Products_Configuration_Guide_Chapter.html#wp1125529
Available from: IOS 12.2(4)T (EVENT-MIB), 12.3(7)T (SNMPset in TCL script), 12.4(20)T (CLI)
Platforms: x8xx ISR,x900x ISR, 72xx, 73xx, 76xx
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 37
 Service Planning
EXPRESSION-& EVENT-MIB

 Simply capacity planning example: if my link utilization is above

50% for an hour, it’s time to upgrade the link
 Steps: Expression-MIB
1. Create an Expression
Utilization = (Δ ifInOctets + Δ ifOutOctets) * 8 * 100 / hour / ifSpeed

2. Create an Event Event-MIB

If utilization > 50%  generate an Event

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 38
 Service Planning
EXPRESSION-& EVENT-MIB
 Simply capacity planning example: Calculate link
utilization on all the interfaces in the router
Router# show running | beg expression
snmp mib expression owner administrator name exp3
expression ($1*800)/$2
enable
object 1
id ifInOctets
wildcard
object 2
NMS% snmpwalk -c public -v 2c <router> expValueCounter32Val
id ifSpeed
SNMPv2-SMI:: expValueCounter32Val.7.109.97.114.105.115.111.108.4.101.120.112.51.0.0.1 = Counter32:
214800 wildcard
SNMPv2-SMI:: expValueCounter32Val.7.109.97.114.105.115.111.108.4.101.120.112.51.0.0.2 = Counter32:
0
SNMPv2-SMI:: expValueCounter32Val.7.109.97.114.105.115.111.108.4.101.120.112.51.0.0.4 = Counter32:
0
SNMPv2-SMI:: expValueCounter32Val.7.109.97.114.105.115.111.108.4.101.120.112.51.0.0.5 = Counter32:
0
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 39
 Service Planning
Adding a Custom MIB Variable
 Problem: Collect data via SNMP, even if there is no MIB support
currently available.

 Solution: Expression-MIB provides the capability to process data

into more relevant information via SNMP
– Expression-MIB can be configured using SNMP directly since 12.0(5)T.
– Initially Cisco Implementation was based on OID 1.3.6.1.4.1.9.10.22 but current
Cisco implementation is based on RFC2982-MIB, OID 1.3.6.1.2.1.90.
– In 12.4(20)T Expression-MIB feature is enhanced to add CLIs to configure
expressions.

 Expression-MIB can gather data from Command Line Interface (CLI

show commands), even if there is no MIB support
 EEM 3.1 provides similar capability without the need to involve
Expression-MIB or Event-MIB
See: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_cfg_snmp_sup.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 40
 Service Planning
Adding a Custom MIB Variable

Is the
Is a certain value from a No Expression-MIB(1)
"CLI show command" Supported in your EEM 3.1
supported in your device via SNMP? Device?
No

Yes
Yes
Running
Script #1
Yes 12.4(20)T or
higher? EEM policy based on CLI Expression-MIB

No Script #2
Reference: Yes
http://www.cisco.com/go/mibs EEM policy based on the RFC2982-MIB
•SNMP Object Navigator
Support for
•Cisco IOS MIB Locator
RFC2982-MIB?

No Script #3
EEM policy based on the Expression-MIB
See: This is available as an EASy package from CiscoBeyond
http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=1961

For the ASR 1000 version

http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=2283
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 41
Service Planning
Custom MIB – EASy Package
Embedded Automation Systems (EASy)
Custom MIB EASy Package:
 Periodically evaluate a show command
 Update a custom MIB Variable
 Trigger Syslog and/or custom actions

To use the Package:

1. Browse and Download EASy Package
www.cisco.com/go/easy

2. Make Sure to also download EASy Installer

3. Watch VOD and/or read documentation

www.cisco.com/go/easy

4. Customize and tailor to your needs

5. Install and Use

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 42
What about Traffic
Flows?
 What is NetFlow ?
 Developed and patented at Cisco®
Systems in 1996
 NetFlow is the defacto standard for
acquiring IP operational data
 Provides network and security
monitoring, network planning,
traffic analysis, and IP accounting
 NetFlow v9 (RFC3954) serves as
the basis for IETF IPFIX Standard
(RFC5101 & RFC5102)
Network World article – NetFlow Adoption on the Rise:
http://www.networkworld.com/newsletters/nsm/2005/0314nsm1.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 44
 Service Planning
Flexible NetFlow (FNF)

 Traditional NetFlow with the v5, v7, or v8 NetFlow export

 NetFlow Version 9 (RFC3954)

Advantages: extensibility
Integrate new technologies/data types quicker Exporting
(MPLS, IPv6, BGP next hop, etc.) Process
Integrate new aggregations quicker
Basis for IETF IPFIX Standard (RFC5101 & RFC5102)

 Flexible NetFlow
Metering
Advantages: cache and export content flexibility
Process
User selection of flow keys
User definition of the records

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 45
 Flexible NetFlow
Multiple Monitors with Unique Key Fields
Traffic Flow Flow
Monitor Monitor
1 2

Key Fields Packet 1 Non-Key Fields Key Fields Packet 1 Non-Key Fields
Source IP 3.3.3.3 Packets Source IP 3.3.3.3 Packets
Destination IP 2.2.2.2 Bytes Dest IP 2.2.2.2 Timestamps
Source Port 23 Timestamps Input Interface Ethernet 0
Destination Port 22078 Next Hop Address SYN Flag 0
Layer 3 Protocol TCP - 6
TOS Byte 0
Input Interface Ethernet 0

Security Analysis Cache

Traffic Analysis Cache
Source Dest. Source Dest. Input … Source IP Dest. IP Input I/F Flag … Pkts
Protocol TOS Pkts
IP IP Port Port I/F
… 3.3.3.3 2.2.2.2 E0 0 … 11000
3.3.3.3 2.2.2.2 23 22078 6 0 E0 1100

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 46
 Flexible NetFlow
Configuration – Example
1. Configure the Exporter
Router(config)# flow exporter my-exporter
Where do I want my data sent?
Router(config-flow-exporter)# destination 1.1.1.1

2. Configure the Flow Record

Router(config)# flow record my-record
Router(config-flow-record)#
What data do
Router(config-flow-record)#
Imatch
want ipv4 destination address
to meter?
match ipv4 source address
Router(config-flow-record)# collect counter bytes

3. Configure the Flow Monitor

Router(config)# flow monitor my-monitor
How do I want to cache
Router(config-flow-monitor)# Information?
exporter my-exporter
Router(config-flow-monitor)# record my-record

4. Apply to an Interface
Router(config)# interface s3/0
On which Interface do I want to monitor?
Router(config-if)# ip flow monitor my-monitor input

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 47
 Flexible Flow Record: Key Fields
Flow IPv4 IPv6
Sampler ID IP (Source or
Payload Size IP (Source or
Destination) Payload Size
Direction Destination)
Prefix (Source or Packet Section Prefix (Source or Packet Section
Interface Destination) (Header) Destination) (Header)
Input Mask (Source or Packet Section Mask (Source or Packet Section
Destination) (Payload) Destination) (Payload)
Output
Minimum-Mask Minimum-Mask
Layer 2 (Source or TTL (Source or DSCP
Destination) Destination)
Source VLAN
Options
Protocol Protocol Extension Headers
Dest VLAN bitmap
NEW Dot1q VLAN Fragmentation
Version Traffic Class Hop-Limit
Flags
Dot1q priority Fragmentation Flow Label Length
Precedence
Offset
Source MAC Option Header Next-header
address Identification DSCP
Header Length Version
Header Length TOS
Destination
MAC address Total Length Payload Length

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 48
 Flexible Flow Record: Key Fields
NEW

Routing Transport Application

src or dest AS Destination Port TCP Flag: ACK Application ID*
Peer AS Source Port TCP Flag: CWR
Traffic Index ICMP Code TCP Flag: ECE
Forwarding ICMP Type TCP Flag: FIN
Multicast
Status Replication
IGMP Type* TCP Flag: PSH
IGP Next Hop Factor*
TCP ACK Number TCP Flag: RST
BGP Next Hop RPF Check
TCP Header Length TCP Flag: SYN
Drop*
Input VRF TCP Sequence Number TCP Flag: URG
Name Is-Multicast
TCP Window-Size UDP Message Length
NEW
TCP Source Port UDP Source Port
TCP Destination Port UDP Destination Port

TCP Urgent Pointer *: IPv4 Flow only

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 49
 Flexible Flow Record: Non-Key Fields

Counters Timestamp IPv4 IPv4 and IPv6

sysUpTime First Total Length Total Length
Bytes Minimum (*) Minimum (**)
Packet
sysUpTime First Total Length Total Length
Bytes Long
Packet Maximum (*) Maximum (**)
Bytes Square Sum TTL Minimum

Bytes Square Sum Long TTL Maximum

Packets

Packets Long

 Plus any of the potential “key” fields: will be the value from
the first packet in the flow
(*) IPV4_TOTAL_LEN_MIN, IPV4_TOTAL_LEN_MAX
(**)IP_LENGTH_TOTAL_MIN, IP_LENGTH_TOTAL_MAX

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 50
Service Planning
Three Types of FNF NetFlow Caches

 Normal cache (traditional NetFlow)

More flexible active and inactive timers: one second minimum
 Immediate cache
Flow accounts for a single packet
Desirable for real-time traffic monitoring, DDoS detection, logging
Desirable when only very small flows are expected (ex: sampling)
Caution: may result in a large amount of export data
 Permanent cache
To track a set of flows without expiring the flows from the cache
Entire cache is periodically exported (update timer)
After the cache is full (size configurable), new flows will not be
monitored
Uses update counters rather than delta counters

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 51
 Service Planning
Core Traffic Matrix with Flexible NetFlow
Problem: Network wide capacity planning requires the traffic matrix
Solution: Use Flexible NetFlow with a permanent cache

flow record traffic-matrix-record

match interface input
match ipv4 dscp
match routing next-hop address ipv4 bgp
collect counter bytes long
collect timestamp sys-uptime first
collect timestamp sys-uptime last
We must define the
flow monitor traffic-matrix-monitor maximum number
record traffic-matrix-record of entries for the
cache entries 1000 permanent cache
cache type permanent
exporter capacity-planning-collector

interface pos3/0
ip flow monitor traffic-matrix-monitor

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 52
 Service Planning
Configuration Using EEM + Cron + CLI

Problem: No synchronized NetFlow export across routers

Solution: Use Flexible NetFlow with a permanent cache

Router(config)# event manager applet periodicexport

Router(config-applet)# event timer cron name
"everyhour" cron-entry "0 * * * *"
Router(config-applet)# action 1.0 cli command
"clear flow monitor traffic-matrix-record force-export"

 Export the content of the permanent cache every one hour

 If time is synchronized across routers (NTP), we have a
synchronized export (snapshot)

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 53
 Service Planning
Flexible NetFlow TopTalkers

show flow monitor monitor-name cache filter options

… aggregation options sort options

 Flow filtering, aggregation and sorting can be combined

to select what information and how it will be displayed

 Top ten protocols observed:

Router# show flow monitor <monitor> cache
aggregate ipv4 protocol sort highest counter bytes top 10

Available from: IOS 12.4(22)T

Platforms: x8xx ISR,x900x ISR, 72xx, ..
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 54
 Service Planning
Flexible NetFlow Top Talkers - Examples
 Top ten IP addresses that are sending the most packets
Router# show flow monitor <monitor> cache
aggregate ipv4 source address
sort highest counter bytes top 10
 Top five destination addresses to which we're routing most traffic
from the 10.10.10.0/24 prefix
Router# show flow monitor <monitor> cache
filter ipv4 destination address 10.10.10.0/24
aggregate ipv4 destination address
sort highest counter bytes top 5

 5 VLAN's that we're sending the least bytes to:

Router# show flow monitor <monitor> cache
aggregate datalink dot1q vlan output
sort lowest counter bytes top 5
 Top 20 sources of 1-packet flows:
Router# show flow monitor <monitor> cache
filter counter packet 1
aggregate ipv4 source address
sort highest flow packet top 20
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 55
 Service Planning
Flexible NetFlow Top Talkers – Example

TCP Servers’
SYN network
attacks 10.10.10.0/24

Router# show flow monitor <monitor> cache

filter ipv4 destination address 10.10.10.0/24
counter packet regex[1-2]
aggregate ipv4 source address
ipv4 destination address
sort highest flow top 100

 The top 100 pairs of IP addresses with one or two packet(s) that
are destined for my servers' network

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 56
Flexible NetFlow
Example: Cat4500 Supervisor Engine 7-E
 Auto Smart Ports
 Embedded Event Manager (EEM) 3.2
 Flexible Netflow and NetFlow v9 support for
New IPv4, IPv6, L2
 Generic Online Diagnostics (GOLD)
 In-Service Software Upgrade (ISSU)
 Smart Call Home

Anomaly Detection using EEM and FNF

srcIf SrcIPadd DstIf DstIPadd MAC TCP bytes
Flags
Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 … …. 3465

Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 Anomaly

…. …. 300
detected!
Fa1/0 173.1.1.2 Fa0/0 10.0.277.1 …. …. 1000

• Policy Action:
EEM (Port Shutdown,
ACL, QoS, …)
FNF
• Custom Syslog

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 57
What about Trending
and Graphical Views?
 NAM 5.0 Interactive Reports New
Jan 20
11
Analyze Performance/Usage Trends and Patterns
Export Data Descriptive
Statistics

Filter by Specific Site, Zoom/Pan to

Host, VLAN, Data specific patterns
Source or Time or time intervals
Interval

• Analyze data over last month or more

• Define custom time interval for analysis

• Export data in raw format for consumption by external

management application

• Drill-down to analyze related trends to support

planning decisions

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 59
But my Teleworkers don’t
have a Cisco Router yet!
 New
Cisco Visual Networking Index
Problem: Sometimes we need trending and forecasting info beyond
our current reach and/or where there is no IOS-based network yet
Solution: Visual Networking Index

 Global initiative to analyze

and forecast
IP network growth
 Mobile and PC-based
data collection
 Graphical data summaries
publicly available
 Individual network usage reports
available to service provider
participants
See: www.ciscovnipulse.com
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 61
How To Analyze Transient
Conditions?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 62
 Service Planning
Embedded Event Manager (EEM)
*Not all available in all releases

IOS.sh TCL
Applets
Policies Policies
3. An EEM Policy is activated that initiates a pre-
defined set of actions

Policy

Embedded Event 2. An EEM Event Detector receives notification

Manager

Event Detector

1. Something happens on the causing an

Event to trigger

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 63
Service Planning
Embedded Event Manager (EEM) Versions
 Embedded monitoring of different components of the system
via a set of software agents (event detectors)
 Event detectors (ED) notify EEM when an event of interest occurs;
based on this, a policy will trigger an action to be taken
 Advantages: Local programmable actions, triggered by specific
events – growing set of detectors and actions:
– EEM 1.0 introduced in 12.0(26)S, 12.3(4)T
– EEM 2.0 introduced in 12.2(25)S
– EEM 2.1 introduced in 12.3(14)T
– EEM 2.2 introduced in 12.4(2)T
– EEM 2.3 introduced in 12.4(11)T
Adds multi-event correlation
– EEM 2.4 introduced in 12.4(20)T
Adds programmatic Applets
– EEM 3.0 introduced in 12.4(22)T
– EEM 3.1 introduced in 15.0(1)M
– EEM 3.2 introduced in 12.2(52)SE
– stay tuned ...
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 64
 Service Planning
EEM Architecture

email SNMP set SNMP SNMP Reload or Application CLI IOS.sh TCL
Syslog
notification Counter get notification switch-over specific Applets Policies Policies

Actions

EEM Applets
multi-event-correlation
Embedded Event
Manager

Event Detectors
Interface XML CDP
Syslog SNMP Timer none HW Watchdog CLI OIR ERM EOT RF GOLD NetFlow IPSLA Route 802.1x MAC
Counter RPC LLDP
ED EDs EDs ED EDs ED ED ED ED ED ED ED ED ED ED ED ED
ED ED ED

Remote:
• Fan
• Notification • Cron Process Interface
Syslog • Temp
Local: • Count Scheduler Descriptor
Event • Env
• Notification down Database Blocks
• ...
• Get/Set

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 65
 Service Planning
EEM Applets and Policies

CLI IOS.sh TCL

Applets Policies Policies

 Part of the Cisco IOS  Separate ASCII File  Separate ASCII File
Configuration my-policy.sh my-policy.tcl
 Based on CLI  Based on Cisco IOS  Based on Cisco IOS
Commands CLI and Shell CLI and Safe TCL
Commands Commands
 Simple Actions  Effective shell-like  Flexible and powerful
simple scripting scripting capabilities
 Programmatic Applet  Registered via the  Registered via the
Extensions Cisco IOS Config Cisco IOS Config

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 66
Service Planning
Example: Trigger a Config Change – 1/3
 Problem: a PKI related config change on a remote device should only
happen once NTP has successfully synched the time
Router(config)# ntp logging
Router(config)# ntp update-calendar
Router(config)# ntp server 172.16.154.40 prefer

 Solution I: use EEM Syslog Event Detector and a CLI Applet to trigger the
change
CLI Applet
event manager applet config_upon_ntp
event syslog pattern ".*%NTP-5-PEERSYNC.*"
action 10 syslog msg "Starting ..."
:
... Your Config Changes Here ...
:
action 30 syslog msg "... done"

Dec 10 13:03:57.746: %NTP-5-PEERSYNC: NTP synced to peer

172.16.254.40
Dec 10 13:03:57.750: %HA_EM-6-LOG: config_upon_ntp: Starting ...
Dec 10 13:03:57.750: %HA_EM-6-LOG: config_upon_ntp: ... done

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 67
Service Planning
Example: Trigger a Config Change – 2/3
 Solution II: use EEM Syslog Event Detector and an IOS.sh Policy to trigger
the change
IOS.sh Policy
##::cisco::eem::event_register_syslog pattern .*%NTP-5-PEERSYNC.*
send log "Starting ..."
enable
conf t
hostname $new_hostname
:
... Your Config Changes Here ...
:
end
send log "... done"
# End of IOS.sh Policy demo script
router#
*Dec 22 18:27:09.659: %HA_EM-6-LOG: sl_cfg_ntp.sh: Starting ...
*Dec 22 18:27:09.801: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:sl_cfg_ntp.sh)
*Dec 22 18:27:09.927: %HA_EM-6-LOG: sl_cfg_ntp.sh: Set hostname from router to it-worked
*Dec 22 18:27:09.927: %HA_EM-6-LOG: sl_cfg_ntp.sh: ... done
it-worked#

 Solution III: use EEM Syslog Event Detector and a TCL Policy to trigger the
change …
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 68
 TCL Policy

Policy runtime
Default = 20 seconds
Increase this value if you see
a “Process Forced Exit” message
from the router.

router#
*Dec 10 10:43:29.061: %HA_EM-6-LOG: config_upon_ntp.tcl: Starting ...
*Dec 10 10:43:29.197: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:config_upon_ntp.tcl)
*Dec 10 10:43:29.329: %HA_EM-6-LOG: config_upon_ntp.tcl: Set hostname from router to it-worked
*Dec 10 10:43:29.329: %HA_EM-6-LOG: config_upon_ntp.tcl: ... done
it-worked#

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 69
 EEM
Getting Started with TCL Policies

1. Define directory Router(config)#event manager directory user policy flash:

Router#copy tftp flash:

Address or name of remote host []? 10.1.88.9
Source filename []? foobar.tcl
2. Copy Tcl script to flash Destination filename [tcl]? foobar.tcl
Accessing tftp://10.1.88.9/foobar.tcl...!
1232 bytes copied in 0.620 secs (1987 bytes/sec)

3. Configure any required event manager environment _email_server 172.27.121.177

environment variables event manager environment _email_from [email protected]
event manager environment _email_to [email protected]

4. Configure any IOS

features EEM may
Examples include IP SLA, ERM and Embedded Object Tracking
depend on (optional)
5. Register Tcl policy Router(config)#event manager policy foobar.tcl type user

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 70
 IOS
Event Description EEM Version in IOS IOS XR NX-OS
XE
Detector (ED Triggers, based on ...)
1.0 2.0 2.1 2.2 2.3 2.4 3.0 3.1 3.2 3.6 4.0 2.1 2.2 4.0 4.1

Availability of Event Detectors

Syslog
SNMP Notif
RegExp match of local syslog message
SNMP MIB Variable Threshold


















  


  
Watchdog IOS process or subsystem activity events            
Interface Counter (Interface) Counter Threshold            
Timer Designated Time or Interval            
Counter Change of a designated counter value            
Application specific An IOS subsystem or policy script            
CLI RegExp match of input via command line interface           
OIR Hardware online insertion and removal OIR             
none No trigger, used in conjunction with exec command           
ERM Embedded Resource Manager (ERM) events      
EOT Enhanced Object Tracking variable (EOT) events          
RF IOS Redundancy Facility (switchover)        
GOLD Generic Online Diagnostics (GOLD) events       
SNMP Proxy Incoming remote SNMP Notification    
XML RPC Incoming XML message    
Routing State change of Routing Protocols   
Netflow Traffic Flow information from Netflow   
IPSLA IPSLA events (supersedes EOT for EEM / IPSLA)   
CLI enhanced Integrates CLI Ed with the XML PI   
SNMP Object Intercept SNMP GET/SET requests  
Neighbor Disco CDP, LLPD, Link up/down events 
Identity 802.1x and MAB authentication events 
MAC MAC Address Table entry changes 
Hardware Register for environmentla monitoring hardware  
Statistics Threshold crossing of a statistical counter  
Sysmgr Process start and stop events  
Fan (absent / bad) Presence and State of a Fan  
Module failure Occurence of a Module Failure Event  
Storm Control Occurence of a Storm Control Event  
Temperature
EASy IntroTemperature Sensor Thresholds  
– bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 71
Viewing EEM Available System TCL Policies

 Use the show event manager policy available

system command to get a list of available System
Policies for a given IOS release
Router# show event manager policy available system
No. Type Time Created Name
1 System Thu Feb 7 01:28:15 2036 ap_perf_test_base_cpu.tcl
2 System Thu Feb 7 01:28:15 2036 cl_show_eem_tech.tcl
3 System Thu Feb 7 01:28:15 2036 no_perf_test_init.tcl
4 System Thu Feb 7 01:28:15 2036 sl_intf_down.tcl
5 System Thu Feb 7 01:28:15 2036 tm_cli_cmd.tcl
6 System Thu Feb 7 01:28:15 2036 tm_crash_reporter.tcl
7 System Thu Feb 7 01:28:15 2036 tm_fsys_usage.tcl

 System Policies live under tmpsys:/lib/tcl/

eem_scripts and can be viewed with the more
command
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 72
 EEM 2.0: Timer Event Detector
EEM 2.1: CLI Action
Minute (0 59)
Export a Permanent Hour (0 23)
Flexible NetFlow Cache on Day of the month (1 31)
Month of the year (1 12)
regular basis Day of the week (0 6 with 0=Sunday)

Router(config)# event manager applet periodicexport

Router(config-applet)# event timer cron name
"everyhour" cron-entry "0 * * * *"
Router(config-applet)# action 1.0 cli command
"clear flow monitor traffic-matrix-record force-export"

Router# debug flow exporter event

Router#
Nov 6 17:00:00.763: FLOW EXP: Exporting packet
(ID: 256, Exporter: capacity-planning-collector)

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 73
 EEM 2.0: EOT Event Detector
Problem: A Notification is required upon
failure of a specific route
D 1.1.1.1 [90/297372416] via 192.168.1.1, 1w6d, Gig

X
Solution: Track the Route using
Enhanced Object Tracking (EOT) and
Embedded Event Manager (EEM) email EOT/EEM 1.1.1.1/32
172.27.121.177

track 400 ip route 1.1.1.1/32 reachability

delay down 10 up 10
!
event manager environment my_server 172.27.121.177
event manager environment my_from [email protected]
event manager environment my_to [email protected]
event manager environment my_route 1.1.1.1/32
!
event manager applet email_track_iproute
event track 400 state down
action 1.0 syslog msg "Prefix to [$my_route] has been withdrawn!"
action 1.1 mail server "$my_server" to "$my_to" from "$my_from“
subject “EEM: Prefix to Remote Site [$my_route] is DOWN" body ""
action 1.2 syslog msg “EEM: Path Failure alert email sent!"

Note: New Routing Event Detector in EEM 3.0

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 74
EEM 2.4: Proxy Event Detector

 Router or switch can RECEIVE

an SNMP trap
 EEM event upon trap receipt
EEM EEM
 Execute (trigger) EEM script to
take local action
 Script sees varbind info
 Example:
UPS on battery backup
===> Shut non-critical POE
Uninterruptible
ports to conserve power
Power Supply
Only 5 minutes remaining
===> Shutdown service
SNMP trap
modules gracefully On Battery
5 Min Remaining!
 Example: managed Services
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 75
 EEM 2.4: Multiple Event Correlation
 Previous to EEM v2.4, there was a
one-to-one correspondence between a
single event and the triggered policy
 In other words, a policy could only be
triggered by a single event and any
event correlation had to be coded by
the user Event Correlation
 Multiple Event Support ushers in an Capabilities
event correlation specification such
that multiple events may be
considered together to trigger a
policy
 For example:
If (Event 1 OR Event 2) AND Event 3,
then
Trigger Policy A
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 76
 EEM 2.4: Multiple Event Correlation
Problem: A Syslog message is required upon state change of either
Ethernet1/0 or Ethernet1/1
Solution: Use Embedded Event Manager (EEM) Multiple Event
Correlation with a correlate statement within the trigger block to define the
logic between individual events and optional occurs clauses to define the
number of times a specific event must be raised before being used in the
correlation (inner level), or the number of times the total correlation must
be true before invoking the action (outer level):

event manager applet example

event tag e1 syslog pattern ".*UPDOWN.*Ethernet1/0.*"
event tag e2 syslog pattern ".*UPDOWN.*Ethernet1/1.*"
trigger occurs 1
correlate event e1 or event e2
attribute e1 occurs 1
attribute e2 occurs 1
action 1.0 syslog msg "Critical interface status change"
set 2.0 _exit_status 0
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 77
 Service Planning
EEM 3.0: Programmatic Applet Example
event manager applet route-watch
event routing network 10.1.1.0/24 type add protocol ospf
action 001 cli command "enable"
action 002 set done 0
action 003 while $done eq 0
action 004 wait 5
action 005 cli command "ping ip 10.1.1.1"
action 005 regexp "!!!!!" "$_cli_result"
action 006 if $_regexp_result eq 1
action 007 cli command "config t"
action 008 cli command "int Tunnel0"
action 009 cli command "shut"
action 010 cli command "end"
action 011 set done 1
action 012 end
action 013 end

 The applet will trigger when the route 10.1.1.0/24 is learned via OSPF
 The applet will try and ping host 10.1.1.1, and when it is successful,
it will take down the backup tunnel interface
Question: how many ping attempts will be made ?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 78
 Service Planning
EEM 3.1: SNMP Notification + Description

Router(config}# event manager applet test_trap

router(config-applet)# description test snmp notification unmanaged service
router(config-applet)# event snmp-notification oid 1.3.6.1.6.3.1.1.4.1.0
oid-val "1.3.6.1.6.3.1.1.5.3" op eq src-ip-address 10.51.89.176
direction incoming
router(config-applet)# action 1.0 … snmpTrapOID
router(config-applet)# action 2.0 …

“snmp-notification” can intercept incoming or outgoing

notifications, but outgoing only for locally generated
notifications

Note: SNMPv2c notification contains the snmpTrapOID OID, which

contains an unique value per notification type

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 79
 Service Planning
Example: Synchronizing EEM Scripts 1/2
Problem: Synchronize EEM Policy .tcl files from a central Repository
Solution I: Use event manager update commands
1. Configure the default Repositiory:
router(config)# event manager directory user repository tftp://172.16.64.1

2. Single exec command to download, un-register and re-register:

router# event manager update user policy name my
%EEM: Update will use the repository path: tftp://172.16.64.1
%EEM: Attempting to copy tftp://172.16.64.1/my.tcl to flash:/eemtcl/my.tcl
Loading my.tcl from 172.16.64.1 (via FastEthernet0): !
[OK - 647 bytes]
%EEM: Copied 647 bytes from tftp://172.16.64.1/my.tcl to flash:/eemtcl/my.tcl
%EEM: Policy my.tcl has been successfully copied and re-registered

*Dec 10 20:12:43.198: %HA_EM-6-FMPD_UPDATE_POLICY_COPY: Policy update has copied 647 bytes from tftp:
*Dec 10 20:12:43.230: %HA_EM-6-FMPD_UPDATE_POLICY_REGISTER: Policy update has successfully re-registe
2bis. Can also synch entire groups, based on regular expression match:
router# event manager update user policy group m.*

3. Verify using show command:

router# show event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 script user syslog Off Wed Dec 10 20:12:43 2008 my.tcl
occurs 1 pattern {.*%NTP-5-PEERSYNC.*}
nice 1 queue-priority low maxrun 90.000 scheduler rp_primary
Available from: IOS 12.4(20)T
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 80
 Service Planning
Example: Synchronizing EEM Scripts 2/2
Solution II: Use new event manager update command

1. Single exec command to specify repository, download, un-register and re-register:

router# event manager update user policy name my.tcl repository tftp://10.1.1.1/
%EEM: Update will use the repository path: tftp://10.1.1.1
%EEM: Attempting to copy tftp://10.1.1.1/my.tcl to flash:/eemtcl/my.tcl
Loading my.tcl from 10.1.1.1 (via FastEthernet0): !
[OK - 647 bytes]
%EEM: Copied 647 bytes from tftp://10.1.1.1/my.tcl to flash:/eemtcl/my.tcl
%EEM: Policy my.tcl has been successfully copied and re-registered

*Dec 16 22:09:11.303: %HA_EM-6-FMPD_UPDATE_POLICY_COPY: Policy update has copied 647 bytes from tftp://10
*Dec 12 22:09:11.329: %HA_EM-6-FMPD_UPDATE_POLICY_REGISTER: Policy update has successfully re-registered
1bis. Can also synch entire groups, based on regular expression match:
router# event manager update user policy group m.*

2. Verify using show command:

router# show event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 script user syslog Off Wed Dec 10 20:12:43 2008 my.tcl
occurs 1 pattern {.*%NTP-5-PEERSYNC.*}
nice 1 queue-priority low maxrun 90.000 scheduler rp_primary

Available from: IOS 15.0(1)M

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 81
Service Planning
Using EEM step-by-step
1. Which problem do you want to solve?
2. Which event detector and action do you need?
– Upgrade to the right IOS image
– show
Use event manager detector <detector-type> detailed

3. Check whether a suitable script/applet is available already

– http://www.cisco.com/go/ciscobeyond
– http://www.cisco.com/go/eem
– http://www.cisco.com/go/easy

4. Work from an existing example

5. Deploy and Monitor

– CiscoWorks LMS (from 3.1) via RME
http://www.cisco.com/go/lms
– Davra Networks EEMLive
http://www.davranetworks.com/

6. If customization/new development/testing is required

– “Network Programming Advisors“ http://www.progrizon.com/
– Cisco Advanced Services

7. Don’t forget to ask to (and share with) the EEM forum

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 82
Agenda

Introduction & Overview

Service Planning
 Service Deployment & Activation
Service Testing, Verification & Assurance
Troubleshooting & Optimization
Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 83
Introduction & Overview
‘Configuration‘ in a Service Life Cycle
scripts and tools network engineer support staff applications

scripts

IOS MOH & IVR xDM

*.mdf *.tcl config images files files

device groups individual devices large scale

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 84
Introduction & Overview
Definition of Activities

Move physical network equipment into

Deployment it‘s operating location

Make new network equipment ready for

Commissioning use and reachable by operations, NMS

Focus
hostname pe-south

Configure a network element depending

!

Configuration
enable password c
!
mpls ip
!
interface Loopbac
ip address 10.10
on it’s role and function in the network

Configure portions of a network for the

Provisioning purpose of a specific user and/or service

Activation Enable users to start using a service

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 85
Introduction & Overview
The Human Factor ...

interface Serial1/0.121 point-to-point

description " NOC Bastelstunde "

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 86
Command Line Interface (CLI)

1
– The Basics

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 87
Command Line Interface (CLI) – Modes
Line Configuration Sub-Mode
router(config-line)#
Global Configuration Mode Routing Configuration Sub-Mode
router(config)# router(config-router)#

hostname interface Interface Configuration Sub-Mode

ip route router(config-if)#

interface ... shutdown Running

ip address
...
Configuration
do ..

encapsulation ...
...
conf t

Priviledged EXEC Mode User EXEC Mode

router# router>
show show (limited) Startup
ping ping
Configuration
debug enable
enable
... ...

ROM Monitor Diagnostic Boot (only on ASR)

Config Register
rommon # > router(diag)#

See: www.cisco.com/en/US/docs/ios/preface/usingios.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 88
Command Line Interface (CLI) – Basics 1/2
A Series of usability features are available in IOS:
 Exec Commands from within Config Mode (from 12.0(21)S, 12.2(8)T)

Issue Exec commands without leaving Config Mode

router# conf t
router(config)# do copy run start
Destination filename [startup-config]?
Building configuration...
[OK]
router(config)#

 Command Aliases (from 10.3, 12.2(33)SRA)

Pre-defines Aliases are available on the CLI

router# show aliases
Exec mode aliases:
Custom Aliases can be defined per (Sub-)Mode h help
router# conf t lo logout
Enter configuration commands, one per line. End with CNTL/Z. p ping
router(config)# alias exec shib show ip interface brief r resume
router(config)# alias exec shru show running-config s show
router(config)# alias exec shrb show running-config | begin u undebug
router(config)# alias configure h hostname un undebug
Router(config)# alias interface nsh no shutdown w where

Note: ROM Monitor also provides an alias command

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 89
 Command Line Interface (CLI) – Basics 2/2
 Interface Ranges and Macros (from 12.1(5)T, 12.1(1)E, IOS XE 2.1)

Define Interface Ranges / Groups

Apply Config to Interface Ranges / Groups

router(config)# interface range FastEthernet 1 - 3
router(config-if-range)# no shut
Consequtive Range
Define and Use immediately

router(config)# define interface-range mylist FastEthernet 2 , FastEthernet 4 -

6
router(config)# interface range macro mylist
router(config-if-range)# no shut
Arbitrary Group
Define Once
Use multiple times

router(config)# interface range FastEthernet 5/1.1 – FastEthernet 5/1.4

router(config-if-range)# encapsulation dot1Q 220
router(config-if-range)# no shut Works on
Subinterfaces and
This will apply: VLAN Ranges too
VLAN ID 220  FastEthernet 5/1.1 from 12.2(8)T
VLAN ID 221  FastEthernet 5/1.2
VLAN ID 222  FastEthernet 5/1.3
VLAN ID 223  FastEthernet 5/1.4
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 90
 Where to start with CLI ?
Feature Navigator:
http://www.cisco.com/go/fn

Command Lookup Tool: http://tools.cisco.com/Support/CLILookup/

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 91
Command Line Interface (CLI)

2
– More Advanced

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 92
Command Line Interface (CLI)
– More Advanced

Son: Dad, why are there always 2 Pilots ?

Dad: one has to prevent the other from doing stupid things
Son: which one is doing the stupid things ?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 93
 Deployment & Activation
IOS CLI Configuration ‚Safety‘ Features
 Contextual configuration diff utility (from 12.3(4)T, 12.2(25)S)
Easily show differences between running and startup configuration
Compare any two configuration files

 Config change logging and notification (from 12.3(4)T, 12.2(25)S)

Tracks config commands entered per user, per session
Notification sent indicating config change has taken place—changes can be
retrieved via SNMP

 Configuration replace and rollback (from 12.3(7)T, 12.2(25)S)

Replace running config with any saved configuration (only the diffs are applied)
to return to previous state
Automatically save configs locally or off box

Config Rollback Confirmed Change (from 12.4(23)T, 12.2(33)S)

 Configuration locking (from 12.3(14)T, 12.2(25)S)

Ensures exclusive configuration change access
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 94
 Deployment & Activation
Example: Using Config Rollback
 Problem: critical config change to a remote router may result in loss of
connectivity, requiring a reload
 Solution: replace the running configuration with the latest good
archive after two minutes – unless the change made is confirmed
router# show archive
There are currently 4 archive configurations saved.
The next archive file will be named disk0:/config-archive-4
Archive # Name
0
1 disk0:/config-archive-1
2 disk0:/config-archive-2
3 disk0:/config-archive-3 <- Most Recent

router# config replace disk0:/config-archive-3 time 120

:
... your Config Change work here ...
:
router# no config replace disk0:/config-archive-3
Available from: IOS 12.3(7)T, 12.2(25)S
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 95
 Deployment & Activation
Example: Using Config Revert
 Problem: critical config change to a remote router may result in loss of
connectivity, requiring a reload
 Solution: revert the running configuration after two minutes – unless
the change made is confirmed
router# config terminal revert time 2
Rollback Confirmed Change: Backing up current running config to flash:bk-2

Enter configuration commands, one per line. End with CNTL/Z.

:
... your Config Change work here ...
:
router# hostname oops
oops(config)# end
oops# Rollback Confirmed Change: Rollback will begin in one minute. Enter
"configure confirm" if you wish to keep what you've configured

oops# Rollback Confirmed Change: oops# config confirm

rolling to:flash:bk-2 or oops#
Total number of passes: 1
Rollback Done
router#

Available from: IOS 12.4(23)T, 12.2(33)S

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 96
What if I need a simple script?

3
IOS Shell Scripting

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 97
 IOS Shell
 Problem: Sometimes we need more than what Interface ranges,
Macros, Auto SmartPorts and other CLI features already offer.
But we may not want all the power and complexity of Tcl Scripting or
Embedded Event Manager

 Solution: Use IOS Shell (IOS.sh)

IOS Shell offers
 Environment Variables MY_VAR=value, %n
 Pipe and Redirection |
 Condition Testing if […]; then else fi
 Loops
IOS.sh # _
 Built-in Functions show shell functions
shell exec <function>
 Custom Function Definitions function <name>(…){…}

Phase I Available from: IOS 12.2(52)SE

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 98
 IOS Shell - Example
 The pre-built shell functions for Auto SmartPorts are a good starting point:
switch# show shell functions CISCO_AP_AUTO_SMARTPORT

function CISCO_AP_AUTO_SMARTPORT () {
if [[ $LINKUP -eq YES ]]; then
conf t
interface $INTERFACE
macro description $TRIGGER
switchport trunk encapsulation dot1q
switchport trunk native vlan $NATIVE_VLAN
switchport trunk allowed vlan ALL
switchport mode trunk
switchport nonegotiate
auto qos voip trust
mls qos trust cos
exit
end
fi
if [[ $LINKUP -eq NO ]]; then

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 99
What if SmartPorts are
almost good enough?
 Example: Automated Port Provisioning
 Problem: When a new device connects, we want to trigger a
sequence of events and configurations in a customizable way and
based on the type of device
 Solution: use Embedded Event Manager (EEM):

 Trigger based on CDP/LLDP/MAC Event

 Send custom Event to NMS (Syslog, SNMP)
 Trigger Authentication (802.1x, MAB)
 Fetch/Build Configuration Parameters
 Apply Port Configuration Template

802.1x
CDP LLDP

See: http://www.cisco.com/go/eem MAC Add

NMS station
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 101
Beyond Auto SmartPorts
Take Full Control With EEM
 Auto SmartPorts are powered by EEM
 When a printer is added to the network, use an
EEM applet to create a new ASP event

event manager applet dectect-printer

event neighbor-discovery interface regexp FastEthernet.* cdp add
action 001 regexp ".*LasterJet.*" "$_nd_cdp_platform"
action 002 if $_regexp_result eq 1
action 003 cli command "enable"
action 004 cli command "config t"
action 005 cli command "interface $_nd_local_intf_name"
action 006 cli command "switchport access vlan $printer_vlan"
action 007 cli command "switchport mode access"
action 008 cli command "switchport port-security"
action 009 cli command "switchport port-security violation restrict"
action 010 cli command "switchport port-security aging time 2"
action 011 cli command "switchport port-security aging type inactivity"
action 012 cli command "spanning-tree portfast"
action 013 cli command "spanning-tree bpduguard enable"
action 014 cli command "end"
action 015 syslog msg "New printer added: $_nd_cdp_entry_name , type:
$_nd_cdp_platform"
action 016 end

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 102
How about a smarter
deployment procedure?
 Smart Install

 Smart Install is a plug-and-play configuration

DHCP
and image-management feature that provides Server
zero-touch deployment for new switches.
 In a Smart Install network, the switch selected TFTP
as the director provides a single management Server
Director
point for images and configuration of client
switches.
 When a client switch is first installed into the Aggregation Layer

network, the director automatically detects the Access Layer

new switch, and identifies the correct Cisco

IOS image and the configuration file for
downloading. It can allocate an IP address
and host name to a client.
 The director can also perform on-demand Client Switches
configuration and software image updates of a
switch or a group of switches in the network.

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 104
Smart Install
Supported Devices and Images

Device Type Minimum Software

Catalyst 3750, 3750v2, 3750E 12.2(52)SE
Catalyst 3560, 3560v2, 3560E, 3560 12 12.2(52)SE
port, 3560 8 port

Catalyst 2960, 2960 8 port 12.2(52)SE

Catalyst 2975 12.2(52)SE

Catalyst 2918 12.2(52)SE

 The director in a Smart Install network must be running Cisco IOS release
12.2(52)SE or later.

 The director can be a Catalyst 3750E, 3750, 3560E, or 3560 switch. Catalyst
2960 and 2975 switches cannot be Smart Install directors currently.

 A client switch can be an intermediate switch connected to another client

switch. A client can be a standalone switch or a switch stack
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 105
Driving the bigger
Workflow?
LMS 4.0 Auto Smart Ports Work Center

Assess
Device
Readiness

Provision
ASP macros

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 107
LMS 4.0 Auto Smart Ports Provisioning

1. Select device
2. Associate macros to
events and configure
macros
3. Deploy

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 108
LMS 4.0 Smart Install Work Center

Assess
Device
Readiness

Configure
the Director

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 109
LMS 4.0 Smart Install Configuration
1. Select Director device
2. Specify software and
config
3. Configure DHCP pool
4. Deploy

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 110
How to be triggered by a Config
Change ?

Embedded Event Manager

(EEM)
5
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 111
 Using EEM to trigger upon config change

Two Options:
 Syslog Event Detector upon any potential config change
 CLI Event Detector upon specific CLI command
– Asynchronous:
• Trigger Policy and then execute CLI command
• Trigger Policy and skip CLI command
– Synchronous:
• Trigger Policy and execute/skip based on exit status
_exit_status == 0  skip CLI command (default)
_exit_status == 1  execute CLI command

event [tag event-tag] cli pattern regular-expression

{[default] [enter] [questionmark] [tab]}
[sync {yes | no skip {yes | no}]
[mode variable]
[occurs num-occurrences] [period period-value]
[maxrun maxruntime-number]

Available from: EEM 2.1,© 2009

EASy Intro – bklauser
integrated with XML PI from EEM
Cisco Systems, Inc. All rights reserved.
3.0
Cisco Public 112
 Example: Using EEM CLI Event Detector
 Problem: VLAN 380 should not be accidentally removed from a trunk
Other Examples:
 Solution: use EEM CLI Event Detector: • no mpls ip
Option a: Don’t prevent anything, just issue a syslog notification: • no router isis
event manager applet cli-async • debug all
event cli pattern "switchport trunk allowed vlan remove.*380.*" sync no skip no
action 1.0 syslog msg "Removing VLAN 380"

Option b: Prevent the entire command and issue a syslog notification:

event manager applet cli-async-skip
event cli pattern "switchport trunk allowed vlan remove.*380.*" sync no skip
yes
action 1.0 syslog msg "Will NOT remove VLAN 380"
Option c: Ask for confirmation, then allow or prevent the entire command:
event manager applet cli-sync
event cli pattern "switchport trunk allowed vlan remove.*380.*" sync yes
action 1.0 puts "Confirm removing VLAN 380 [yes|no]:"
action 2.0 gets response
action 3.0 if $response eq yes goto 5.0
action 4.0 puts "NOK - VLAN 380 will NOT be removed"
action 4.1 exit 0
action 5.0 puts "OK - VLAN 380 will be removed"
action 5.1 exit 1

Caveats: command may be (much) bigger than what you match! Ranges!
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 113
Editing Files on the CLI

6
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 114
Editing Files
 Problem: Often ASCII files are being used when using Device
Manageability Instrumentation in IOS:
 Tcl scripts and EEM Tcl Policies
 EMM Menu Definition Files
 Config Templates and other text files

During Development and Test it would be useful to be able to

edit these files directly from IOS.
But: IOS does not include an ASCII Editor ...

 Solution: Use a Tcl implementation of an Editor in IOS

The GNU <ed> editor is a very simple,
line-based editor available as Tcl
implementation
see: http://en.wikipedia.org/wiki/Ed_(Unix)
see: http://www.gnu.org/software/ed/ed.html

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 115
Editing Files – Using ed.tcl
1. Copy ed.tcl and a simple test file to the flash:
router# show flash
:
8 27091 Nov 19 2008 10:51:26 ed.tcl
9 68 Nov 19 2008 11:00:12 testfile.txt

2. Define an Alias for simplicity:

router(config)# alias exec ed tclsh flash:/ed.tcl

3. Edit the file using ed:

router# ed flash:/testfile.txt a a – add lines
65 1,$p – print lines 1 to last and here are
1,$p yet another two lines
line one of the test file . . – end adding
line two of the test file ,n
another line ,p – print all lines 1 line one of the test file
,p 2 line two of the test file
line one of the test file 3 another line
line two of the test file 4 and here are
another line ,n – numbered print all lines 5 yet another two lines
,n w
1 line one of the test file 99 w – write file
2 line two of the test file q
3 another line router#
q – quit
Available from www.cisco.com/go/ciscobeyond (http://tinyurl.com/ed-on-ios)
( See http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=1461 )
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 116
Archiving and keeping Files up

7
to date

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 117
 Deployment & Activation
Example: Archiving Configuration – 1/6
Problem: Device configurations must be archived periodically, collecting
them from the outside should not be the only answer.

Solution 0: Manually create meaningful copies of the running config:

nexus-7000# copy run bootflash:/$(TIMESTAMP)-$(SWITCHNAME).conf

nexus-7000# dir bootflash:

29796 Apr 27 17:38:16 2009 2009-04-27-17.38.16-nexus-7000.conf

nexus-7000# show cli variable

VSH Variable List
-----------------
SWITCHNAME=“nexus-7000"
TIMESTAMP="2009-04-27-17.47.48"

Note: from IOS 12.3T onwards, refer to $h and $t variables within archive config path option

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 118
 Deployment & Activation
Example: Archiving Configuration – 2/6
Solution 1: Archive the running configuration once every day locally:

archive
path disk0:/config-archive
maximum 7
time-period 1440

View the content of the archive:

Router#show archive
There are currently 3 archive configurations saved.
The next archive file will be named disk0:config-archive-3
Archive # Name
0
1 disk0:config-archive-1
2 disk0:config-archive-2 <- Most Recent
3
4
5
6
7
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 119
 Deployment & Activation
Example: Archiving Configuration – 3/6
Solution 2: Archive the running configuration to tftp upon write:
archive
path tftp://10.1.1.1
write-memory

Note: Config can also be archived on-demand:

Router#archive config

Solution 3: Use Kron to schedule periodic archiving (plus other activity)

archive
path tftp://10.1.1.1
!
kron policy-list backupconfig
cli archive config
!
kron occurrence backup-occur at 23:23 recurring
policy-list backupconfig
multiple policy-lists possible

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 120
 Deployment & Activation
Example: Archiving Configuration – 4/6

Solution 4: Use Embedded Event Manager (EEM) with a Syslog Event

Detector and a TCL Applet to only archive configs if there
was a change
Define EEM Environment Variable

Router(config)# event manager environment filename <myfile.txt>

Router(config)# event manager directory user policy "flash:/TCL"
Router(config)# event manager policy archive.tcl type user

Router(config)# archive
Router(config-archive)# path flash:disk0
Router(config-archive)# maximum 14
Register EEM TCL Script

Configure Archive Location and Size

This script is available from www.cisco.com/go/ciscobeyond

( See http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=1103 )
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 121
 Deployment & Activation
Example: Archiving Configuration – 5/6
::cisco::eem::event_register_syslog pattern ".*%SYS-5-CONFIG.*" maxrun 90
#########################################################
# EEM TCL Script to archive the config upon change
#
# Developed by Marisol Palmero Sylog Event
#
# The following EEM environment variable is used:
# - filename: name of the file specified in the path command within
#
# Lets check if all the variable exists, otherwise quit
#########################################################
if {![info exists filename]} {
set result "Policy cannot be run: variable filename not set"
error $result $errorInfo
} Policy runtime
Default = 20 seconds
namespace import ::cisco::eem::* Increase this value if you see
a “Process Forced Exit” message
namespace import ::cisco::lib::* from the router.

if [catch {cli_open} result] {

puts stderr $result
exit 1
} else {
array set cli1 $result
}
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 122
 Deployment & Activation
Example: Archiving Configuration – 6/6
if [catch {cli_exec $cli1(fd) "en"} result] {
puts stderr $result
exit 1
}

set showarchive [cli_exec $cli1(fd) "show archive"]

set lines [split $showarchive "\n"]

foreach line $lines {

set result [regexp {<- Most Recent} $line ]
if {$result != 0} {
set result1 [regexp {^\s+\d+\s+(.+)-(\d+)\s+<-} $line -> path extension]
set output [cli_exec $cli1(fd) "show archive config differences
system:/running-config flash:$filename-$extension"]
if { [regexp "!No changes were found" $output] } {
break
} else {
cli_exec $cli1(fd) "archive config"
break
} Archive if there was a
} } change of if there was
if {$result == 0} { no archived version yet
cli_exec $cli1(fd) "archive config"
}
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 123
 Example: Synchronizing EEM Scripts 1/2
Problem: Synchronize EEM Policy .tcl files from a central Repository
Solution I: Use event manager update commands
1. Configure the default Repositiory:
router(config)# event manager directory user repository tftp://172.16.64.1

2. Single exec command to download, un-register and re-register:

router# event manager update user policy name my
%EEM: Update will use the repository path: tftp://172.16.64.1
%EEM: Attempting to copy tftp://172.16.64.1/my.tcl to flash:/eemtcl/my.tcl
Loading my.tcl from 172.16.64.1 (via FastEthernet0): !
[OK - 647 bytes]
%EEM: Copied 647 bytes from tftp://172.16.64.1/my.tcl to flash:/eemtcl/my.tcl
%EEM: Policy my.tcl has been successfully copied and re-registered

*Dec 10 20:12:43.198: %HA_EM-6-FMPD_UPDATE_POLICY_COPY: Policy update has copied 647 bytes from tftp:
*Dec 10 20:12:43.230: %HA_EM-6-FMPD_UPDATE_POLICY_REGISTER: Policy update has successfully re-registe
2bis. Can also synch entire groups, based on regular expression match:
router# event manager update user policy group m.*

3. Verify using show command:

router# show event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 script user syslog Off Wed Dec 10 20:12:43 2008 my.tcl
occurs 1 pattern {.*%NTP-5-PEERSYNC.*}
nice 1 queue-priority low maxrun 90.000 scheduler rp_primary
Available from: IOS 12.4(20)T
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 124
 Example: Synchronizing EEM Scripts 2/2
Solution II: Use new event manager update command

1. Single exec command to specify repository, download, un-register and re-register:

router# event manager update user policy name my.tcl repository tftp://10.1.1.1/
%EEM: Update will use the repository path: tftp://10.1.1.1
%EEM: Attempting to copy tftp://10.1.1.1/my.tcl to flash:/eemtcl/my.tcl
Loading my.tcl from 10.1.1.1 (via FastEthernet0): !
[OK - 647 bytes]
%EEM: Copied 647 bytes from tftp://10.1.1.1/my.tcl to flash:/eemtcl/my.tcl
%EEM: Policy my.tcl has been successfully copied and re-registered

*Dec 16 22:09:11.303: %HA_EM-6-FMPD_UPDATE_POLICY_COPY: Policy update has copied 647 bytes from tftp://10
*Dec 12 22:09:11.329: %HA_EM-6-FMPD_UPDATE_POLICY_REGISTER: Policy update has successfully re-registered
1bis. Can also synch entire groups, based on regular expression match:
router# event manager update user policy group m.*

2. Verify using show command:

router# show event manager policy registered
No. Class Type Event Type Trap Time Registered Name
1 script user syslog Off Wed Dec 10 20:12:43 2008 my.tcl
occurs 1 pattern {.*%NTP-5-PEERSYNC.*}
nice 1 queue-priority low maxrun 90.000 scheduler rp_primary

Available from: IOS 15.0(1)M

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 125
 Example: Install Embedded Automations
Problem: Embedded Automations based on Tcl Scripting or Embedded
Event Manager may include multiple scripts, policies, configurations,
variables and pre-requisites. How can we install (and un-install) all of
these in a consistent manner?
Solution: Create a package and use the EASy Installer
Router# easy-installer tftp://10.1.1.1/my-package.tar flash:/easy

-----------------------------------------------------------------------
Configure and Install EASy Package ‘my-package'

-----------------------------------------------------------------------
1. Display Package Description
2. Configure Package Parameters
3. Deploy Package Policies
4. Verify Installed Package
5. Exit

Enter option:

See: http://www.cisco.com/go/easy

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 126
How to pre-commission new
Cisco Devices ?

AutoInstall (DHCP Opt 150)

8
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 127
How to deal with new routers ...

?
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 128
 How to deal with new routers – Auto Install
IOS AutoInstall Feature consists of:
 Ethernet Interface up
 DHCP Client + Option 150

Combined with external

 DHCP and TFTP Server
this enables a new router to
 automatically retrieve a default configuration
 without manual interaction via console cable or telnet

See: http://www.cisco.com/en/US/docs/ios/12_1t/12_1t5/feature/guide/dt_dhcpa.html
Available from: IOS 12.1(5)T, IOS-XE 2.1.0
Platforms: ASR 1000, x8xx ISR, x9xx ISR, 37xx, ME3400, ME4900, Cat4k, Cat6k, 76xx, 10k, UC520
See also: Smart Install
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 129
 Deployment & Activation
Example: Automated Pre-Commissioning
Problem: How to automatically pre-commission a new Cisco ISR without
manual intervention on the Console
Solution: Use the AutoInstall Feature combined with an external DHCP
and TFTP server

0. Power up the CPE and

connect to Ethernet
1. CPE sends DHCP Discover
2. DHCP Server replies with Offer
3. CPE sends DHCP Request
4. DHCP Server replies with option 150
5. CPE requests hostname-confg
file from TFTP
6. TFTP erver sends hostname-
config file to CPE
 CPE is now pre-commissioned
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 130
Deployment & Activation
Example: Automated Pre-Commissioning
NE is connected
to the Network

NE gets an IP address via

BOOTP, SLARP or DHCP

NE gets network-
What exactly happens in Step 5
config file from TFTP

IP maps to
Default config
Reverse DNS No hostname in No No
file exists on
successful? network-
TFTP?
config file?
Yes
Yes
NE attempts to get Yes
hostname-config or NE gets
AutoInstall
hostname.cfg from TFTP router-config or
Fails
router.cfg from TFTP

File exists on No
TFTP?
AutoInstall
Yes Completes
AutoInstall
AutoInstall Fails
Completes manual config
completion

copy run start

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 131
How to automate entire
deployment / maintenance
scenarios ?

Zero Touch Deployment

9
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 132
Scaling Robust Remote Deployment ...

?
Telnet
Router>
Router> enable
Router# conf t revert time 2
Router(config)#

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 133
Deployment & Activation
Sometimes we need to automate ...
Typical Challenges:
• Large Scale
- more than just a few 12 image updates
- more than a few 100 config or file updates

• Robustness
- unreliable / un-managed access
- interruptions, outages

• Security
- authentication, privacy,
- trust and skills of on-site staff
- unknown hostnames / ip addresses

• Time
- de-coupling of deployment and activation
- many devices within small time window

• Cost
- manual, skilled labour cost vs. automated solution

 Automate initial and partial configuration, image upgrades or

distribution of files (any file, any place)
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 134
 Deployment & Activation
Zero-Touch Deployment Methods
Cisco IOS External Mediation
Method Notes
Deployment Agents Server
Cisco Broadband For Cable Modem Access Only
DOCSIS DOCSIS
Access Center (BAC) Widely Standardized
For DSL Access
Cisco Broadband Standard Is Work in Progress with Currently
TR-069 TR-069
Access Center (BAC) Loose Definition, Check
Interop Test from Plugfest
Flexibility for Scenarios Not
Embedded Event Covered by Any Other Method
EEM FTP, TFTP, SCP,…
Manager Sometimes Used in Concert
with Other Methods
Kron Kron and TCL FTP, TFTP, SCP,… When EEM Is Not Available
Agnostic of Access Technology
DHCP Cisco Network
DHCP Partially Standardized,
(AutoInstall) Registrar, TFTP
Multiple Options Used
CNS Config Agent Most Secure and Robust
CNS Image Agent Cisco Configuration
CNS Agnostic of Access Technology
CNS Inventory Agent Engine
CNS Event Agent Agnostic of IP Addressing

Zero-Touch Deployment = Embedded Agents + External Mediation

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 135
 Deployment & Activation
Example: Zero-Touch Deployment – 1/3
Problem: A large number of Teleworker Routers have to be deployed.
Access Technology and Service Provider vary; IP Addressing is not known
in advance
Solution: Pre-Configure Routers with a generic boostrap config
This config ensures initial IP connectivity, identifies the device and
communicates back to Configuration Engine for appropriate config
Router # cns id hardware-serial
Router # cns config initial MyConfigEngine 80 event no-persist
Router # cns id hardware-serial event
Router # cns event MyConfigEngine 11011

Note: Many other options for ID

exist and are often used instead
of hardware-serial:

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 136
 Deployment & Activation
Example: Zero-Touch Deployment – 2/3
1. CPE sends DHCP Discover
2. DHCP Server replies with Offer
Warehouse

3. CPE sends DHCP Request

4. DHCP Server replies with option 150
5. CPE requests bootstrap-confg file
via TFTP
6. TFTP server sends CPE bootstrap-
config file
⇒ CPE is shipped to Customer Site
⇒ Customer Order linked to CPE ID

7. CPE sends HTTP request to CNS-CE

Customer Premise

8. CNS-CE verifies object ID

9. CNS-CE verifies Device ID
10. CNS-CE reads template from File System
11. CNS-CE sends Config
(= template + parameters from LDAP)
12. Successful event
13. Publish success event

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 137
Custom Interactive Menus on
the CLI

12
Menu Command and
Embedded Menu Manager
(EMM)

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 138
 Interactive Menus on the CLI
Problem: How to make some CLI commands available in a guided way
(for example to 1st Line Support, Local IT, Field Force, etc)
Solution I: Configure a Menu using the old <menu> commands
Solution II: Define a custom Menu in Embedded Menu Manager (EMM)

IOS menu Command Embedded Menu Manager (EMM)

 easy to learn, simple to use  easy to learn, simple to use
 limited functionality and flexibility  very flexible
 menu only, cli only  menus and wizards, cli and tcl
 selections only  selections, inputs, actions, help texts
 part of the IOS config  separate MDF file(s)
 widely available  recent development – 12.4(20)T

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 139
 Menu Config Command – 1/2
Simple Menu Defined in the Config
 Custom ASCII Menus
 Part of IOS Config
 Simple CLI Actions
Menu name
menu OldMenu title ^C
A simple example of the OLD menu command^C
menu OldMenu prompt ^C Menu Title
Please select a menu item:^C
menu OldMenu text 1 Run a ping test Menu Item Label
menu OldMenu command 1 ping 10.1.1.1
menu OldMenu options 1 pause Menu Item Action
menu OldMenu text 9 Exit
menu OldMenu command 9 exit
menu OldMenu status-line

 Caveats:
– Remember to provide an <exit> option
– Simple menus and actions only
– No user input other than menu items
– Part of the running- and startup-config
Available from: IOS 10.0,
EASy Intro – bklauser
12.2(33)S
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 140
Menu Config Command – 2/2
router# menu OldMenu
Server “router" Line 0 Terminal-type (unknown)

A simple example of the OLD menu command

1 Run a ping test

9 Exit

Please select a menu item: 1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
--More—
Server “router" Line 0 Terminal-type (unknown)

A simple example of the OLD menu command

1 Run a ping test

9 Exit

Please select a menu item:

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 141
 Embedded Menu Manager (EMM)
Programmable Menu Framework
 Custom ASCII Menus
 XML based Menu Definition Files (MDF)
 Range / Type Checking
 TCL Scripting Actions
 Nested and Sequential Menus (Wizards)
================================================================================
Branch Router Operations Menu on branch-99
Enter ? for help or ?# for item help
--------------------------------------------------------------------------------
1. Install Diagnostic Scripts
2. Change Hostname
3. Run CPU Diagnostic Script
4. Run Memory Diagnostic Script
5. Run WAN Diagnostic Script
6. Instant World Peace
7. Exit
Enter selection [6]:

Available from: IOS 12.4(20)T

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 142
 EMM Menu Definition File Example – 1/2
Menu name and required
<?xml version="1.0"?>
schema version
<Menu MenuName="NMS" schemaVersion="1.1">
<MenuTitle>
<EmbTCLValue>
<TCLCommand>
return " Branch Router Operations Menu on [hostname]"
</TCLCommand>
Title can be constant or generated
</EmbTCLValue>
with Tcl
</MenuTitle>
<HelpString>
<Constant String="View and modify some common Network Management
configuration parameters"/>
</HelpString> The menu and each item can have
<GlobalTCL> its own help text
<TCLCommand>
proc get_config { regex } {
set config [exec "show run | inc $regex"]
return $config
}
</TCLCommand>
</GlobalTCL> Optional global Tcl section to store procs
: used throughout menu
:
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 143
 EMM Menu Definition File Example – 2/2
From simple menu choices to complete customized wizards
:
:
<Item ContinuePrompt="true" ItemJustification="LEFT">
<ItemTitle>
<Constant String=“Change Hostname" />
</ItemTitle>
<HelpString>
<Constant String="This selection lets you type a new hostname" />
</HelpString>
<Wizard>
<QueryPrompt>
<Constant String="What hostname do you suggest?" />
</QueryPrompt>
<FreeForm />
</Wizard>
<IOSConfigCommand>
"hostname $r(1)"
</IOSConfigCommand>
:
:

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 144
 Example: Combining ERM, EEM and EMM

Embedded Menu
Manager (EMM)
==========================================================
ERM Diagnostics
Policy Enter ? for help or ?# for item help
----------------------------------------------------------
1. Install Diagnostic Scripts
2. Set Global Variables (email parameters)
Embedded Event
Manager (EEM) 3. Deploy CPU Diagnostic Script
4. Deploy Memory Diagnostic Script
Event Detector 5. Deploy Buffer Diagnostic Script
6. Display Diagnostic Policy Configuration
7. Remove Diagnostic Policies
8. Exit
Embedded Resource
Manager (ERM) Enter selection [8]:

This MDF file and Tcl scripts are available from www.cisco.com/go/ciscobeyond
( See http://forums.cisco.com/eforum/servlet/EEM?page=eem&fn=script&scriptId=1363 )
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 145
Custom Interactions via
HTTP
 Extensible HTTP Server in IOS
Problem: Sometimes we may event want to (or need to) provide a web-
based custom interaction with IOS
Solution: Customize the EASy HTTx Package – which provides an
extensible HTTP Server running on IOS

c1812-easy#
c1812-easy# httx-start
c1812-easy#
*Jan 6 11:02:44.649: %HA_EM-6-LOG: no_easy_httx_start.tcl: Accepting connection from 10.55.146.51:3235
*Jan 6 11:02:44.669: %HA_EM-6-LOG: no_easy_httx_start.tcl: "GET flash:/easy/easy-httx_public/html/index
*Jan 6 11:02:44.825: %HA_EM-6-LOG: no_easy_httx_start.tcl: Accepting connection from 10.55.146.51:3236
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 147
EASy HTTx Package – Extensible HTTP
Embedded Automation Systems (EASy)
HTTx EASy Package Provides:
 Interactive Installation
 Light-weight HTTP Server in Tcl
 Ability to trigger Tcl scripts on IOS
 Extensible Framework

To use the Package:

1. Browse and Download HTTx EASy Package
www.cisco.com/go/easy

2. Make Sure to also download EASy Installer

3. Watch VOD and/or read documentation

www.cisco.com/go/easy

4. Customize and tailor to your needs

5. Install and Use

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 148
Wrap-Up & Close
In Summary

 All ‘Configuration’ tasks

are NOT equal

 There are a Range of Users / Applications with different

configuration Skills and Needs

 It‘s not only about telnet and running-config

 Cisco IOS offers a plethora of configuration features to

address the specific needs

 Always choose the best fit

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 149
Agenda

Introduction & Overview

Service Planning
Service Deployment & Activation
 Service Testing, Verification & Assurance
Troubleshooting & Optimization
Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 150
Testing, Verification & Assurance
Two Types of Questions
 Is it working ? Testing and Verification
Verify planning and design assumptions were valid
Ensure Deployment & Activation Phase was successful
Proactively eliminate well-known potential problems
Periodically verify design assumptions

 Are we meeting SLA ? Service Assurance

Ensure business objectives and service level agreements are met
on an ongoing basis
Proactively mitigate well-known potential incidents

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 151
Testing, Verification & Assurance
Two Types of Connectivity
 Connectivity, Yes/No Testing and Verification
If the user can reach the IP endpoint the service is available
Can be calculated using basic availability equation

 Bounded Criteria Connectivity Service Assurance

The user can reach the IP endpoint within some bounded criteria agreed
upon between the service provider and customer
Connectivity is a prerequisite for bounded crieria connectivity

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 152
IP Service Level Agreements (IP SLA)
 Active probing by injecting synthetic test traffic
 Experience and Adoption across markets and technology domains
 Vast range of Cisco and 3rd Party NMS tool support

Metrics
Latency Jitter Packet Loss Connectivity

Domains
IP Ethernet MPLS VoIP Services Medianet

Operations ICMP ICMP UDP TCP 802.1ag LSP PWE3 H.323 SIP HTTP DNS
Echo Jitter PathEcho Connect Jitter Trace VCCV GD GD

ICMP UDP UDP 802.1ag LSP LSP H.323 SIP

PathEcho Echo Jitter Echo Ping Tree CS CS DHCP FTP

IPIPSLA
IP SLA
SLA
MIB
MIB Operation
Operation
MIB Operation
Data
Data
Data

IP SLA Source IP SLA Responder

See: www.cisco.com/go/ipsla
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 153
 Testing, Verification & Assurance
IPSLA – Introduction 2/2
 Cisco IOS feature available on most platforms Accessible via CLI and SNMP
(CISCO-RTTMON-MIB)
 Measure Delay, Jitter, Loss Probability
 IPSLAs responder and ICMP echo probe were available within IP Base in
12.4(6)T and above
 IPSLAs functionality is available in IPVoice and above packages
 In 12.3T a customer can still obtain the old package types and use
IPSLAs
Since IOS
As of 11.2 the old packages have been removed
12.4T
12.2(15)T2, 12(3)3, 12.2(25)S
time

Engine: Engine 1 Engine 2

Feature Name: RTR SAA IPSLAs

CLI: rtr… ip sla mon… ip sla …

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved.
12.3(12)T
Cisco Public
12.4(6)T 154
 Testing, Verification & Assurance
IPSLA – ICMP and UDP Jitter Examples

RouterA

RouterC

RouterA(config)#
ip sla 1
RouterD
icmp-echo RouterC
timeout 500
frequency 10
ip sla schedule 1 start-time now

ip sla 10
udp-jitter RouterD 16384 num-packets 1000 interval 20
request-data-size 172
tos 20
frequency 60
ip sla schedule 10 start-time now
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 155
Testing, Verification & Assurance
IPSLA – ICMP Echo Operation

Router#show ip sla sta mon 1

Round trip time (RTT) Index 1
Latest RTT: 1 ms
Latest operation start time: *05:26:00.226 UTC Fri Jan 4 2008
Latest operation return code: OK
Number of successes: 1
Number of failures: 0
Operation time to live: 188 sec

Router#sh ip sla sta 1 detail

Round trip time (RTT) Index 1
Latest RTT: 1 ms
Latest operation start time: *05:26:30.224 UTC Fri Jan 4 2008
Latest operation return code: OK
Over thresholds occurred: FALSE
Number of successes: 2
Number of failures: 0
Operation time to live: 155 sec
Operational state of entry: Active
Last time this entry was reset: Never
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 156
Testing, Verification & Assurance
IPSLA – UDP Jitter Operation

Router#sh ip sla statistics 10

Round trip time (RTT) Index 10
Latest RTT: 1 ms
Latest operation start time: *05:43:28.720 UTC Fri Jan 4 2008
Latest operation return code: OK RTT Values
Number Of RTT: 10
RTT Min/Avg/Max: 1/1/1 ms
Latency one-way time milliseconds
Number of one-way Samples: 0
Source to Destination one way Min/Avg/Max: 0/0/0 ms
Desination to source one way Min/Avg/Max: 0/0/0 ms
Jitter time milliseconds
Number of Jitter Samples: 9
Source to Destination Jitter Min/Avg/Max: 20/20/23 ms
Destination to Source Jitter Min/Avg/Max: 22/21/24 ms
Packet Loss Values
Loss Source to Destination: 0 Loss Destination to
Source: 0
Out Of Sequence: 0 Tail Drop: 0 Packet Late
Arrival: 0
Number of successes: 1
Number of failures: 0
Operation time to live: 3567 sec
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 157
Design Decisions and Factors
 Topology
- partial mesh based on traffic matrix
- full mesh
- hub and spoke

 Scheduling
- minimize the number of concurrent operations
- minimize resource competition

 Use the same operation across various classes of

service to generate comparable metrics.

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 158
Full Mesh
Nodes Operation
2 1
3 3
4 6
5 10
6 15
7 21
8 28
… …
100 4950
• Number of operations is
proportional to the square
of the number of nodes
• Does not scale
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 159
 Full Mesh CE-to-CE [Example]

CE CE

PE Core PE

PE

 Accurate: direct measurement from end-to-end, best

user-perspective view
CE
 Expensive: for n nodes, requires n(n-1)/2 operations
 In certain cases, it might be difficult to poll the results with SNMP
on the CE
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 160
 Partial Mesh

London
 Full mesh is not
Amsterdam San Jose always desirable
 Select only critical
path, like branch
offices to
headquarters
Raleigh Paris
 Dramatically reduces
the number of probes

Brussels

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 161
 Composite SLA for Delay [Example]

CE CE

PE Core PE

PE

 Easy: Total delay can be easily calculated by adding the

measured delay along the path

CE
 Flexible: You can split the measurement for Core
Edge, and total
 Measurements are less accurate, as each measurement carry
its own error tolerance (typically ± 1 ms per measurement)

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 162
 Composite SLA for Packet Drop [1/2]

 A trivial solution might is to consider the sum of drop

probabilities; this is conservative
 A more accurate approach is to invert the probability of a
successful packet delivery
 If Πx is the loss probability across section x, then the total
loss probability is:

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 163
 Composite SLA for Packet Drop [2/2]

Example: We Have Three Sections with

Various Drop Probabilities:

Π1 = 0.01 Π2 = 0.02 Π3 = 0.03

r1 r2 r3 r4

 First solution:
0.01+0.02+0.03=0.06 (6%)
 Second solution:
1-[(1-0.01).(1-0.02).(1-0.03)]=0.058906 (5.8%)

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 164
Composite SLA for Jitter

2 ms 4 ms 3 ms

Can We Add a Jitter Value to a Jitter Value?

 Short answer: NO!

 This is not a valid approach to calculate total jitter based on
measured jitter
(jitter is not additive)
 Too many factors: positive jitter, negative jitter, percentile-95
of jitter, average jitter,…
 You’d better measure it, not calculate it

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 165
Testing, Verification & Assurance
IPSLA – Recurring Scheduling

 You can schedule a single IPSLAs operation to start

automatically at a specified time and for a specified duration
every day:
The life value for a recurring IPSLAs operation should be less than
one day.
The ageout value for a recurring operation must be "never" (which
is specified with the value 0, this is the value by default), or the
sum of the life and ageout values must be more than one day.

 Example:
Router(config)# ip sla schedule 5 start-time 12:00:10
life 3600 recurring

*12.3(8)T
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 166
Testing, Verification & Assurance
IPSLA – Multiple Operations Scheduling

 Operations of the same type and same frequency should be

used with IPSLA multiple operations scheduling:
Notion of group, it lets you start many operations at once
Reduced load on the network
If you do not specify a frequency, the default frequency will be the
same as that of the schedule period)
 Example, start operations 1 to 3 within the next 20 seconds

Router (config)# ip sla 1

Router (config)# icmp-echo RouterC
Router (config)# ip sla 2
Router (config)# icmp-echo RouterD
Router (config)# ip sla 3
Router (config)# icmp-echo RouterE

Router (config)#ip sla group schedule 1 1-3 sch 20 start now

Router #show ip sla group schedule

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public *12.3(8)T167
 Testing, Verification & Assurance
IPSLA – Random Scheduling
Problem: Strictly periodically starting IPSLA operations might be subject
to ‘synchronization effects’ with other processes (ie. routing updates),
leading to inaccurate data.
Solution: Use IPSLA Random Scheduling to randomize start time

This example starts operation 1 to 3 within the next 44 seconds, and each
operation will have a random frequency varying between 10 and 15 seconds:
Router(config)#ip sla group schedule 1 1-3 schedule-period 44 frequency range
10-15 start-time now life forever

Router#sh ip sla op | i start

Latest operation start time: *12:56:12.243 PST Fri Jan 4 2008
Latest operation start time: *12:56:06.323 PST Fri Jan 4 2008
Latest operation start time: *12:56:07.743 PST Fri Jan 4 2008
router#sh ip sla op | i start
Latest operation start time: *13:00:19.423 PST Fri Jan 4 2008
Latest operation start time: *13:00:15.895 PST Fri Jan 4 2008
Latest operation start time: *13:00:21.015 PST Fri Jan 4 2008 *12.4(2)T
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 168
What about SLA in
dynamic networks?
 New
15.1T
Auto IP SLA – Don‘t touch your Hub
Some IP SLA Topologies …
 … are naturally Hub and Spoke
 … have a large number of Spokes with similar IP SLA requirements
 … consist of dynamically joining / disappearing Spokes
ip sla auto template type ip udp-jitter my-ipsla-
template
parameters
request-data-size 64
num-packets 1000
ip sla auto schedule my-ipsla-schedule
frequency 45
start-time now
ip sla auto endpoint-list type ip my-ipsla-endpoints
discover
ageout 36000
ip sla auto group type ip my-ipsla-group
schedule my-ipsla-schedule
template udp-jitter my-ipsla-template
destination my-ipsla-endpoints

ip sla responder auto-register 10.10.10.2 endpoint-list my-ipsla-endpoints

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 170
Service Testing, Verification and Assurance
Example: Network Automation with IPSLA – 1/4

Problem
 Need to monitor IP SLA
 Trigger actions upon violation of SLA

Solutions
 IP SLAs Thresholds
 Using EEM and the EOT Event Detector
 Using EEM 3.x and the IP SLA Event Detector

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 171
Service Testing, Verification and Assurance
Example: Network Automation with IPSLA – 2/4
Solution 1: Using IP SLA reaction triggers:
RouterA(config)#
ip sla 10
icmp-echo 3.3.3.3
frequency 10
ip sla reaction-configuration 10 react timeout threshold-type consecutive 3
action-type trapAndTrigger
ip sla schedule 10 life forever start-time now
ip sla reaction-trigger 10 20

logging on
ip sla logging trap

snmp-server host nms_server version 2c public

snmp-server enable traps syslog

Sending SNMP trap with IP SLAs embedded threshold

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 172
 Service Testing, Verification and Assurance
Example: Network Automation with IPSLA – 3/4
IP SLA Embedded Object Tracking (EOT)
ip sla 10 track 10 rtr 10 reachability
icmp-echo 3.3.3.3 delay down 10 up 20
timeout 500
frequency 3
ip sla schedule 10 life forever start-time now

IP SLA/EOT/EEM

X
Environment Variables
($_* variables to be defined)

EEM Applet
email
event manager applet email_server_unreachable 3.3.3.3
event track 10 state down
action 1.0 syslog msg "Ping has failed, server unreachable!"
action 1.1 cli command "enable"
action 1.2 cli command "del /force flash:server_unreachable"
action 1.3 cli command "show clock | append server_unreachable"
action 1.4 cli command "show ip route | append server_unreachable"
action 1.5 cli command "more flash:server_unreachable"
action 1.6 mail server "$_email_server" to "$_email_to" from "$_email_from" subject "Server Unreachable: ICMP-Echos
Failed" body "$_cli_result"
action 1.7 syslog msg "Server unreachable alert has been sent to email server!"
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 173
Service Testing, Verification and Assurance
Example: Network Automation with IPSLA – 4/4
Solution 3: Using Embedded Event Manager 3.0 IP SLA Event Detector:

Router(config)# ip sla 10
Router(config-ip-sla)# icmp-echo 3.3.3.3

Router(config)# ip sla enable reaction-alerts

Router(config)#ip sla reaction-config 1 react Timeout

action-type none threshold-type consecutive 3

Router(config)# ip sla schedule 10 start now

Router(config}# event manager applet test

router(config-applet)# event ipsla operation-id 10 reaction-type Timeout
router(config-applet)# action 1.0 syslog priorities emergencies
msg “IP SLA operation $_ipsla_oper_id to server XYZ has timed out”

Trigger an Embedded Event Manager Applet when the

IP SLA operation threshold is crossed
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 174
EASy Package: Custom High-Availability
Problem: We need a failover from primary to secondary link – but with
flexibility and custom notification beyond what a simple routing protocol based
solution provides

Solution: Automate based on IP SLA, EOT and Embedded Event Manager

Upon State Change

Did
IP SLA
succeed Operation timeout

Tracked object is up, Tracked object is down,

Execute up commands Execute down commands

Is Is
No No
up-syslog down-syslog
set? set?
Yes Yes

Send up syslog Send down syslog

done

See: Available as an EASy Package:

http://www.cisco.com/go/easy
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 175
Agenda

Introduction & Overview

Service Planning
Service Deployment & Activation
Service Testing, Verification & Assurance
 Troubleshooting & Optimization
Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 176
Be Prepared – Some Good
Practices

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 177
Be Prepared – Some Good
Practices

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 178
Troubleshooting & Optimization
Good Practice: Reserve Memory for Cons.
Problem: Network or Device Problems may consume a lot of
Memory and/or Memory may become extensively fragmented –
potentially there won’t be enough Memory left for the Console …
Solution: Reserve Memory for the console ahead of time, on
every device

Router(config)# memory reserved console <number-of-kilobytes>

Rule of Thumb: for the number of kilobytes use a value greater than 3
times the NVRAM size

 IOS Default is 256 kilobytes

 available since 12.0(22)S, 12.2(28)SB (7300), 12.4(15)T

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 179
Where to start with MIBs ?
MIB Locator:
http://www.cisco.com/go/mibs

SNMP Object Navigator:

http://www.cisco.com/go/mibs
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 180
 Troubleshooting & Optimization
Good Practice: Check SNMP OID Statistics
Which OIDs are my NMS Apps (CiscoView) polling ?
Router#show snmp statistics oid

time-stamp #of times requested OID

16:16:50 CET Jan 12 2005 97 sysUpTime
16:16:50 CET Jan 12 2005 9 cardTableEntry.7
16:16:50 CET Jan 12 2005 9 cardTableEntry.1
16:16:50 CET Jan 12 2005 4 cardTableEntry.9
16:16:50 CET Jan 12 2005 16 ifAdminStatus
16:16:50 CET Jan 12 2005 16 ifOperStatus
16:16:50 CET Jan 12 2005 6 ciscoEnvMonSupplyStatusEntry.3
16:16:50 CET Jan 12 2005 17 ciscoFlashDeviceEntry.2
16:16:50 CET Jan 12 2005 8 ciscoFlashDeviceEntry.10
16:16:50 CET Jan 12 2005 2 ltsLineEntry.1
16:16:50 CET Jan 12 2005 2 chassis.15
16:16:27 CET Jan 12 2005 11 ciscoFlashDeviceEntry.7
16:16:27 CET Jan 12 2005 2 cardIfIndexEntry.5
16:16:24 CET Jan 12 2005 1 ciscoFlashDevice.1

Available from: IOS 12.0(22)S, 12.4(20)T

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 181
Troubleshooting & Optimization
Good Practice: IfIndex Persistence – 1/3

 Feature which can make ifIndex persist across reboots (In

Switches is on by default)
 ifIndex persistence means that the mapping between the
ifDescr (or ifName) and ifIndex object values from the IF-MIB
is retained across reboots.
 Useful:
SNMP: monitoring the interfaces counters
NetFlow: reporting of the interface ifIndex
RMON: events/alarms based on specific interfaces
 25 bytes of NVRAM used by this feature per interface.
Applying ifIndex persistence to
all interfaces
Router(conf)# snmp-server ifindex persist
Router(config-if)# snmp-server ifindex persist
Applying ifIndex persistence to
EASy Intro – bklauser
an specific interface
© 2009 Cisco Systems, Inc. All rights reserved. 182
Cisco Public
Troubleshooting & Optimization
Good Practice: IfIndex Persistence – 2/3
Now there is a show command:

Router# show snmp mib ifmib ifindex

Ethernet0/0: Ifindex = 1
Loopback0: Ifindex = 39
Null0: Ifindex = 6
:

Router# snmp mib ifmib ifindex loopback 0

Loopback0: Ifindex = 39

Introduced in 12.0(7)S, 12.2(2)T

http://www.cisco.com/en/US/customer/products/sw/iosswrel/ps1839/
products_feature_guide09186a0080087b0d.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 183
Troubleshooting & Optimization
Good Practice: IfIndex Persistence – 3/3
Router(config)# snmp-server ifindex persist
Router(config)# snmp mib persist event EVENT-MIB
Router(config)# snmp mib persist expression
Router(config)# snmp mib persist circuit EXPRESSION-MIB
Router(config)# snmp mib persist cbqos CIRCUIT-MIB

CISCO-CLASS-BASED-QOS-MIB

 You must perform a copy running starting

command to persist the newly assigned ifIndex
values.
copy running start!
Router # dir nvram:ifIndex-table
Directory of nvram:/ifIndex-table
2 -rw- 283
0 <no date> ifIndex-table
126968 bytes total (114116 bytes free)
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 184
Reliable Delivery and Filtering
of Syslog

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 185
Troubleshooting & Optimization
Reliable Delivery and Filtering of Syslog

 Provides for reliable and secure delivery for syslog

messages using Blocks Extensible Exchange Protocol
(BEEP)
RFC 3195, “Reliable Delivery for syslog”

 Provides a filtering mechanism per syslog session,

called a message discriminator
 Provides a rate-limiter per syslog session
 Integrated in 12.4(11)T, even if the BEEP framework
was supported for quite some time, 12.4(2)T
 Which syslog servers support BEEP?
http://www.syslog.cc/ietf/rfcs/3195.html

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 186
 Troubleshooting & Optimization
Example: Filtering of Syslog – 1/2
Production SyslogD

BEEP for highest

severities, with a UDP for lowest
maximum rate-limit severities, with a
rate-limit of 100/s
(10000/s)
Troubleshooting
SyslogD

UDP for the debug

syslog
syslog messages
messages
with OSPF in the
message body
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 187
 Troubleshooting & Optimization
Example: Filtering of Syslog – 2/2

Router(config)# logging discriminator filter1

severity includes 0,1,2,3 rate-limit 10000
Router(config)# logging discriminator filter2
severity includes 4,5,6,7 rate-limit 100
Router(config)# logging discriminator filter3 msg-
body includes debug includes facility OSPF

Router(config)# logging trap debugging

Router(config)# logging host <production> transport

beep discriminator filter1
Router(config)# logging host <production> transport
udp port 1471 discriminator filter2
Router(config)# logging host <troubleshooting>
discriminator filter3
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 188
What about Syslog messages
indicating
an ACL hit ?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 189
 Troubleshooting & Optimization
ACL Syslog Correlation
Problem: ACL hits can produce a Syslog message – but often in the NOC
or SOC we want to know which specific line of an ACL (ie.: ACE – Access
Control Entry) was kicking-in ...
Solution: Make use of IOS ACL Tags and Syslog Correlation

1. Define Tags for your ACEs:

ip access-list extended access-control
permit ip any host 10.10.10.100 log red-server
permit ip any host 10.10.10.200 log blue-
server
permit ip any any

2. Tags will be appended to Syslog Messages:

*Apr 13 16:31:18.958: %SEC-6-IPACCESSLOGDP: list access-control
permitted icmp 192.168.1.100 -> 10.10.10.100 (0/0), 11 packets [ red-
server ]
*Apr 13 16:32:18.953: %SEC-6-IPACCESSLOGDP: list access-control
permitted icmp 192.168.1.100 -> 10.10.10.200 (0/0), 3 packets [ blue-
server ]
See: http://www.cisco.com/en/US/partner/docs/ios/security/configuration/guide/sec_acl_syslog.html
Available from: IOS 12.4(22)T
Platforms: 18xx, 28xx, 38xx, 72xx, 73xx, 76xx
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 190
Troubleshooting & Optimization
Example: ACL Syslog Correlation and EEM
Problem: Let’s assume we not only need a syslog message, but also
want to take specific actions ...
Solution: Combine ACL Syslog Correlation with EEM
1. Define Tags for your ACEs:
access-list 100
deny tcp host 10.0.2.2 host 10.0.2.181 eq 9000 log ThisIsBlocked
permit ip any any

2. Define an EEM Applet to match the Tag and take action:

event manager applet catch-an-ace-tag
event syslog pattern "ThisIsBlocked"
action 1.0 syslog priority emergencies msg “Start... "
:
Your Actions Here
:
action 9.0 syslog priority emergencies msg "... done"

3. A matching packet will generate a syslog message, which will in turn trigger EEM :
*Apr 13 16:58:06.386: %SEC-6-IPACCESSLOGDP: list 100 denied tcp 10.0.2.2
(56273) 10.0.2.181(9000), 1 packet [ThisIsBlocked]
*Apr 13 16:58:06.394 UTC: %HA_EM-0-LOG: catch-an-ace-tag: Start ...
*Apr 13 16:58:07.025 UTC: %HA_EM-0-LOG: catch-an-ace-tag: ... done
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 191
Good to know: any traffic with
low TTL?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 192
Troubleshooting & Optimization
Good Practice: Know about low-TTL
TTL is:
 An IP Header field used to limit packet life time (upon routing loops)
 Each routing hop along a packet’s path decrements this value
 Upon TTL==0 the packet is dropped

Low TTL Values are also (mis-)used:

 to establish bidirectional TCP sessions across NAT
(aka TCP hole-punching)
 by some applications (multicast, load-balancing, …)
 by security attacks (denial of service, break-in, ...)

Hence Low TTL can indicate:

 Routing issues
 Funny application behaviour
 Security incidents (what is my normal low-TTL traffic ?)

 How to report on low-TTL in my network ?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 193
 Troubleshooting & Optimization
Example: Monitor low-TTL Traffic ...
Problem: We want to know about low-TTL traffic
Solution: Use Flexible Netflow and Embedded Event Manager 3.0 to
detect traffic flows with TTL < 5
1. Configure flexible Netflow to match on TTL, Source- and Destination Address
flow record <my-record> Baseline Cache …
match ipv4 ttl
match ipv4 source address
match ipv4 destination address
: -Top (unexpected) Talkers with low-TTL traffic ?
flow monitor <my-monitor> - Deviation from Normal ?
record <my-record> - Senders with many low-TTL flows ?
: - Take Actions (block suspicious senders) ?

2. Configure the Netflow Event Detector in EEM to notify upon a new flow record
event manager applet my-ttl-applet
event nf monitor-name "my-ttl-monitor" event-type create event1
entry-value "5" field ipv4 ttl entry-op lt
action 1.0 syslog msg “Low-TTL flow from $_nf_source_address"

3. Syslog message and/or use show flow monitor <my-monitor> cache command
*Dec 2 17:39:31.221: %HA_EM-6-LOG: my-ttl-applet: Low-TTL flow from 192.168.2.248
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 194
What if I need a
Packet Capture – I?
 Troubleshooting & Optimization
Embedded Packet Capture (EPC)
Problem: Sometimes a Packet Capture would be useful for
Troubleshooting, Security or Application Analysis, Baselining, etc.
BUT: deploying Packet Sniffers is slow, expensive and requires local skills
and equipment ...
Solution: Make use of IOS Embedded Packet Capture to capture PCAP
format data and/or analyze on the device
1. Defining a capture buffer on the device
Router# monitor capture buffer …

2. Defining a capture point

Router# monitor capture point … Capture .pcap
Buffer File
3. Associate capture point to buffer
Router# monitor capture point associate …
4. Start / Stop capture points
Router# monitor capture point start …
Capture
5. Show and/or Export the content of the buffer Point
Router# monitor capture buffer <tracename> export
See: http://www.cisco.com/go/epc
Available from: IOS 12.4(20)T
Platforms: 8xx, 18xx,
EASy Intro – bklauser 28xx,
© 2009 38xx
Cisco Systems, ISRs,
Inc. All 72xx
rights reserved. Cisco Public 196
 Troubleshooting & Optimization
Example: process-switched traffic – 1/2
We want to capture process-switched traffic:
1-3. Define a capture buffer, capture point and associate the two
Router# monitor capture buffer my-buffer size 100 max-size 1000 circular
Router# monitor capture point ip process-switched my-capture in
Router# monitor capture point associate my-capture my-buffer
4. Start capturing traffic
Router# monitor capture point start all
*Nov 25 10:00:58.990: %BUFCAP-6-ENABLE: Capture Point my-capture enabled.
5. Show / Analyze on the router …
Router# show monitor capture buffer all parameters
Capture buffer my-buffer (circular buffer)
Buffer Size : 102400 bytes, Max Element Size : 1000 bytes, Packets : 28
Allow-nth-pak : 0, Duration : 0 (seconds), Max packets : 0, pps : 0
Associated Capture Points:
Name : my-capture, Status : Active We have. some traffic
Configuration:
monitor capture buffer my-buffer size 100 max-size 1000 circular
monitor capture point associate my-capture my-buffer
Router# show monitor capture buffer my-buffer dump
10:14:05.914 UTC Nov 25 2008 : IPv4 Process : Fa0/0 None
66A3C5B0: FFFFFFFF FFFF0001 64FF4C01 ........d.L.
66A3C5C0: 080045C0 00300000 00000111 0B5AACA1 [email protected],!
66A3C5D0: 0103FFFF FFFF02C7 02C7001C 85F60001 .......G.G...v..
66A3C5E0: 0010AC12 01020000 5D4C0F03 0004AC12 ..,.....]L....,.
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 197
 Troubleshooting & Optimization
Example: process-switched traffic – 2/2
5. … or export as PCAP file and analyze externally
Router# monitor capture buffer my-buffer export tftp://10.10.10.10/mypcap

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 198
 Troubleshooting & Optimization
EPC – Additional Considerations
 Capture stop criteria:
– manual stop
– after a specified time interval
– after given number of packets

 Capture point:
– IPv4 or IPv6
– CEF (drop, punt) or process switching
– interface specific or all interface
– Direction: in, out, both, from-us (process-switched specific)
– multicast: only ingress packets are captured, not the replicated egress packets
– MPLS: does not capture MPLS encapsulated frames today

 Buffer can be defined as linear or circular

 Buffer filter based on an access-list

Router# monitor capture buffer my-buffer filter access-list 10

 Buffer export options: FTP, HTTP, HTTPS, RCP, SCP, or TFTP

Note: exec mode commands only, nothing in the configuration

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 199
What if I need a
Packet Capture – II?
Diagnosing Transient Problems
Problem: you are seeing VPN tunnel drops on your VPN head-end
router at 3:00 am every day. The tunnels continue to flap until the
physical interface is reset. You want to analyze the traffic on the wire
at that time.

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 201
EPC – EASy Package
Embedded Automation Systems (EASy)
EPC EASy Package Supports:
 Interactive Installation
 Timed or manual capture start
 Linear or circular buffer
 Buffer Export

To use the Package:

1. Browse and Download EPC EASy Package
www.cisco.com/go/easy

2. Make Sure to also download EASy Installer

3. Watch VOD and/or read documentation

www.cisco.com/go/easy

4. Customize and tailor to your needs

5. Install and Use

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 202
What if I need a
Packet Capture – III?
NAM 5.0: Smart Capture Analysis New
Jan 20
11
Highlights observed anomalies in packet traces

NAM enables:
 Packet trace analysis
highlighting observed protocol/
packet level anomalies
 One-click targeted packet
captures
 Combined application visibility,
traffic analysis and smart
packet capture analysis
NAM benefits:
 Improves operational efficiency
with on-demand captures
 Smart analysis pinpoints root-
cause much faster than
manually analyzing or
scanning the packet traces

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 204
NAM 5.0: Troubleshooting Workflow New
Jan 20
Isolate Source of Application Performance Degradation 11

1. Analyze application
performance over time

2. Zoom to investigate
specific performance
issues
Time-based Filter

3. Identify the Top N clients 4. Isolate the servers with 5. Drill-down to select
affected by the degradation high response time server to analyze
activity

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 205
 NAM 5.0: WAN Optimization Analysis New
Jan 20
Monitor Client Experience and Optimization Improvements 11

Select Branch Site,

Server Site/Server, Analyze performance
Application, and application traffic
Reporting Interval (Optimized vs. Passthru)

Examine number of
Examine Traffic Volume
Concurrent Connections
(Client, WAN) and achieved
(Optimized vs. Passthru)
Compression Ratio

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 206
Preventive Maintenance
– anyone ?
 POST (Power-On Self-Test) is a great thing ...

... but some errors you prefer to know while

the system is still running ...

... and: can you afford to power-cycle a box

after OIR just for POST to run ?

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 208
 Troubleshooting & Optimization
Generic OnLine Diagnostics (GOLD)
CLI and scheduling for Functional Runtime Diagnostics
 Bootup Diagnostics (upon bootup and OIR)
Good Practice: schedule all
 Periodic Health Monitoring (during operation) non-disruptive tests
periodically
 OnDemand (from CLI)
 Scheduled Testing (from CLI)
 Test Types include:
– Packet switching tests
• Are supervisor control plane & forwarding plane
functioning properly?
• Is the standby supervisor ready to take over?
• Are linecards forwarding packets properly?
• Are all ports working?
• Is the backplane connection working?

– Memory Tests
– Error Correlation Tests
 Complementary to POST
Available from: CatOS 8.5(1), IOS 12.2(14)SX
Platforms: CBS 3xxx, Cat 3560, 3750, 6500, ME6524, 72xx, 10k, CRS
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 209
 Troubleshooting & Optimization
Example: The effect of wear and tear – 1/2
Problem: Repeated insertion and removal of Modules can lead to wear
and tear damage on connectors. This in turn can cause failures … how do
you find out during operation, without power-cycling the box ?
Solution: Use GOLD to verify functionality of a mis-behaving module
1) Let’s see which GOLD tests are available and scheduled for our Module:
Router# show diagnostic content module 3
Module 3:

Diagnostics test suite attributes:

M/C/* - Minimal level test / Complete level test / Not applicable
B/* - Bypass bootup test / Not applicable
P/* - Per port test / Not applicable
D/N/* - Disruptive test / Non-disruptive test/ Not applicable
S/* - Only applicable to standby unit / Not applicable
X/* - Not a health monitoring test / Not applicable
F/* - Fixed monitoring interval test / Not applicable
E/* - Always enabled monitoring test / Not applicable
A/I - Monitoring is active / Monitoring is inactive

ID Test Name Attributes (day hh:mm:ss.ms)

==== ================================== ============ =================
1) TestScratchRegister -------------> *B*N****A 000 00:00:30.00
2) TestSPRPInbandPing --------------> *B*N****A 000 00:00:15.00
:
18) TestL3VlanMet -------------------> M**N****I not configured
:

See: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/diagtest.html
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 210
 Troubleshooting & Optimization
Example: The effect of wear and tear – 2/2
2) Now let’s run TestL3VlanMet on-demand for Module 3:
Router# diagnostic start module 3 test 18
:
00:09:59: %DIAG-SP-3-MINOR: Module 3: Online Diagnostics detected a
Minor Error. Please use 'show diagnostic result <target>' to see
test results.
show diagnostics result module 3 detail
3) Then check the test results:
Router# show diagnostic result module 3
Module 3: CEF720 48 port 1000mb SFP SerialNo : xxxxxxxx

Overall Diagnostic Result for Module 3 : MINOR ERROR

Diagnostic level at card bootup: minimal

Test results: (. = Pass, F = Fail, U = Untested)

1) TestTransceiverIntegrity:
Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
----------------------------------------------------------------------------
U U U U U U U U U U U U U U U U U U U U U U U U

Port 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
----------------------------------------------------------------------------
U U U U U U U U U U U U U U U U U U U U U U U U

:
:
18) TestL3VlanMet -------------------> F
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 211
Troubleshooting & Optimization
GOLD and Embedded Event Manager

Combine GOLD and

Embedded Event Manager

 GOLD Event Detector: to trigger EEM actions

based on GOLD test results
(custom alerts, failover, diagnostics, ...)

 OIR or CLI Event Detector: to trigger an

on-demand GOLD test as post-validation of
deployment or maintenance work

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 212
 Troubleshooting & Optimization
GOLD and CiscoWorks LMS
CiscoWorks LMS supports GOLD configuration and monitoring

Good Practice: schedule all

non-disruptive tests
periodically

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 213
 Troubleshooting & Optimization
Smart Call Home – CCO Application
Before
 Personalized Reports
– Messages, diagnostics and
recommendations
– Inventory and configuration for
all Call Home devices
– Security alerts, Field notices,
and End-of-Life notices
– Configuration Sanity Analysis
– PDF and XLS Export

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 214
Troubleshooting & Optimization
Smart Call Home
Customer

Cisco Smart Cisco TAC

Call Home

Diagnostics
Rules

Installed Base

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 215
Troubleshooting & Optimization
Smart Call Home with a Partner
Customer Partner

Cisco Smart Cisco TAC

Call Home

Diagnostics
Rules

Installed Base

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 216
Troubleshooting & Optimization
Smart Call Home – Transport Gateway
Customer Partner

Customer Cisco Smart Cisco TAC

DMZ Call Home

?
Transport
Gateway Diagnostics
mailbox
Rules

Linux, Solaris, Installed Base

Windows OS

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 217
 Troubleshooting & Optimization
Smart Call Home
Network EMS, NMS Network Network Support
Device Software Operator Engineer Engineer From
 Late Surprises
 Multiple Manual
SNMP, Syslog Escalation Steps
UI, email

TTS, email, voice

 Iterative Problem
Isolation
CRM, email, voice

CRM, email, voice  Phone, Email based

Data Exchange

Registered Network Network SmartCallHome Support

Device Operator Engineer on cisco.com Engineer To

 Early Warnings
 Automated Flow
Smart Call Home Message

CRM  Pinpoint Detailed Events

CRM, email, voice
 Reporting and Exports
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 218
 Troubleshooting & Optimization
Smart Call Home – Transport Gateway
Registered Mail Transport SmartCallHome
Device Server Gateway on cisco.com

1
SMTP
2
POP / IMAP
3
HTTPS

 Platform Support
– Redhat Linux
– Solaris
– Microsoft Windows

 Free Download and Install Guide

www.cisco.com/go/smartcall
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 219
 Troubleshooting & Optimization
Smart Call Home – How to get started ...
Before Available

 Verify Device Coverage Committed

Planned

– Across segments
– Platform support

 Enroll TG first if needed

 Step-by-Step Quick Start

Enrollment Guides:
www.cisco.com/go/smartcall

 Complete Enrollment by providing

Security Token received via email

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 220
Agenda

Introduction & Overview

Service Planning
Service Deployment & Activation
Service Testing, Verification & Assurance
Troubleshooting & Optimization
 Summary

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 221
References
 Q&A
References – Instrumentation
Device Manageability Instrumentation (DMI) www.cisco.com/go/instrumentation
 Embedded Event Manager (EEM): www.cisco.com/go/eem
 Cisco Beyond – EEM Community: www.cisco.com/go/ciscobeyond
 Embedded Menu Manager (EMM): http://tinyurl.com/emm-in-124t
 Embedded Packet Capture (EPC): www.cisco.com/go/epc
 Flexible NetFlow: www.cisco.com/go/netflow and www.cisco.com/go/fnf
 GOLD: http://www.cisco.com/en/US/products/ps7081/products_ios_protocol_group_home.html
 IPSLA (formerly SAA, formerly RTR): www.cisco.com/go/ipsla
 Network Analysis Module: http://www.cisco.com/go/nam
 Network Based Application Recognition (NBAR): www.cisco.com/go/nbar
 Security Device Manager (SDM): http://www.cisco.com/go/sdm
 Smart Call Home: www.cisco.com/go/smartcall
 Web Services Management Agents (WSMA): http://tinyurl.com/wsma-in-150M
 Cisco Configuration Engine (CCE): www.cisco.com/go/ciscoce

 Feature Navigator: www.cisco.com/go/fn

 MIB Locator: www.cisco.com/go/mibs

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 223
References
Embedded Automation Systems (EASy)
Embedded Automation Systems (EASy)
1. Browse and Download EASy Packages
www.cisco.com/go/easy

2. Make Sure to also download EASy Installer

3. Browse Other Embedded Automations

www.cisco.com/go/ciscobeyond

4. Learn About The Technology Under The Hood

www.cisco.com/go/instrumentation
www.cisco.com/go/eem
www.cisco.com/go/pec

5. Discuss, Ask Questions, Suggest Answers

supportforums.cisco.com

6. Upload your own Examples to CiscoBeyond

www.cisco.com/go/ciscobeyond

7. Engage via [email protected]

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 224
References
Embedded Automation Systems (EASy)
www.cisco.com/go/easy www.cisco.com/go/ciscobeyond

www.cisco.com/go/instrumentation supportforums.cisco.com
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 225
References
Network Automation @ Turn It On
www.cisco.com/go/turniton
Turn It On Program
includes several small Network Automation Examples around Cisco IOS unique features

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 226
 References
TechWise TV Episode 73 (August 2010)

Featuring Cisco
Solutions Experts Robb
Boyd and Jimmy Ray
Recorded at CiscoLive 2010 Purser and Borderless
Targeted at Geeks and TDMs Networks Host Jennifer
Geisler
Small, real, practical, engaging examples
Special Guests:
Joe Clarke
Cult Status of Robb and Jimmy Ray  Tracy Jiang
Matt Lambert
Bruno Klauser
David Lin

See: http://www.cisco.com/en/US/solutions/ns340/ns339/ns638/ns914/html_TWTV/twtv_episode_73.html
Localized Events / Broadcasts are possible …
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 227
References
Network Automation @ CiscoLive 2011

1. Navigate to
http://bit.ly/cSMV3N

2. Search for ‚Automation‘

3. Enjoy !

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 228
Q&A
References – Conferences and Events
 Cisco Networkers and CiscoLive
– Life Events in Europe, Emerging Markets and US
– Virtual Events and Recordings

 Network Automation
November 8-10th 2011, Paris

 NEMA
October 28th 2011, Paris
http://nema.networkembedded.org/

 SASO and IEEE SelfMan

– Contributions since 2006

 EMANICS
 CNSM, AIMS, IM and NOMS
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 229
References
Network Automation Hands-On Lab – PEC

1. Navigate to PEC
http://www.cisco.com/go/pec

2. Click on >Launch

3. Search for ‚%EASy%‘ in the Title

4. Enjoy !

Until Oct 2010

V1: EASy DMI Lab
From Feb 2011
V2: Network Automation made EASy
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 230
 Q&A
Recommended Reading

Published: Jun 9, 2010

www.ciscopress.com/title/1587059452

EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 231
Questions?
Wrap-Up & Close
In Summary
Network Automation …
Based on Embedded Automation Systems (EASy)
Device Manageability Instrumentation (DMI)

… is a Paradigm Change
… offers opportunities far beyond
'just' OPEX savings
… extends beyond the traditional
operational life cycle
… is EASy to adopt now

How will You use

Network Automation?
EASy Intro – bklauser © 2009 Cisco Systems, Inc. All rights reserved. Cisco Public 233
 Thank You
Please Complete Your Evaluations
[email protected]