DETECTION & TSHOOT in METRO ETHERNET

Hai Phuong Doan

High Touch Engineer
24-Sept-2018
Agenda

• Control Plane
• Service Plane
• State of devices
• Q&A

2
Control Plane
HCM Metro Ethernet Physical Topology
BSC/RNC BSC/RNC BSC/RNC

Gi0/9 PE_ASR9006BDI01C/2C PE_ASR9010C3011C/2C PE_ASR9006C3001C/2C

Gi0/9
Gi0/8
Gi0/8
Gi0/8

Gi0/0/0/0

Gi0/9
Gi0/0/0/1
AGG-01 MTC-
Gi0/8 HCMQ1011

6x10GE

Gi0/9

6x10GE

Gi0/8
MTC-
HCMQ0331

Gi0/9

Gi0/8
6x10GE
Gi0/0/0/0
Gi0/9 MTC-
HCMQ0911

Gi0/0/0/1 AGG-02
Gi0/8

Gi0/9
Gi0/9
Gi0/8 PE_SSR8004Q0901E/2E PE_ASR9006Q0901C/2C
Gi0/9 Gi0/8

GE BSC/RNC BSC/RNC
PE-RAN PE-RAN
TenGE ASBR-
4x10GE HCM-
nx10GE ASR9010-P-01
HCMQ1011
4x10GE
nx10GE

4x10GE C30
MTC-HCMQ1011
4x10GE
HCM-
6x10GE 4x10GE ASR9010-P-02
ASBR-
HCMQ1012

6x10GE
MTC-HCMQ0331 HCM-
ASR9010-
1GE/2xGE ASBR- 4x10GE
Q091C
HCMQ0911
4x10GE
6x10GE
4x10GE Q09
MTC-HCMQ0911
4x10GE
1GE/2xGE HCM-
4x10GE ASR9010-
ASBR- Q092C
HCMQ0912
2GGW 2GGW
Multi-AS Unified MPLS Transport Model
OSPF
BGP

RR01

PE_Q10

AGG MC_Q10 ASBR_Q10

MC_Q03
CSG
PE_Q09

AGG
MC_Q09 ASBR_Q01

RR02
BGP IPv4 LU
BGP IPv4
iBGP advertise
2GGW RR-1
eBGP advertise

AGG MC
ASBR-PE PE
MTG MTG
PE RAN PE RAN RR RR
AGG AGG MTG MTG
MC MC ASBR_PE ASBR_PE

MC MC
PE_RAN PE_RAN
AGG AGG AGG
CSG

NHS NHS
NHS

Redistribute
OSPF to BGP PE RAN
Set RR-2
(65355:1000,
65355:1222,
no-export)
BGP VPNv4
BGP VPNv4
2GGW RR-1

NHU

CSG AGG MC
ASBR-PE PE

NHS NHU

PE RAN
RR-2

NHU
Service Plane
2G TDM
2G TDM
CSG-H01196#show cem circuit inter cem 0/0

CEM0/0, ID: 0, Line: UP, Admin: UP, Ckt: ACTIVE

Controller state: up, T1/E1 state: up
Idle Pattern: 0xFF, Idle CAS: 0x8
Dejitter: 20 (In use: 0)
Payload Size: 256
Framing: Unframed
CEM Defects Sel2tr
None

Signalling: No CAS
RTP: No RTP

Ingress Pkts: 1580005 Dropped: 0

Egress Pkts: 1580005 Dropped: 0

CEM Counter Details

Input Errors: 0 Output Errors: 0
Pkts Missing: 0 Pkts Reordered: 0
Misorder Drops: 0 JitterBuf Underrun: 0
Error Sec: 0 Severly Errored Sec: 0
Unavailable Sec: 0 Failure Counts: 0
Pkts Malformed: 0 JitterBuf Overrun: 0
 2G TDM
CSG-H01196#show mpls l2transport vc 100010246

Local intf Local circuit Dest address VC ID Status Last local AC circuit status sent: No fault
------------- -------------------------- --------------- ---------- ---------- Last local PW i/f circ status rcvd: No fault
CE0/0 SATOP E1 0 10.251.158.9 100010246 STANDBY Last local LDP TLV status sent: No fault
CE0/0 SATOP E1 0 10.251.158.10 100010246 UP Last remote LDP TLV status rcvd: No fault
Last remote LDP ADJ status rcvd: No fault
Local interface: CE0/0 up, line protocol up, SATOP E1 0 up MPLS VC labels: local 46, remote 492
Destination address: 10.251.158.10, VC ID: 100010246, VC status: up Group ID: local 59, remote 64
Output interface: Vl4, imposed label stack {27565 492} MTU: local 0, remote 0
Preferred path: not configured Remote interface description: ## STM1 connect to BSG012E
Default path: active Sequencing: receive disabled, send disabled
Next hop: 10.251.2.9 Control Word: On (configured: autosense)
Create time: 04:23:51, last status change time: 04:15:13 Dataplane:
Last label FSM state change time: 03:25:43 SSM segment/switch IDs: 41074/8292 (used), PWID: 14
Signaling protocol: LDP, peer 10.251.158.10:0 up VC statistics:
Targeted Hello: 10.251.128.124(LDP Id) -> 10.251.158.10, LDP is UP transit packet totals: receive 5544088, send 5544089
Graceful restart: configured and enabled transit byte totals: receive 1419286528, send 1419286784
Non stop routing: not configured and not enabled transit packet drops: receive 0, seq error 0, send 0
Status TLV support (local/remote) : enabled/supported
LDP route watch : enabled
Label/status state machine : established, LruRru
Last local dataplane status rcvd: No fault
Last BFD dataplane status rcvd: Not sent
Last BFD peer monitor status rcvd: No fault
Last local AC circuit status rcvd: No fault
2G TDM
2GGW-HCMQ1001#show aps 2GGW-HCMQ1002#show aps
SONET 0/3/1 APS Group 10: working channel 1 (Active) (HA) SONET 0/3/1 APS Group 10: protect channel 0 (Inactive) (HA)
Protect at 10.251.94.162 Working channel 1 at 10.251.158.208 (Enabled) (HA)
PGP timers (from protect): hello time=1; hold time=3 bidirectional, non-revertive
PGP timers (configured): hello time=1; hold time=3 PGP timers (configured): hello time=1; hold time=3
SDH framing hello fail revert time=30
Remote APS configuration: (null) SDH framing; SDH MSP signalling by default
Received K1K2: 0x01 0x00
No Request (working)
Transmitted K1K2: 0x00 0x00
No Request (Null)
Remote APS configuration: (null)
3G / 2G Abis over IP
4G
State of devices
Standard for tranceivers optical fiber
CPU Utilization
• Threshold alarm: 95%

• Cause: SNMP, DDoS, …

• Hang, crash processes

CPU Utilization
Router#sh proc cpu sorted
CPU utilization for five seconds: 94%/83%; one minute: 93%; five minutes: 93%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
308 4632548761560333811 296 4.31% 5.34% 4.73% 0 IP Input
23 707740656 610768913 1158 2.55% 1.44% 1.27% 0 IPC Seat Manager
502 6531629722626881502 0 1.27% 1.17% 1.16% 0 Port manager per
11 231048684 543080787 425 0.79% 0.66% 0.87% 0 ARP Input
130 2564 676 3792 0.31% 0.20% 0.08% 1 SSH Process
376 45725284 72875477 627 0.23% 0.32% 0.32% 0 CEF: IPv4 proces
331 52835481557024368 3 0.15% 0.16% 0.15% 0 Ethernet Msec Ti
121 24044040 39194753 613 0.15% 0.07% 0.07% 0 IPC LC Message H
302 45168824 119816713 376 0.15% 0.23% 0.23% 0 CDP Protocol
563 19969416 31554926 632 0.15% 0.10% 0.08% 0 DHCPD Receive
Memory Utilizaation
• Threshold alarm: 95%

• %HA-HA_WD-4-MEMORY_ALARM : Memory threshold crossed

• Cannot commit configuration

Memory Utilizaation
Router#show processes memory sorted
Processor Pool Total: 219882976 Used: 218388996 Free: 1493980
I/O Pool Total: 134217728 Used: 32551060 Free: 101666668

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 198185764 76290240 117510748 0 0 *Init*
365 0 105839944 0 105847240 0 0 qos_mon_periodic
392 0 1032442840 366419728 6235528 421888 0 OSPF-1 Router
382 0 424563084 3173912 3274312 0 0 LCON Main
0 0 302772828 269888696 2960652 13489163 0 *Dead*
Status of components
• Command: Show platform

• Notify in log buffer

Status of components
RP/0/RSP0/CPU0:Router#admin show platform
Fri Apr 20 03:20:34.981 UTC
Node Type State Config State
-----------------------------------------------------------------------------
0/RSP1/CPU0 A9K-RSP440-SE(Active) IOS XR RUN PWR,NSHUT,MON
0/FT0/SP ASR-9006-FAN READY
0/FT1/SP ASR-9006-FAN READY
0/0/CPU0 A9K-2T20GE-L IN-RESET PWR,NSHUT,MON
0/2/CPU0 A9K-MOD80-SE IOS XR RUN PWR,NSHUT,MON
0/2/0 A9K-MPA-4X10GE OK PWR,NSHUT,MON
0/2/1 A9K-MPA-20X1GE OK PWR,NSHUT,MON
0/PS0/M0/SP A9K-3KW-AC READY PWR,NSHUT,MON
Statistics of IPv4 prefix
• ASR901: 20k

• ASR903(RSP2): 20k

• ASR9K w/ Typhoon LC: 4M

Statistics of IPv4 prefix
RP/0/RSP0/CPU0:ASR9006-A# show route summ
Sun Sep 30 02:50:46.466 UTC
Route Source Routes Backup Deleted Memory(bytes)
connected 17 1062 0 151060
local 1079 0 0 151060
local LSPV 1 0 0 140
local SMIAP 1 0 0 140
local VRRP 0 0 0 0
static 13 0 0 1820
ospf 500 0 0 0 0
ospf 20 0 0 0 0
ospf telstra 0 0 0 0
rip 0 0 0 0
isis 88 0 0 0 0
isis 100 0 0 0 0
isis COLT 0 0 0 0
isis 1 2 2 0 560
isis polar 0 0 0 0
bgp 8151 0 0 0 0
dagr 0 0 0 0
application 0 0 0 0
Total 1113 1064 0 304780
Statistics of VPNv4 prefix
• ASR901: 4k

• ASR903 (RSP2): 20k

• ASR9k w/ Typhoon LC: 2M

Statistics of VPNv4 prefix
RP/0/RSP0/CPU0:ASR9006-F#show bgp vpnv4 unicast summary
<omitted>
Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 26 26 26 26 26 0

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd

166.105.229.235 1 45090 0 0 0 0 0 00:00:00 Idle
172.16.1.5 1 65001 6227 6350 26 0 0 1d07h 2
172.16.18.126 1 65001 1414 1311 0 0 0 2d21h Idle
172.16.18.254 1 65001 1446 1314 0 0 0 2d21h Idle
Enviroment
• Command: show environment

• Temperature, voltage sensors and status of led alarm

Enviroment
RP/0/RSP0/CPU0:ASR9010-D#show
environment
Sun Mar 29 08:34:37.146 UTC LED Information
---------------------------------------------
Temperature Information
--------------------------------------------- R/S/I Modules LED Status
0/RSP0/*
R/S/I Modules Sensor (deg C) host Critical-Alarm Off
0/2/* host Major-Alarm Off
ep0 Inlet0 31.5 host Minor-Alarm Off
ep0 Hotspot0 37.6 host ACO Off
host Fail Off
0/RSP1/*
ep1 Inlet0 28.4 host Critical-Alarm Off
ep1 Hotspot0 33.1 host Major-Alarm On
host Minor-Alarm Off
<omitted> host ACO Off
host Fail Off

BMC Information
---------------------------------------------
Label Utilization
• Command:
• show mpls label table summary
• show platform hardware cef hw-resource
• ASR901:4K

• ASR903 (RSP2): 28.5k

• ASSR9k w/ Typhoon LC: 1M

Label Utilization
RP/0/RSP0/CPU0:ASR9010-D#show mpls label ASR901#show platform hardware cef hw-resource
table summary
No of IPv6 Route Entries : 0
Sun Mar 29 08:43:01.646 UTC
No of IPv6 Host Entries : 0
Application Count No of IPv4 Defip Entries : 1658
No of Outgoing Labels : 1656
---------------------------- -------
No of In-Labels(Label Entries): 19
LSD(A) 4 No of Label Replace(operations): 0
No of l3intf Replace(operations): 0
LDP(A) 6
BFD(A) 2
BFD(S) 1
L2VPN(A) 6
LDP(S) 6
BGP-VPNv4(A):bgp-default 1
---------------------------- -------
TOTAL 19
Bridge-domain Utilization
• Command:
• show bridge-domain stats
• show l2vpn bridge-domain summary
• ASR901: 4k

• ASR903 (RSP2): 4k

• ASR9k w/ Typhoon LC: 8k

Bridge-domain Utilization
RP/0/RSP0/CPU0:ASR9010-D#show l2vpn bridge-domain summary
Sun Mar 29 08:50:43.705 UTC
Number of groups: 1, bridge-domains: 2, Up: 2, Shutdown: 0, Partially-
programmed: 0
Default: 2, pbb-edge: 0, pbb-core: 0
Number of ACs: 4 Up: 0, Down: 4, Partially-programmed: 0
Number of PWs: 0 Up: 0, Down: 0, Standby: 0, Partially-programmed: 0
Number of P2MP PWs: 0, Up: 0, Down: 0, other-state: 0
Number of VNIs: 0, Up: 0, Down: 0, Unresolved: 0
BFD session Utilization
• Command:
• show bfd session
• show bfd summary
• ASR901: 10

• ASR902 (RSP2): 1023

• ASR9k: depending on echo bfd packet’s duration

BFD session Utilization
RP/0/RSP0/CPU0:ASR9010-D#show bfd summary
Sun Mar 29 08:59:27.310 UTC
Node All PPS usage MP PPS usage Session number
% Used Max % Used Max Total MP Max
---------- --------------- --------------- ------------------
0/2/CPU0 0 1 27000 0 1 9600 1 1 8000
0/3/CPU0 0 10 28000 0 10 9600 1 1 8000
0/4/CPU0 0 0 28000 0 0 9600 0 0 8000
0/5/CPU0 0 0 28000 0 0 9600 0 0 8000
0/7/CPU0 0 66 28000 0 0 9600 1 0 8000
Q&A