Monitor & Support

VMware SD-WAN

Confidential │ ©2020 VMware, Inc.

Agenda ▪ SD-WAN Enterprise
▪ VCO Portal Monitoring
▪ Remote Diagnostics
▪ External Tools
▪ SNMP
▪ Netflow
▪ Syslog
▪ Webhooks
▪ vRNI Integration
▪ Edge Network Intelligence
▪ API

▪ SD-WAN Infrastructure / Operations

▪ Gateways
▪ Orchestrator
Confidential │ ©2020 VMware, Inc. 2
VMware SD-WAN Cloud-Delivered SD-WAN
VMware SD-WAN’s network service consists of 3 key components

VMware SD-WAN

1 Orchestrator VMware SD-WAN

Orchestrator
SaaS
2
Public
Internet
Branch site
with VMware Enterprise data center
2 Cloud Gateway SD-WAN Edges
Dynamic Multipath
via VMware SD-WAN
Optimization
Gateway

3
Private
Network/MPLs

3 Edge Enterprise data center

with on-premises VMware
SD-WAN Edge

Confidential │ ©2020 VMware, Inc. 3

VMware SD-WAN Cloud-Delivered SD-WAN
VMware SD-WAN’s network service consists of 3 key components
API. SNMP ,
Remote VCO Portal
Syslog and
VMware SD-WAN Diagnostics Monitoring
vRNI

1 Orchestrator VMware SD-WAN

Orchestrator
SaaS
2
Public
Internet
Branch site
with VMware Enterprise data center
2 Cloud Gateway SD-WAN Edges
Dynamic Multipath
via VMware SD-WAN
Optimization
Gateway

3
Private
Network/MPLs

3 Edge Enterprise data center

Netflow and with on-premises VMware
ENI SD-WAN Edge

Confidential │ ©2020 VMware, Inc. 4

Orchestrator Monitoring

Confidential │ ©2020 VMware, Inc. 5

SD-WAN Orchestrator Monitoring
From Orchestrator Portal

1 2 3

Statistics Events Alerts

Confidential │ ©2020 VMware, Inc. 6

Edge & Link Status
▪ Exportable tables to CSV
▪ Context aware search capability

▪ Configurable columns

Confidential │ ©2020 VMware, Inc. 7

Edge Monitoring
Overview tab

Confidential │ ©2020 VMware, Inc. 8

QoE Scores
Determine link quality in one glance

ISP 1

ISP 2

Hover over time slice for more detail

Confidential │ ©2020 VMware, Inc. 9

Path Visibility ▪ site-to-site metrics (jitter, latency, packet loss)
See how the paths are doing ▪ Data usage statistics
▪ Quickly diagnose user experience impact
▪ Historical network insights and real-time
monitoring
▪ Generate a printed report

Confidential │ ©2020 VMware, Inc. 10

VMware SD-WAN Network Monitoring
Measure and Report per Transport Link

▪ Detailed metrics of transport links across time

▪ Metric readings available in real-time or historical stats, to diagnose an existing link issue or perform capacity
planning analysis
▪ Show TCP and UDP details in ”Live Monitoring” to understand distinct L4 protocols data usage

Confidential │ ©2020 VMware, Inc. 11

VMware SD-WAN Network Monitoring
Identify SD-WAN top applications

▪ L7/Application-level visibility/reporting
▪ Obtain top applications based on data usage
▪ Click an application to gather top users, top destinations and which link
▪ Historical data usage trends

Confidential │ ©2020 VMware, Inc. 12

VMware SD-WAN Network Monitoring
SD-WAN Top Talkers

▪ Obtain per device application usage trends

▪ Devices identified by IP address, MAC address or hostname (manually assign a hostname if DNS
resolution not available)
▪ Each client device identified by a color on the graph
▪ Identify anomalous network usage and pin point the culprit source

Confidential │ ©2020 VMware, Inc. 13

Visibility behind Layer3 Switch
• Provide visibility options: keyed by MAC or IP address
• Statistics are collected based accordingly
• Available for control in device settings (profile & edge)

▪ Select “Visibility by MAC Address” ▪ Select “Visibility by IP”

▪ Clients are behind L2 SW ▪ Clients are behind Layer3 switch
▪ Client MAC, IP and Hostname(if applicable) will be shown ▪ SW MAC, Client IP and Hostname(if applicable) will be shown
▪ OS can be fingerprinted if Edge is DHCP server ▪ OS can be fingerprinted if DHCP Relay enabled on Switch

Layer3 Switch MAC

Confidential │ ©2020 VMware, Inc. 14

Privacy Controls
Limiting upstream visibility

▪ Available at Administration | System Settings

▪ Applicable to both the Enterprise as well as the MSP

▪ Tiered delegation

▪ If enabled, host details will be aggregated at the OS level

▪ No hostnames will be displayed, IP addresses and MAC addresses

▪ Operators by default will not be able to create Enterprise user accounts

Confidential │ ©2020 VMware, Inc. 15

VMware SD-WAN Network Visibility
Analyze SD-WAN Top Destinations
▪ Visualize traffic to destination server(s) for a given
application
▪ View destinations stats by domain, FQDN or IP
address
▪ Quick insights on destination data usage based
on color coded measurements
▪ Click a destination to see served application and
top users

Confidential │ ©2020 VMware, Inc. 16

VMware SD-WAN Network Visibility
Business Policy

Confidential │ ©2020 VMware, Inc. 17

VMware SD-WAN Network Visibility
System Level Health

Confidential │ ©2020 VMware, Inc. 18

Overlay Flow Control
Central Routing Monitoring

• Single pane enterprise view for routing

• Both underlay and overlay routes
• Quickly identify incorrectly learned routes
• Identify, click and fix

Confidential │ ©2020 VMware, Inc. 19

Orchestrator Reporting

Confidential │ ©2020 VMware, Inc. 20

Enterprise Reporting
Overview
▪ Enterprise Reporting is available on the new Orchestrator UI

▪ Users with following roles can run and schedule a report

▪ Operator: SuperUser, Standard Admin, Business Specialist

▪ Partner: SuperUser, Standard Admin, Business Specialist

▪ Enterprise: SuperUser, Standard Admin

▪ Reports created for an enterprise can be viewed by operator, partner managing the enterprise
and enterprise admin

▪ To view a report, access to the Orchestrator is required

Confidential │ ©2020 VMware, Inc. 21

Enterprise Reporting Configuration Workflow 3. Customize report

1. One Click Report or Customized Report

4. Select Edges

2. One time or Scheduled Reports

5. Send via Email

Confidential │ ©2020 VMware, Inc. 22

Enterprise Reports
Use Case: Top Applications and Top sites

Confidential │ ©2020 VMware, Inc. 23

Orchestrator Portal Monitoring

1 2 3

Statistics Events Alerts

Confidential │ ©2020 VMware, Inc. 24

SD-WAN Orchestrator Events

Events track all changes of SD-WAN state whether they are network events or configuration

events, e.g. link up or down, edge failure, edge HA failover, etc…

These events are stored in a log that can be accessed by anyone with the correct authorization

The Events feature is useful for obtaining the following information:

• Audit trail of user activity

• Historical record of activity at a given site

• Record of outages and significant network events

• Analysis of degraded ISP performance

Confidential │ ©2020 VMware, Inc. 25

Event Monitoring

▪ Filtering available on event types and content via context aware search
▪ Events include severity as well as the user originating the event

Confidential │ ©2020 VMware, Inc. 26

Orchestrator Portal Monitoring

1 2 3

Statistics Events Alerts

Confidential │ ©2020 VMware, Inc. 27

Common Alert Types

E
Edge Link VPN HA VNF

Confidential │ ©2020 VMware, Inc. 28

Complete List of Edge Alerts

E E

Confidential │ ©2020 VMware, Inc. 29

 Heartbeat Mechanism

Highlights
Legend
1 Heartbeats are sent via overlay 4 VCO Tracks state and sends email • VCE=Edge
alert
2 Heartbeats are sent every 30s • VCG=Gateway
5 Heartbeats can contain LinkStats
3 VCG forwards heartbeats via Internet to VCO • VCO=Orchestrator
Information

HTTPS in Overlay HTTPS Email Alert

SD-WAN Internet
Public Overlay
Email Server

VCE Primary VCG VCO

Confidential │ ©2020 VMware, Inc. 30

Edge Status Transition Alerts

E E E
Edge (Down/Up) state Down Alert—triggered Up Alert—triggered
is tracked by VCO and from Orchestrator if >120s upon receipt of
is determined by VCO since last successful first successful heartbeat
with the presence of heartbeats heartbeat
from the Edge

Confidential │ ©2020 VMware, Inc. 31

Edge Status Transition Alerts

Management Plane Data plane can be fully intact Common alert for single WAN
driven alert while this alert is triggered link site

Confidential │ ©2020 VMware, Inc. 32

What to check during an Edge Down Alert
Heartbeat every 30s

HTTPS in Overlay HTTPS

SD-WAN Internet
Public Overlay

VCE Primary VCG VCO

Does a hub site have paths to Edge? Use list paths Has Edge lost underlay transport to Internet?

Issue with Primary VCG? Has Edge lost power?

Peering issue?

Confidential │ ©2020 VMware, Inc. 33

Link Alerts

Link—Up Link—Down

Confidential │ ©2020 VMware, Inc. 34

Link Status Transition Alerts

Link (Down/Up) state is tracked by VCO

At least one working link to send
and is determined upon link info included
heartbeat is required
in the heartbeat

Down Alert—Two consecutive heartbeats

where link information indicates Common alert—especially for broadband links
a link is “down”

Up Alert—One heartbeat where link

info indicates a link is “up”

Confidential │ ©2020 VMware, Inc. 35

VPN Alert Type

VPN Tunnel Down

Confidential │ ©2020 VMware, Inc. 36

VPN Tunnel Down Alert

Only for routed-based VPN No VPN Tunnel Up alert at this time

to non-VeloCloud site

Tunnel Down is included in next gateway

heartbeat if all paths on the routed-based Very rare alert
VPN are down to non-VeloCloud site

Up to 4 paths possible depending

on configuration

Confidential │ ©2020 VMware, Inc. 37

VPN Tunnel Down Alert
With redundant tunnels in place

SD-WAN
Public
Overlay
VCE
VCG Non-VC Site

SD-WAN
Public
Overlay
VCE
VCG
Non-VC Site

SD-WAN
Public
Overlay
VCE
VCG
Non-VC Site

Confidential │ ©2020 VMware, Inc. 38

What to check during a VPN Tunnel Down Alert

SD-WAN
Public
Overlay
VCE Click to add text
VCG Non-VC Site

Pull gateway diagnostic

bundle
Check on Network Services
Monitoring

Check with non-VC site owner

Confidential │ ©2020 VMware, Inc. 39

HA Alert Type

Edge HA Failover

Confidential │ ©2020 VMware, Inc. 40

What to check during a HA Failover Alert

Primary Edge Interface state change?

HA Link Primary Edge service fail?

Primary Edge lost power?
Pull diagnostic bundle and check edged.log
VCE VCE

Confidential │ ©2020 VMware, Inc. 42

VNF Alert Type

Edge VNF
Edge VNF Insertion
Virtual Machine Deployment

Confidential │ ©2020 VMware, Inc. 43

Monitor VNF for an Edge
Monitor the status of VNFs and the VMs

Confidential │ ©2020 VMware, Inc. 44

Configure VNF Alerts
From Orchestrator configure to receive alerts and notifications related to the VNF events.
• Edge VNF Virtual Machine Deployment – Receive an alert when there is a change in the Edge VNF virtual machine
deployment state.
• Edge VNF Insertion – Receive an alert when there is a change in the Edge VNF deployment state.
• Edge VNF Image Download Event – Receive an alert when there is a change in the Edge VNF image download state.

Confidential │ ©2020 VMware, Inc. 45

Alert Configurations
As an operator

▪ Can be enabled/disabled (enterprise) account-wide

▪ Pre-notifications provide delay of notification to the enterprise admin
▪ Operators will received the alert first, providing reaction and diagnostic time
▪ After a configurable delay, enterprise admin will be notified
▪ Enterprise admin will only be notified if the alert hasn’t been cleared or resolved by the operator

Confidential │ ©2020 VMware, Inc. 46

Remote Diagnostics
VMware SD-WAN Orchestrator Portal

Confidential │ ©2020 VMware, Inc. 47

Available Troubleshooting tools
VMware SD-WAN Orchestrator Portal

1 2 3 4

Remote Remote Edge Remote Packet

Actions Diagnostics Diagnostic Capture
Bundles

Confidential │ ©2020 VMware, Inc. 48

Remote Actions

Restart Service: Reboot: Full System Reboot Service Restart Is Sufficient

Restarts VeloCloud Application (10s) (3-10 Minutes) 95% of the Time

Confidential │ ©2020 VMware, Inc. 49

Remote Diagnostics Available Tools
Multiple tests can be executed simultaneously

Confidential │ ©2020 VMware, Inc. 50

Remote Diagnostics Example
Available Tests

Multiple tests can be executed

simultaneously

Confidential │ ©2020 VMware, Inc. 51

Remote Diagnostics Example
List Active Flows

▪ Valuable diagnostic tool

▪ Will identify:
▪ Detected application
▪ Associated policy
▪ Associated route
▪ Can filter in src/dst IP

Confidential │ ©2020 VMware, Inc. 52

Remote Diagnostics Example
List Active Paths

▪ Separate tests for paths to:

▪ Gateways
▪ Hubs
▪ Includes
▪ Path state
▪ Bandwidth (up/down)
▪ Latency (tx/rx)
▪ Packetloss (tx/rx)
▪ Jitter (tx/rx)
▪ Volume (tx/rx)
▪ QOE scores represent path to
closest gateway

Confidential │ ©2020 VMware, Inc. 53

 Diagnostic Bundle Request

Edge Diagnostic Bundle Gateway Diagnostic Bundle

Operator & Partner Accounts, not for enterprise admins Operators Accounts only

Confidential │ ©2020 VMware, Inc. 54

Diagnostic Bundle Request
Orchestrator Diagnostic Bundle – Operator Only

Confidential │ ©2020 VMware, Inc. 55

Anatomy of the Edge Diagnostic Bundle

Debug Command Output

▪ debug.py output
▪ System command output (ifconfig, dmesg,
conntrack)

Ephemeral – Overwritten when rebooted

/var/log all system logs

▪ Persistent Velocloud logs that are in /var/log

▪ Core files under velocloud/core (if there are
any)
▪ When edge is configured as HA, the standby devices
diags will be located in:
velocloud/diagtmp/standby.zip

Confidential │ ©2020 VMware, Inc. 56

Most useful files

ifconfig-a L3 interface stats

lsusblsusb-tlsusb-v USB Device stats

optvcbindebugpy--flow_stats Flow stats

optvcbindebugpy--handoffq Handoff queue (looking for drops)

optvcbindebugpy--remote_routesall Local SD-WAN routing table

optvcbindebugpy-v--biz_pol_dump Dump of active business policies

optvcbindebugpy-v--link_stats Dump of link stats

optvcbindebugpy-v--path_stats Dump of path stats

optvcbinethtool_dumpsh Physical interface stats

top-H-n1-b.out CPU stats

Confidential │ ©2020 VMware, Inc. 57

Log Directory
/velocloud/log (Example output)

Capturing all Velocloud logs

• GENERAL places to check Heath
• edged
• mgd
• activation
• messages
• vc_procmon
• SPECIFIC services / features:
• mgdha
• bgpd
• ospfd
• pimd
• vnfd
• Etc. (other files will exist depending on
features enabled)

Confidential │ ©2020 VMware, Inc. 58

Available Troubleshooting tools
From Orchestrator Portal

1 2 3 4

Remote Remote Edge Remote Packet

Actions Diagnostics Diagnostic Capture
Bundles

Confidential │ ©2020 VMware, Inc. 59

Packet Capture Capabilities

5 to 120s WAN, LAN, USB interfaces Stored for 60 days

Confidential │ ©2020 VMware, Inc. 60

60
Packet Captures

▪ Automatic age out (and removal) of old packet captures

▪ Can be downloaded on admin host system

Confidential │ ©2020 VMware, Inc. 61

External Tools Monitoring
SNMP & Netflow & Syslog & Webhook

Confidential │ ©2020 VMware, Inc. 62

External Monitoring Tools
From edge to external tool

SNMP Netflow Syslog Webhook

Confidential │ ©2020 VMware, Inc. 63

 SNMP Monitoring

SNMP can be enabled by selecting the desired SNMP

Version. Enabled on Edge or Profile Level on Monitor CPU, Memory, Interface Error Counter,
Orchestrator. Running Process
Following are the MIB groups that you can export

▪ SNMP MIB-2 System

▪ SNMP MIB-2 Interfaces For SNMP v3, you can set Port, Name, Password,
▪ HOST-RESOURCES-MIB, from RFC 1514 and Privacy (DES or AES) settings. For SNMP v2c,
you can set Port, Community, and Allowed IP settings
▪ Polled by querying management IP address of Edge
▪ NMS must have IP access to Edge, Gateway or Orchestrator

Available on Edge, Gateway & Orchestrator

Confidential │ ©2020 VMware, Inc. 64

Enabling SNMP
Available at the profile level

▪ SNMP managers can query Edges directly

▪ Public Collector : through a public interface (open up the firewall ; whitelist manager address)
▪ DC Hosted Collector : through the VPN (attached to the Edge management IP address)

▪ Both v2c and v3 versions are supported and can be enabled simultaneously
▪ v3 allows for privacy (additional DES or AES encryption)
▪ v3 is the best choice for access over public internet
▪ v2 should only be transported over the VPN

Confidential │ ©2020 VMware, Inc. 65

External Monitoring Tools
From edge to external tool

1 2 3 4

SNMP Netflow Syslog Webhook

Confidential │ ©2020 VMware, Inc. 66

VMware SD-WAN Network Visibility
NetFlow IPFIX NetFlow Export Stream

SD-WAN Edge
(NetFlow Exporter)
Orchestrator

Gateway
Internet

SD-WAN Edge
(NetFlow Exporter)
VMware SD-WAN Hub Edge
MPLS (NetFlow Exporter) NetFlow Collectors
SD-WAN Edge
(NetFlow Exporter)

• NetFlow format is based on standard IPFIX templates

• Only VMware SD-WAN Edges will support NetFlow export
• An external NetFlow collector is required to ingest NetFlow data and represent network stats
• NetFlow is used for finding bandwidth hogs, hunting down network threats and isolating application slowness issues
• SD-WAN NetFlow data will include flow application recognition, data stream statistics, network path ID and SD-WAN remediation applied

Confidential │ ©2020 VMware, Inc. 67

Enabling NetFlow Export
Available at both profile level & edge level

▪ NetFlow stats are exported from each Edge

▪ Supporting NetFlow v10

▪ Sending to collectors
▪ Sourced from the management IP address
▪ To configured destinations on Internet or on VPN
▪ Recommending collectors to be placed inside the corporate VPN

▪ Subset of data that flow stats are collecting on the Orchestrator

▪ Policy can only be applied to the global segment
▪ Flows from all segments will be exported

Confidential │ ©2020 VMware, Inc. 68

Netflow Collector Settings under Network Services
Netflow Config Step 1

• Configuring a NetFlow Collector has moved to ‘Network Services’

• You can configure multiple Netflow collectors and assign up to 8 collectors per Edge
• Netflow collector destination is segment aware
• Support filter rules to apply to a profile or edge
• Filter rules allow for specific network telemetry to be streamed to netflow collector

Confidential │ ©2020 VMware, Inc. 69

NetFlow Collector Assignment to Profile or Edge
NetFlow Config Step 2

Confidential │ ©2020 VMware, Inc. 70

External Monitoring Tools
From edge to external tool

1 2 3 4

SNMP Netflow Syslog Webhook

Confidential │ ©2020 VMware, Inc. 71

Syslog

Orchestrator Edge

• 2 syslog servers per segment can be defined

• 10 syslog servers per edge across all segments
• Ensure reachability to syslog server via underlay or overlay. Enable cloud VPN (branch to
branch VPN settings) if required for overlay reachability

Confidential │ ©2020 VMware, Inc. 72

Positioning Syslog Server
WAN Underlay Collector

MPLS
Collector A

LAN Side Collector Internet

WAN Overlay Collector

Collector C
Collector B

• VCO bound events originating from VCE will be sent to one or more syslog collectors
• Syslog collector should reside in the enterprise network
• rsyslogd will be the underlying utility to forward logs to the syslog collector.
• Supported transport protocols : TCP and UDP

Confidential │ ©2020 VMware, Inc. 73

Syslog Monitoring
Syslog for Orchestrator

Syslog on the Orchestrator are available for three Most verbose output compared to API & Event log in
services VCO

Property Service
Uses standard syslog RFC
log.syslog.backend Backend
log.syslog.portal Portal
log.syslog.upload Upload Only available to partners with on-premise
deployment as vco operator access is required

Confidential │ ©2020 VMware, Inc. 74

Syslog Export
Syslog configuration for Orchestrator
▪ Capability to export syslog events
▪ to an external collector (Enable multiple Syslog
collectors only applicable from Release 3.3.1 onwards)

▪ Is done VCO wide

▪ no segregation by customer

▪ Has to be configured as a system property

▪ no UI available
▪ log.syslog.<service>

▪ Require access to the VCO being configured as an

account with superuser operator privileges.
▪ On-prem vs. Hosted

▪ Integrate with Splunk, ELK stack

Confidential │ ©2020 VMware, Inc. 75

Configuration Workflow

• Configure Syslog Collector

• Profile->Device->Syslog Settings

1. • Set the “Role” field to “Firewall Event” or “Edge and

Firewall Event”

• Configure Firewall Rules

• Check the “Log” checkbox in the FW rule
2.

Confidential │ ©2020 VMware, Inc. 76

 UI Workflow – Configure Syslog server for FW logs
Configure -> Profiles -> Device -> Syslog Settings TLS

1 3 2

1. Protocol - Add another option: TLS. When TLS is selected

• Port: Set default port to 6514
• Certificate: Add a field for Client certificate. Provide a button for uploading a client certificate
that can be used to authenticate the edge to the Syslog server and establish the TLS session

2. Roles – Add a new role called “Firewall Log”

3. Move Facility Code to the specific line item as there can be different facility codes for
event logs and firewall logs.

Confidential │ ©2020 VMware, Inc. 77

Edge Events
Edge events & Firewall logging

• EDGE EVENT
• FIREWALL EVENT
• EDGE AND FIREWALL EVENT

Confidential │ ©2020 VMware, Inc. 78

Edge Firewall Logging
Key Highlights
▪ Every firewall policy action (allow or drop) has log option
▪ Stateful Firewall policy with allow and log option creates two
type logs for every session (session-open and session-close)
▪ A drop policy creates session-drop log
▪ TCP / UDP encapsulated standard syslog format firewall
logging to the external log server or SIEM
▪ Improves new connection per seconds on Edges

Available Fields in Firewall Logs

Session-Open Session-Close Session-Deny

▪ Date and time ▪ All fields in Session- ▪ Date and time
Open message and
▪ SRC and DST IP ▪ SRC and DST IP
▪ Duration
▪ SRC and DST Port ▪ SRC and DST Port
▪ Total bytes
▪ IP Proto ▪ IP Proto
▪ Reason
▪ Firewall Policy Name ▪ Firewall Policy Name
▪ Incoming Interface ▪ Incoming Interface
▪ L7 Application ▪ L7 Application

By default, the Syslog Forwarding button is available on the Firewall page of the Profile or Edge configuration and is disabled.

Confidential │ ©2020 VMware, Inc.

External Monitoring Tools
From edge to external tool

1 2 3 4

SNMP Netflow Syslog Webhook

Confidential │ ©2020 VMware, Inc. 80

Webhooks for Alerts
What are webhooks?
• Webhooks are data and executable commands sent from one app to another over HTTP.
• They're called webhooks since they're software hooks—or functions that run when something
happens—that work over the web.
– Reference: https://zapier.com/blog/what-are-webhooks/

What is the Feature?

• Currently we can notify Alerts via Email, SMS, and SNMP.
• We are adding webhooks as an additional notification method for Alerts.

How can I use it?

• Users will now be able to automate actions in response to VCO-triggered alerts. Ex. Create a
ticket for “Edge Down”

Security
• HTTPS is used to post the message. And HMAC is computed using the secret key entered and
sent to the recipient for message authentication.

Confidential │ ©2020 VMware, Inc. 81

Webhooks for Alerts
Working Flow

Pre-applied
Webhook URL

VCO POST Defined JSON Webhook provider

requests payload response
Yes

Pre-defined as Secret
Edge Down (Optional)
an “Alert”

No
No Action

Confidential │ ©2020 VMware, Inc. 82

How to Deploy

1.[Customer View] Configuration -> Alerts & Notification

Select the “Alerts” you want to trigger, only checked Alert can trigger Webhook request

Confidential │ ©2020 VMware, Inc. 83

How to Deploy

2.[Customer View] Configuration -> Alerts & Notification Placeholder

"{{alertTime}}"
Configure Webhooks at bottom in this page "{{alertType}}"
• Input Webhook URL "{{customer}}"
"{{entityAffected}}"
• Struct JSON payload "{{lastContact}}"
"{{message}}"
"{{vco}}"

Struct JSON payload here

Put your Webhook URL here

Struct JSON payload here

Put your Webhook URL here

Confidential │ ©2020 VMware, Inc. 84

Slack Integration
Slack Webhook https://api.slack.com/messaging/webhooks
1. Apply “Incoming Webhook”, you will get 2. Copy URL to VCO configuration page and
webhook URL and OAuth token struct JSON payload

JSON payload:
{
"text": "{{alertTime}} {{alertType}}
[Enterprise]:{{customer}}
[Edge]:{{entityAffected}}\n{{message}}\nVCO:
https://{{vco}}\nlast contact: {{lastContact}}"
}

3. Reboot edge to trigger Alert, Slack Webhook bot will print pre-defined information (Compared VCO Monitor Alerts on the right)

Confidential │ ©2020 VMware, Inc. 85

Edge Network Intelligence

Confidential │ ©2020 VMware, Inc. 86

Introducing VMware Edge Network Intelligence
A vendor agnostic AIOps solution
VMware SD-WAN Multi-
Digital
Cloud
Workspace

IaaS
Mobile

SaaS
Edge Network Intelligence
Branch
End user and IoT device experience for a distributed and secure enterprise

Private Cloud
Campus
WLAN LAN Network Secure Security Application
SD-WAN
Services Access Services

Edge Cloud
Home

Confidential │ ©2020 VMware, Inc. 87

 Extend SD-WAN Visibility into the Branch, Campus
Analytics, visibility, troubleshooting, and remediation

Home Office

Cable Edge Network UCaaS / SaaS / IaaS via Non-Business

Modem/Wi-Fi VMware SD-WAN Internet Traffic
Intelligence Analytics VMware SD-WAN Gateway
Router Orchestrator
Engine
Analytics
Edge
Wireless clients
VMware SD-WAN Edge
Internet /
Branch or Campus Site Web

Wired
Closet Switches
Fabric VMware SD-
WAN
Gateway

Wireless clients
Access Wired clients
Points

VMware Dynamic Multi-Path Optimization

SD-WAN VMware Enterprise Data Center
Edge SD-WAN
IoT Edge Hub

Confidential │ ©2020 VMware, Inc. 88

Fault Isolation

Issue footprint and

categorization allows System auto-learns app
prioritization and faster performance baselines &
root cause analysis does anomaly detection

Potential root causes and

suggested next steps are
provided

Confidential │ ©2020 VMware, Inc. 89

API

Confidential │ ©2020 VMware, Inc. 90

Orchestrator Portal = web frontend for the API

HTML5 ReST API

Portal Users Orchestrator

Browsers, Curl

▪ API First Design Principle

▪ No function is available in the portal that isn’t available in the API
▪ API’s are a superset of the functionality
▪ Contains additional calls that ease integration into existing applications

▪ Browsers are instructed to run API calls to fetch data

▪ Data driven solution

▪ Inspection of the webpages will reveal the ReST calls that are made

Confidential │ ©2020 VMware, Inc. 91

Orchestrator ReST API Overview
Interactions with other components

HTML5
ReST API

Portal Users
Browsers, Curl SD-WAN Orchestrator

ReST API

API Users

▪ JSON-RPC API over HTTPS transport between the client and the Orchestrator
▪ Client can be web browser, curl
▪ Use cases: monitoring, provisioning, configuration

Confidential │ ©2020 VMware, Inc. 92

What to use the API for?
Example use cases

▪ Integration into existing OSS/BSS system

▪ E.g. ECOMP is used by AT&T to Orchestrate the SD-WAN service
▪ Other vendors include Amdocs, NetCracker, OpenDayLight

▪ Custom user portals

▪ E.g. Earthlink/Windstream incorporates QoE events into an end-customer portal

▪ Custom reporting
▪ E.g. Generate monthly edge activation report to feed into billing system
▪ Enterprise customers can generate reports not directly available through the web UI

▪ High volume tasks

▪ E.g. add a standard firewall rule to all enterprise customers

▪ Small feature enhancements

▪ E.g. Time based policy changes. Higher priority for backups at night.

Confidential │ ©2020 VMware, Inc. 93

Chrome Developer Tools
Showing ReST responses
Useful for determining
Press ctrl+shift+I or Navigate to Use the Console to do
which APIs are used for
More Tools → Developer Tools a quick test of API call
a particular VCO page

Confidential │ ©2020 VMware, Inc. 94

SD-WAN Orchestrator API

API reference, Documentation, scripts can be found on

https://code.vmware.com/apis/1037/velocloud-sdwan-vco-api

Confidential │ ©2020 VMware, Inc. 95

Real-world Scenarios
Common Troubleshooting cases

Confidential │ ©2020 VMware, Inc. 96

Identify common POC issues and Workflow

Overlay Application
Activation Routing Performance
Tunnels

Troubleshooting Troubleshooting
Scenario Symptoms
Tools Steps

Confidential │ ©2020 VMware, Inc. 98

Symptoms:
Internet Status Unavailable Scenario 1
Activation
DNS Not Reachable
Unsuccessful
VCO Unreachable

Confidential │ ©2020 VMware, Inc. 99

Activation Unsuccessful: Symptom #1
Internet Status: Unavailable

Confidential │ ©2020 VMware, Inc. 100

 Activation Unsuccessful: Symptom #1
Getting the MAC Address for Public WAN Interface from Edge UI

Confidential │ ©2020 VMware, Inc. 101

Activation Unsuccessful: Symptom #1
Get MAC address of interface from Edge UI and Check ARP entry on Upstream switch
1 2

MAC Address match up for Edge Device Public WAN

interface with Upstream Switch 00:BA:BE:78:81:B7

Confidential │ ©2020 VMware, Inc. 102

 Activation Unsuccessful: Symptom #1
Screenshot from Diagnostic bundle from an un-activated Edge UI
3

ARP entry from Diagnostic Bundle arp-a.out.txt file

Confidential │ ©2020 VMware, Inc. 103
Activation Unsuccessful: Symptom #1
Screenshot 3: Decoder for Activation Link to get IP address

Verify assigned IP address/Subnet Mask/Gateway

Use a decoder to get IP Information Configure | Edge | Devices
Confidential │ ©2020 VMware, Inc. 104
Edge Activation Email - Sample
Two form factor authentication for Edge Onboarding

Confidential │ ©2020 VMware, Inc. 105

Symptoms:
Scenario 1
#1 Internet Status Unavailable
Activation
#2 DNS Not Reachable Unsuccessful
#3 VCO Unreachable

Confidential │ ©2020 VMware, Inc. 106

Activation Unsuccessful: Symptom #2
Orchestrator Unreachable

Confidential │ ©2020 VMware, Inc. 107

Activation Unsuccessful Symptom # 2
Screenshot for VCO Unreachable

Change VCO FQDN to

IP address

Confidential │ ©2020 VMware, Inc. 108

Activation Unsuccessful Symptom # 2
Screenshot for Certificate Error during Activation

Enable checkbox for

Check Ignore Certificate

Confidential │ ©2020 VMware, Inc. 109

Activation Unsuccessful Symptom #2
Orchestrator (VCO) Unreachable/SSL error

Troubleshooting Steps
1. Change VCO FQDN to IP Address
✓ nslookup/dig to get the IP address for the VCO

2. Check ignore Certificates with VCO IP address

3. WAN Interface order

✓ If there are multiple WAN interfaces, ensure Public IP is set on lowest numbered interface.
✓ Reset VCE config and regenerate activation link with public IP assigned to lowest interface

4. What to check on Firewall(FW): Next Hop as FW

✓ Verify there is no destination filter on FW for “VCO IP && outbound TCP 443”
✓ Check FW logs for blocked traffic

Confidential │ ©2020 VMware, Inc. 110

Symptoms:
Scenario 2
#1 Gateway Service Unreachable
Overlay Tunnels
Unsuccessful

Confidential │ ©2020 VMware, Inc. 111

Gateway Service unreachable
Banner on Monitor > Edges > Dashboard page shows link to GW as Degraded

Confidential │ ©2020 VMware, Inc. 112

Gateway Service unreachable
Troubleshooting Steps

1. Check Test and Troubleshoot > Remote Diagnostics > List Path > From Peer drop down

✓ Verify path to VCG (Gateway)

2. If no path, Run Packet capture on Edge device WAN interface.

✓ You will notice outbound UDP 2426 packets towards VCG but no return packets from VCG.

Confidential │ ©2020 VMware, Inc. 113

Gateway Service unreachable
1 Remote Diagnostics from VCO

List Path shows no path towards VCG

Confidential │ ©2020 VMware, Inc. 114

 Gateway Service unreachable
Troubleshooting Tools

2 Diagnostic Bundle 3 Packet Capture

Confidential │ ©2020 VMware, Inc. 115

Gateway Service unreachable
Packet Capture from Edge Device WAN interface

Packet Capture

Confidential │ ©2020 VMware, Inc. 116

Symptoms:
Scenario 3
#1 BW measured < than what’s expected Bandwidth
Measurement
Unsuccessful

Confidential │ ©2020 VMware, Inc. 117

Bandwidth Measurement
BW measured < than what is expected

Confidential │ ©2020 VMware, Inc. 118

Bandwidth Measurement Symptom# 1
BW measured is < than what is expected

Troubleshooting Steps

1. Force Re-measurement
BW test via Edge Remote Diagnostics

2. BW Measurement Mode
Change the BW measurement mode from Slow start to Burst mode.

✓ Use Slow Start up to 200 Mbps

✓ Use Burst mode up to 500 Mbps

✓ Above 500 Mbps, set the BW statically

175Mbps measured Bandwidth, will trigger a retest in

burst mode

Confidential │ ©2020 VMware, Inc. 119

Bandwidth Measurement
BW Measurement Mode

From VCO -> Configure-> Edges->

Devices -> WAN Settings ->
Advanced
1

• Use Slow Start up to 200 Mbps

• Use Burst mode up to 500 Mbps
• Above 500 Mbps, set the BW
statically
2

Confidential │ ©2020 VMware, Inc. 120

Bandwidth Measurement
Bandwidth Re-measurement

Confidential │ ©2020 VMware, Inc. 121

Symptoms:
Scenario 4
#1 Ping fails between Hub and Spoke Routing
not working

Confidential │ ©2020 VMware, Inc. 122

Pings fails between Hub and Spoke
Cannot Ping Branch Edge to DC Hub & Vice-Versa

Troubleshooting Steps
1. Check Cloud VPN settings under Global and other segments

2. Check Tunnel status between Edge and VCG

✓ Test and Troubleshoot -> Remote Diagnostics -> List Path
✓ Packet Capture from VCE WAN interface for bi-directional packet flows

3. Advertise Flag under Static route/vlan

Confidential │ ©2020 VMware, Inc. 123

Pings fails between Hub and Spoke
Monitor | Edges | Edge device | Link status is GREEN/UP

Confidential │ ©2020 VMware, Inc. 124

 Ping fails between Hub and Spoke

Troubleshooting Steps Troubleshooting Steps

1. Check List Path 2. Check Cloud VPN settings under Global and
other segments
Test and Troubleshoot -> Remote Diagnostics -> List Path

If No path, then run packet capture from Branch site edge device
WAN interface, notice for outbound UDP 2426 and no return traffic
from DC Hub device.

Peer dropdown missing Hub Path

Confidential │ ©2020 VMware, Inc. 125

Pings fails between Hub and Spoke

Troubleshooting Steps

3. Advertise Flag under Static route/vlan

Confidential │ ©2020 VMware, Inc. 126

Pings fails between Hub and Spoke
Verify that ICMP is enabled if pinging Edge LAN interface

▪ Check that ICMP is Enabled

▪ Ensure that NAT Direct is unchecked

Confidential │ ©2020 VMware, Inc. 127

 Pings fails between Hub and Spoke
Troubleshooting Tools

1 Remote Diagnostics from VCO 2 OFC Table

Confidential │ ©2020 VMware, Inc. 128

 Pings fails between Hub and Spoke
Troubleshooting Tools

3 Diagnostic Bundle 4 Packet Capture

Confidential │ ©2020 VMware, Inc. 129

Symptoms:
Scenario 5
App performance is degraded App Performance
User reports slowness in general not Optimal

Confidential │ ©2020 VMware, Inc. 130

Application Performance
Symptom # 1 Application Performance degraded

1 Remote Diagnostics from VCO

Confidential │ ©2020 VMware, Inc. 131

Application Performance
Symptom # 1 Application Performance degraded

Troubleshooting Steps
Incorrect Biz Policy match

Test and Troubleshoot | List Flows | shows application traffic matching incorrect manually
created Biz Policy instead of default.

Change Business policy for application from Direct to Multi-Path

Confidential │ ©2020 VMware, Inc. 132
Scenario: Direct vs. Multi-Path (Overlay)

VCE VCG

VCO

Internet

Confidential │ ©2020 VMware, Inc. 133

Scenario: Corrective Measure

VCE VCG

VCO

Internet

Confidential │ ©2020 VMware, Inc. 134

Application Performance
Symptom # 1 Application Performance degraded
Troubleshooting Tools: VCO UI > Monitor > Edge > QoE
1 QoE

Confidential │ ©2020 VMware, Inc. 135

 Application Performance
Symptom # 2 User Reports slowness in general. Performance was better before SDWAN
Troubleshooting Steps Troubleshooting Steps
1. Check QoE Manually Change LAN interface set from
Half-duplex to Full-duplex.
2. If QoE shows degraded, then check the interface
status under Test and Troubleshoot-> Remote 1 Configure >Edges> Devices > LAN Interface
diagnostics > Interface Status

Confidential │ ©2020 VMware, Inc. 136

Edge Diagnostic Commands
Shell Access and Command lines

Confidential │ ©2020 VMware, Inc. 137

Edge Shell Access

▪ From the LAN side, any gateway address assigned to a VLAN

▪ Can also be accessed from the WAN IP addresses
▪ Must open the firewall up and specify a source host address
▪ Should be done only on a temporary basis

▪ Default username: root

▪ Note that virtual Edges (including marketplace edges) require different access
▪ Keypair needs to be set during instantiation
▪ Works of dedicated management interface. Other interfaces will not work.

▪ Central command to obtain state& status from:

▪ In /opt/vc/bin/debug.py -h

*** Add 169.254.129.1 to the firewall tab under support access

Confidential │ ©2020 VMware, Inc. 139

How to login to an SD-WAN Edge from the Gateway
https://ikb.vmware.com/s/article/57322?lang=en_US

Confidential │ ©2020 VMware, Inc. 140

How to login to an SD-WAN Edge from the Gateway
Login Event from VCO

Confidential │ ©2020 VMware, Inc. 141

Virtual Edge Console Access
Verifying installation via the console

• Open Console (username: vcadmin / password: set as part of deployment, there is no default)
• Verify Activation with /opt/vc/bin/is_activated.py
• Manual activation with /opt/vc/bin/activate.py –s <vco_fqdn> <activation_key>

Confidential │ ©2020 VMware, Inc. 142

Troubleshooting activations
Confirming if the edge can connect to VCO

velocloud hub:/opt/vc/bin# is_activated.py

False

In case need to reset the edge to blank

configuration state:

velocloud hub:/opt/vc/bin# reset_config.sh -H

Standard Linux commands can be used to verify

basic configuration
• For example: ifconfig, route –n can be used
to verify the interface has an IP and the
edge has a default route

Scan through the event sequence in

/var/log/activation.log

Confidential │ ©2020 VMware, Inc. 143

Link vs. Tunnel vs. Path

Confidential │ ©2020 VMware, Inc. 144

Most useful debugging commands
ifconfig-a L3 interface stats

lsusblsusb-tlsusb-v USB Device stats

/opt/vc/bin/debug.py--flow_stats Flow stats

/opt/vc/bin/debug.py--handoffq Handoff queue (looking for drops)

/opt/vc/bin/debug.py –v routes Local SD-WAN routing table

/opt/vc/bin/debug.py -v --biz_pol_dump Dump of active business policies

/opt/vc/bin/debug.py -v --link_stats Dump of link stats

/opt/vc/bin/debug.py -v --path_stats Dump of path stats

/opt/vc/bin/ethtool_dumpsh Physical interface stats

top-H-n1-b.out CPU stats

Confidential │ ©2020 VMware, Inc. 145

Support Review

Confidential │ ©2020 VMware, Inc. 146

Post-Sales Support
Support may be contacted via the following two channels:

Web https://my.vmware.com/ (Customer Connect)

Phone Visit Support Phone Numbers to find your local phone number

How to open an SR:

https://kb.vmware.com/s/article/53907

Confidential │ ©2020 VMware, Inc. 147

 Production Support
Create a Support Request

Step 2: From the dropdown, select “VMware SD-WAN by VeloCloud”

Step 1: Log into https://my.vmware.com (Customer
Connect Portal) using your designated customer account

• How to open SRs for SASE KBs?

VMware SASE – Support (53907)
VMware Edge Network Intelligence - Support (81330)
VMware Secure Access – Support (83703)
VMware SD-WAN – Support (83702)
VMware Cloud Web Security – Support (83701)

Confidential │ ©2020 VMware, Inc. 148

Supplying a Minimum Data Set (MDS)
When opening a support case, there is specific information that support needs. Providing this
information at the time of opening a case helps speed up the troubleshooting process.
Item Example

Partner/Customer SR Number: This may be a ticket in the requestors system

Customer Name: Acme Inc

Partner Name: BigTel

Orchestrator URL: Link to the Orchestrator

Customer Business Impact: 3 branch sites are unable to access time-clock system

VMware SD-WAN Edge Name(s): Chicago, Dallas, Cleveland

VMware SD-WAN Gateway Name(s): vcg123

Edge or Gateway Diagnostic Bundles or PCAP: Link to partner/customer storage to download the diagnostic bundle

Problem Statement: 1 to 2 sentence description

Analysis: Full details of the issue. See below for suggestions.

Requested Assistance: What specifically is being asked of support?

Confidential │ ©2020 VMware, Inc. 149

Recommended Data Set
In addition to MDS
As part of the analysis, we recommend the following piece of information:
✓ Issue Start / Stop time(s) in UTC

✓ Example Impacted Flow Source / Destination IP address(es)

✓ Example Impacted Flow Source / Destination port number(s)

✓ Flow Path (Edge-to-Edge, Edge-to-Gateway, Direct)

✓ Relevant Log Information

✓ Customer or partner deadlines.

✓ Describe the problem symptoms.

✓ Describe the recurrence pattern. (Problem increases with increase/decrease of system load)

✓ Steps that can be performed to reproduce the issue

✓ If a workaround has been put in place, describe those steps and the outcome.

Note on Diagnostic Bundle:

Diagnostic bundles are extremely important. SDWAN Support team highly recommends that diagnostics bundles are taken
during the issue. If that's not possible, it should be taken immediately after the issue. This will ensure the maximum amount of
information can be captured.

Confidential │ ©2020 VMware, Inc. 150

Case Severity Definition
Sev1 – Sev4
Severity Level Definition
Production device or other mission critical system(s) are down, and no workaround
is immediately available.

• All or a substantial portion of your mission critical data is at a significant risk of

loss or corruption.
Critical (Sev. 1)
• You have had a substantial loss of service.
• Your business operations have been severely disrupted.

Severity 1 support requires you to have dedicated resources available to work on

the issue on an ongoing basis during your contractual hours.
Major functionality is severely impaired.

• Operations can continue in a restricted fashion, although long-term productivity

Major (Sev. 2)
might be adversely affected.
• A major milestone is at risk. Ongoing and incremental installations are affected.
• A temporary workaround is available.
Partial, non-critical loss of functionality of the software.

Minor (Sev. 3) • Impaired operations of some components but allows the user to continue using
the software.
• Initial installation milestones are at minimal risk.

Cosmetic (Sev. 4) Cosmetic issues, including errors in the documentation.

VMware SD-WAN Support 24-hour support for critical SD-WAN deployments

Confidential │ ©2020 VMware, Inc. 151
When to escalate
You can escalate a support request at any time. The escalation process is especially
appropriate in the following situations
• Your production system goes down during an upgrade or other implementation

• You need to communicate a critical business impact to VMware SD-WAN Support

Management

• You are dissatisfied with the responsiveness to or resolution of a support request

We highly recommend that you contact VMware SD-WAN Support by telephone for
escalations. This will ensure that your request is directed to the appropriate resources as
soon as possible to achieve a successful resolution.

Confidential │ ©2020 VMware, Inc. 152

How to escalate

By Web By Phone

Log into the https://my.vmware.com/ Call VMware SD-WAN Support. (New

portal Severity 1 Cases Only)

Select your pre-existing SR Provide your SR number and request that

your SR be escalated. The agent will
Click ‘Request manager assistance’ engage the Manager on Duty

Enter a description of what you may need

and your contact information.

Confidential │ ©2020 VMware, Inc.

MyVMware Portal
‘Request manager assistance’ option

Confidential │ ©2020 VMware, Inc. 154

 Thank You

Confidential │ ©2020 VMware, Inc.