Troubleshooting a failed Discovery

Symptoms

 No device found
 Too many devices found
 The wrong device is matched
 The CMDB is not populated
 Device lacks the necessary information
 Duplicate records appear
 Not enough records are present
 Discovery stops

Resolution

1. Determine if Discovery failed during the Are you there phase(PORT

SCANNING)

Troubleshooting the Shazzam Phase in Discovery

Discovery reacts to the Shazzam port scan as follows:

 If Discovery finds an IP address with a state of OPEN for any port communicating
over WMI, SSH, or SNMP, Discovery lists the IP address in the Shazzam returns
(result="open") and begins the classification phase.
 If Discovery does not receive a response from a port, it does not list the IP address in
the returns from Shazzam.
 If Discovery receives a response from the scan that refuses the connection, Discovery
lists the IP address and the result (result="refused") in the Shazzam returns.

Common problems that may occur during the Shazzam phase, also known as a port scan, in
Discovery, and possible solutions include the following:

Shazzam cannot find any devices at all

 It could be a network connectivity issue. Perform ping.

Shazzam cannot find any particular type of devices (SNMP/SSH/WMI)

 If it is not a network issue, first check the port_probe_spec Shazzam output

payload and make sure your port is being scanned. If it is not scanned, then check
the behavior of the schedule(check the below screenshot of ECC Queue output XML
file)
  Also, it is possible that the port probes are not prioritized property.

. The out-of-box system prioritizes the use of port probes as follows:

 1 - WMI
 2 - SSH
 3 – SNMP

You can find this on instance discovery->discovery definitions->Port Probes

Shazzam cannot find any SNMP devices

 If the solutions above do not work, credentials for SNMP devices are needed for
SNMP. (Add credentials / SNMP String in the credential table)

Shazzam is not finding some devices

 Make sure the device type port is open and the service is up and running on that
device (for example, WMI service, SSH server).

 Shazzam has probe parameters that may need to be fine tuned. For example, if
Shazzam is not finding SSH devices,
the GenericTCP_waitForConnectMS and BannerTCP_waitForConnect
MS wait time may need to be extended.
 Reducing the shazzam_chunk_size from 100 to a lower number may also help.

The MID Server is not turning information for a certain network segment, or for the
entire network

 Check network connectivity using PING, TRACEROUTE, TELNET, and SNMP scanning
tools.
 Make sure the MID Servers that Discovery is using can reach the desired segments of
the network.
 2. Determine if Discovery failed during the What are you phase.
Troubleshooting the Classification Phase in Discovery.

After the information is returned from the Classify probe, Discovery undergoes the
classification phase, which determines the device type for each probe protocol that is
supported. This is the first time Discovery uses credentials to query a target. Discovery
classifies computers based on Operating System (OS) and Network Devices based on their
capabilities. Currently, we support the following protocols for classification: SSH, WMI,
SNMP.

Under each protocol type, Discovery goes through the various classification types to
determine what type the target device is. If a device matches the criteria for a given classifier,
it is classified as such, and Identity probes specific to that device are then triggered.

For example, if the target can communicate through SSH, it could be a Linux, Mac, AIX,
Solaris, etc. Each classification type then dictates the probes that Discovery runs for the
Identification phase and Exploration phase.

If multiple ports are open to be classified, based on the classification priority, certain
protocols are tried before others.

In the base system, the classification priority is in this order: WMI > SSH > SNMP > CIM.

For example, if a device has both SNMP and SSH available for Discovery, the SSH classify
probe is tried first. If it succeeds, then Discovery continues with SSH until the device is fully
discovered. If it fails, then Discovery tries launching the SNMP classify probe to see if it is
possible to explore with SNMP.
Troubleshooting Classification

Errors and warnings found during classification are available in the Discovery log for each
Discovery. These warnings are typical of probing in general. Make sure that the property
glide.discovery.debug.classification is set to true.

If a classifier probe cannot connect to a device, the Discovery status lists it as Active,
couldn’t Classify.

Common problems include:

 Authentication: The proper probe credentials must be available on the instance. The
credentials provided must have the right to perform the probes in question.

 Connectivity: Socket timeouts and other forms of IO issues may be due to network
configuration, congestion, or firewalls. Placing a MID in the same LAN as the target device
may solve some of these issues.

 Classifier: A custom classifier may need to be created for devices not supported in the base
system. This is rare.

 SNMP OIDs: If SNMP is being used, the device’s system OID may need to be added to System
OIDs table on the instance.

 WMI application: If the Windows application stops performing as expected when all else has
been validated, restart the operating system to eliminate any residual potential issues (if
possible).
 Unix systems: When a UNIX Classify probe is having problems, the most likely cause is
credentials. There are rare cases in which a UNIX system might have issues with local ACLs
similar to Network devices or even logical firewalls, but this is not common. Use a third party
application such as puTTY from the MID Server host to confirm the credentials Discovery is
attempting to use.
 Access control lists (ACLs): Routers, powering devices, switches, can use Access Control lists
that limit which IP addresses are allowed to make queries to the device. Ensure that your
MID Server’s IP address is in that list. Since you cannot telnet to a UDP port, use a third party
query tool such as SNMP WALK from the MID Server host to validate the proper credentials
and ACL access.
 Logical firewall: In WMI communication, traffic only initiates on port 135. When two
Windows operating systems communicate with WMI, they actually negotiate unused high
ports to finish the conversation. This presents a problem for logical firewalls in Windows
systems. Normally, firewalls allow port 135 to be seen as it is with the Discovery port scan
probe, but block the high ports that Discovery needs to communicate. To overcome this, use
either of the these options:
o Configure firewalls to allow any port to use any protocol from the MID Server’s host,
using the WMI script that is run locally.
o Lock down WMI to specific ports.
 3. Determine if Discovery failed during the Have I seen you before
phase.Troubleshooting the Identification Phase in Discovery.

The CI Identification phase is where Discovery attempts to determine whether the target device
that is being discovered already has a record in the CMDB. If the answer is yes, then Discovery
continues by launching the exploration probes. If there answer is no, then Discovery creates a
record for it in the CMDB before launching the exploration probes.

By design, each identity probe attempts to gather as much identifying information about the
target as possible. This typically includes things like hostname, serial numbers, and network
adapters

Discovery runs them through a list of identifiers until it comes to a stopping point, which
could be:

1. An identifier has found one and only one match.

2. No active identifiers are able to find one and there is only one match in the
CMDB. Here are the possible outcomes:
a. No identifier was able to find any match at all.
b. At least one of the identifiers found more than one match.

In the case of 1, an identifier has found one and only one match in the CMDB. Then
Discovery declares that it knows exactly which CI record it is.
In the case of 2-a, no identifiers are able to find any match in the CMDB. Therefore, the
device being discovered must be a new CI. Discovery creates a record for it in the CMDB.
In the case of 2-b, the best case any of the identifiers is able to do is to find multiple
matches. In this case, Discovery stops because it is unable to determine which record it needs
to update. In other words, until someone steps in and cleans up the duplicate records,
Discovery is not able to continue to the exploration phase.

Identification Troubleshooting

If the identity probes are somehow not launched, check the following:

1. Did the classification probe run successfully? If not, check the credential
against the target.
2. Find the appropriate classification (UNIX, Windows, SNMP) and check if the
identify probe is included in the Triggers probe related list. If not, add it back
in.

Discovery is overwriting the incorrect CI record

This could happen if, during the identification phase, an identifier finds a match in the
CMDB that is, in fact, incorrect. For example, if someone created or imported a CI and
entered an incorrect serial number or a serial number that is too generic, such as a chassis
serial number, Discovery identifies it by the serial number and overwrites the record by
mistake. This can be remedied either by not using the serial number identifier, or by
modifying the serial number input so that the serial number is always unique.
I have different CIs that are overwriting each other in the same record
This is typically due to an identifier mistaking the same record in the CMDB for different
devices. For example, there are two devices with the exact same name and they somehow
have no serial number as the identifiable information. Discovery always ends up
creating/updating the same record for both devices. If both devices happen to be in the same
Discovery run, it would even look like they are the same CI with multiple IPs and one gets
the Identified, ignored extra IP message.

A discovered device matches more than one existing CI

Navigate to Discovery Definition > Properties and select Yes for the CI identification
debugging property. This property writes additional data for each IP address to the Discovery
log during the CI identification process. Consult this log to resolve issues with duplicate CIs.

Discovery stops after the identity probe has run

This could be due to identifiers finding multiple matches in the CMDB. In other words,
Discovery has found matches in the CMDB, but is not able to determine which one is the
correct record to update. In this case, human intervention to the CMDB is needed. The
duplicate records must be cleaned up for Discovery to continue properly. Here is an example
of what it looks like in the log:

4. Determine if Discovery failed during the What else can you tell me
about yourself phase. Troubleshooting the Exploration Phase in
Discovery.

During exploration, most probes use the same credentials used during classification and
identification, however there are probes that have additional requirements.

1. VMware vCenter and ESX/ESXi

o While discovering a Windows Server, if an active process is classified as vCenter, the
VMware - vCenter probe is launched. The credential used for this probe is of
type=VMware.
o During the processing of the results from the VMware - vCenter probe, for each ESX
server that is found, a CIM - ESX Chassis Serial Number probe is launched. This probe
uses the credential type=CIM
 2. Microsoft SQL
o While discovering a Windows Server, if an active process is classified as Microsoft
SQL Server, the Windows - MSSQL probe is triggered.
3. SSH commands that require sudo:
o Certain SSH probes require elevated privileges and leverage the use of sudo.

Troubleshooting Exploration

The same commands within Discovery probes can be executed outside of the ServiceNow
instance on the MID Server host. Typically this is the best way to troubleshoot.

WMI

 Use the command line tool wmic to target WMI Objects and registry paths.
 Use the command line tool cscript to run javascript against a remote machine.

Powershell

 Within Powershell, use gwmi to target Managed Objects and registry paths.

SSHCommand

 Use an SSH client and connect to the target machine with the same credential that Discovery
should be using.
 Once connected, execute the same command or script.

SNMP

 Use a command line tool like snmpwalk to target OIDs on a remote device.

Common Exploration Phase Errors

 WMI and Powershell

o The impersonation of the user failed.

o Ensure that the domain is specified, along with the username in the
credentials.
 Connection failed to WMI service and other common Windows (WMI/Powershell) error
messages:

Error: The remote server machine does not exist or is unavailable

Failed to access target system. Please check credentials and firewall settings on the target
system to ensure accessibility: Access is denied. (Exception from HRESULT: 0x80070005
(E_ACCESSDENIED))

Failed to access target system. Please check credentials and firewall settings on the target
 system to ensure accessibility: The RPC server is unavailable. (Exception from HRESULT:
0x800706BA)

o WMI, does the mid server service account have access to the targeted
machine? What if a domain admin account is used as the mid server service
account?
o From the command prompt on the mid server host, execute for
runner_type=WMI

wmic /node:"<target>" /user:"<user>" /password:"<password>" path

win32_operatingsystem

o From within a Powershell console on the mid server host, execute for
runner_type=Powershell

gwmi win32_operatingsystem -computer <ip> -credential '<username>'

o It is possible that probe is timing out while waiting for a response. If the
command is successful from a command prompt, try extending the
wmi_timeout value of the probe.

 vCenter Discovery
o Unable to establish connection to https://10.249.17.207/sdk
o No VMWare type credential is stored in the credential table.
o The user name being used is a domain account and needs to be prefixed with a
domain.

 CIM_RegisteredProfile{{RegisteredName='Base
Server'}}.CIM_ElementConformsToProfile{{ResultClass:'CIM_ComputerSystem'}}.CIM_Compu
terSystemPackage{{ResultClass:'CIM_Chassis',PackageType='3'}}.* - CIM_RegisteredProfile -
Authentication failed.
o No CIM type credential is stored in the credential table. Also, see vCenter: Setting up
CIM read-only access: Creating a local read only user
 MSSQL
o Cannot find type [Microsoft.SqlServer.Management.Smo.Server]: make sure the
assembly containing this type is loaded
o You need to install Microsoft SQL Server management library (SMO)