Microsoft Sentinel Handbook
Module 3: Data Connectors: Ingesting Security Data
This module delves into the critical first step of any SIEM solution: getting your
security data into Microsoft Sentinel. Data Connectors are the conduits that bring logs
and events from various sources into your Log Analytics Workspace, making them
available for analysis, detection, and investigation.
3.1. Overview of Data Connectors
Data connectors are pre-built integrations or mechanisms that allow Microsoft
Sentinel to collect security data from different products, services, and systems.
Without data flowing into Sentinel, it cannot perform its core functions of detection
and analysis.
3.1.1. Why Data Ingestion is Crucial
● Visibility: The more data sources you connect, the more comprehensive your
view of the organization's security posture becomes. This "single pane of glass" is
essential for effective security monitoring.
● Detection Capability: Analytics rules and threat hunting queries can only
operate on the data that has been ingested. A wider range of data sources leads
to more effective and accurate threat detection.
● Incident Context: When an incident occurs, having related data from various
systems (e.g., identity, endpoint, network, cloud apps) provides crucial context for
a thorough investigation.
3.2. Common Microsoft Service Connectors
Microsoft Sentinel offers a rich set of built-in connectors for Microsoft services, which
are often the easiest to configure due to native integration.
3.2.1. Azure Active Directory (AAD)
● Purpose: Ingests sign-in logs and audit logs from Azure AD.
● Value: Essential for monitoring user authentication, access attempts,
administrative changes, and detecting identity-based attacks (e.g., brute-force,
suspicious sign-ins).
● Data Tables: SigninLogs, AuditLogs.
3.2.2. Microsoft 365 Defender (M365D) / Microsoft Defender XDR (MDX)
● Purpose: Integrates alerts and raw event data from various Microsoft Defender
products. This is a powerful connector as it brings in high-fidelity security data.
○ Microsoft Defender for Endpoint: Endpoint detection and response (EDR)
data from workstations and servers (process creation, file activity, network
connections).
○ Microsoft Defender for Identity: Identity-based attack detections from
on-premises Active Directory.
○ Microsoft Defender for Office 365: Email and collaboration threat protection
(phishing, malware in email).
● Value: Provides deep insights into endpoint activity, identity threats, and
email-borne attacks, crucial for comprehensive XDR (Extended Detection and
Response).
● Data Tables: Various tables prefixed with Device, Identity, Email, etc. (e.g.,
DeviceProcessEvents, IdentityLogonEvents, EmailEvents).
3.2.3. Azure Activity Logs
● Purpose: Captures events related to Azure resource management operations
(e.g., creating virtual machines, modifying network security groups, deleting
resources).
● Value: Critical for monitoring administrative activity within your Azure
environment, detecting unauthorized changes, and ensuring compliance.
● Data Table: AzureActivity.
3.2.4. Azure Firewall, WAF, NSG Flow Logs
● Purpose: Ingests network traffic logs from Azure networking components.
○ Azure Firewall Logs: Traffic passing through your Azure Firewall.
○ Web Application Firewall (WAF) Logs: Web traffic filtered by WAFs (e.g.,
Azure Application Gateway WAF).
○ Network Security Group (NSG) Flow Logs: Network flow information for
traffic allowed or denied by NSGs.
● Value: Provides visibility into network communication patterns, potential
intrusions, data exfiltration attempts, and policy violations.
● Data Tables: Various tables like AzureDiagnostics (for Firewall, WAF) and
AzureNetworkAnalytics_CL (for NSG Flow Logs).
3.2.5. Microsoft Defender for Cloud Apps (MDCA)
● Purpose: Collects logs and alerts from cloud applications (SaaS apps like
Salesforce, Dropbox, etc.) and provides cloud access security broker (CASB)
capabilities.
● Value: Helps monitor user activity in cloud apps, detect shadow IT, enforce data
loss prevention (DLP) policies, and identify unusual behavior in sanctioned cloud
applications.
● Data Tables: McasShadowITReport, McasActivity, McasAlerts.
3.3. Non-Microsoft (Third-Party) Connectors
Sentinel is not limited to Microsoft services; it can ingest data from a wide array of
third-party security solutions and infrastructure components.
3.3.1. Syslog, Common Event Format (CEF)
● Purpose: Standard protocols for sending log messages from network devices
(firewalls, routers), Linux servers, and many security appliances.
● Value: Enables collection from a vast ecosystem of non-Microsoft devices and
applications. CEF is a standardized format built on Syslog, making parsing easier.
● Mechanism: Typically requires a Linux-based Log Analytics agent (often called a
"Syslog forwarder" or "CEF collector") to receive logs and forward them to
Sentinel.
● Data Table: CommonSecurityLog.
3.3.2. Custom Logs (Log Analytics Agent)
● Purpose: For applications or systems that generate logs in unique formats (e.g.,
custom application logs, specific server logs not covered by other connectors).
● Value: Provides flexibility to ingest almost any text-based log file.
● Mechanism: The Log Analytics agent is installed on the source machine,
configured to monitor specific file paths, and then parses the custom logs based
on definitions you provide.
● Data Table: Custom tables with _CL suffix (e.g., MyCustomAppLog_CL).
3.3.3. API-based Connectors
● Purpose: Some third-party services offer APIs (Application Programming
Interfaces) to pull security data directly.
● Value: Allows for direct, often richer, integration with services that don't support
Syslog/CEF.
● Mechanism: Sentinel provides specific connectors for popular services (e.g.,
AWS CloudTrail, Google Workspace, Palo Alto Networks Firewall via API).
3.4. Managing and Monitoring Data Connectors
Once data connectors are configured, ongoing management is essential to ensure
continuous and healthy data ingestion.
3.4.1. Health Monitoring
● Sentinel provides a "Data connectors" blade where you can see the status of
each connector (e.g., "Connected," "Disconnected," "Errors").
● Monitor daily ingestion volume to detect sudden drops or spikes that might
indicate a problem.
● Use built-in workbooks (like "Workspace Usage Report") to track data ingestion
trends.
3.4.2. Troubleshooting Common Ingestion Issues
● Permissions: Ensure the identity used by the connector (e.g., Azure AD app,
managed identity) has the necessary permissions to access the source data.
● Network Connectivity: Verify firewalls, network security groups, and proxies are
not blocking communication between the data source and the Log Analytics
Workspace.
● Agent Status: For agent-based connectors (Syslog, Custom Logs), ensure the
Log Analytics agent is running and healthy on the source machine.
● Configuration Errors: Double-check connector-specific configurations (e.g.,
correct API keys, Syslog port settings, log file paths).
● Source Logs: Confirm that the source system is actually generating the logs you
expect.
By effectively configuring and managing your data connectors, you lay the essential
groundwork for Microsoft Sentinel to provide comprehensive security visibility and
threat detection.
