Analyzing Log Data
• Use Common Tools to Analyze Logs
• Use SIEM Tools for Analysis
Copyright © 2021 CertNexus, Inc. All rights reserved. 1
USE COMMON TOOLS TO ANALYZE LOGS
Copyright © 2021 CertNexus, Inc. All rights reserved. 2
Preparation for Analysis
• You need to turn collected raw data into a form that is useful for analysis.
• Can be done manually, or through automation tools.
• Programming, shell scripting, and batch file writing can help develop tools.
• Regular expressions can also help you search for patterns.
• SIEMs are also helpful as an automated tool for log analysis.
Copyright © 2021 CertNexus, Inc. All rights reserved. 3
Guidelines for Preparing Data for Analysis
Filter out unnecessary or duplicate data.
Combine sources.
Synchronize events logged in different sources.
Normalize data formats.
Store data securely.
Copyright © 2021 CertNexus, Inc. All rights reserved. 4
Log Analysis Tools
• Linux tools
• grep
• cut
• diff
• Windows tools
• findstr
• WMIC
• Event Viewer
• Scripting languages
• Bash (Linux)
• Windows PowerShell
Copyright © 2021 CertNexus, Inc. All rights reserved. 5
The grep Command
• Searches text files for specific string.
• Useful for log analysis in the following ways:
• Search for specific facility codes.
• Search for specific process IDs.
• Search for specific details of an event.
• Search for specific IP addresses/hostnames.
• Search for specific dates and times.
• Search multiple log files.
Copyright © 2021 CertNexus, Inc. All rights reserved. 6
The cut Command
• Trims the characters returned for each line of a text file.
• Useful for removing the "noise" of too much info.
• Basic example: cut -c5 syslog
• Delimiter example:
Copyright © 2021 CertNexus, Inc. All rights reserved. 7
The diff Command
• Takes two text files and returns how they differ.
• Does this line by line.
• Useful when you need to correlate disparate logs.
• Compare time values for each entry.
• Ensure logs haven't been tampered with.
• After line 1, line 2 from second file needs to be added.
• Line 3 from first file needs to be deleted so line 3 in both files
matches up.
Copyright © 2021 CertNexus, Inc. All rights reserved. 8
The findstr Command
• Windows version of the grep command.
• Has a slightly different syntax.
• Includes most of the same options.
• Basic example:
Copyright © 2021 CertNexus, Inc. All rights reserved. 9
WMIC for Log Analysis
• Can be used to analyze logs on remote Windows machines.
• Use the NTEVENT alias to return log entries matching parameters.
• Example:
• Useful for identifying specific events and details without being at the machine.
Copyright © 2021 CertNexus, Inc. All rights reserved. 10
Event Viewer
• Severity of event:
• Information
• Warning
• Error
• Audit Success/Failure
• Info about event:
• Subject
• Error details
• Event ID
• Source
• Description
• Various management options.
Copyright © 2021 CertNexus, Inc. All rights reserved. 11
Bash
Bash: A shell and scripting language for Unix-like systems.
• Comes standard in most Linux distros.
• Can run complex scripts.
• Scripts are useful for automating analysis
tasks.
• Saves you time.
Copyright © 2021 CertNexus, Inc. All rights reserved. 12
Windows PowerShell
PowerShell: The default Windows command shell and scripting language.
• A scripting language and command
shell built on .NET Framework.
• Useful for both local and remote
management of Windows systems.
• Functions through the use of cmdlets.
• Like Bash, can automate log analysis
and increase efficiency.
Copyright © 2021 CertNexus, Inc. All rights reserved. 13
Additional Log Analysis Tools
• awk
• tail
• SEC
• Microsoft Log Parser
• Logwatch
• Kiwi Syslog Server
• Visualization tools
• Big data analysis tools
Copyright © 2021 CertNexus, Inc. All rights reserved. 14
Long Tail Analysis
Long tail analysis: The process of culling low-frequency events to identify anomalies.
• Simplified application of the "long tail" in statistics and business contexts.
• Assumption: More common events are less useful, and vice versa.
• Certain data points are too noisy and will overwhelm analysis efforts.
• Discover which data points are rare.
Copyright © 2021 CertNexus, Inc. All rights reserved. 15
Guidelines for Using Linux- and Windows-Based Tools for Log Analysis
• Linux
Know the format of common Linux logs.
Use grep when you need to search a log for specific strings.
Consider the different options for grep.
Use cut to manage the length of logs.
Create a delimiter with cut to return more accurate results.
Use diff to examine the ways two logs diverge in content.
Use piping to run multiple commands together.
Use Bash shell scripts to automate entering these commands.
• Windows
Know the format of common Windows logs.
Use findstr when you need to search a log for specific strings.
Consider the different options for findstr.
Use NETEVENT with WMIC to pull logs from a remote computer.
Use Event Viewer to filter logs and create custom views.
Use Windows PowerShell scripts to automate log retrieval tasks.
Copyright © 2021 CertNexus, Inc. All rights reserved. 16
Activity: Analyzing Linux Logs for Security Intelligence
• You'll use grep and cut to find specific information and make it more readable.
• This will make your log analysis efforts easier.
Copyright © 2021 CertNexus, Inc. All rights reserved. 17
USE SIEM TOOLS FOR ANALYSIS
Copyright © 2021 CertNexus, Inc. All rights reserved. 18
Security Intelligence Correlation
• Events that seem completely valid may reveal a security problem when looked at in combination.
Copyright © 2021 CertNexus, Inc. All rights reserved. 19
Security Information and Event Management (SIEM)
SIEM: The real-time or near-real-time analysis of security events.
• Provides insight into intrusions through intelligence aggregation and correlation.
• Can be software, hardware, or outsourced services.
• Greatly increases analysis productivity.
• Has various considerations.
Copyright © 2021 CertNexus, Inc. All rights reserved. 20
SIEM Analysis
• Intelligence loses value over time. SIEM
• Gathering and analyzing intelligence takes a lot of effort.
• SIEMs help to automate the tedious processes.
• Still significant gaps that can't be automated.
• Conduct front-end analysis to save time. Planning and
Collection
Direction
• Configure SIEM to focus on important events identified
through risk analysis.
Dissemination
and Processing
Integration
Analysis and
Production
Copyright © 2021 CertNexus, Inc. All rights reserved. 21
Agent-Based vs. Agentless SIEMs
• Agent-based:
• Agent service installed on each host.
• Agentless:
• SIEM monitors hosts remotely.
• Cannot replace human analysis entirely.
Copyright © 2021 CertNexus, Inc. All rights reserved. 22
Guidelines for Using SIEMs for Security Intelligence Analysis
• Support compliance
Preserve data as required in its original forms.
Configure alerts to generate in a form that is conducive to follow-up, like support tickets.
Review your logs on a frequent, regular basis.
Ensure SIEM can generate documentation that verifies frequent scans and reviews.
• Configure for comprehensive security
Configure SIEM to aggregate data from many boundary, network, and data defenses.
Configure SIEM to identify unauthorized assets and software.
Use SIEM to monitor compliance and generate alerts for misconfigurations.
Use SIEM to monitor wireless devices.
Use SIEM to monitor rules and other configurations for network devices.
Configure SIEM to report on unnecessary use of admin privileges.
Correlate user activities with rights/roles to reveal violations of least privilege.
Configure SIEM to perform continuous vulnerability assessment and remediation.
Copyright © 2021 CertNexus, Inc. All rights reserved. 23
Activity: Incorporating SIEMs into Security Intelligence Analysis
• There are many logs coming in from all over the network.
• You need a centralized analysis platform like a SIEM.
• You decide to test Splunk.
• You'll analyze your server's event logs.
Copyright © 2021 CertNexus, Inc. All rights reserved. 24
Reflective Questions
1. What are some of the tools you use most often to analyze log data?
2. How could/does a SIEM in your organization or an organization you're familiar with help increase
productivity in the security intelligence lifecycle processes?
Copyright © 2021 CertNexus, Inc. All rights reserved. 25
