Analyzing Log Data

• Use Common Tools to Analyze Logs

• Use SIEM Tools for Analysis

Copyright © 2021 CertNexus, Inc. All rights reserved. 1

 USE COMMON TOOLS TO ANALYZE LOGS

Copyright © 2021 CertNexus, Inc. All rights reserved. 2

 Preparation for Analysis

• You need to turn collected raw data into a form that is useful for analysis.
• Can be done manually, or through automation tools.
• Programming, shell scripting, and batch file writing can help develop tools.
• Regular expressions can also help you search for patterns.
• SIEMs are also helpful as an automated tool for log analysis.

Copyright © 2021 CertNexus, Inc. All rights reserved. 3

 Guidelines for Preparing Data for Analysis

 Filter out unnecessary or duplicate data.

 Combine sources.
 Synchronize events logged in different sources.
 Normalize data formats.
 Store data securely.

Copyright © 2021 CertNexus, Inc. All rights reserved. 4

 Log Analysis Tools

• Linux tools
• grep
• cut
• diff
• Windows tools
• findstr
• WMIC
• Event Viewer
• Scripting languages
• Bash (Linux)
• Windows PowerShell

Copyright © 2021 CertNexus, Inc. All rights reserved. 5

 The grep Command

• Searches text files for specific string.

• Useful for log analysis in the following ways:
• Search for specific facility codes.
• Search for specific process IDs.
• Search for specific details of an event.
• Search for specific IP addresses/hostnames.
• Search for specific dates and times.
• Search multiple log files.

Copyright © 2021 CertNexus, Inc. All rights reserved. 6

 The cut Command

• Trims the characters returned for each line of a text file.

• Useful for removing the "noise" of too much info.
• Basic example: cut -c5 syslog
• Delimiter example:

Copyright © 2021 CertNexus, Inc. All rights reserved. 7

 The diff Command

• Takes two text files and returns how they differ.

• Does this line by line.
• Useful when you need to correlate disparate logs.
• Compare time values for each entry.
• Ensure logs haven't been tampered with.

• After line 1, line 2 from second file needs to be added.

• Line 3 from first file needs to be deleted so line 3 in both files
matches up.
Copyright © 2021 CertNexus, Inc. All rights reserved. 8
 The findstr Command

• Windows version of the grep command.

• Has a slightly different syntax.
• Includes most of the same options.
• Basic example:

Copyright © 2021 CertNexus, Inc. All rights reserved. 9

 WMIC for Log Analysis

• Can be used to analyze logs on remote Windows machines.

• Use the NTEVENT alias to return log entries matching parameters.
• Example:

• Useful for identifying specific events and details without being at the machine.

Copyright © 2021 CertNexus, Inc. All rights reserved. 10

 Event Viewer

• Severity of event:
• Information
• Warning
• Error
• Audit Success/Failure
• Info about event:
• Subject
• Error details
• Event ID
• Source
• Description
• Various management options.

Copyright © 2021 CertNexus, Inc. All rights reserved. 11

 Bash

Bash: A shell and scripting language for Unix-like systems.

• Comes standard in most Linux distros.

• Can run complex scripts.
• Scripts are useful for automating analysis
tasks.
• Saves you time.

Copyright © 2021 CertNexus, Inc. All rights reserved. 12

 Windows PowerShell

PowerShell: The default Windows command shell and scripting language.

• A scripting language and command

shell built on .NET Framework.
• Useful for both local and remote
management of Windows systems.
• Functions through the use of cmdlets.
• Like Bash, can automate log analysis
and increase efficiency.

Copyright © 2021 CertNexus, Inc. All rights reserved. 13

 Additional Log Analysis Tools

• awk
• tail
• SEC
• Microsoft Log Parser
• Logwatch
• Kiwi Syslog Server
• Visualization tools
• Big data analysis tools

Copyright © 2021 CertNexus, Inc. All rights reserved. 14

 Long Tail Analysis

Long tail analysis: The process of culling low-frequency events to identify anomalies.

• Simplified application of the "long tail" in statistics and business contexts.

• Assumption: More common events are less useful, and vice versa.
• Certain data points are too noisy and will overwhelm analysis efforts.
• Discover which data points are rare.

Copyright © 2021 CertNexus, Inc. All rights reserved. 15

 Guidelines for Using Linux- and Windows-Based Tools for Log Analysis

• Linux
 Know the format of common Linux logs.
 Use grep when you need to search a log for specific strings.
 Consider the different options for grep.
 Use cut to manage the length of logs.
 Create a delimiter with cut to return more accurate results.
 Use diff to examine the ways two logs diverge in content.
 Use piping to run multiple commands together.
 Use Bash shell scripts to automate entering these commands.
• Windows
 Know the format of common Windows logs.
 Use findstr when you need to search a log for specific strings.
 Consider the different options for findstr.
 Use NETEVENT with WMIC to pull logs from a remote computer.
 Use Event Viewer to filter logs and create custom views.
 Use Windows PowerShell scripts to automate log retrieval tasks.

Copyright © 2021 CertNexus, Inc. All rights reserved. 16

 Activity: Analyzing Linux Logs for Security Intelligence

• You'll use grep and cut to find specific information and make it more readable.
• This will make your log analysis efforts easier.

Copyright © 2021 CertNexus, Inc. All rights reserved. 17

 USE SIEM TOOLS FOR ANALYSIS

Copyright © 2021 CertNexus, Inc. All rights reserved. 18

 Security Intelligence Correlation

• Events that seem completely valid may reveal a security problem when looked at in combination.

Copyright © 2021 CertNexus, Inc. All rights reserved. 19

 Security Information and Event Management (SIEM)

SIEM: The real-time or near-real-time analysis of security events.

• Provides insight into intrusions through intelligence aggregation and correlation.

• Can be software, hardware, or outsourced services.
• Greatly increases analysis productivity.
• Has various considerations.

Copyright © 2021 CertNexus, Inc. All rights reserved. 20

 SIEM Analysis

• Intelligence loses value over time. SIEM

• Gathering and analyzing intelligence takes a lot of effort.
• SIEMs help to automate the tedious processes.
• Still significant gaps that can't be automated.
• Conduct front-end analysis to save time. Planning and
Collection
Direction
• Configure SIEM to focus on important events identified
through risk analysis.
Dissemination
and Processing
Integration

Analysis and
Production

Copyright © 2021 CertNexus, Inc. All rights reserved. 21

 Agent-Based vs. Agentless SIEMs

• Agent-based:
• Agent service installed on each host.
• Agentless:
• SIEM monitors hosts remotely.
• Cannot replace human analysis entirely.

Copyright © 2021 CertNexus, Inc. All rights reserved. 22

 Guidelines for Using SIEMs for Security Intelligence Analysis

• Support compliance
 Preserve data as required in its original forms.
 Configure alerts to generate in a form that is conducive to follow-up, like support tickets.
 Review your logs on a frequent, regular basis.
 Ensure SIEM can generate documentation that verifies frequent scans and reviews.
• Configure for comprehensive security
 Configure SIEM to aggregate data from many boundary, network, and data defenses.
 Configure SIEM to identify unauthorized assets and software.
 Use SIEM to monitor compliance and generate alerts for misconfigurations.
 Use SIEM to monitor wireless devices.
 Use SIEM to monitor rules and other configurations for network devices.
 Configure SIEM to report on unnecessary use of admin privileges.
 Correlate user activities with rights/roles to reveal violations of least privilege.
 Configure SIEM to perform continuous vulnerability assessment and remediation.

Copyright © 2021 CertNexus, Inc. All rights reserved. 23

 Activity: Incorporating SIEMs into Security Intelligence Analysis

• There are many logs coming in from all over the network.
• You need a centralized analysis platform like a SIEM.
• You decide to test Splunk.
• You'll analyze your server's event logs.

Copyright © 2021 CertNexus, Inc. All rights reserved. 24

 Reflective Questions

1. What are some of the tools you use most often to analyze log data?

2. How could/does a SIEM in your organization or an organization you're familiar with help increase
productivity in the security intelligence lifecycle processes?

Copyright © 2021 CertNexus, Inc. All rights reserved. 25