LogRhythm University

Custom MPE Rules

Using Regular Expression
LogRhythm v7
Custom MPE Rules Using Regular
Expression
LogRhythm v7
LogRhythm University

v1.3
© LogRhythm, Inc. All rights reserved.
This document contains proprietary and confidential information of LogRhythm, Inc., which is protected
by copyright and possible non-disclosure agreements. The Software described in this Guide is furnished
under the End User License Agreement or the applicable Terms and Conditions (“Agreement”) which
governs the use of the Software. This Software may be used or copied only in accordance with the
Agreement. No part of this Guide may be reproduced or transmitted in any form or by any means,
electronic or mechanical, including photocopying and recording for any purpose other than what is
permitted in the Agreement.

Disclaimer
The information contained in this document is subject to change without notice. LogRhythm, Inc.
makes no warranty of any kind with respect to this information. LogRhythm, Inc. specifically disclaims
the implied warranty of merchantability and fitness for a particular purpose. LogRhythm, Inc. shall not
be liable for any direct, indirect, incidental, consequential, or other damages alleged in connection with
the furnishing or use of this information.

Trademark
LogRhythm is a registered trademark of LogRhythm, Inc. All other company or product names
mentioned may be trademarks, registered trademarks, or service marks of their respective holders.

LogRhythm Inc.
4780 Pearl East Circle
Boulder, CO 80301
(303) 413-8745
www.logrhythm.com

LogRhythm Customer Support

support@logrhythm.com
CONTENTS
Chapter 1: A Review of Log Processing 1
Course Overview 2

LogRhythm Platform 3

Log Source Definitions 4

Data Processor Functions 5

Metadata Processing 6

Unidentified Logs 8

What Is RegEx? 10

RegEx Resources 11

RegEx in LogRhythm 13

Exercises 15

Exercise 1: Review Questions 16

Exercise 2: Writing and Testing Regular Expressions 18

Chapter 2: Introduction to Regex 21

Understanding RegEx 22

Literal Characters 23

Positional Characters 24

Using Literals With Positional Characters 26

Matching Characters 27

Character Sets 29

Repetition Characters 31

Custom MPE Rules Using Regular Expression │ Contents iii

 Wildcard Characters 32

Character Groups 33

Optional Matches 35

Reserved Characters 36

Common Regular Expressions Used In LogRhythm 38

Exercises 39

Exercise 1: Write a RegEx Using Literal Characters 40

Exercise 2: Write a RegEx that does not Match Certain Numbers or Letters 41

Exercise 3: Match Multiple Strings Using Literal Characters 42

Exercise 4: Write a RegEx Using a Character Set 44

Exercise 5: Using a Character Set with Optional Matches 45

Exercise 6: Challenges: Advanced Regular Expressions 46

Chapter 3: RegEx in LogRhythm 53

Using RegEx in LogRhythm 54

Greedy vs. Non-Greedy Quantifiers 55

Capture Groups 58

Using Named Capture Groups to Parse Metadata 60

Overloading Tags in LogRhythm 63

Exercises 65

Exercise 1: Reading Regular Expressions 66

Exercise 2: LogRhythm MPE Rule Builder Parsing Guide 67

Exercise 3: Identify Metadata in a Log Message 68

Exercise 4: Create an Overloaded Tag 70

Exercise 5: Create an Overloaded Tag 71

Chapter 4: Creating Custom MPE Rules 73

The MPE Rule Builder 74

iv Custom MPE Rules Using Regular Expression │ Contents

 General Settings 76

Processing Settings 78

Default Policy Settings 80

Log Message Source Type Associations 81

Base-Rule Regular Expressions 83

Catch-All Rules 84

Test Center 86

General RegEx Tips 88

Writing RegExes For Use In LogRhythm 89

Considerations For MPE Rule Creation 92

Steps To Create New Base-Rules 93

Walk Through Example Part One: Creating A Base-Rule 95

Exercises 105

Exercise 1: Access the LogRhythm Training Environment 106

Exercise 2: Create a New MPE Rule 110

Exercise 3: Create Another New MPE Rule 120

Exercise 4: Create Another New MPE Rule 126

Exercise 5: Challenge: Create a New MPE Rule with less instruction 131

Exercise 6: Challenge: Create Another New MPE Rule 133

Chapter 5: Enabling Custom Rules 137

The Final Steps for MPE Rule Creation 139

Sub-Rules 140

Creating Sub-Rules 142

The Sub-Rule Tags 144

Sub-Rule Example 146

Walk Through Example Part Two: Adding Sub-Rules 151

MPE Rule Processing Logic 157

Custom MPE Rules Using Regular Expression │ Contents v

 Changing The Sort Order 159

Rule Library Browser 162

MPE Policy Settings 164

Enabling Custom MPE Rules 168

Log Source Type Manager 169

The Date Format Manager 171

Date and Time Parsing 173

Steps After MPE Rule Creation 174

Exercises 175

Exercise 1: Chapter Review Questions 176

Exercise 2: The Rule Library 177

Exercise 3: Create an MPE Rule with Sub-rules 180

Exercise 4: Create an MPE Rule with Sub-rules 185

Exercise 5: Create an MPE Rule with Sub-rules 187

Exercise 6: Challenge: Create an MPE Rule with Sub-rules 189

Exercise 7: Enable New MPE Rules 191

Chapter 6: Best Practices 197

Rule Performance 198

Reviewing Processing Performance 200

The Efficiency Of A Regular Expression 204

RegEx Recommended Practices 207

The Whole Process For Custom MPE Rules 209

Exercises 211

Exercise 1: The Final Challenge 212

Glossary 229

vi Custom MPE Rules Using Regular Expression │ Contents

CHAPTER 1
A Review of Log Processing
Course Overview 2
LogRhythm Platform 3
Log Source Definitions 4
Data Processor Functions 5
Metadata Processing 6

What Is RegEx? 10
RegEx Resources 11

RegEx in LogRhythm 13

Exercises 15
Exercise 1: Review Questions 16

Exercise 2: Writing and Testing Regular Expressions 18

Custom MPE Rules Using Regular Expression │ A Review of Log Processing 1

 Course Overview
In this course we will introduce you to Regular Expression statements used to parse data from log
messages. We will practice building Regular Expression statements, and apply those skills to write a
parsing rule for a new Log Source Type. At the end of this course you should be able to confidently
configure and successfully implement parsing rules for new or custom Log Sources.

We will cover the following topics:

A review of how LogRhythm processes log messages

An introduction to Regular Expressions
How Regex is used in LogRhythm
o The MPE Rule Builder tool
Creating custom MPE Rules
o Base-rule development
o Sub-rules
Enabling Custom Rules in LogRhythm

2 Chapter 1 | A Review of Log Processing

LogRhythm Platform

The image above illustrates the LogRhythm Platform. This course is focused on the processing of log
data, as outlined in red in the image. While this is a small piece of the work that LogRhythm performs, it's
imperative that processing is done correctly to allow for analysis and output of the data.

Let's first quickly review how LogRhythm processes log data.

Chapter 1 | A Review of Log Processing 3

 Log Source Definitions

LogRhythm processes log data according to rules and policies specified for each Log Source. In this
course, we will review the message processing rules and policies, how they function, and how to create
custom rules and policies for custom Log Sources.

Before a log can be processed, LogRhythm must know from what device the log is coming and how it
should be processed.

Every log is associated to a single Log Source which LogRhythm uses to determine the origin of that log
message. A Log Source is a unique supplier of log data collected from a host. Hosts can have one or
more Log Sources.

A Log Source Type is assigned to each Log Source. Log Source Types are used to group logs that come
from common hardware types, have the same data format, and have the same processing rules applied.
A Log Source Type is also assigned to rules defined in the Log Processing Policies. These assigned Log
Source Types enable LogRhythm to determine which Log Processing rules are applied to each log
message. Assigning the Log Source Type improves processing performance because those logs are only
processed against rules for that Log Source Type; rules for other Log Source Types are automatically
skipped.

Log Processing Policies, or Message Processing Engine (MPE) policies, determine which Log
Processing Rules will be applied to a Log Source for processing and how matching log messages are
managed. The policies dictate how long the log remains online for reporting, known as Time-To-Live
(TTL), if the log should be archived, and if a copy of the log should be forwarded to the Platform Manager.

Log Processing Rules, also known as MPE Rules, are applied to each log message to parse data. The
MPE Rules are records associated to specific Log Source Types with an assigned Common Event,
Base-rule regular expression, Sub-rules, and other processing and policy settings used for processing
logs.

In summary, each Log Source is assigned a Log Source Type, which is associated to a Log Processing
Policy (MPE Policy). The Log Processing Policy contains Log Processing Rules (MPE Rules) which are
applied to all Log Messages received from the associated Log Source.

4 Chapter 1 | A Review of Log Processing

Data Processor Functions

Logs flow from the Log Sources to the System Monitors which then forwards the logs to the Data
Processor (DP). Logs are processed by the Message Processing Engine (MPE), contained on the DP.

When the MPE service is started:

It loads all MPE Policies assigned to active Log Sources.

It loads all MPE Rules referenced by the loaded MPE Policies.

The log messages are processed by applying the MPE Rules enabled within the MPE Policy for the Log
Source where the log originated. LogRhythm utilizes multiple log processing threads that can execute at
the same time, making the MPE capable of processing thousands of messages per second.

Chapter 1 | A Review of Log Processing 5

 Metadata Processing

During processing, LogRhythm parses, calculates, and derives metadata from log messages.
LogRhythm metadata fields store network and host information parsed from the log message. The
metadata fields go into a database to help speed performance when you use the LogRhythm search tools,
the Personal Dashboard, and Reporting.

Below is a an example of a log message.

Example
Raw Log:

08 13 2009 15:21:19 1.1.2.1 <LOC4:ERRR> Aug 13 2009 15:21:19 GSC-

Internet-FW: %ASA-3-106014: Deny inbound icmp src INSIDE:4.2.1.3 dst
INSIDE:4.1.1.1 (type 0, code 0)

After processing this log, the following information is available in LogRhythm: derived metadata and
parsed metadata.

Information derived from log

This information is not explicitly contained in the log message, but is derived based on contents of the log,
the associated Log Source Type, and the defined MPE Rule properties.

Log Source = Cisco ASA

Matched MPE Rule = PIX-X-106014: Denied Packet
Common Event = Traffic Denied by Network Firewall
Classification = Network Deny
Classification Type = Operations

Metadata fields parsed from log

This information is referred to as contextual metadata. This metadata is contained in the log message.

6 Chapter 1 | A Review of Log Processing

 Vendor Message ID = 106014
Protocol = ICMP
Origin Host = 4.2.1.3
Impacted Host = 4.1.1.1

LogRhythm is able to process log messages, deriving and parsing this information, using regular
expression pattern matching. We refer to these regular expressions as MPE Rules.

MPE Rules are records associated to specific Log Source Types with a common event, base rule regular
expression, sub rules, and other processing and policy settings used for processing logs. The MPE Rule
Builder, a tool available in the LogRhythm Client Console, is used to view, create and edit new MPE
base rules and sub-rules.

Pictured below is the regex rule, as seen in the MPE Rule Builder, used to process the log shown above:

The MPE processes collected logs against the regular expression rule to:

Normalize the log message

o Identify the log message
o Classify the activity described in the log message
o Parse important information from the log message into metadata fields
Determine actions to be taken on the log
o Archive
o Index
o Forward to the Platform Manager

The number of messages a deployment can process per second is measured by the MPS. The speed at
which a log is processed can greatly affect the performance of LogRhythm. This is important to remember
when we begin building custom regex rules later in the course. Poorly written regexes in MPE Rules can
directly impact performance and cause the published MPS to fail.

Chapter 1 | A Review of Log Processing 7

 Unidentified Logs
Unidentified logs are those logs sent through the Message Processing Engine (MPE) that were not
identified when evaluated by the MPE Rules. Unidentified logs can typically be expected after adding a
new Log Source.

If LogRhythm is unable to identify a log, a new custom MPE Rule will need to be created to identify,
process, and parse data from that log. Unidentified logs, or metadata that is not parsed correctly from
logs, is the reason for this course.

Unidentified logs will be displayed in the following locations:

Deployment Monitor

8 Chapter 1 | A Review of Log Processing

Log Volume Report

Logs that have no Classification assigned in the Personal Dashboard

Chapter 1 | A Review of Log Processing 9

 What Is RegEx?

LogRhythm uses regular expression (regex) rules for identifying and processing log messages. RegEx
rules find and parse metadata from log messages. So, what is regex? RegEx is a line of code that defines
a pattern that will be compared against a string of text (any string) and will return “true” if the pattern
matches the string, or will return “false” if it does not match the string.

A regex is:

A flexible way of identifying one or more meaningful portions of text.

A match identification that determines if a string fits a particular format.
A combination of text and regular expression reserved characters.
Recognized, non-proprietary industrial standard used in many systems.

According to Wikipedia: A regular expression, regex or regexp (sometimes called a rational expression)
is, in theoretical computer science and formal language theory, a sequence of characters that define a
search pattern. Usually this pattern is then used by string searching algorithms for "find" or "find and
replace" operations on strings. Regular expressions are used in search engines, search and replace
dialogs of word processors and text editors, in text processing utilities such as sed and AWK and in
lexical analysis. Many programming languages provide regex capabilities, built-in, or via libraries.1

The text book definitions above are correct, but simply put, using regex is like a foreign language. The
more you practice using it, the better, and more fluent you'll become in speaking and writing this language.

This text is written in regular expression:

[aeglru]{7}\s[einsprsx]{10}

In this course, you'll learn to read it and understand that it could be translated to read:

Regular Expression

1https://en.wikipedia.org/wiki/Regular_expression

10 Chapter 1 | A Review of Log Processing

RegEx Resources

Today in class we will utilize some online resources to help you learn about Regex, before we start using
Regex in LogRhythm.

This website provides detailed information about regex, and can provide additional information about
using regex:

http://www.regular-expressions.info/

This website provides a quick reference for regex including symbols, ranges, grouping, assertions and
some sample patterns to get you started.

https://www.cheatography.com/davechild/cheat-sheets/regular-expressions/

This website has a great testing tool that will be used during this training class to test your regex to make
sure it actually matches the intended data:

https://regex101.com/

Notice the panels on the page at regex101.com.

Regular Expression: this is where you'll enter your regex

Test String: this is where you'll enter the strings that your regex needs to match
Explanation: review this panel for a detailed definition of what your regex characters will match
Match Information: review this panel to verify if your regex matches your sting successfully or not

Notice the list of Regex Flags. We will use the global, multi line, and insensitive flags today.

Chapter 1 | A Review of Log Processing 11

 Global: this flag will be used to find all possible matches, not stopping at the first match
Multi line: this flag will be used when attempting to match more than one line of text
Insensitive: this flag is used to ignore the case (upper or lower) of letters

12 Chapter 1 | A Review of Log Processing

RegEx in LogRhythm
RegExes are defined in the MPE Rules. The regex defines the information that belongs in a specified
metadata field.

The regular expression, pictured above, was used to read and parse the metadata in the Log Information
pictured below.

Metadata fields are searchable within LogRhythm and become powerful when used in searches, alarms,
and reports. The metadata captured by an MPE Rule can be user defined, however those custom
metadata fields can not be used as filters in searches, alarms, and reports.

Chapter 1 | A Review of Log Processing 13

 This page intentionally left blank.

14 Custom MPE Rules Using Regular Expression

EXERCISES
Regex Overview
Exercise 1: Review Questions 16
Exercise 2: Writing and Testing Regular Expressions 18

Chapter 1 | A Review of Log Processing 15

 Exercise 1:
Review Questions

The purpose of this exercise is to check your knowledge of the information presented in this chapter.

1. True or False.

A host can have one or more Log Sources.

2. ___________________________ are used to group logs that come from common hardware types, have
the same data format, and have the same processing rules applied.

a. Log Source Types

b. Log Sources

c. MPE Rules

d. Data Processors

3. __________________________ are applied to each log message to parse data.

a. Log Source Types

b. Metadata

c. MPE Rules

d. Data Processors

4. LogRhythm is able to process log messages, deriving and parsing metadata information, with the use
of ___________________________________________________________________________________.

a. the Data Processor

b. regular expression pattern matching, or regexes

c. the MPE Rule Builder

d. Security Intelligence and Analytics

16 Chapter 1 | A Review of Log Processing

Chapter 1 | A Review of Log Processing 17
 Exercise 2:
Writing and Testing Regular Expressions

The purpose of this exercise is to practice using the regular expression test tool. We've not yet learned
how to read or write regular expressions, but we'll be using a test tool throughout the remainder of the day.
This exercise will allow you to explore the tool and learn how it is used.

Part One: Using the RegEx Test Tool

For training purposes, we will use a regex testing tool on a third party web site. LogRhythm does not
endorse this site or any other regex web site. If you know of a regex testing tool that you prefer, please
feel free to use it. However, for this course we will continue to refer you to the site of our choosing.

1. From you personal workstation, open a browser window and navigate to the regex testing tool
located at: regex101.com
2. Copy and paste the following regex into the Regular Expression text box found at that site:
^\d\d\d \w\w\w\w\s\D\D\D\D\D\D, \w\w\w\w\w$
3. Copy and paste the following data string into the Test String text box.
123 Fake Street, 90210
4. Review the panel on the right-side of the window. If the test tool was able to successfully match
the patten of the string, you should see a message indicating a Full Match was made in the Match
Information panel. Your screen should look like this:

a. If you do not see a Full Match of your pattern, you may have copied an extra space, or
missed pasting a character, in one of the strings. Try again until your screen looks like the
image above.
5. Scroll down in the Explanation panel on this website. Review the list of characters and the
description of the characters it will match. Become familiar with the regex characters matched in
this example.
6. Delete the existing text in the Regular Expression field. Paste this regex in the Regular
Expression field:
^\w\w\w\W\w\w\w\w Street, \S.\S.\S$

18 Chapter 1 | A Review of Log Processing

 7. Does this regex produce a full match of the string? _______________________________________
8. If so, why do both regexes match the same string? _______________________________________
________________________________________________________________________________

Chapter 1 | A Review of Log Processing 19

 Part Two: Using the RegEx Test Tool
Review the information in the green box on the top of screen. What does it say about your regex?

1. How many matches were found? ____________________________________________________

2. How many steps did it take to make a match? __________________________________________
3. How long did it take to make a match? ________________________________________________

This information can help verify the speed or performance of your rule. The faster the better. The less
steps the more efficient. Less efficient rules will ultimately impact the performance of the Data Processer
in LogRhythm.

20 Chapter 1 | A Review of Log Processing

CHAPTER 2
Introduction to Regex
Understanding RegEx 22
Literal Characters 23
Positional Characters 24
Using Literals With Positional Characters 26
Matching Characters 27
Character Sets 29
Repetition Characters 31
Wildcard Characters 32
Character Groups 33
Optional Matches 35
Reserved Characters 36
Common Regular Expressions Used In LogRhythm 38
Exercises 39
Exercise 1: Write a RegEx Using Literal Characters 40

Exercise 2: Write a RegEx that does not Match Certain Numbers or

Letters 41

Exercise 3: Match Multiple Strings Using Literal Characters 42

Exercise 4: Write a RegEx Using a Character Set 44

Exercise 5: Using a Character Set with Optional Matches 45

Exercise 6: Challenges: Advanced Regular Expressions 46

Custom MPE Rules Using Regular Expression │ Introduction to Regex 21

 Understanding RegEx
Before developing a Regular Expression, it’s important to understand how the regex engine works to
identify the text within a string.

Regular Expressions are sets of code used to compare a structured format against an actual line of text. It
can determine if the structured format matches the provided string of text.

Regular Expressions use characters to represent patterns of data in a string. In general, regular expres-
sions are written to read the text from left to right. They try to match characters in the text in the order
defined by the pattern matching format. After finding the start of the match, the engine continues to match
until it hits a character that is not matched by the pattern.

The most simple regex pattern consists of a single character. Regex will look at a string until that one
character is matched.

Example
Regex: a
Input String: It's all good all the time.
Text Matched: a The regex stops searching for matches after it finds the first match. It will not
match the second "a" in the string, unless explicitly written to do so.

Example
Regex: all
Input String: It's all good all the time.
Text Matched: all The regex stops searching for matches after it finds the first match. It will not
match the second "all" in the string, unless explicitly written to do so.

Note: Regular Expressions by default are always case sensitive, however LogRhythm has
configured it's engine to ignore case for ease of use. Characters are not case sensitive in the
LogRhythm RegEx Engine.

22 Chapter 2 | Introduction to Regex

Literal Characters
Regular Expressions use characters to represent patterns of data in a string. A literal is a type of
character used when it is expected to be an exact match in the string. These characters are the easiest to
match in a string of text and should be used as often as possible to match characters that are known to be
consistent in a string of text.

All characters used in a Regular Expression, that are not special characters, are interpreted automatically
by regex as literal characters.

Example
The regex “boo” would literally match any 3 letter combination containing the letters “b” “o” “o” in a
row such as:
Boo
boomerang
taboo

Example
The regex “1000” would match any string with “1000” in it, such as:
1000
1000000
01010001000

Chapter 2 | Introduction to Regex 23

 Positional Characters

Positional characters, also referred to as anchors, match a position in the string, not an actual character
like a literal. The positional characters are used to specify where to start and where to end the match.

These two positional characters are sometimes used in LogRhythm:

^ Indicates the left most position in the string; the beginning of a line.
$ Indicates the right most position in the string; the end of a line.

Let's review the same example from before adding the ^ positional character at the beginning of the
regular expression.

Example
A regex of “^boo” would match any 3 letter combination containing the letters “b” “o” “o” in a row
starting at the beginning of a string:
boo
boomerang
taboo (no match; boo is not positioned at the beginning of the string)

24 Chapter 2 | Introduction to Regex

 Example
A regex of “^1000” would match any 3 letter combination containing the numbers “1” “0” “0” “0” in a
row starting at the beginning of a string:
1000
1000000
01010001000 (no match; 1000 is not positioned at the beginning of the string)

Chapter 2 | Introduction to Regex 25

 Using Literals With Positional Characters
Let's review the same example from before adding the ^ positional character at the beginning and the $
positional character to the end of the regular expression. This regex is looking to match the characters
only if they are positioned at the beginning and end of the string.

Example
A regex of “^boo$” would match any 3 letter combination containing the letters “b” “o” “o” in a row
starting at the beginning of a string and ending at the end of the string.
boo
boomerang
taboo

Example
A regex of “^1000$” would match any 3 letter combination containing the numbers “1” “0” “0” “0” in a
row starting at the beginning of a string and ending at the end of the string.
1000
1000000
01010001000

26 Chapter 2 | Introduction to Regex

Matching Characters

While literal characters look to match the exact characters, matching characters, also known as
metacharacters, have special meanings. Matching characters represent other characters.

The matching characters are significant in regex because you likely will not know the exact data string to
be matched, but the will know the pattern of the data string to be matched. Matching characters indicate
specific types of characters to be matched in a string. Matching characters allow for great flexibility in the
pattern of the regex.

Example
The character \w will match all the alpha numeric characters (a through z and 0 through 9) and the
underscore _ character.
Regex: \w
Input String: Hello World!
The regex matched the first occurrence of any alpha numeric character.

Example
Regex: \w\w\w\w\w\s\w
Input String: Hello World!
The regex matched 5 consecutive alpha numeric characters and one space followed by an
additional alpha numeric character.

Note: Using the \s character instead of a literal space character can be helpful in determining
that a space actually exists in the regex. Some font types make it difficult to determine if a space
has been typed or not.

Chapter 2 | Introduction to Regex 27

 Keep in mind, there really are no right or wrong answers when writing regex patterns; use what matches
best, or what you're most comfortable using at your current skill level. As you get more "fluent" in the
regex language, you'll learn better, more efficient, ways to create matching patterns. Notice in the
example shown here, there is more than one possible matching regex for the same string of text.

Example
Regex: ^\d\d\d \w\w\w\w\s\D\D\D\D\D\D, \w\w\w\w\w$
Input string: 123 Fake Street, 90210

Regex: ^\w\w\w\W\w\w\w\w Street, \S.\S.\S$

Note: Because \w matches a huge list of characters, using [a-z0-9] character sets will perform
substantially better. We'll review character sets and performance efficiency later in this course.

28 Chapter 2 | Introduction to Regex

Character Sets
Notation Characters Matched Example

[] Matches any one character between the [abc] matches only a or b or c

brackets (basically an OR statement) and only matches one of those
characters

[^ ] Matches any character except the characters [^abc] matches anything

appearing after the ^ and before the end bracket ] except a, b, or c

[A-D] Matches any one character contained in the [A-Z0-9] matches any one
range of A through D alphanumeric character
Specifying a range of characters can only be [^0-4] matches any one
done in character sets character except 0, 1, 2, 3, or 4

Using a character set, or character class, enables the regex to match only one out of several characters.
Enter the character set between square brackets, and the regex will match one of the specified
characters.

Use the ^ as the first character inside the square brackets to match anything except the characters
specified inside the square brackets.

Use the - to specify a range of characters to be matched.

Example
Regex: gr[ae]y
Will match either the word gray or grey

Chapter 2 | Introduction to Regex 29

 Example
Regex: 0x[a-f0-9][a-f0-9]
Will match a valid two character hex number, 0x00 through 0xFF

Example
Regex: 8030[^5-9]
Will match an area code from 80300 to 80304.
Will not match area codes from 80305 to 80309 because it will match anything except this range

30 Chapter 2 | Introduction to Regex

Repetition Characters
RegEx will only match the left-most character match, or the first pattern match found, unless explicitly
written into the pattern. Repetition characters indicate the number of times an expression should be
matched. Repetition characters, also known as quantifiers or qualifiers, enable the regex to continue to
look for additional pattern matches after the first one was found. Quantifiers specify how many times a
character must be present in the string for a match to be found.

Example
Regex: [al]{3} (Means match 3 "a" and/or "l" letters in a row)
Input String: It's all good all the time.
Text Matched: all The regex stops searching for matches after it finds the first match. It will not
match the second "all" in the string.

Example
Regex: [A-Z]{3} (Means match 3 letters in a row)
Input String: 123 ABC 456 DEF
Text Matched: ABC Regex stops searching for matches at the "space" which is not included in the
pattern. It will not match DEF.

Chapter 2 | Introduction to Regex 31

 Wildcard Characters
The period ., which matches any character (except for new line), and the star *, which matches the
previous item 0 or more times, are often used together like a wildcard pattern. The .* wildcard characters
are often used when the strings have the potential for different text that needs to be matched.

Example
Input string: 123 Fake Street, 90210
Input string: 45678 Fake Street Lane, 80304
Matching regex: ^\d+\s+.*,\s+\d+$

32 Chapter 2 | Introduction to Regex

Character Groups
Character Function

( Start of character group

) End of character group

| Logical OR operator

A set of parentheses ( ), or round brackets, is used to group a portion of regex together. The open paren-
thesis, (, is a reserved character used to represent the start of a group of characters. The close paren-
thesis, ), is a reserved character used to represent the end of a group of characters.

These groups are often used with repetition characters to apply a quantifier to the entire group.

Character groups can be used to:

Match a group of text that repeats one or more times, using the + repetition character

Example
Regex: (\d\d\s\d\d\s\d\d\d\d\s)+

Will match the highlighted portion of this input string:

these dates 04 05 2017 04 06 2017 04 07 2017 are good options

Match an optional group of text zero or one times, using the ? repetition character

Example
Regex: (logrhythm\\)?\w+

Will match all of the following input strings:

logrhythm\jamesh
jamesh
chrisp

Create an OR condition, using the | character

o The | character is only used in Capture Groups; it specifies that either option could be found
in the string and that either option is an acceptable match.

Chapter 2 | Introduction to Regex 33

 Example
Regex: (this|that) book

Will match both of the following input strings:

this book
that book

34 Chapter 2 | Introduction to Regex

Optional Matches
The regex character “?” makes the previous regex character an optional match. The optional character
may or may not exist in the pattern. This is often used to account for words with alternate spellings or
alternate words used to state the same information.

Example
Regex: Flavou?r

Will match both Strings:

Flavour
Flavor

Example
Regex: User (\w+\\)?\w+ logged in

Will match all Strings:

User Pete logged in
User Securityco\Pete logged in
User Securityco\Jim logged in

Chapter 2 | Introduction to Regex 35

 Reserved Characters
Some regex characters are Reserved and have special meaning when processed by the regex engine.
These characters do not take their literal meaning, but rather are Reserved to perform a specific function
in a regex string.

The following are Reserved characters:

Character Reserved Function

\ Escape character

^ Beginning of a line

$ End of a line

. Match ANY character

| Logical OR operator

? Match the previous character (0 or 1 times)

* Match the previous character (0 or more times)

+ Match the previous character (1 or more times)

( Start of character group

) End of character group

[ Start of character class/set

] End of character class/set

Example
Use reserved characters to perform a function
Regex:22$
Will match the characters 22 at the end of a line
My new T-shirt cost $22

36 Chapter 2 | Introduction to Regex

If you need a Reserved character to be used as a Literal character, it must be escaped from its functional
use, by specifying the escape character, \, in front of it.

Example
To match a reserved character, escape it.
Regex: \$22
Will match the $ and 22 as a literal characters
Input String: My new T-shirt cost $22

Example
^10\+200 would match 10+200
^\$1000 would match $1000
^c:\\finance would match c:\finance

Chapter 2 | Introduction to Regex 37

 Common Regular Expressions Used In
LogRhythm
The following regex characters are commonly used in the MPE Rules in LogRhythm. This list is contained
in the MPE Rule Builder Guide and can be used as a reference when building your own regexes.

Characters Function

^ Start of the log (positional)

$ End of the line or log message (positional)

. Matches anything except for characters on a new line

? Makes preceding character optional

(|) OR statement

\w Matches any alphanumeric character (0-9 or A-Z)

\d Matches a single digit

\s Matches a single whitespace

* Matches preceding value zero or more times

+ Matches preceding value one or more times

\ Escapes a reserved character

38 Chapter 2 | Introduction to Regex

EXERCISES
Regex Overview
Exercise 1: Write a RegEx Using Literal Characters 40
Exercise 2: Write a RegEx that does not Match Certain Numbers or Letters 41
Exercise 3: Match Multiple Strings Using Literal Characters 42
Exercise 4: Write a RegEx Using a Character Set 44
Exercise 5: Using a Character Set with Optional Matches 45
Exercise 6: Challenges: Advanced Regular Expressions 46

Chapter 2 | Introduction to Regex 39

 Exercise 1:
Write a RegEx Using Literal Characters

The purpose of this exercise is to practice writing and testing regular expressions.

For training purposes, we will use a regex testing tool on a third party web site. LogRhythm does not
endorse this site or any other regex web site. If you know of a regex testing tool that you prefer, please
feel free to use it. However, for this course we will continue to refer you to the following site:

Regex101.com

Input String: Robert, Janet and Joseph went to the grocery store.

1. Create a Regular Expression that matches the whole string, but only match the comma "," and the
word "and" using literal characters.

Utilize the regex testing website.

Enter the string and work on a regex until you get a complete match.

Hint
There are multiple ways to accomplish this task.

40 Chapter 2 | Introduction to Regex

 Exercise 2:
Write a RegEx that does not Match Certain Numbers or
Letters

The purpose of this exercise is to practice writing and testing regular expressions.

Continue to use the following site: Regex101.com

Input String:

John Q Public, Denver, Colorado, 80213

Regular Expression:

1. Create one Regular Expression that:

a. Matches the whole string but only uses literal characters for the commas
b. Does not match names that have numbers
c. Does not match zip codes with letters

Utilize the regex testing website

Enter the string and work on a regex until you get a complete match for the string
o Make sure you do not get a match if you modify the name in your Test String to contain a
number
o Make sure you do not get a match if you modify the zip code in your Test String to contain a
letter

Hint
There are multiple ways to accomplish this task.
\D matches any non-digit, \d matches any digit

Chapter 2 | Introduction to Regex 41

 Exercise 3:
Match Multiple Strings Using Literal Characters

The purpose of this exercise is to practice writing and testing regular expressions.

Continue to use Regex101.com

Input Strings:

Apple is the word of the day

Banana is the word of the day

is the word of the day

Regular Expression:

1. Create one Regular Expression that matches all three strings but only uses literal characters for the
word “is” and the word “day”; do not use literals for any other characters in the string.

Utilize the regex testing website

o Modify the Regex Options, otherwise your string will not be matched
l Click the flag icon on the right-side of the Regular Expression box

42 Chapter 2 | Introduction to Regex

 l Click to set the multi line and insensitive flags, your Regex Flags should look like
this:

Enter the string and work on a regex until you get a complete match for all three strings

Hint
There are multiple ways to accomplish this task.

Chapter 2 | Introduction to Regex 43

 Exercise 4:
Write a RegEx Using a Character Set

The purpose of this exercise is to practice writing and testing regular expressions.

Continue to use Regex101.com

Input String:

Today we’re learning regular expressions.

Regular Expression:

1. Create a character set that will match only the characters used within the string.

Utilize the regex testing website

Enter the string and work on a regex until you get a complete match for the string

Hint
There are multiple ways to accomplish this task.
Be sure to use the escape character to ensure “.” is recognized as a Literal, instead of it
matching any character.

44 Chapter 2 | Introduction to Regex

 Exercise 5:
Using a Character Set with Optional Matches

The purpose of this exercise is to practice writing and testing regular expressions.

Continue to use Regex101.com

Input Strings:

303-555-1212

(303) 123-LOGS

800-911-HELP

Regular Expression:

1. Create one Regular Expression that:

a. Matches all three formats of telephone numbers
b. Accounts for the fact that the last 4 digits can be either numbers or letters and that
telephone numbers do not use the letters Q or Z

Utilize the regex testing website.

Enter the strings and work on a regex until you get a complete match for all the strings.
o Change one of the letters in a string to a Q or Z. Does it still match? It shouldn't.

Hint
There are multiple ways to accomplish this task.
The ? character will match the previous item once or not at all (Optional).
Verify your Regex Flags are set to account for multiple lines of text.

Chapter 2 | Introduction to Regex 45

 Exercise 6:
Challenges: Advanced Regular Expressions

The purpose of this exercise is to practice writing and testing more advanced regular expressions.
Complete as many Challenges as you are able.

Challenge One:
Input Strings:

Employee ID 0551 entered the building

Employee ID 0021 exited the building

Employee ID 0032 granted access to the building

Regular Expression:

1. Create one regular expression

2. Use a Character Group that matches any of the types of building access noted in the strings.

Hint
Use an OR condition in a Character Group. (this|that|the other)

46 Chapter 2 | Introduction to Regex

Challenge Two:
Input Strings:

Session ID [0xa487c11-1111] started

Session ID [0x33a0a56-75] ended

Session ID [0x77df0011-322] terminated

Regular Expression:

1. Create one Regular Expression that matches all three strings by using the following:

A Character set [ ] with the “^” not operation that matches the Session ID value.
Character groups ( ) to match “started”, “ended” and “terminated” as literal characters.

Hint
Use an OR statement.
Review the Character Set and Character Group topics for more information.

Chapter 2 | Introduction to Regex 47

 Challenge Three:
Input Strings:

(719) 555-1212

1-719-555-1212

+1 719 555-1212

+1 (719) 555-1212 x234

Regular Expression:

1. Create one Regular Expression that:

Matches all the telephone number strings.

Uses Repetition Characters in Character Groups to match all digits.
Accounts for the fact that the area code and first three digits have to be numbers, the last 4 digits
can be either numbers or letters, and that telephone numbers don’t use the letters Q or Z.

Hint
Review the Repetition Characters and Character Groups topic for more information.

48 Chapter 2 | Introduction to Regex

Challenge Four:
Input Strings:

Package 123456789 has arrived

Package 12345678901 (special order) has arrived

Package 123456789012 has arrived

Regular Expression:

1. Create one Regular Expression that:

Matches all of the strings

Matches all of the text that is identical in each string with Literal characters
Uses Repetition characters to match only package numbers between 9 and 12 characters in
length
o Add a few more digits to the last package number string. Does your regex find a match? It
should not.

Hint
Review the Repetition characters topic for more information about matching a specific number of
characters.
Use a Character Group inside a Character Group to handle the “special order” condition.
Use ? to ignore the presence of the “special order” substring by putting it in a Character Group.

Chapter 2 | Introduction to Regex 49

 Challenge Five:
Input Strings:

2/16/2009 2:13 AM

06 22 2008 17:22:28 10.6.93.99

Regular Expression:

1. Create Regular Expressions:

Create two separate regular expressions; one to match each string.

Match the date, time, and IP Address field as applicable.
For the date and time fields, use Repetition Characters to allow for 1 – 2 characters.
For the IP Address, use Repetition Characters to allow each octet to range from 1 – 3 characters.

Hint
Review the Repetition Characters topic for more information about matching a specific number
of characters.

50 Chapter 2 | Introduction to Regex

Chapter 2 | Introduction to Regex 51
 This page intentionally left blank.

52 Custom MPE Rules Using Regular Expression

CHAPTER 3
RegEx in LogRhythm
Using RegEx in LogRhythm 54
Greedy vs. Non-Greedy Quantifiers 55
Capture Groups 58
Using Named Capture Groups to Parse Metadata 60
Overloading Tags in LogRhythm 63

Exercises 65
Exercise 1: Reading Regular Expressions 66

Exercise 2: LogRhythm MPE Rule Builder Parsing Guide 67

Exercise 3: Identify Metadata in a Log Message 68

Exercise 4: Create an Overloaded Tag 70

Exercise 5: Create an Overloaded Tag 71

Custom MPE Rules Using Regular Expression │ RegEx in LogRhythm 53

 Using RegEx in LogRhythm

In this chapter we'll build upon our basic knowledge of writing regular expressions. We'll introduce
additional regex characters, learn how to improve the efficiency of regular expressions, how to capture
information into metadata fields, and begin to match patterns contained in log messages.

In essence, we'll learn how LogRhythm is able to quickly process the log messages into usable metadata
fields.

54 Chapter 3 | RegEx in LogRhythm

Greedy vs. Non-Greedy Quantifiers
Regular Expressions will stop looking for matches as soon as the first match is found, even if a better
match can be found later in the text string, unless explicitly written into the pattern. Quantifiers are written
into the regex pattern so that it continues looking for additional matches.

Quantifiers, sometimes referred to as qualifiers, are used to specify matching the preceding character
zero or more times. Quantifiers specify how many times a character must be present in the string for a
match to be found.

The *, +, ?, {, and } characters are interpreted as quantifiers, unless they are included in a character
group.

There are two types of quantifiers used in regex to find additional matches:

Greedy quantifiers: will stop searching only after the maximum number of instances of the
pattern is satisfied. It wants to find as many pattern matches as possible.
o The two repetition characters * and + are, by default, greedy because they inherently specify
an unlimited number of matches.
l The * can be too "hungry" for more data and cause a rule to become less efficient. You
may need to tell it to stop searching for matches using the non-greedy quantifier.
Non-greedy quantifiers: (also referred to as lazy) will stop searching as soon as the minimum
number of instances is satisfied. It wants to find as few matches as possible. A ? is added to the
greedy quantifier to limit the number of matches to the minimum, in effect making it non-greedy.
The ? tells it to stop after the minimum number of matches is found.
o The non-greedy quantifier is generally only a concern for the regex character . which will
match anything.
o You'll almost always use the non-greedy quantifiers .*? or .+? instead of just .* or .+ .

Greedy Characters Non-Greedy Function

* *? Match the previous item 0 or more times

+ +? Match the previous item 1 or more times

? ?? Match the previous item 0 or 1 times

{n} {n}? Match exactly n times

{n,} {n,}? Match at least n times

{n,m} {n,m}? Match from n to m times

Chapter 3 | RegEx in LogRhythm 55

 Example
Greedy Regex: ^.*
Input String: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.jonas user
account locked out

Everything matches (greedy)

^ position at start of string
.*? matches any character between zero and unlimited times, as many times as possible

Example
Non-Greedy Regex: ^.*?
Input String: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.jonas user
account locked out

The minimum number of matches is satisfied with the first character

^ position at start of string
.*? matches any character between zero and unlimited times, as few times as possible

Example
Greedy Regex: ^.*user\s
Input String: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.jonas user
account locked out

^ position at start of string

.* matches any character 0 to unlimited times, as many times as possible followed by the literal
characters u s e r; it continues to look for the same pattern over and over again, which is why it
matched the second occurrence of the word "user"
\s matches any whitespace

Example
Non-Greedy Regex: ^.*?user\s
Input String: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.jonas user
account locked out

^ position at start of string

.*? matches any character 0 to unlimited times, as few times as possible; it stops searching for
pattern matches after it matches once
user matches the literal characters u s e r
\s matches any whitespace

56 Chapter 3 | RegEx in LogRhythm

 Example
Greedy Regex: ^\d+\/\d+\/\d+\s+\d+:\d+:\d+\s+MsgID=\d+
Input String: 02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user joe.jonas user
account locked out

In this example it is appropriate to use the greedy qualifier because the non-greedy option would not
have matched the entire ID number

^ position at start of string

\d+ matches digits 0-9 unlimited number of times
\/ matches / as a literal character ( the reserved character is escaped using the \)
\d+ matches digits 0-9 unlimited number of times
\/ matches / as a literal character
\d+ matches digits 0-9 unlimited number of times
\s+ matches whitespace unlimited number of times
\d+ matches digits 0-9 unlimted number of times
: matches the literal character :
\d+ matches digits 0-9 unlimited number of times
\s+ matches whitespace unlimited number of times
MsgID= matches the literal characters m s g i d = (regex in LogRhythm is case insensitive)
\d+ matches digits 0-9 unlimited number of times

Chapter 3 | RegEx in LogRhythm 57

 Capture Groups
In the previous chapter we were introduced to character groups. Using the parentheses ( ) in a regular
expression, characters are grouped together allowing you to apply operations to the entire set. In this topic
we'll build on that foundation.

When information is grouped, regex will automatically capture that information in memory for reuse.
These are called capture groups.

Example

58 Chapter 3 | RegEx in LogRhythm

Named Capture Group
Capture groups can also be assigned names, these are called named capture groups. Using the
following format, you can access the data matched by the regex by its given capture group name:
(?<name>regex)

Example

Example
The following regex uses a named capture group to extract metadata from a log message.

In this regex:
EVID: indicates a literal match of characters in the string.
The ? character at the beginning of the group says to name the group as specified by the information
contained inside the angle brackets, <vmid>, and regex uses these characters \d+ to match the
text contained in the string.

This is how LogRhythm uses named capture groups to parse matching data contained in log messages
into metadata fields.

Chapter 3 | RegEx in LogRhythm 59

 Using Named Capture Groups to Parse
Metadata
The named capture groups used in LogRhythm are the metadata fields. LogRhythm has written standard
regular expressions for each metadata field to match the information found in log messages. These
standard, or default, regexes are referred to as tags. A tag is simply a label used to identify, or specify,
information.

The LogRhythm tags are responsible for putting the information matched in the string into its associated
metadata field. LogRhythm parses information from logs using associated tags in regular expressions.

Pictured below is a sample of metadata fields, their associated tags, and the default regex used to match
data. These tables can be found in the LogRhythm MPE Rule Builder Parsing Guide.

Example
The tag <login> uses the default regex value \w+ to parse information into the User (Origin)
metadata field.
The tag <sender> uses the regex value [^\s]+@[^\s]+ to parse information into the Sender
metadata field.

60 Chapter 3 | RegEx in LogRhythm

Refer to the LogRhythm MPE Rule Builder Parsing Guide, located in LogRhythm's Community site for a
complete list of metadata fields, their associated tags, and default regular expressions.

Note: All mapping and parsing tags are lower case.

These fields are available in all product features beginning with the release of LogRhythm v7.3.

The fields in the tables in the LogRhythm MPE Rule Builder Parsing Guide are grouped according to their
appearance in the Web Console. If you do not see a field in the Web Console under the same tab as the
LogRhythm MPE Rule Builder Parsing Guide, you may have flagged the field as a favorite, in which case
the field will appear in the Favorites tab instead.

Chapter 3 | RegEx in LogRhythm 61

 Using Named Capture Groups to Parse Metadata: Example
Because of the default regex association, using tags makes writing regular expression patterns in
LogRhythm simple. When writing regex in LogRhythm you can leave out the group characters used to
match the string, (? and the regex characters), because LogRhythm already knows the default regex
associated to the metadata tag.

Example
The image below is taken from LogRhythm. Using this simple regex, EVID:<vmid>, LogRhythm is
able to match the 0044 ID number and capture it into the VMID metadata field.

62 Chapter 3 | RegEx in LogRhythm

Overloading Tags in LogRhythm
Instead of using the default regex LogRhythm has assigned for a tag, if necessary, you can specify
different regex characters to use for that tag; this is referred to, in LogRhythm, as overloading a tag.
Overloading gives you more control over the regex and allows you to explicitly define the regex used to
parse data into the metadata field. You might think of overloading as replacing regexes.

An overloaded tag uses the standard regex format for Named Capture Groups: (?<name>regex).

The ? at the beginning of an overloaded tag, in LogRhythm, says to replace the default regex for the
<tag>, and use these specified regex characters instead, to match the text in the string.

Overloading Example
By default in LogRhythm, the <login> tag uses the value \w+ .

This tag would not match the login of James.Hendrix. James might be parsed, and Hendrix might be
parsed, but not the . character. This would lead to incorrect login information parsed from log messages.
Because you can’t predict every variation of a login name, you can overload the tag to match different
login formats:

JamesHendrix
JHendrix
JamesH
James.Hendrix

Overloading the regex syntax for a tag would look like this: (?<login>.*?). Which says, instead of
using the default regex, \w+, use the regex .*? to match characters for the <login> value. The .*? will
match any character zero or more times and then stop at the end of the pattern; remember it is non-
greedy.

When to Replace Default RegExes

You can replace the default regex if the source data does not conform to the default pattern. You only
need to replace the default regex when it:

will not properly parse the correct data out of the log message.
is not the optimal regex from a performance perspective.

To replace the default regex, the following syntax should be used:

(?<tag_name>regex)

Chapter 3 | RegEx in LogRhythm 63

 Example
Suppose your regex needs to match file names with a specific extension such as this sample log:
User john.doe opened AnnualReport.pdf

If the regex was written as:

User <login> opened <object>

The value parsed for login would be john and the value for object would be AnnualReport. The
period is not a word character and the default regex of \w+ would only match up to the period. This
default regex should be replaced and the new regex should be:

User (?<login>\w+\.?\w*) opened (?<object>\w+\.pdf)

Now, the regex will parse anything for login starting with a word character that optionally contains a
period followed by additional characters; and will parse anything for object starting with a word and
the literal .pdf characters.

Important!
Do not ever overload these tags in LogRhythm:
<sip>, <dip>, <snatip>, or <dnatip>
These tags use explicitly written regex patterns to specifically match IPv4 and IPV6 addresses.

64 Chapter 3 | RegEx in LogRhythm

EXERCISES
Introduction
Exercise 1: Reading Regular Expressions 66
Exercise 2: LogRhythm MPE Rule Builder Parsing Guide 67
Exercise 3: Identify Metadata in a Log Message 68
Exercise 4: Create an Overloaded Tag 70
Exercise 5: Create an Overloaded Tag 71

Chapter 3 | RegEx in LogRhythm 65

 Exercise 1:
Reading Regular Expressions

The purpose of this exercise is to practice reading a regular expression to better understand what it will
match.

Answer the questions below about the following Regular Expression:

User\s(?<login>.*?)\s

As applied to the following input strings taken from log messages:

User johnw authenticated

User domain\peter has logged on

User eddie@domain.logrhythm.com has been authenticated

1. This portion of the regex, (?<login>.*?), is known as a:

___________________________________________________________________________

2. Would the regex match the user name contained in each of the logs listed above?

3. Would this regex match the entire string of data contained in each of the logs listed above?
________________________________________________________________________________

66 Chapter 3 | RegEx in LogRhythm

 Exercise 2:
LogRhythm MPE Rule Builder Parsing Guide

Part One: Review the Rule Builder Parsing Guide

The purpose of this exercise is to familiarize yourself with the usage of the LogRhythm MPE Rule Builder
Parsing Guide.

1. Navigate to LogRhythm's Learning Management System (LMS).

a. Locate the Training Materials section for this course.
b. Click to open the LogRhythm-MPE-RuleBuilderGuide PDF document.
c. Save this document to your Desktop. We'll use it again in future Exercises.
2. Where else could you find this document? ______________________________________________

3. Review the document. On what page does the Regular Expression Characters and Practices
section begin?____________________________________________________________________

4. What does the Display Field contain? ________________________________________________

_____________________________________________________________

5. Locate the Subject metadata field in the Display Field. What is the Description?______________
________________________________________________________________________________

6. What is the Default Regex of the Subject Display Field, and what pattern will that match? ______
________________________________________________________________________________
_______________________________________________________________________________

Part Two: Use the Rule Builder Parsing Guide

Suppose you need to parse data from a log message into the following metadata fields:

Object __________________________________________________________________________
Object Name _____________________________________________________________________
IP Address (Origin) ________________________________________________________________
Hostname (Impacted) ______________________________________________________________
User (Origin) _____________________________________________________________________

What are the associated tagsfor each?

Chapter 3 | RegEx in LogRhythm 67

 Exercise 3:
Identify Metadata in a Log Message

Part One: Parsing Information

The purpose of this exercise is to practice making decisions about parsing data from logs.

Review the following log message:

06 08 2017 15:39:12 192.168.99.10 <LOC0:INFO> Jun 8 15:39:12 192.168.99.129

EVID:0047 172.10.1.15 : Group created, user: User008 Group: Group006 From:
192.168.1.23

1. What information can you identify in this log, that you would probably want to be parsed? ________
________________________________________________________________________________

2. How would you know which IP address is the origin of activity and which is the impacted
IP address?
________________________________________________________________________________
________________________________________________________________________________

3. The following metadata was parsed from the log; however the information highlighted in the image
is not included in the log message.

How was LogRhythm able to parse that data?___________________________________________

68 Chapter 3 | RegEx in LogRhythm

Part Two: Use the Rule Builder Parsing Guide
Review the following log message:

06 09 2017 08:07:07 192.168.99.10 <LOC0:INFO> Jun 9 08:07:07 192.168.99.142

EVID:0011 172.10.1.20: failed object access for user User023 from
192.168.1.26. Object = c:\Program Files\ConfigFile021.ini Session ID:
09fbd070-d4cd-40bd-a92a-6167ebac69fd

The following data was parsed from the log:

0011
192.168.1.26
172.10.1.20
User023
c:\Program Files\ConfigFile021.ini
09fbd070-d4cd-40bd-a92a-6167ebac69fd

1. What are the associated metadata fields for the data listed above? _________________________
________________________________________________________________________________

2. What tags would have been used in the regular expression to parse the data into the metadata
fields? __________________________________________________________________________

Chapter 3 | RegEx in LogRhythm 69

 Exercise 4:
Create an Overloaded Tag

The purpose of this exercise is to practice writing regex for use in LogRhythm.

Review the following log message:

06 09 2017 11:14:54 192.168.99.10 <LOC0:INFO> Jun 9 11:14:54 192.168.99.135

EVID:0045 172.10.1.10 : Account added to group, account: Julia.Louis-Dryfus
By user: User014 Group: Group005 From: 192.168.1.13

1. What Tag would be used to parse the Account's name from this log? ________________________

2. What is the Default Regex used by LogRhythm for that tag? ______________________________

3. What does that regex say, or what characters will it match?

4. Will this regex match and capture the entire Account name in the metadata field?_______________

5. Write an example of an Overloaded Tag that would be used to capture Account names in the
format found in the log above.________________________________________________________

6. What other tags might have also been used in the regular expression to parse the data into the
metadata fields? __________________________________________________________________
________________________________________________________________________________

70 Chapter 3 | RegEx in LogRhythm

 Exercise 5:
Create an Overloaded Tag

The purpose of this exercise is to practice writing regex for use in LogRhythm.

Review the following log message:

06 25 2017 15:18:04 1.1.1.1 <LOC0:INFO> Jun 25 15:18:04 1.1.1.13 EVID:0044

Email tracking: Sender: User013@DHost1 Recipient: User003@DHost3 Subject:
Don't ask me how we can do this!! It's Magic!!!!

1. What Tag would be used to parse the Sender's name from this log? ________________________

2. What is the Default Regex used by LogRhythm for that tag? ______________________________

3. What does that regex say, or what characters will it match?

4. Will this regex match and capture the entire Sender name in the metadata field?_______________

5. What tag would be used to parse the Subject from this log? _______________________________

6. What is the default regex associated with that tag?

7. Will the default regex for that tag match and capture the entire Subject text found in the log
message?
________________________________________________________________________________

8. Write an example of an Overloaded Tag that would be used to capture the Subject text in the
format found in the log above.________________________________________________________

9. What other tags might also be used in a regular expression to parse data into metadata fields?
________________________________________________________________________________
________________________________________________________________________________

Would these tags need to be overloaded or would the default regex parse the data correctly?
________________________________________________________________________________

Chapter 3 | RegEx in LogRhythm 71

 This page intentionally left blank.

72 Custom MPE Rules Using Regular Expression

CHAPTER 4
Creating Custom MPE Rules
The MPE Rule Builder 74
General Settings 76

Processing Settings 78

Default Policy Settings 80

Log Message Source Type Associations 81

Base-Rule Regular Expressions 83

Catch-All Rules 84

Test Center 86
General RegEx Tips 88
Writing RegExes For Use In LogRhythm 89
Considerations For MPE Rule Creation 92
Steps To Create New Base-Rules 93
Walk Through Example Part One: Creating A Base-Rule 95
Exercises 105
Exercise 1: Access the LogRhythm Training Environment 106

Exercise 2: Create a New MPE Rule 110

Exercise 3: Create Another New MPE Rule 120

Exercise 4: Create Another New MPE Rule 126

Exercise 5: Challenge: Create a New MPE Rule with less instruction 131

Exercise 6: Challenge: Create Another New MPE Rule 133

Custom MPE Rules Using Regular Expression │ Creating Custom MPE Rules 73
 The MPE Rule Builder

In the previous chapters we learned how to read and write regular expressions because that is how
LogRhythm processes logs. LogRhythm's Message Processing Engine (MPE) processes each log
message using the regexes specified in the many MPE Rules written for many different Log Source
Types.

In the next few chapters we will learn how to create regular expressions for use in LogRhythm. MPE
Rules can be viewed and created in the MPE Rule Builder tool. The LogRhythm engineering team uses
this tool to develop MPE Rules available out-of-the box, also known as "System" rules. LogRhythm
administrators can use this same tool to develop and deploy "Custom" MPE Rules. LogRhythm analysts
can also use the MPE Rule Builder tool to develop new MPE Rules, however analysts can not deploy
their rules into production.

In this course, you will use the MPE Rule Builder to view, create, and edit custom MPE Rules. The MPE
Rule Builder is located in the Client Console under the Tools menu by selecting Knowledge > MPE Rule
Builder.

74 Chapter 4 | Creating Custom MPE Rules

In this chapter we'll review the following elements that comprise LogRhythm's MPE Rule Builder:

General Settings
Processing Settings
Default Policy Settings
Log Message Source Type Associations
Base-rule Regular Expressions
The Test Center

Chapter 4 | Creating Custom MPE Rules 75

 General Settings

The General settings area in the MPE Rule Builder is where you'll begin. Enter a Rule Name and select a
Common Event for your new custom MPE Rule.

Rule Name
When naming a rule, follow these accepted best practices:

Use the VMID in the rule name. When the matching log message contains a vendor message ID
such as an event ID in Windows Event Logs, it is good to include the ID in the name of the rule.
This makes searching for the rule easier and also makes the rule more descriptive of the log that it
matches.

Use the service in the rule name. If the rule matches a log from a system that generates logs for
a wide variety of services, such as the Windows Application Event Log, the service that generated
the log message should be included in the rule name.

All rule names should contain a brief description of the action described by the log. For
example: EVID 528 : Failed Authentication : Bad Username or Password

Common Event
MPE Rules are assigned a Classification and Common Event that describes the activity contained in the
associated log messages. Using the Common Event Manager found in the MPE Rule Builder tool,
you can view the complete list of more than 29,000 Common Events. When creating custom MPE Rules,
use your best judgment to select a Common Event that matches the description of the activity occurring
in your log messages.

76 Chapter 4 | Creating Custom MPE Rules

It is good practice to select an appropriate Classification first. Use your best judgment to determine if the
activity in the log describes an Audit, Security, or Operations type event. Then select one of the Classi-
fications under that category. This will narrow the list of Common Events, making it easier to select one
that best matches the activity described in your log. You can also search for keywords under All Items to
help narrow your selection of Common Events.

Important!
Do not create new Common Events. LogRhythm recommends using one that exists in the list.

Rule Status
There are three statuses used in the MPE Rule Builder:

Development: While creating new rules, they should be set to Development status
Test: This allows the rule to be applied to log messages by the MPE for testing purposes, but:
o The logs that match this rule will not show in the Personal Dashboard
o Alarms will not be triggered based on logs that match the MPE Rule
o The MPE Rule will be automatically disabled by the MPE after one of the following
conditions is met:
l The MPE Rule was successfully tested against 1000 log messages
l The MPE experiences performance issues while processing logs against the rule
l Review the scmedsvr.log file for messages about performance issues
Production: This allows the rule to be applied to log messages by the MPE in LogRhythm
o The logs that match the MPE Rule can be displayed in the Personal Dashboard
o Custom MPE Rules can be edited, even in Production mode; however System MPE Rules
(created by LogRhythm) can not be edited

Chapter 4 | Creating Custom MPE Rules 77

 Processing Settings
The Processing Settings are used by the MPE to aid in the interpretation of the regex rule. The first few
lines in the Processing Settings tab contain check-boxes to enable additional options for the regex rules.

You might recall seeing some of these same "flags" used, at the regex101.com web site, to specify
additional regex options for reading multiline strings and case sensitivity.

The last few lines in the Processing Settings tab contain drop-down boxes to aid in identifying the host or
application found in a log. These options assist in determining the Source and Destination context. There
are a couple of options in each drop-down box that help determine if tags are used in the regex and how
the tags are handled.

78 Chapter 4 | Creating Custom MPE Rules

The following table should be used as a guideline when configuring the Processing Settings in the
MPE Rule Builder tool.

Note: The stored value for the host fields are often derived from values parsed from IP address
and hostname tags. For example, the value parsed for the Source IP tag is stored in the Host
(Origin) field , and the value parsed for the Destination IP tag is stored in Host (Impacted) field.
However, system logs do not consistently store the origin/source or impacted/destination values
in the same respective parsing field, as is often the case for network devices reporting on
network flow data.

Chapter 4 | Creating Custom MPE Rules 79

 Default Policy Settings

The Policy Settings are used by the MPE to aid in data management. The options in the Default Policy
Settings tab in the MPE Rule Builder are not selected by default. These options can be selected here,
however, these settings are almost always overwritten by the settings in the Log Processing Policies
tab in the Deployment Manager in LogRhythm. We will configure these settings using the MPE Policy
Rule Editor instead.

80 Chapter 4 | Creating Custom MPE Rules

Log Message Source Type Associations

LogRhythm supports, and has already created, Log Source Types for hundreds of Log Sources. Log
Source Types identify the device or application from which these logs are generated. When creating a
new custom MPE Rules, you must associate a Log Source Type with your new MPE Rule. Selecting the
correct Log Source Type is critical when adding new MPE Rules.

Use the scroll bar to review the Log Message Source Type Associations list. The list of Log Source
Types is divided by log collection method and device type. You'll see entries for the following types of
logs: API, Flat File, J-Flow, LogRhythm, Microsoft Event, MS Windows, Netflow, OPSEC LEA, SNMP
Trap, Syslog, UDLA, VLS, and more.

When you locate a valid Log Source Type that correctly matches your log collection method, click to
place a checkmark inside the box to associate it your new MPE Rule.

Chapter 4 | Creating Custom MPE Rules 81

 If a device or application is custom to your organization or very newly released, you may need to create a
new Custom Log Source Type before you can create new MPE Rules for it. See Chapter 5 for more
information about creating new Log Source Types.

82 Chapter 4 | Creating Custom MPE Rules

Base-Rule Regular Expressions

In the MPE Rule Builder, the Base-rule Regular Expression tab is used to enter the regex for your new
custom MPE Rule. In general, Base-rules identify log messages by matching the metadata fields to a
specific log format or pattern. Base-rules contain a tagged regular expression used to identify the pattern
of a log and isolate interesting pieces of metadata. Using tags, these data strings can be directed to
metadata fields.

The image above shows a Base-rule regular expression. The LogRhythm tags used for the named capture
groups are automatically highlighted in red.

Scratch Pad
The Scratch Pad tab can be used while writing new regular expressions. Use it to copy and paste parts of
your regex, or another similar regex you'd like to borrow from, before entering it in the Base-rule Regular
Expression tab.

Chapter 4 | Creating Custom MPE Rules 83

 Catch-All Rules
Catch-all rules are a significant function of the Base-rule. Catch-all rules are Base-rules used to parse
any logs that may not match any of the other rules written for that Log Source Type. Catch-all rules are
used to parse very common pieces of information found in every log from that Log Source Type. Catch-all
rules may be able to:

Identify the log message type.

Classify the log message type (Information / Warning / Error).

Catch-all rules can be created to process log messages, which you have not identified yet, that could
possibly be received from this same Log Source. Perhaps the log is in a different format, or contains
different message verbiage; the Catch-all rule will still be able to identify the log and parse some common
metadata.

If the vendor creates a new ID or Message in their logs, the Catch-all would be able to identify the log, and
you could then create a new rule to capture that new piece of metadata created by the vendor. You should
monitor for the occurrence of Catch-all rules and can create new rules to parse important information, as
needed. Catch-all rules will be named as such, by default, and you should use the same format when you
create new Catch-all rules to ensure consistency in reporting and searching:

Catch All : NAME_HERE

Note: Catch-all rules created by LogRhythm often contain Levels 1-4 in the rule name. These
Levels are no longer used; however, you will still see some of the original Catch-all rules with
that naming convention.

84 Chapter 4 | Creating Custom MPE Rules

Pictured below is a Catch-all rule for the Log Source Type, MS Event Log for XP/2000/2003 - Applic-
ation.

Example
For this Windows Event Log message:
5/11/2010 1:35 PM TYPE=Information USER= COMP=XENON SORC=Windows Update
Agent CATG=Installation EVID=18 MESG=Installation Ready

The Catch-all rule will parse the TYPE, USER, COMP, SORC, and EVID from this log. The MESG
verbiage contained in this log does not match any of the options contained in the regex rule, so a
new rule would need to be created to collect the MESG information from this log message.

Chapter 4 | Creating Custom MPE Rules 85

 Test Center

The Test Center is key to building new MPE Rules. You'll use the Test Center tab to verify your regular
expression successfully parses data from your log samples. The picture, shown above, illustrates a regex
that was tested to successfully parse the VMID and VendorInfo metadata from a log message.

The first step is to import log samples from your desired Log Source Type into the Test Center. Logs can
be imported into the Test Center a few different ways. In this course, we'll practice importing log message
samples from within the LogRhythm Log Viewer, but you can also import logs manually. Right-click in
the Test Center tab to see the menu, then select the desired import method.

Select Import Log Messages Manually to copy and paste log messages into the Test Center.
Select Import LogRhythm Log Export (llx) File to import logs in an LLX file format.
From this menu, you can also select to export log messages and clear unwanted log messages
from the Test Center panel.

86 Chapter 4 | Creating Custom MPE Rules

To use the Test Center tab to verify your regular expression successfully parses data from your log
samples, use the buttons in the Test Center. Click Test Selected log samples, or Test All log samples.
The results of test will be displayed in a pop-up window:

Compare the number of logs processed with the number of logs that matched your regex rule. Make sure
the average number of logs processed per second generates a favorable rating (Fantastic or Great) for
best MPE performance. If you notice rules with an average rating of less than Great, you may need to
tune your regex to increase it's performance; slow rules will decrease the number of messages
LogRhythm's Mediator can process per second (MPS).

Chapter 4 | Creating Custom MPE Rules 87

 General RegEx Tips
Consider the following tips, when creating new regular expressions for use in LogRhythm MPE Rules:

If you have a reliable Literal character match that is common in all logs, start with that.
o Match Literal characters on each side of the information contained in tags.

Example
l
Instead of: ^.*?EVID:<vmid>
Use: EVID:<vmid>

Skip over “unimportant” information in the log using the non-greedy characters: .*?
Stop matching after all your important information has been captured.
o You don’t need to match the entire log all the way to the end.
Match the tags with the log data a little bit at a time, test it, and review the metadata fields to see
what has been captured, then continue to add more to your regex.

88 Chapter 4 | Creating Custom MPE Rules

Writing RegExes For Use In LogRhythm
In this section we'll review a sample log and illustrate how you would attempt to parse metadata from logs
for use in LogRhythm.

Suppose you've noticed the following log message showing in a report as an Unidentified log or a log
which has no Classification assigned in LogRhythm.

10/11/2008 08:15:00.0154 EVID:4140 – A ‘catastrophic failure’ has occurred.

Your network is hosed.

You did some research and determine that the log contains a new error code recently created by the
vendor of your firewall device. You realize that because this is a brand new EVID code, it does not yet
have an associated MPE Rule in LogRhythm.

You called LogRhythm Support and asked about it, but have been told a ticket has been submitted to the
Labs team for creation of a new MPE Rule but there is no estimated time of completion. Your manager
says you need to have this type of log identified by LogRhythm today. You'll need to create your own
MPE Rule.

You begin by importing this log message into the MPE Rule Builder tool.

To create a regex for use in LogRhythm, you need to begin by thinking in terms of metadata fields. You
know you want to match the new 4 digit EVID code and parse it into a metadata field. You'll use the
LogRhythm MPE Rule Builder Guide to locate the associated metadata field display name, tag, and
default regex characters. You download this document from LogRhythm's Community, under the Other
tab on the Product documentation & documents page.

Chapter 4 | Creating Custom MPE Rules 89

 You see Vendor Message ID listed in the Display Field, <vmid> is the Tag which uses the Default
Regex \w+ to match the text for the EVID number.

Since LogRhythm has already created the Default Regex and associated Tags, it makes writing regexes
for use in MPE Rules easy. You only have to specify where in the string the regex will find matching data
and then enter the tag.

You review the log message again:

10/11/2008 08:15:00.0154 EVID:4140 – A ‘catastrophic failure’ has occurred.

Your network is hosed.

You note that the Vendor Message ID number appears after the text string "EVID:" and there is a space
where the number ends. The tag, <vmid>, contains the Default Regex which will match the Vendor
Message ID and parse it into the specified metadata field. The Base-rule Regular Expression for your new
MPE Rule will look like this:

EVID:<vmid>

You Test the selected log message and notice the ID number is successfully parsed into the VMID
metadata field.

90 Chapter 4 | Creating Custom MPE Rules

Since you also want to capture the verbiage in the error message, you use the Rule Builder Guide again to
locate the appropriate Display Field and associated tag:

You review the log message again:

10/11/2008 08:15:00.0154 EVID:4140 – A ‘catastrophic failure’ has occurred.

Your network is hosed.

You notice there is a space preceding the error text, but you also notice the periods, apostrophes, and
spaces contained within the text. The default regex, \w+, will only match word characters. You will need
to overload the second tag in your regex to account for the additional characters that need to be matched
in the string.

You work on the second half of the Base-rule Regular Expression to specify where in the string the regex
will find matching text and enter the second tag. Your Regular Expression now looks like this:

EVID:<vmid>\s(?<vendorinfo>.*?)$

When you Test the selected log message the VMID number and VendorInfo are both successfully parsed
into their respective fields.

Chapter 4 | Creating Custom MPE Rules 91

 Considerations For MPE Rule Creation
When creating a Custom MPE Rule, there are a few considerations to ensure you create it correctly. The
following can be used as a reference.

1. Research your custom Log Source

a. Log collection method
b. Log message formats
c. Log and severity levels
d. Determine what Events and Alarms are needed for the log messages
2. Verify a Log Source Type exists
a. Review the Log Source Type Manager for a list of existing Log Source Types
b. Assign the new MPE Rule to a valid Log Source Type or, if one does not already exist,
create a new Custom Log Source Type (we'll cover this in Chapter 5)
i. Assign the Log Format that correctly matches the log collection method
i. Syslog
ii. Text File
3. Create your new Custom MPE Rule
a. Create a new MPE Rule for your custom Log Source
i. Name the rule something specific
ii. Assign a Common Event from the list of over 29,000 available options
b. Create a Base-rule and test your regex for successful matching results
i. Create Sub-rules and test for successful matching results
i. Assign a Common Event for each Sub-rule
c. Create a Catch-all rule for unknown logs that might be generated by this Log Source

92 Chapter 4 | Creating Custom MPE Rules

Steps To Create New Base-Rules
Below is an outline of steps required for new MPE Base-rule building (we'll review and then practice these
steps during the remainder of this course):

1. In the LogRhythm Client Console, navigate to the Tools menu, select Knowledge, then MPE
Rule Builder. The MPE Rule Builder window will open.
2. Click the New button. The Rule Builder appears.

3. Ensure you are in the General tab in the top-left panel and assign a Rule Name.
4. Click the Common Event Manager button to associate the new Rule with a Common Event.
5. Click the Rule Status you want. You should start with Development and change to Production
when ready to go live with your new rule.
6. Add a brief Description for the rule, if desired.
7. Click the Processing Settings tab and complete the fields, if necessary.

Chapter 4 | Creating Custom MPE Rules 93

 8. In the Log Message Source Type Associations section in the top-right panel scroll through the
folder tree to select a Log Source Type.

9. In the center of the Rule Builder window, enter the Base-rule Regular Expression to determine
what items are parsed from the log message.
10. Use the Test Center tab to test the Regular Expression on log samples.
11. Save your new MPE Rule.

94 Chapter 4 | Creating Custom MPE Rules

Walk Through Example Part One: Creating
A Base-Rule

Let's walk through creating a new Base-rule for this sample log.

02 20 2011 00:00:30 1.1.1.1 <LOC0:INFO> Feb 20 00:00:51 1,2011/02/20

00:00:51,0002C100617,TRAFFIC,end,33,2011/02/20
00:00:45,1.6.5.5,1.1.1.4,0.0.0.0,0.0.0.0,Inbound_vWire,,,ping,vsys1,un-
trust,trust,ethernet1/1,ethernet1/2,LogRhythmForwarder,2011/02/20
00:00:51,176828,1,0,0,0,0,0x0,icmp,allow,120,120,120,2,2011/02/20
00:00:37,6,any,0

Open the MPE Rule Builder tool.

Click to create a new rule.

Enter a Rule Name.

Assign an appropriate Classification and Common Event.

In this example, I know this log was generated by a Palo Alto firewall device, and the log above
contains a description of network traffic.
o So I selected the Classification of Network Traffic.

Chapter 4 | Creating Custom MPE Rules 95

 o I filtered the list of Common Events by the word general and found a match that seems to
match the activity in this log appropriately: General Network Traffic.

Next, associate the new rule to the appropriate Log Source Type.

96 Chapter 4 | Creating Custom MPE Rules

Import the log message into the Test Center.

Chapter 4 | Creating Custom MPE Rules 97

 Determine the data to parse from the log and their associated LogRhythm metadata fields. We will use
the following:

TRAFFIC – Vendor Message ID

1.6.5.5 – IP Address (Origin)
1.1.1.4 – IP Address (Impacted)
0 – TCP/UDP Port (Origin)
0 – TCP/UDP Port (Impacted)
icmp – Protocol
allow – One variation of what happened with this traffic; will be used in a Sub-rule
o For example, this traffic activity could have also been described as deny or fail
l Sub-rules will be created with a different Common Event for each type of activity
(discussed in the next chapter)

Note: This information is defined by the vendor. If you don't know which IP address or Port is
Impacted versus Origin, for example, check with the vendor of the Log Source, or use your best
judgment.

Use the LogRhythm MPE Rule Builder Parsing Guide to locate the associated tags.

TRAFFIC - Vendor Message ID – vmid

1.6.5.5 – IP Address (Origin) - sip
1.1.1.4 – IP Address (Impacted) - dip
0 – TCP/UDP Port (Origin) - sport
0 – TCP/UDP Port (Impacted) - dport
icmp – Protocol - protname
allow – to be used in Sub-rule - tag1

Note: We will discuss Sub-rules in the next chapter.

98 Chapter 4 | Creating Custom MPE Rules

Chapter 4 | Creating Custom MPE Rules 99
 Begin entering the Base-rule Regular Expression.

Use wildcards to skip over unnecessary information. Use literal characters to match text before
and after the data to be parsed into the desired metadata field.
Click to Test Selected log message. Verify the data was parsed correctly into the metadata field.

100 Chapter 4 | Creating Custom MPE Rules

Enter additional regex and tags to parse data into another metadata field. Click Test Selected log
message.

Repeat until all required metadata fields are parsed correctly.

Chapter 4 | Creating Custom MPE Rules 101

 Skip unnecessary information using wildcards until the literal characters,1, are matched, then parse
matched information for sport and dport tags.

Skip unnecessary information using wildcards until literal characters are matched, then parse information
for protname and tag1 for Sub-rule tags. Notice all metadata fields contain the information parsed from
the log message.

102 Chapter 4 | Creating Custom MPE Rules

Be sure to click Save for your new custom MPE Rule.

Chapter 4 | Creating Custom MPE Rules 103

 This page intentionally left blank.

104 Custom MPE Rules Using Regular Expression

EXERCISES
Introduction
Exercise 1: Access the LogRhythm Training Environment 106
Exercise 2: Create a New MPE Rule 110
Exercise 3: Create Another New MPE Rule 120
Exercise 4: Create Another New MPE Rule 126
Exercise 5: Challenge: Create a New MPE Rule with less instruction 131
Exercise 6: Challenge: Create Another New MPE Rule 133

Chapter 4 | Creating Custom MPE Rules 105

 Exercise 1:
Access the LogRhythm Training Environment

The purpose of this exercise is to ensure that you are able to connect to your Training Virtual Machine
(VM).

1. Open a browser on your workstation and navigate to: logrhythm.instructorled.training

2. Your Instructor will provide your Access code. Enter your unique Access code here.

3. Enter your First and Last Name and click OK. You do not need to enable password protection.

4. Click the Lab tab on the tool bar on the top of the page.

106 Chapter 4 | Creating Custom MPE Rules

 5. Click the icon in the middle of the page that says Connect to the lab.

6. The Remote Desktop session will start, and will probably default to the "rtadmin" login account.
Click the back-arrow and select the Administrator account.

Login to the virtual host using the Administrator account with the password logrhythm!1

Chapter 4 | Creating Custom MPE Rules 107

 7. You should now see the Desktop of your virtual training server.

8. Click the logo to open the LogRhythm Client Console.

108 Chapter 4 | Creating Custom MPE Rules

 9. Enter the logrhythmadmin userID and logrhythm!1 password.

Chapter 4 | Creating Custom MPE Rules 109

 Exercise 2:
Create a New MPE Rule

The purpose of this exercise is to create a new custom MPE rule that matches the vendor ID message,
EVID0012, found in logs. Follow the steps below to complete this exercise.

Part One: Locate Log Messages Containing EVID 0012

You'll need to Run and Save an Investigation in LogRhythm's Client Console to find log messages that
your new custom regex will need to match. To do so, follow the steps below.

1. From within the Client Console, click the Investigate button

2. Click the radio button to Configure New Investigation and then click Next
3. Click the radio button to perform a Data Processor Search
a. Set the Date Range to In the Last 1 Days

b. Then click Next

4. Click the radio button next to Selected Log Sources then click the Add... button
5. Click the browse button next to the Log Source Types field

a. In the Record Type Filter panel click System

110 Chapter 4 | Creating Custom MPE Rules

 b. In the Log Source Type panel, scroll down, click to select Syslog - LogRhythm Syslog
Generator and then click OK

c. You should now see the Log Source Criteria Add window again, click the Search button
d. When the Log Sources appear at the bottom of the window, right-click and select Check All
Displayed and then click OK
e. You should now see the Investigator Wizard window again
f. Right-click and select Check All Displayed again
g. Click Next
6. Click Next, there is no need to specify an Event
7. Click Next to accept the default Log Repositories
8. In the Name field, type My Saved Investigation
a. Click Save
b. Click Launch
9. When the Investigation is complete, click to view the Log Viewer tab on the top of the panel
10. Enter the following text into the Log Message filter field: EVID:0012

Chapter 4 | Creating Custom MPE Rules 111

 11. Drag your mouse down the column on the far-left of the window to select (highlight) all of the
displayed logs

12. Right-click and select Copy Selected Logs to Rule Builder

You will probably see this message pop-up, just click Yes to proceed

112 Chapter 4 | Creating Custom MPE Rules

Part Two: Use the MPE Rule Builder tool to create a new MPE Rule
1. The Rule Builder window should open, enter the following rule name: EVID 0012
2. Open the Common Event Manager by clicking the Browse button next to the Common Event field

3. To select a Common Event:

a. Highlight the Classification named: Audit: Access Success
b. The Common Events under that Classification are now listed on the right side
c. Scroll down (or apply a filter) to find Object Accessed
Click OK to save the Common Event

4. Ensure the Rule Status is set to Development

5. To associate the new rule with a Log Source Type:

Chapter 4 | Creating Custom MPE Rules 113

 a. In the Log Message Source Type Associations tab in the upper right panel, expand the
System Log Source Types directory tree
b. Scroll down to locate the Syslog – LogRhythm Syslog Generator
i. Click to insert a checkmark in the box next to it

114 Chapter 4 | Creating Custom MPE Rules

Part Three: Determine Tags Required to Parse Metadata from Logs
The purpose of this exercise is to practice locating the LogRhythm tags associated with the metadata
fields that you want to have parsed from the log messages. The Tags and their associated metadata
Display Fields can be referenced in the LogRhythm-MPE-RuleBuilderGuide PDF document. These
tags will be used in your regex rule.

Your new MPE Rule must parse the following metadata from the log messages containing EVID:0012

Vendor Message ID
Host (Impacted)
User (Origin)
Host (Origin)
Object
Session ID

1. Review the LogRhythm-MPE-RuleBuilderGuide PDF document.

2. Look in the Display Field column to locate each of the metadata names listed above.
a. What are the associated LogRhythm Tag(s) you will need use in your regex to parse the
required metadata from the log messages?
i. Vendor Message ID _____________________________________________________
ii. Host (Impacted) ________________________________________________________
iii. User (Origin) ___________________________________________________________
iv. Host (Origin) ___________________________________________________________
v. Object ________________________________________________________________
vi. Session ID ____________________________________________________________

Hint
You will need to reference additional metadata Display Fields for some of the metadata.
Some Display Fields will require the use of two tags, as noted in the Rule Builder Guide.

You'll use OR statements (|) in your regex to account for the two possible tags for that metadata.

Chapter 4 | Creating Custom MPE Rules 115

 Part Four: Write a new Regular Expression for your Custom MPE Rule
The purpose of this exercise is to practice writing a regular expression in LogRhythm that parses
metadata.

1. Click to open the Test Center tab located on the bottom panel of the window
a. You should see the log messages you copied from your Investigation results
2. Begin to write a regular expression to match any and all of the log messages containing EVID:0012
a. You might find it helpful to write part of a regex and test, then continue writing and test again
until the entire regex has parsed the required metadata.
i. Enter your regex in the Base Rule Regular Expressions tab in the middle panel of
the window

i. Click the Save button after regex has been entered

3. Click the Test All button to ensure your new rule matches all of the log messages that were
imported into the MPE Rule Builder tool

116 Chapter 4 | Creating Custom MPE Rules

 4. Review your results in the Rule Builder Test Results pop-up window

a. How many of your logs matched the rule?____________________________________

b. Does your new rule match 100% of the logs?______________________________________
i. If your rule does not match 100% of your logs, you should work to fine tune the rule so
that it does
c. Review the Regex Average Logs/Second (total): line. What does it say about the
performance of your rule? Does it say (Fantastic)?__________________________________
d. If you notice rules with an Average rating of less than Fantastic, you may need to tune your
regex to increase it's performance; slow rules will decrease the number of messages
LogRhythm's Mediator can process per second (MPS)
5. Review the metadata field columns on the right side of the Test Center panel

a. Are the SIP, DIP, SName, DName, Login, Object, and VMID fields populated for every
log?_______________________________________________________________________
b. Why or Why not?_____________________________________________________________

Hint
When finished creating the regular expression it should resemble:
EVID:<vmid>\s(<dip>|(?<dname>.*?)):.*?user\s <login>\sfrom\s(<sip>|(?<sname>.*?))\.\sOb-
ject\s=\s(?<object>.*?)$

Chapter 4 | Creating Custom MPE Rules 117

 Part Five: Configure Additional Settings for your Custom MPE Rule
1. To configure Processing Settings, select the Processing Settings tab in the upper-left panel. The
following settings can remain in their default state:
a. Ensure the Match multi-line log messages option is unchecked, since this is not a
multiline log message
b. In order to simplify regex development, and to handle possible changes in text by the device
generating the log message, ensure the Ignore case when evaluating log message option
is checked
c. The Disable rule performance monitoring option can remain unchecked
d. Since this rule has both source and destination host tags the Host Context field should be
set to Tags Normal, meaning source tags (<sip> or <sname>) will be used to determine the
source host and destination tags (<dip> or <dname>) will be used to determine destination
host
e. Again, since this rule has both source and destination tags the Origin Host determined
via: field and Impacted Host determined via: field should be set to Tags
f. Since this rule is not parsing destination port or protocol, and it is not immediately obvious
what application is generating the log, the Impacted Application determined via: setting
should be set to N/A.

2. All Default Policy Settings settings can remain in their default state

118 Chapter 4 | Creating Custom MPE Rules

 3. Click to SAVE your Rule again.

4. Click the lower X to close the Rule Builder window (not the X on the top blue bar).

Chapter 4 | Creating Custom MPE Rules 119

 Exercise 3:
Create Another New MPE Rule

The purpose of this exercise is to create a new custom MPE rule that matches the vendor ID message,
EVID 0011, found in logs.

Part One: Locate Log Messages Containing EVID 0011

Review your Saved Investigation in LogRhythm's Client Console to find log messages that your new
custom regex will need to match.

1. From within the Client Console, click the Window menu button
2. Click to select the Investigator (My Saved Investigation) window

3. When the Investigator window opens, ensure you are viewing the Log Viewer tab
4. Enter the following text into the Log Message filter field: EVID:0011
5. Drag your mouse down the column on the far-left of the window to select (highlight) all of the
displayed logs
6. Right-click and select Copy Selected Logs to Rule Builder

120 Chapter 4 | Creating Custom MPE Rules

You will probably see this message pop-up, just click Yes to proceed

Chapter 4 | Creating Custom MPE Rules 121

 Part Two: Use the MPE Rule Builder tool to create a new MPE Rule
1. The Rule Builder window should open, enter the following rule name: EVID 0011
2. Open the Common Event Manager by clicking the Browse button next to the Common Event field
3. To select a Common Event:
a. Highlight the Classification named: Audit: Access Failure
b. The Common Events under that Classification are now listed on the right side
c. Scroll down (or apply a filter) to find Access Object Failure
d. Click OK to save the Common Event
4. Ensure the Rule Status is set to Development
5. To associate the new rule with a Log Source Type:
a. In the Log Message Source Type Associations tab in the upper right panel, expand the
System Log Source Types directory tree
b. Scroll down to locate the Syslog – LogRhythm Syslog Generator
i. Click to insert a checkmark in the box next to it

122 Chapter 4 | Creating Custom MPE Rules

Part Three: Determine Tags Required to Parse Metadata from Logs
Your new MPE Rule must parse the following metadata from the log messages containing EVID:0011

Vendor Message ID
Host (Impacted)
User (Origin)
Host (Origin)
Object
Session ID

Part Four: Write a new Regular Expression for your Custom MPE Rule
1. Write a regular expression to match any and all of the log messages containing EVID:0011
a. Enter your regex in the Base Rule Regular Expressions tab in the middle panel of the
window
i. Click the Save button after regex has been entered

2. Click to open the Test Center tab located on the bottom panel of the window
3. Click the Test All button to ensure your new rule matches all of the log messages that were
imported into the MPE Rule Builder tool
4. Review your results in the Rule Builder Test Results pop-up window
a. How many of your logs matched the rule?____________________________________
b. Does your new rule match 100% of the logs?______________________________________
i. If your rule does not match 100% of your logs you should
i. Enure you are only testing the rule against logs that contain EVID:0011
ii. Work to tune the rule so that it matches all logs that contain EVID:0011
c. Review the Regex Average Logs/Second (total): line. What does it say about the
performance of your rule? Does it say (Fantastic)?__________________________________
d. If you notice rules with an Average rating of less than Fantastic, you may need to tune your
regex to increase it's performance; slow rules will decrease the number of messages
LogRhythm's Mediator can process per second (MPS)
5. Review the metadata field columns on the right side of the Test Center panel

Chapter 4 | Creating Custom MPE Rules 123

 a. Are the SIP, DIP, SName, DName, Login, Object, and VMID fields populated for every
log?_______________________________________________________________________
b. Why or Why not?_____________________________________________________________
___________________________________________________________________________
___________________________________________________________________________

Hint
When finished creating the regular expression it should resemble:
EVID<TAG_HERE>\s+(<TAG_HERE>|(?<TAG_HERE>.*?)):.*?for user (?<TAG_HERE>.*?)
from (<TAG_HERE>|(?<TAG_HERE>.*?))\.\s+Object = (?<TAG_HERE>.*?)$

124 Chapter 4 | Creating Custom MPE Rules

Part Five: Configure Additional Settings for your Custom MPE Rule
1. To configure Processing Settings, select the Processing Settings tab in the upper-left panel.
Ensure the following settings are configured:

2. Leave the Default Policy Settings tab at the defaults.

3. Click to SAVE your Rule again.

4. Click the lower X to close the Rule Builder window (not the X on the top blue bar).

Chapter 4 | Creating Custom MPE Rules 125

 Exercise 4:
Create Another New MPE Rule

The purpose of this exercise is to create a new custom MPE rule that matches the vendor ID message,
EVID0008, found in logs.

Part One: Locate Log Messages Containing EVID 0008

1. Review your My Saved Investigation in LogRhythm's Client Console
2. Enter the following text into the Log Message filter field: EVID:0008 to find log messages that your
new custom regex will need to match
3. Highlight all of the displayed messages
4. Right-click again and select Copy Selected Logs to Rule Builder

You will probably see this message pop-up, just click Yes to proceed

126 Chapter 4 | Creating Custom MPE Rules

Part Two: Use the MPE Rule Builder tool to create a new MPE Rule
1. Enter the following rule name: EVID 0008
2. Open the Common Event Manager by clicking the Browse button next to the Common Event field
To select a Common Event:
a. Highlight the Classification named: Audit: Account Created
b. The Common Events under that Classification are now listed on the right side
c. Scroll down (or apply a filter) to find User Account Created
d. Click OK to save the Common Event
3. Ensure the Rule Status is set to Development
4. Associate the new rule with the Log Source Type: Syslog – LogRhythm Syslog Generator

Chapter 4 | Creating Custom MPE Rules 127

 Part Three: Determine Tags Required to Parse Metadata from Logs
Your new MPE Rule must parse the following metadata from the log messages containing EVID:0008

Vendor Message ID
Host (Impacted)
User (Origin)
User (Impacted)
Session ID

1. What are the associated LogRhythm Tag(s) you will need use in your regex to parse the required
metadata from the log messages?
a. Vendor Message ID _____________________________________________________
b. Host (Impacted) ________________________________________________________
c. User (Origin) ___________________________________________________________
d. User (Impacted) ________________________________________________________
e. Session ID ____________________________________________________________

Hint
You will need to reference the MPE Rule Builder Guide for metadata fields we have not used yet.

128 Chapter 4 | Creating Custom MPE Rules

Part Four: Write a new Regular Expression for your Custom MPE Rule
1. Write a regular expression to match any and all of the log messages containing EVID:0008
a. Enter your regex in the Base Rule Regular Expressions tab in the middle panel of the
window
i. Click the Save button after regex has been entered
2. Click to open the Test Center tab located on the bottom panel of the window
3. Click the Test All button to ensure your new rule matches all of the log messages that were
imported into the MPE Rule Builder tool
4. Review your results in the Rule Builder Test Results pop-up window
a. How many of your logs matched the rule?____________________________________
b. Does your new rule match 100% of the logs?______________________________________
c. Review the Regex Average Logs/Second (total): line. What does it say about the
performance of your rule? Does it say (Fantastic)?__________________________________

Hint
The first two IP addresses appearing in the logs will not be parsed from these messages. The
third IP address appearing in some of the logs is the Host (Impacted).

Chapter 4 | Creating Custom MPE Rules 129

 Part Five: Configure Additional Settings for your Custom MPE Rule
1. Leave the Processing Settings at the default settings
2. Leave the Default Policy Settings tab at the defaults
3. a. Click to SAVE your Rule again.

b. Click the lower X to close the Rule Builder window (not the X on the top blue bar).

Hint
When completed, your regex should look something like this:
EVID:(<vmid>)\s(<dip>|(?<dname>.*?)): user account (?<account>.*?) created by (?<login>.*?)
Session ID: (?<session>.*)$

130 Chapter 4 | Creating Custom MPE Rules

 Exercise 5:
Challenge: Create a New MPE Rule with less instruction

The purpose of this exercise is to practice creating a new custom MPE rule that matches the vendor ID
message, EVID0009. Since this is a Challenge, you will be provided with less instruction. Good luck.

Part One: Locate Log Messages Containing EVID 0009

1. Review your My Saved Investigation in LogRhythm's Client Console and copy log messages that
contain EVID 0009 into the Rule Builder

Part Two: Use the MPE Rule Builder tool to create a new MPE Rule
1. Enter the following rule name: EVID 0009
2. Use the Classification: Audit: Account Deleted
3. Select the Common Event named User Account Deleted
4. Ensure the Rule Status is set to Development
5. Associate the new rule with the Log Source Type: Syslog – LogRhythm Syslog Generator

Part Three: Determine Tags Required to Parse Metadata from Logs

Your new MPE Rule must parse the following metadata from the log messages containing EVID:0009

Vendor Message ID
Host (Impacted)
User (Origin)
User (Impacted)
Session ID

1. What are the associated LogRhythm Tag(s) you will need use in your regex to parse the required
metadata from the log messages?
a. Vendor Message ID _____________________________________________________
b. Host (Impacted) ________________________________________________________
c. User (Origin) ___________________________________________________________
d. User (Impacted) ________________________________________________________
e. Session ID ____________________________________________________________

Part Four: Write a new Regular Expression for your Custom MPE Rule
1. Write a regular expression to match any and all of the log messages containing EVID:0009
a. Enter your regex in the Base Rule Regular Expressions tab in the middle panel of the
window
i. Click the Save button after regex has been entered

Chapter 4 | Creating Custom MPE Rules 131

 2. Click the Test All button to ensure your new rule matches all of the log messages that were
imported into the MPE Rule Builder tool
3. Review your results in the Rule Builder Test Results pop-up window
a. Does your new rule match 100% of the logs containing EVID:0009?____________________

Part Five: Configure Additional Settings for your Custom MPE Rule
1. Leave the Processing Settings tab at the defaults
2. Leave the Default Policy Settings tab at the defaults
3. Don’t forget to Save your rule again
4. Close the window when you're done

132 Chapter 4 | Creating Custom MPE Rules

 Exercise 6:
Challenge: Create Another New MPE Rule

The purpose of this exercise is to practice creating a new custom MPE rule that matches the vendor ID
message, EVID0045. Since this is a Challenge, you will be provided with less instruction. Good luck.

Part One: Locate Log Messages Containing EVID 0045

1. Copy log messages that contain EVID 0045 into the Rule Builder tool

Part Two: Use the MPE Rule Builder tool to create a new MPE Rule
1. Rule name: EVID 0045
2. Classification: Audit: Access Granted
3. Common Event: Account Added To Group
4. Rule Status: Development
5. Log Message Source Type Association: Syslog – LogRhythm Syslog Generator

Chapter 4 | Creating Custom MPE Rules 133

 Part Three: Determine Tags Required to Parse Metadata from Logs
Your new MPE Rule must parse the following metadata from the log messages containing EVID:0045

Vendor Message ID
Host (Impacted)
User (Origin)
User (Impacted)
Group
Host (Origin)

1. What are the associated LogRhythm Tag(s) you will need use in your regex to parse the required
metadata from the log messages?
a. Vendor Message ID _____________________________________________________
b. Host (Impacted) ________________________________________________________
c. User (Origin) ___________________________________________________________
d. User (Impacted) ________________________________________________________
e. Group ________________________________________________________________
f. Host (Origin) ___________________________________________________________

134 Chapter 4 | Creating Custom MPE Rules

Part Four: Write a new Regular Expression for your Custom MPE Rule
1. Write a regular expression to match any and all of the log messages containing EVID:0045
2. Does your new rule match 100% of the logs containing EVID:0045?____________________

Part Five: Configure Additional Settings for your Custom MPE Rule
1. Configure Processing Settings
2. Leave the Default Policy Settings tab at the defaults
3. Don’t forget to Save your rule again
4. Close the window when you're done

Chapter 4 | Creating Custom MPE Rules 135

 This page intentionally left blank.

136 Custom MPE Rules Using Regular Expression

CHAPTER 5
Enabling Custom Rules
The Final Steps for MPE Rule Creation 139
Sub-Rules 140
Creating Sub-Rules 142

The Sub-Rule Tags 144

Sub-Rule Example 146

Walk Through Example Part Two: Adding Sub-Rules 151

MPE Rule Processing Logic 157

Changing The Sort Order 159

Rule Library Browser 162

MPE Policy Settings 164
Enabling Custom MPE Rules 168

Log Source Type Manager 169

The Date Format Manager 171

Steps After MPE Rule Creation 174

Exercises 175
Exercise 1: Chapter Review Questions 176

Exercise 2: The Rule Library 177

Exercise 3: Create an MPE Rule with Sub-rules 180

Exercise 4: Create an MPE Rule with Sub-rules 185

Exercise 5: Create an MPE Rule with Sub-rules 187

Custom MPE Rules Using Regular Expression │ Enabling Custom Rules 137
 Exercise 6: Challenge: Create an MPE Rule with Sub-rules 189

Exercise 7: Enable New MPE Rules 191

138 Custom MPE Rules Using Regular Expression │ Enabling Custom Rules
The Final Steps for MPE Rule Creation
In the previous chapter we reviewed the MPE Rule builder and how to create new Base-Rule Regular
Expressions. In this chapter we'll build on that foundation to cover the final steps for custom MPE Rule
creation with the following topics:

Creating Sub-rules
MPE Processing Logic
Browsing the Rule Library
MPE Policy Settings
Enabling MPE Rules
Creating new Log Source Types

Chapter 5 | Enabling Custom Rules 139

 Sub-Rules
Sub-rules are used to parse specific data that may not be contained in every log message generated from
a Log Source Type. Sub-rules help differentiate log messages that match the same Base-rule, but contain
particular values in part of log. Sub-rules are often used to identify information such as an event ID
number, a message, a user, or group name. Sub-rules include a regex, that only applies to a string of
data for a specific field. Think of it as parsing a more specific message to process that particular log
differently.

Sub-rules are associated with one Base-rule, but there can be many Sub-rules for a single Base-rule.
When possible, a Base-rule should have Sub-rules added to further normalize the log data. Rules are
created for every type of log that a Log Source can generate, which means there can be a lot of Base-rules
and even more Sub-rules!

Example
If a system shutdown is identified in the Base-rule, the Sub-rules would identify the specific reason
for the shutdown.

During log processing, the MPE will evaluate logs against Base-rules, and if matched:

The MPE will evaluate the log against Sub-rules associated with that Base-rule
o If a Sub-rule is matched, the MPE will associate the log with that Sub-rule
If there are no Sub-rules or no Sub-rules matched, the MPE will associate the log with the Base-rule

140 Chapter 5 | Enabling Custom Rules

The image below shows a list of Sub-rules created for the Catch-all Base-rule associated to the Log
Source Type: MS Event Log for XP/2000/2003 - Application. Notice the data in each of the columns:

Name - Each Sub-rule is given a unique name that reflects the activity described in the log
message.
Classification - Each Sub-rule is assigned its own Classification allowing for differentiation of
Event priority. Some logs from this Log Source Type will have a higher priority such as Warning
messages, while other logs may just contain Informational messages of a lower priority.
Common Event - Each Sub-rule is assigned its own Common Event allowing for granularity of
descriptions for each log message. When creating Sub-rules, use your best judgment to select a
Common Event that matches the description of the activity occurring in your log messages.

Chapter 5 | Enabling Custom Rules 141

 Creating Sub-Rules
To create a new Sub-rule you must first have an existing Base-rule. Right-click inside the Sub-rule tab and
select New.

Note: You can also Clone existing Sub-rules if you are creating multiples with the same Classi-
fication and Common Event, but perhaps with slight variations of VMIDs.

The Sub-rule Properties window opens, allowing you to configure your new Sub-rule:

Here, you'll specify a Rule Name (which will automatically be copied into the Full Name field), and assign
a Common Event by clicking on the Common Event Manager icon.

142 Chapter 5 | Enabling Custom Rules

Sub-rules are configured using Mapping Tags which specify the Field values that must match in order
for a log to be identified with the Common Event in this Sub-rule. The Match column contains the
following options used to match the criteria found in the metadata field as specified in the Value column:

Anything is paired with the wildcard character *

Nothing is used to represent an empty Value
Pattern is used to match a Value using a regex pattern
o For example: (this|that|the other)
Equal To (=) is used to specify an exact Value
!= (Not Equal To) is used to match anything other than the specified Value

The Mapping Tags tab, shown in the image below, contains a list representing all of the tags used in the
associated Base-rule Regular Expression. This Sub-rule is used to match the log messages containing a
vmid Value equal to 12289. Notice the Sub-rule is concerned only with the data in the vmid Field, all other
Fields can contain a Value of anything.

Chapter 5 | Enabling Custom Rules 143

 The Sub-Rule Tags
Did you notice these Field names: tag1, tag2, and tag5? Those are not metadata fields in LogRhythm.

Five additional tags are available for identifying data in a log specifically for use in Sub-rules. These tags
do not parse text into metadata fields and are not visible to analysts in LogRhythm. These tags are only
used to identify portions of a log message that will be used in Sub-rules. You will only use these Sub-rule
tags in Base-rules if you intend on creating Sub-rules that use the values captured into the associated
fields.

144 Chapter 5 | Enabling Custom Rules

Understanding Sub-Rule Tags
To understand this concept, let's look at this example of a Base-rule regular expression:

^.*? (?:\d){2}:(?:\d){2}:(?:\d){2} .*? <process>\[\d*\]: FTP LOGIN

(?<tag1>SUCCEEDED|FAILED) FROM (<sip>|(?<sname>)(\s|$)

Notice the OR statement with two options (in bold font) in the rule above. This statement is used to
capture data from log messages into the <tag1> named capture group:

10 11 2008 22:54:53 1.1.1.2 <FTPD:NOTE> Oct 12 05:54:53 ftpd[87428]:

FTP LOGIN SUCCEEDED FROM 21.1.1.1

10 11 2008 22:54:55 1.1.1.2 <SAU2:NOTE> Oct 12 05:54:55 ftpd[87428]:

FTP LOGIN FAILED FROM 21.1.1.1, apache

The Base-rule should have two Sub-rules created for the <tag1> values captured from the logs:

One Sub-rule will match logs containing SUCCEEDED login attempts

One Sub-rule will match logs containing FAILED login attempts

Chapter 5 | Enabling Custom Rules 145

 Sub-Rule Example
Below is an example of a log message describing a general Microsoft SQL Server message:

Example
LOG MESSAGE:
<Event xmlns-
='http://schemas.microsoft.com/win/2004/08/events/event'><System><Prov-
ider Name='SQLISPackage100'/><EventID Qualifiers='16385'>
12289InformationLRVM-XMNT AUTHORITY\SYSTEMLR Backup
Cleanup</EventID><Level> </Level><Task>None</Task><Key-
words>Classic</Keywords><TimeCreated SystemTime='2017-06-
04T06:00:07.000000000Z'/><EventRecordID>25582324</EventRecordID><Chann-
el>Application</Channel><Computer></Computer><Security UserID=''/>
</System><EventData>Package "" finished success-
fully.</EventData></Event>

The MPE Rule that matches this log message, named SQL Server Messages, uses the following Base-
rule regular expression to parse data from the log message:

Example
BASE-RULE REGEX:
Name='(?<tag2>.*?(SQLSERVER|SQLISPackage100|SQLAgent|SQL\$).*?)'.*?<EventID.*?>
(?<vmid>\d+)<.*?Level>(?<severity>.*?)<.*?Com.*?>(?<dname>.*?)(\..*?)?</C((.*?(\r|\n))
{3}succ.*?:(?<tag3>.*?)(\r|\n)(.*?(\r|\n)){2}sess\S+:(?<session>\d+)(.*?(\r|\n)){5}obj\S+:
(?<objectname>\w+)(.*?(\r|\n)){3}ser\S+name:((?<domain>.*?)\\)?(?<login>\S+)(.*?(\r|\n)){9}object_
name:(?<object>.*?)(\r|\n)sta\S+:(?<command>.*?)(\r|\n))?(.*?pr.*? id of (?<processid>\d+))?
(.*?Sch.*? Job '(?<object>.*?)')?(.*?Status: (?<tag3>\w+))?(.*?for user '((?<domain>.*?)\\)?
(?<login>.*?)'(.*?database '(?<object>.*?)')?.*?\[CLIENT:\s+(<sip>|(\<)?(?<sname>.*?)(\>)?)\])?
(.*?UserID='((?<domain>[^/']+)\\)?(?<login>.*?)')?(.*?Pac\w+ "(?<object>.*?)")?(.*?com\S+
(?<command>.*?)\.)?(.*?\((?<object>.*?)\.\).*?for '(?<protname>\w+)://<sip>:(?<sport>\d+)')?(.*?
database '(?<object>.*?)')?(.*?Alert for '(?<object>.*?)'.*?cur\w+ value of '(?<objectname>.*?)')?
(.*?(?<tag1>(?:LooksA.*? req\w+|checkserv\S+: returning \w+)|(IsAlive request)))?

This Base-rule will match the pattern of the log and will capture the following data:

SQLISPackage100 into the <tag2> field

12289 into the <vmid> field
Information into the <severity> field
LRVM-XM into the <dname> field
NTAUTHORITY into the <domain> field
SYSTEM into the <login> field
LR Backup Cleanup into the <object> field

146 Chapter 5 | Enabling Custom Rules

 Note: There is obviously much more in the Base-rule regular expression that could optionally
match data contained in similar log messages generated by this Log Source Type.

Pictured below is a Sub-rule associated to the Base-rule in this example. This Sub-rule is looking to match
the specific vmid value, 12289. The log message above contains the EVID 12289, and therefore matches
the criteria for this Sub-rule.

Chapter 5 | Enabling Custom Rules 147

 The picture below shows this Sub-rule applied to the matching log message. This log was first matched
by the Base-rule SQL Server Messages, then because the specific vmid is equal to 12289 it is matched
by the Sub-rule EVID 12289 : Package Finished, and assigned the Common Event of Task Finished.

148 Chapter 5 | Enabling Custom Rules

In the picture below, you can see there are many EVIDs that could be matched by other Sub-rules in
similar logs from this Log Source. Each Sub-rule is assigned a different Common Event equivalent to the
activity described in matching logs.

described in the log message.

Chapter 5 | Enabling Custom Rules 149

 Pictured below is the Log Information window opened from within the Personal Dashboard in LogRhythm.
Notice the <tag2> information, SQLISPackage100, is not listed in the Processed Meta Data Fields tab.
The <tag2> information is not visible to the end user; it is only used by the Sub-rule for processing
purposes.

150 Chapter 5 | Enabling Custom Rules

Walk Through Example Part Two: Adding Sub-Rules
In the Walk Through Example seen in the previous chapter, we identified a description of traffic activity,
but we've discovered that activity could vary from log message to log message (allow, deny, failed). To
account for the different information that might appear in log messages, Sub-rules need to be created for
the new Base-rule. The Sub-rule will be used to identify the different types of Network Traffic described in
logs from this Log Source Type.

In the example log below, we see the identifying information allow describes the activity. We also see
that deny and failed are two additional activities described in log messages generated from this Log
Source Type. We'll create three Sub-rules to account for each of these different events.

Let's walk through creating new Sub-rules for these sample log files:

02 20 2011 00:00:30 1.1.1.1 <LOC0:INFO> Feb 20 00:00:51 1,2011/02/20

00:00:51,0002C100617,TRAFFIC,end,33,2011/02/20
00:00:45,1.6.5.5,1.1.1.4,0.0.0.0,0.0.0.0,Inbound_vWire,,,ping,vsys1,un-
trust,trust,ethernet1/1,ethernet1/2,LogRhythmForwarder,2011/02/20
00:00:51,176828,1,0,0,0,0,0x0,icmp,allow ,120,120,120,2,2011/02/20
00:00:37,6,any,0

02 20 2011 00:00:30 1.1.1.1 <LOC0:INFO> Feb 20 00:00:51 1,2011/02/20

00:00:51,0002C100617,TRAFFIC,end,33,2011/02/20
00:00:45,1.6.5.5,1.1.1.4,0.0.0.0,0.0.0.0,Inbound_vWire,,,ping,vsys1,un-
trust,trust,ethernet1/1,ethernet1/2,LogRhythmForwarder,2011/02/20
00:00:51,176828,1,0,0,0,0,0x0,icmp,deny ,120,120,120,2,2011/02/20
00:00:37,6,any,0

02 20 2011 00:00:30 1.1.1.1 <LOC0:INFO> Feb 20 00:00:51 1,2011/02/20

00:00:51,0002C100617,TRAFFIC,end,33,2011/02/20
00:00:45,1.6.5.5,1.1.1.4,0.0.0.0,0.0.0.0,Inbound_vWire,,,ping,vsys1,un-
trust,trust,ethernet1/1,ethernet1/2,LogRhythmForwarder,2011/02/20
00:00:51,176828,1,0,0,0,0,0x0,icmp,failed ,120,120,120,2,2011/02/20
00:00:37,6,any,0

We will use the Base-rule we created previously in the MPE Rule Builder: Walk Through Example:
Palo Alto.

Chapter 5 | Enabling Custom Rules 151

 In the Sub-Rules tab, right-click to create a New Sub-rule.

The Sub-rule Properties window will be displayed.

Name the rule something you might recognize when it is displayed in the console, such as Palo Alto:
Traffic Allowed.

152 Chapter 5 | Enabling Custom Rules

Select an appropriate Common Event to describe this activity. It should be similar, but more specific than
the Common Event used for the Base-rule. In this example, we've selected General Traffic Allowed.

Chapter 5 | Enabling Custom Rules 153

 Set the Mapping Tags specific to this Sub-rule. Because the Base-rule will parse most of the data, we
only need to select the Mappings for the Tag1 metadata field, which we used in the Base-rule to parse the
allow text from the log.

Click OK to save this new Sub-rule.

Right-click to create a second Sub-rule for the deny type traffic events.

Select an appropriate Common Event to describe this activity. It should be similar, but more specific than
the Common Event used for the Base-rule. In this example, we've selected General ACL Deny Event.

Set the Mapping Tags specific to this Sub-rule. We only need to select the Mappings for the Tag1
metadata field to parse the deny text from the log.

154 Chapter 5 | Enabling Custom Rules

Click OK to save this new Sub-rule.

Right-click to create another new Sub-rule for the failed type traffic events.

Select an appropriate Common Event to describe this activity. It should be similar, but more specific than
the Common Event used for the Base-rule. In this example, we've selected Failed To Send Packet.

Set the Mapping Tags specific to this Sub-rule. We only need to select the Mappings for the Tag1
metadata field to parse the failed text from the log. Click OK to save this new Sub-rule.

Chapter 5 | Enabling Custom Rules 155

 The Sub-rules tab now displays the rules we just created to account for each type of traffic activity
described in the logs generated from this Log Source Type.

156 Chapter 5 | Enabling Custom Rules

MPE Rule Processing Logic

When new custom MPE Rules are created, both Base-rules and Sub-rules, will be sorted into a
processing order. The processing order will naturally occur during the development of new rules because
the first rule created will be the first processed, etc.

Both Base-rules and Sub-rules have a processing order within a rule set. It is important to remember the
logic behind processing when creating new MPE Rules.

The Sorter takes the most descriptive rules and processes logs against those first and least descriptive,
or Catch-all rules, are processed last.

Custom Base-rules are always applied before System (LogRhythm created) Base-rules
Custom Sub-rules are always applied before System Sub-rules

Chapter 5 | Enabling Custom Rules 157

 Base-Rule Sorting
The rule processing sort order for Custom and System Base-rules are as listed:

Custom Base-rules:
o Custom Sub-rules where the value parsed for VMID should be a specific value by Sort Order
o All other Custom Sub-rules by Sort Order
System-Base rules:
o Custom Sub-rules where the value parsed for VMID should be a specific value by Sort Order
o System Sub-rules where the value parsed for VMID should be a specific value by Sort Order
o All other Custom Sub-rules by Sort Order
o All other System Sub-rules by Sort Order

Sub-Rule Sorting
Sorting only affects Sub-rules with wildcards and regexes. Sorting is irrelevant for Sub-rules where each
tag value is specified since the Sub-rule will only match the exact values specified. When using wildcards
and regexes, it is important to understand how their sorting is treated by the MPE.

When processing Sub-rules, the MPE will process in the following order:

Custom Sub-rules where VMID (Vendor Message ID) is equal to a specific value
System Sub-rules where VMID is equal to a specific value
Custom Sub-rules where VMID is not equal to a specific value
System Sub-rules where VMID is not equal to a specific value

The sort order will determine the priority in which a Sub-rule is matched. For instance, if you have a Sub-
rule where all the tags are wildcards, you want to make sure this Sub-rule is the last sorted item, because
sorting it higher would cause it to always match, and rules below would never be tested for specific
matching values.

158 Chapter 5 | Enabling Custom Rules

Changing The Sort Order
Important!
More specific rules should be processed before less specific rules and Catch-all rules should be
sorted last.

LogRhythm uses auto-sorted MPE Rules, but this can be modified as specified here:

System Sub-rules continue to be statically sorted

Custom Sub-rules continue to be statically sorted
Specify static or auto-sorting for each Custom Base-rule
Enforce relative ordering for each auto-sorted Custom Base-rule
Override the relative ordering for each auto-sorted System Base-rule
Specify that an auto-sorted Custom MPE Rule should sort above all System rules

You can change the processing order of rules to ensure that one rule does not cancel the processing of
another rule.

Example
Suppose you have created two Base-rules that seem to work for more than one log type. You need
to ensure they are in the correct sort order so one doesn't cancel the processing of the other;
however, the better option is to re-write both rules into a single rule with better Sub-rules.

Example
By default, Custom rules run before System rules, however if you write a Catch-all rule you don't
want it running before a System rule as it will basically cancel it.

Chapter 5 | Enabling Custom Rules 159

 To view or edit the processing order for a Base-rule, select the associated Log Message Source Type
from the list in the Rule Browser, then click the Edit button and select Edit Base-rule Sorting.

To view or edit the processing order for a Sub-rule, select the associated Base-rule from the list in the
Rule Browser, and when the Rule Builder window opens, right-click in the Sub-Rules tab the Edit
button and select Edit Sub-rule Sorting.

160 Chapter 5 | Enabling Custom Rules

In the Rule Sorter, all rules appear in the grid from top to bottom in ascending order by Sort Order. Select
a rule and use the arrows to move it up or down, or click the button to Automatically Sort the rules, which
will sort based on the Mapping Tag Values.

Sorting only affects Sub-rules with wildcards and regexes. Sorting is irrelevant for Sub-rules where each
Mapping Tag Value is specified since the Sub-rule will only match the exact values specified. If you have
a Sub-rule where all the Mapping Tags are wildcards, you want to make sure this Sub-rule is the last
sorted item, because sorting it higher would cause it to always match, and rules below would never be
applied for matching.

The image below shows an example of wildcards used in Mapping Tags.

Chapter 5 | Enabling Custom Rules 161

 Rule Library Browser

After you create new MPE Rules you can view them in the Rule Library. In the Rule Library you can scroll
through a selection of over 7,000 System MPE Rules (those created by LogRhythm's engineers)
distributed as part of LogRhythm's Knowledge Base (KB). The Rule Library lists the MPE Rules by Log
Message Source Type.

Click to select a Log Source Type to view it's associated MPE Rules in the Rule Browser window.

Hint
You may not see all of the MPE Rules by default. To change the view so you can see the right
rule type and status, click View to open the menu and select the items you wish to see, as
shown in the picture above.

162 Chapter 5 | Enabling Custom Rules

The picture below highlights the System rules for the Log Source Type MS Windows Event Logging -
Application. Notice there are many rules for this one Log Source Type, listed in the panel on the right-
side. There are many different rules because there can be many different log messages generated by this
one Log Source Type.

Chapter 5 | Enabling Custom Rules 163

 MPE Policy Settings
An MPE Policy, also known as a Log Processing Policy, is a collection of MPE Rules for a specific Log
Source Type (such as Cisco PIX or Windows 2003 Security Event Log). Log Processing Policies
determine which rules will be applied to logs from a particular Log Source and how matching log
messages are treated; including the length of time the log remains online for reporting (TTL), whether or
not the log should be archived, and whether or not the log should be forwarded to the Platform Manager.

The MPE Rule determines how LogRhythm parses metadata from the log.

The MPE Policy determines data management and Event management settings for the log.

During the addition and configuration of the Log Sources in your environment, specifying the correct MPE
Policy is imperative. Only logs from Log Sources assigned an MPE Policy will be processed by the MPE.
The LogRhythm Knowledge Base (KB) comes with pre-packaged default MPE Policies for all supported
log types. These Log Processing Policies for the System Log Source Types are all named LogRhythm
Default.

164 Chapter 5 | Enabling Custom Rules

Log Sources can only be assigned:

One MPE Policy

An MPE Policy associated with the same Log Source Type as the Log Source

The MPE Rule contains the default settings for the MPE Policy when the rule is first added, but MPE
Policies can be customized for the Log Source Type to which they will be assigned. MPE Policies can be
edited for each associated MPE Rule, to specify different Data Management Settings, Event
Management Settings. For example, Event forwarding settings for the Windows Security Event Logs on a
file server could be different from those on a domain controller.

MPE Policies can be viewed, edited, deleted, or created from within the Deployment Manager in the
Client Console on the Log Processing Policies tab. You can configure the MPE Policy settings in the
the MPE Policy Rule Editor; this is also where MPE Rules are Enabled and Disabled.

Modifying a Log Processing Policy

Follow these steps to modify an existing Log Processing Policy:

1. Log in to the Client Console using administrator credentials.

2. Open Deployment Manager.
3. Select the Log Processing Policies tab.
4. Select the Log Processing Policies you want to modify.
5. On the File menu, click Properties.
6. The MPE Policy Editor appears.
7. Check the Enabled field for each rule to include in the policy.

Chapter 5 | Enabling Custom Rules 165

 8. To override the default aging and Event settings, check the Edit field of the rule to edit, then right-
click and select Properties from the context menu.
9. The MPE Policy Rule Editor appears.
10. Make your changes. All edits you make will apply to all the rules that are currently selected.
11. Click OK.
12. You'll return to the MPE Policy Editor window.
13. Click OK.

166 Chapter 5 | Enabling Custom Rules

 Note: MPE Policies can be edited to specify different Data Management Settings,or Event
Management settings; however LogRhythm recommends changing these settings in the Global
Log Processing Rule (GLPR) Manager instead. The GLPR Manager can be accessed from the
Deployment Manager via the Tools menu by selecting Aministration - Global Log Processing
Rule Manager.

Chapter 5 | Enabling Custom Rules 167

 Enabling Custom MPE Rules

After a new rule has been created and tested successfully you need to Enable the rule. MPE Rules must
be in either Test or Production status before it can be Enabled.

To enable a rule, navigate to the Deployment Manager, select the Log Processing Policies tab, select the
Log Source Type to which your rule is associated, check the boxes for your new Custom rules, then right-
click and select Properties. The Enabled checkbox is located in the MPE Policy Rule Editor window.

168 Chapter 5 | Enabling Custom Rules

Log Source Type Manager
LogRhythm supports, and has already created, Log Source Types for hundreds of Log Sources.
However, if you have a custom application or a newly released device, LogRhythm may not yet have a
Log Source Type created for it. When creating a new custom MPE Rule, you may need to first create a
new Log Source Type to associate with the new MPE Rule.

Creating new Log Source Types

You can create custom Log Source Types from the Log Source Type Manager found under the Tools -
Knowledge menu in the LogRhythm console.

After clicking the New button, the Log Source Type Properties window allows you to assign a name
and Log Format.

Chapter 5 | Enabling Custom Rules 169

 The Log Format is especially important. You must specify the Log Format to match the type of log being
collected. Each device generates logs in a certain format. Usually when creating Custom Log Source
Types the Log Format will either be a Syslog or Text File format.

Syslog devices have a setting to specify where to send the Syslog Feed, so that is an easy way to
know it is a Syslog format.
Text Files, or Flat File, log formats are plain text files containing log messages from systems,
computers, devices, and applications that cannot transmit their log messages via Windows Event
Logs or Syslogs. Flat Files lack any structural standards, so they can vary widely in format.

Click OK to save your new Log Source Type.

Important!
Use your best judgment to determine the format of your particular custom log messages. If
you've selected the wrong Log Format, the logs will show as Unidentified in LogRhythm; the
MPE will not be able to identify or parse data correctly.

170 Chapter 5 | Enabling Custom Rules

The Date Format Manager

When adding new custom Log Sources, the Date Format Manager is used to select the appropriate regex
used to parsed the date from the log messages generated by your Log Source. The Date Formats are
listed by device type, so you might first try to match your Log Source to one that already exists for your
device; however you may not always find an exact match by Name. In that case, review the date format
in your log messages and find a regex that will match.

Example
This is a sample log from a Flat File with the date information in bold font:
06/28/2017 02:41:06 PM alert action: 1-28255 Win.Trojan.Kuluoz Detected
sHost:150.94.138.231 dHost:195.190.211.155 email:-
frederick@cert.logrhythm.com

Example
This is the associated Date Format used by LogRhythm to parse the date from the Flat File log
message:
<M>/<d>/<yy> <h>:<m>:<s> <t>

The first thing you might notice about the regex is that it is expressed in Regular Expression type
characters and not what you may think of as a typical date format. Date formats are often expressed with
DD representing two digits for the day, MM representing two digits for the month, and YYYY representing
four digits for the year, and so on, as shown here: DD/MM/YYYY HH:MM:SS.

Chapter 5 | Enabling Custom Rules 171

 The Date Formats in LogRhythm are expressed using the following characters, as shown in the Help tab
for the Date Format Properties:

1. The <M> matches the month in our sample log

2. The <d> matches the day in our sample log
3. The <yy> matches the four digit year in our sample log

If you can not find a Date Format that matches the date format in your log, you may need to build your
own, using the information in the Help tab as a guide.

172 Chapter 5 | Enabling Custom Rules

Date and Time Parsing

If you have issues with the Date and Time, verify the Log Source Type configuration settings are correct
for that Log Source.

When you select a Log Source Type during the deployment of a Log Source, the date and time regex is
automatically included. When adding Flat File Log Sources, which vary widely in format, you must select
the format of the date and time found in the log so that the correct regex is applied. You have the option to
choose from many different date and time formats.

Important!
When creating new MPE Rules, do not include the date and time information in the Base-rule
regular expression; it is not needed. The Date and Time information is automatically assigned
during processing for Time Normalization.

Chapter 5 | Enabling Custom Rules 173

 Steps After MPE Rule Creation
When creating a custom MPE Rule, there are a few basic steps that can be followed to ensure you have
completed all configurations needed for your new custom Log Source. The following steps can be used as
a reference after you've created a new Custom MPE Rule.

1. Change the Rule Status, of your new Base-rule and its associated Sub-rules, to Test
2. Assign a Log Processing Policy
a. Review the list of existing Log Processing Policies in the Deployment Manager
b. Select the appropriate Log Source Type
c. Create a new Log Processing Policy if one is not already created
i. Give it a name and select the appropriate Log Source Type
3. Enable the new MPE Rule
a. Edit the properties of the Log Processing Policy
b. Select the Enable option and set the Risk Based Priority value
4. Verify it was tested successfully by the MPE
a. If not, make changes to the rule for improved efficiency
i. Enable it to run the Test again
5. Edit rule Sorting
a. Detailed rules should be on top and more general rules on the bottom
b. Catch-all rules should be at the bottom of all other rules
6. Change the Rule Status to Production in the Rule Builder
7. Enable the new MPE Rules
a. Edit the properties of the Log Processing Policy
b. Enable the new Custom MPE Rules and set the Risk Based Priority value
i. Your MPE Rules and your Log Processing Policy are associated to your custom Log
Source Type
8. Verify the logs are identified and metadata is parsed by the rule as expected
a. Perform an Investigation or review the logs in the Dashboard
9. Make changes, if needed, to ensure the log is processed as desired
a. Repeat verification of log processing and make changes until log is processed as desired
10. If you've just created a new MPE Rule for a device that will likely be used by others, submit it to
the LogRhythm Labs team for review. If approved, the Labs team can add it permanently to the
list of available Log Source Types, MPE Rules, and perhaps create some additional Sub-rules.

174 Chapter 5 | Enabling Custom Rules

EXERCISES
Introduction
Exercise 1: Chapter Review Questions 176
Exercise 2: The Rule Library 177
Exercise 3: Create an MPE Rule with Sub-rules 180
Exercise 4: Create an MPE Rule with Sub-rules 185
Exercise 5: Create an MPE Rule with Sub-rules 187
Exercise 6: Challenge: Create an MPE Rule with Sub-rules 189
Exercise 7: Enable New MPE Rules 191

Chapter 5 | Enabling Custom Rules 175

 Exercise 1:
Chapter Review Questions

The purpose of this exercise is to review the information presented in this chapter.

1. What is the main purpose of a Base-rule?________________________________________________

_______________________________________________________________________________
2. How many Base-rules can a Sub-rule be associated with?_________________________________
3. How many Sub-rules can be associated with a Base-rule?_________________________________

176 Chapter 5 | Enabling Custom Rules

 Exercise 2:
The Rule Library

The purpose of this exercise is to introduce you to the Rule Library in LogRhythm.

1. Navigate to the Tools menu. Select Knowledge - MPE Rule Builder.

2. Click the library icon on the top-left of the window to open the Rule Library window.

3. Click to select any one Log Message Source Type from the Rule Browser list.

Chapter 5 | Enabling Custom Rules 177

 4. The MPE Rules associated with that Log Source Type will be displayed on the right side of the
window. How many MPE Rules are associated with the Log Source Type you selected?_________

5. Double-click to open any one of the Rules listed on the right-side of the window. The Rule's name,
Common Event, and Base-rule Regular Expression will be displayed in the MPE Rule Builder.

178 Chapter 5 | Enabling Custom Rules

 6. Click the Sub-rules tab located on the bottom panel of the window. Are they any Sub-rules
associated with the MPE Rule you selected?___________________________________________

Chapter 5 | Enabling Custom Rules 179

 Exercise 3:
Create an MPE Rule with Sub-rules

The purpose of this exercise is to practice configuring new MPE Sub-rules.

Part One: Create a new Base-rule

1. Perform an Investigation so you can copy log messages that contain the following EVIDs: 0013,
0014, 0015, and 0016 into the Rule Builder tool.
2. Use the Rule Builder tool to create a new Base-rule with the following details:

Rule Name: Operations Logs

Classification: Operations : Error

Common Event: General Error

Risk Rating: 0

Rule Status: Development

Log Source Type: Syslog – LogRhythm Syslog Generator

Be sure to save your rule as you go.

The Base-rule should match logs with any of the following Event IDs:

0013
0014
0015
0016

The Base-rule should parse the following metadata:

Vendor Message ID
Impacted Host
Process Name

180 Chapter 5 | Enabling Custom Rules

Part Two: Create new Sub-rules
Create Sub-rules for your new Base-rule.

EVID Rule Name Classification Common Event

0013 EVID 0013 Operations : Critical General Critical

0014 EVID 0014 Operations : Error General Error

0015 EVID 0015 Operations : Warning General Warning

0016 EVID 0016 Operations : Information General Information

Create a new Sub-rule for each of the EVIDs listed in the table above. Repeat these steps for all EVIDs
listed in the table above.

1. On the Sub-Rules tab, right-click and select New

2. Enter Rule Names and assign Classifications and Common Events for each EVID as listed in
the table above
3. In the Mapping Tags tab, configure the following for the vmid row:
Set the Match column to Equal To (=)
In the Value column type the associated EVID in the vmid row

4. In the Settings tab notice:

The Processing Settings from the Base-rule were automatically applied to the Sub-rule

Chapter 5 | Enabling Custom Rules 181

 The Event Settings option was automatically configured to Forward matching logs to the
Platform Manager
All other settings can remain at their default, as well

5. Be sure to save your rule

182 Chapter 5 | Enabling Custom Rules

Part Three: Create Additional Sub-rules
Add additional Sub-rules to identify operations logs specific to HTTPD processes.

EVID Rule Name Classification Common Event

0013 EVID 0013 HTTPD Operations : Crtical HTTPD Critical Condition

0014 EVID 0014 HTTPD Operations : Error HTTPD Error

0015 EVID 0015 HTTPD Operations : Warning HTTPD Warning

0016 EVID 0016 HTTPD Operations : Information HTTPD Information

Create a new Sub-rule for each of the EVIDs listed in the table above. Repeat these steps for all EVIDs
listed in the table above.

1. On the Sub-Rules tab, right-click and select New

2. Enter Rule Names and assign Classifications and Common Events for each EVID as specified
in the table above
3. In the Mapping Tags tab, configure the following for the vmid row:
Set the Match column to Equal To (=)
In the Value column next to the process row, type HTTPD
In the Value column next to the in the vmid row, type the associated EVID number

Chapter 5 | Enabling Custom Rules 183

 4. Leave all values in the Settings tab at the default
5. Ensure the sort order is configured to avoid logs matching less specific rules first
a. Right-click in the list of Sub-rules and select Edit Sub-rule Sorting
b. Highlight a row and use the arrow buttons to change the sort order

6. Save your rule and close the Rule Builder window

184 Chapter 5 | Enabling Custom Rules

 Exercise 4:
Create an MPE Rule with Sub-rules

The purpose of this exercise is to practice configuring Custom MPE Sub-rules.

Part One: Create a Base-rule

1. Perform an Investigation so you can copy log messages that contain the EVID 0041 into the Rule
Builder tool.
2. Use the Rule Builder tool to create a new Base-rule.
3. Create a Base-rule that matches EVID 0041 (Netflow Logs).

Rule Name: EVID 0041

Classification: Operations : Network Traffic

Common Event: Netflow Summary

Log Source Type: Syslog - LogRhythm Syslog Generator

The Base-rule should parse the following metadata:

Vendor Message ID
Protocol
Host (Origin)
TCP/UDP Port (Origin)
Host (Impacted)
TCP/UDP Port (Impacted)
KBytes Received <bytesin>
KBytes Sent <bytesout>
Packets Received
Packets Sent

Chapter 5 | Enabling Custom Rules 185

 Part Two: Create Two new Sub-rules
Create the following Sub-rules for your new Base-rule:

1. Large Bytes In

Rule Name: Large Bytes In

Criteria: Bytes In > 200
Classification: Network Traffic
Common Event: Packet That Is Too Large Received From Server
Risk Rating: 0

2. Large Bytes Out

Rule Name: Large Bytes Out

Criteria: Bytes Out > 200
Classification: Network Traffic
Common Event: TCP Urgent Data Enforcement
Risk Rating: 0

Your new Custom MPE Rule should look like this when completed:

Be sure to save your rule and close the window.

186 Chapter 5 | Enabling Custom Rules

 Exercise 5:
Create an MPE Rule with Sub-rules

The purpose of this exercise is to practice configuring new MPE Sub-rules.

Part One: Create a Base-rule

1. Perform an Investigation so you can copy log messages that contain Event ID 0037 (Transaction
Logs) into the Rule Builder tool.
2. Use the Rule Builder tool to create a new Base-rule.
3. Create a Base-rule that matches EVID 0037.

Rule Name: EVID 0037

Classification: Audit : Other Audit Success

Common Event: Sales Transaction

Risk Rating: 0

The Base-rule should parse the following metadata:

Vendor Message ID
Host (Impacted)
User (Origin)
User (Impacted)
Amount

Chapter 5 | Enabling Custom Rules 187

 Part Two: Create new Sub-rules
Create one Sub-rule for your new Base-rule that meets the following criteria:

Rule Name: Large Transaction

Classification: Audit : Other Audit Success

Common Event: Sales Transaction

Criteria:

Account = account010 or account011

Amount > $250.00

Your rule might looks something like this when completed:

188 Chapter 5 | Enabling Custom Rules

 Exercise 6:
Challenge: Create an MPE Rule with Sub-rules

The purpose of this exercise is to practice configuring more complex MPE Sub-rules with less instruction.

Part One: Create a Base-rule

Use the Rule Builder tool to create a Base-rule that matches the Event ID 0044 (Email Logs).

Rule Name: EVID 0044

Classification: Information

Common Event: Email Subject Information

Risk Rating: 0

The Base-rule should parse the following metadata:

Vendor Message ID
Sender
Recipient
Subject

Chapter 5 | Enabling Custom Rules 189

 Part Two: Create new Sub-rules
Create one Sub-rule for your new Base-rule that meets the following criteria:

Rule Name: Flagged Email Subjects

Classification: Security : Misuse

Common Event: Unauthorized E-mail

Criteria: Subject contains ‘confidential’ or ‘earnings’

When completed your rule might look something like this:

190 Chapter 5 | Enabling Custom Rules

 Exercise 7:
Enable New MPE Rules

The purpose of this exercise is to practice enabling all the new Custom MPE Rules you created.

Part One: Change the Status of your new MPE Rules from Development to
Production
Follow the instructions below to change the status of each of your new MPE Rules.

Important!
Because this is a Training environment, time and data is limited; therefore, we will not be putting
our Rules into to Test status. Today you will move these Rules directly from Development to
Production status. LogRhythm recommends you Test new MPE Rules in your organization's
environment before moving them into Production status.

1. To find and edit your MPE Rules, click the icon to open the Rule Library (also known as the Rule
Browser)

2. The Rule Browser window will open, but may not display your new Custom MPE Rules. Click the
View menu to change the settings to Show Development Rules and Show Custom Rules.

3. Highlight and double-click to open your first Custom Rule.

4. Click to change the Rule Status from Development to Production.

Chapter 5 | Enabling Custom Rules 191

 5. To change the status of the associated Sub-rules, right-click in the list of Sub-Rules and select
Synchronize With Base-rule.
6. Save your changes.
7. Repeat steps above for all of your new Custom MPE Rules. If you successfully completed all of
the previous Exercises and Challenges you should have the following new MPE Rule Names:
EVID 0012
EVID 0011
EVID 0008
EVID 0009
EVID 0045
Operations Logs
EVID 0041
EVID 0037
EVID 0044
8. When completed with this Exercise, the Rule Status for all of your new Custom MPE Rules
should look like this:

192 Chapter 5 | Enabling Custom Rules

 9. When completed with this Exercise, the Rule Browser will look something like this:

10. Change the Sort Order of your new Custom MPE Rules to ensure they are processed before the
System MPE Rules.
a. Click to open the Edit menu and select Edit Base-rule Sorting.
b. You might see a message about Rule Validation; if so, click OK.
c. Your Custom MPE Rules are probably at the bottom of the list; you'll want to move them to
the top of the Sort Order. Scroll down to locate your rules.
d. Click to place a checkmark in the box under the Sort Above System column for one of your
rules.
e. Click the top arrow button to move it all the way to the top of the Rule Sorter window.

f. Repeat steps d and e until all of your new Custom MPE Rules have been moved.

Chapter 5 | Enabling Custom Rules 193

 Part Two: Enable your new MPE Rules
Follow the instructions below to enable your new MPE Rules in LogRhythm.

1. Open the Deployment Manager.

2. Select the Log Processing Policies tab.
3. Double-click to select the Syslog - LogRhythm Syslog Generator Log Source Type.
4. The MPE Policy Editor appears.
5. Click to put a checkmark the box in the Edit field for each of your new rules so you can include
them in the MPE Policy.
a. Right-click and select Properties from the context menu.
6. The MPE Policy Rule Editor appears.
a. Click to put a checkmark the Enabled box.
i. All edits you make will apply to all the rules that are currently selected.
7. Click OK.
8. You'll return to the MPE Policy Editor window.
9. Click OK.

194 Chapter 5 | Enabling Custom Rules

Part Three: Verify Logs are Identified and Parsed in LogRhythm
You'll want to run an Investigation to ensure your logs are now identified and parsed correctly in
LogRhythm.

1. From the LogRhythm Client Console, click the Investigate button

2. Click to Configure New Investigation to find all logs in the last hour from the Log Source Type:
Syslog - LogRhythm Syslog Generator
3. When the results are displayed, click to review the logs in the Log Viewer tab
a. Review the MPE Rule Name field. Make sure your new MPE Rule has been applied to the
appropriate log messages.
b. Review the following metadata fields to ensure they are populated correctly for the different
logs and their MPE Rules (not all fields will contain data for every log)
i. Common Event
ii. MPE Rule Name
iii. Host (Origin) and (Impacted)
iv. Vendor Message
v. User (Origin) and (Impacted)
vi. Object
vii. Session ID
viii. Group
ix. Process
x. TCP/UDP Port (Origin) and (Impacted)
xi. Protocol
xii. Packets Received or Packets Sent
xiii. Amount
xiv. Account
xv. Sender
xvi. Recipient
xvii. Subject

Chapter 5 | Enabling Custom Rules 195

 This page intentionally left blank.

196 Custom MPE Rules Using Regular Expression

CHAPTER 6
Best Practices
Rule Performance 198
Reviewing Processing Performance 200
The Efficiency Of A Regular Expression 204
RegEx Recommended Practices 207
The Whole Process For Custom MPE Rules 209
Exercises 211
Exercise 1: The Final Challenge 212

Custom MPE Rules Using Regular Expression │ Best Practices 197

 Rule Performance
In this course you've been introduced to regular expressions and how to incorporate them into LogRhythm
for use in MPE Rules. Writing regular expressions is a creative process and you've observed many
different ways to match and parse the same information. It may seem there are not many clear rules for
you to follow. This section contains some guidelines for use in creating custom MPE Rules to help with
processing performance.

It’s important to remember that the Rule Builder Test Results are only one aspect of MPE performance.

There are a lot of other factors along the processing chain that impact performance. LogRhythm's best
advice is to make your Rules process as fast as possible. A Rule Average of 4500 is generally considered
the floor for performance. This is not a great number, but provides some baseline as you are creating
rules.

Easy ways to optimize performance

As you become more experienced with regex, you may want to tune rules for improved processing
performance. Here are some tips you can use when you become more advanced in your regex skills.

Using more explicit Literal characters has better performance than using the Matching characters.
Use Character Sets [ ] when possible.
When testing log messages in the Rule Builder tool, LogRhythm recommends importing a large
volume of logs as well as logs of different formats. Testing logs of different formats gives a better
idea of how the regex will perform once put into Production. Do these logs of different formats
match the regex (they should not)? Is the performance of the rule still good when processing
against non-matching logs?
For non-Capturing Groups you can use ?: at the beginning to instruct the regex engine not to store
the data found in the group in memory. The information will be matched, but not captured; therefore
improving performance. The regex would look like this: (?:item1|item2|item3) or (?:stuff)

198 Chapter 6 | Best Practices

 Example
Windows has two versions of related logs; one for VMID0008, and one for VMID0009. You get the
information you need from VMID number and you don't need to capture the verbiage of the text in the
MESG. You use the ?: inside your character group to ensure the MESG exists in the log, but is not
captured or stored in memory.

Session IDs are almost always hexadecimal numbers. \x is a hexadecimal digit regular expression
character, but using it can cause poor performance. LogRhythm recommends using wildcard
characters .*? to skip over the information instead of using the hexidecimal regex character.

Chapter 6 | Best Practices 199

 Reviewing Processing Performance

This section highlights some resources for verifying or investigating issues with MPE performance. Mitig-
ation of performance issues is an advanced subject and requires knowledge of building, testing, and
adjusting rules in the MPE Rule Builder, as learned in this class, and a fairly robust knowledge of regular
expressions.

LPS Detail Logs

The MPE, a component of the Mediator Server service, keeps a record of how many times a rule has been
compared to a log message and the total amount of time spent processing logs against each rule. The
LPS Detail report generates detailed statistics for a Log Processing Policy for a given period of time and
is displayed in standard text format, readable with any text viewer. The LPS detail is a tool you can use to
identify if there is an issue with processing logs and what rule might be the offender.

The LPS Detail report contains a header and a section for each Log Processing Policy that is active. The
header contains information identifying the report, the date and time it was created, and the ID number of
the license being used to run the Mediator Server service. Each Log Processing Policy section contains
the data field (column) headers, the Log Source Type of the policy, the name of the policy, and then one
line for each Base-rule in the policy.

This report has four columns that are valuable for analyzing the performance of MPE Rule processing:

Base Rule is the name of the rule

Sort order is the order in which the rules will process the logs Automatic (A) or Static (S)

200 Chapter 6 | Best Practices

 Attempts is the total number of logs compared against this Base-rule and any associated Sub-
rules
% Match which shows the percentage of those attempts that were successful in matching the rule.

If you have a high number of Attempts and a low % Match rate then that rule is not efficient.

Chapter 6 | Best Practices 201

 SCMEDSVR Logs
The scmedsvr.log file contains errors, warnings, and data pertaining to System Monitor agent connec-
tions, and network operations. This log can be examined to review:

The number of logs inserted per second into the Indexer by the scmedsvr service.
The number of logs processed by the MPE per second.
The number of logs currently in the MPE Log Processing queue.
The percentage full of the Log Processing queue (logs not yet processed).
And much more.

In LogRhythm v7.2.2 a new feature was released that causes poor performing rules to timeout and lets
the logs fall to Catch-all rules. This option can be used to identify poor performing rules and capture log
samples to send to the Support team. You can verify this option is turned on by looking for the following
line in the scmedsvr.ini file:

"[Optional] MPERuleTimoutEnabled=True"

If this option is not enabled, contact LogRhythm’s Support team to discuss the pros and cons of this
feature before turning it on. If it is enabled, you can run an Investigation for logs with a Common Event of
LogRhythm MPE Rule Performing Poorly; the results will show all the offending rules and their associated
log messages. Look for repeat offenders; a few timeouts doesn't necessarily equal a poor rule or bad logs.
Save a sample of logs in an LLX file and open a Support ticket so the engineers can propose the
necessary improvements.

202 Chapter 6 | Best Practices

SCMPE Logs
The scmpe.log file contains errors, warnings, and data pertaining to the MPE component of the server.
Examine the scmpe.log file for any error messages related to log processing.

If you see repeat offending logs locking threads or timing out then there is likely something wrong with the
logs (format changes, extremely large log size, or excessive noise), or an issue with an MPE Rule (poor
performing regex, improper sort ordering of rules, or the regex just needs an update).

Chapter 6 | Best Practices 203

 The Efficiency Of A Regular Expression
Poorly written regular expressions can lead to poor processing performance in LogRhythm. Fully under-
standing the performance of a regex, requires additional knowledge about how the regex engine works.

The regex engine reviews an input string of data from left to right. When a match is found, it continues to
move through the data looking for additional matches. If the regex does not find another match, it
Backtracks in the string of data to look for another pattern match.

Index: The Index is what allows the regex engine to keep track of it's position in the string of data.
When the regex engine begins to read at the start of the string, it is at index 0. When it is 50
characters into the string, it is at index 49 (remember it starts at 0).
Backtrack: Using .*? creates multiple paths through a regular expression. The regex engine keeps
track of its location and if one path fails to yield a match, the regex engine will backtrack and try
another path.

While .*? is a powerful tool, it can negatively impact performance. If you have a reliable Literal characters
in a log message, tailoring the regex to that structure is preferable.

Example
A comma delimited log parser could use this type of regex ,[^,]*, and would perform far better than
,.*?, as it doesn't depend on back references.

Example
Effectively, the regexes .*? and [^*]+ get the same results; they will both non-greedily match the
same data. The difference between the two is their efficiency, due to backtracking, as shown in the
results below.

204 Chapter 6 | Best Practices

Using .*? causes the regex engine to backtrack over 50 times:

Chapter 6 | Best Practices 205

 Using [^*]+ allows the regex engine to process the string much more efficiently with no backtracking:

Character sets are really powerful for log messages with predictable delimiters. If you have a comma
separated log message, using a negative character set in between delimiters dramatically improves
performance. Character sets are inherently non-greedy.

Example
,[^,]*,
or
\|[^\|]*\|
Works well combined with quantifiers

Advanced Regex: ^(?:[^\|]+\|){4}active_threat

Input String: CEF:0|Confer|Confer_Syslog_Connector|2.0|Active_Threat|

206 Chapter 6 | Best Practices

RegEx Recommended Practices
In this course, you have been provided with a very basic understanding of writing Regular Expressions;
however they are extremely powerful and can become extremely complex. The more you read and play,
the better you'll become at writing RegExes. LogRhythm recommends you research, study, and practice
writing Regular Expressions to improve the processing performance of your Custom MPE Rules.

The following table, which can be found in the MPE Rule Builder Guide, highlights some recommended
practices for regex development in LogRhythm. Utilizing these regex characters could help you improve
the performance of an MPE Rule. If you notice processing issues, review the regex for your Base-rule and
modify, if possible.

Chapter 6 | Best Practices 207

 Note: All regex Examples shown in the table reference the following log message:
02/28/2007 16:55:22 MsgID=1590 : Failed authentication for user
“any.user” user account locked out

208 Chapter 6 | Best Practices

The Whole Process For Custom MPE Rules
The following steps can be used as a reference when creating new Custom MPE Rules for use in your
LogRhythm deployment.

1. Research your custom Log Source

a. Log collection method
b. Log message format
c. Log and severity levels
d. Determine what Event and Alarms are needed for the logs
2. Verify a Log Source Type exists
a. Review the Log Source Type Manager for a list of existing Log Source Types
b. Assign the new MPE Rule to a valid Log Source Type or create a new Custom Log Source
Type, if one does not already exist
i. Assign the correct Log Format that correctly matches the log collection method
i. Syslog
ii. Text File
3. Create your new Custom MPE Rule
a. Create a new MPE Rule for your custom Log Source
i. Name the rule something specific
ii. Assign a Common Event from the list of over 29,000 available options
b. Create a Base-rule and test your regex for successful matching results
i. Create Sub-rules and test for successful matching results
i. Assign a Common Event for each Sub-rule
c. Create a Catch-all rule for unknown logs that might be generated by this Log Source

4. Change the Rule Status, of your new Base-rule and its associated Sub-rules, to Test
5. Assign a Log Processing Policy
a. Review the list of existing Log Processing Policies in the Deployment Manager
b. Select the appropriate Log Source Type
c. Create a new Log Processing Policy if one is not already created
i. Give it a name and select the appropriate Log Source Type
6. Enable the new MPE Rule
a. Edit the properties of the Log Processing Policy
b. Select the Enable option and set the Risk Based Priority value
7. Verify it was tested successfully by the MPE
a. If not, make changes to the rule for improved efficiency
i. Enable to Test again

Chapter 6 | Best Practices 209

 8. Edit Base-rule Sorting
a. Detailed rules should be on top and more general rules on the bottom
b. Catch-all rules should be at the bottom of all other rules
9. Change the Rule Status to Production in the Rule Builder
10. Enable the new MPE Rules
a. Edit the properties of the Log Processing Policy
b. Enable the new Custom MPE Rules and set the Risk Based Priority value
i. Your MPE Rules and your Log Processing Policy are associated to your custom Log
Source Type
11. Verify the logs are identified and metadata is parsed by the rule as expected
a. Perform an Investigation or review the logs in the Dashboard
12. Make changes, if needed, to ensure the log is processed as desired
a. Repeat verification of log processing and make changes until log is processed as desired
13. If you've just created a new MPE Rule for a device that will likely be used by others, submit it to
the LogRhythm Labs team for review
a. If approved, the Labs team can add it permanently to the list of available Log Source Types,
MPE Rules, and perhaps create some additional Sub-rules made available to all customers
i. The LogRhythm Labs engineers can review your logs and MPE Rules at no cost;
however their time and resources are limited, and there is no guaranteed time frame
for response to your request
ii. The LogRhythm Professional Services (PS) engineers can assist with writing regexes
for custom Log Sources in a more timely manner; however their time is billable

210 Chapter 6 | Best Practices

EXERCISES
Introduction
Exercise 1: The Final Challenge 212

Chapter 6 | Best Practices 211

 Exercise 1:
The Final Challenge

This section will be a challenge. You'll apply all you've learned in this training session to collect logs from
a new Log Source. Review the details provided about this scenario and then follow the steps to add this
new device into your LogRhythm deployment.

Scenario Details
Your IT department has purchased and installed a new IDS (Intrusion Detection System) device in your
infrastructure. Your manager has tasked you with bringing logs from this device into LogRhythm.

Important!
Here is a sample log from this device:
06/28/2017 02:41:06 PM alert action: 1-28255 Win.Trojan.Kuluoz
Detected sHost:150.94.138.231 dHost:195.190.211.155 email:-
frederick@cert.logrhythm.com

You've done some research about this new IDS device and your LogRhythm platform. The information
listed here is what you know:

The new IDS device's hostname: IDS001

The System Monitor responsible for gathering logs from this new Log Source: LRVM-XM

Log Message Type: Flat File

Path to log messages: C:\LogRhythm\RuleGenerator\CustomIDSlog.txt

Run Script for Scenario

1. Because this is a Training environment and we are pushing fake logs into our environment, you'll
need to run a script now to generate some new IDS log messages
a. Navigate to the Desktop on your Training VM
b. Double-click on the RulesChallengeDataGenerator.py icon

c. Type 1 and press Enter to execute the script which will generate IDS logs

212 Chapter 6 | Best Practices

Part One: Add the new Log Source to a System Monitor in LogRhythm
1. In the LogRhythm Deployment Manager, open the System Monitors tab
2. Double-click on the LRVM-XM System Monitor to open the Properties window
3. Right-click in the list of Log Sources and select New
4. Click the button next to the Log Message Source Type to make a selection
a. Click to select the System Record Type Filter
b. Browse through the list of Log Source Types. Do you see one listed for Flat File - IDS?
c. Because there is not a Log Source Type specific to your new IDS device, select the generic
Log Source Type:
i. In the Text Filter box, type other and click Apply
ii. Double-click to select Flat File - Other from the list
i. This generic Log Source Type can be used to identify many types of Flat File
log messages
5. In the Log Message Source Name box, type IDS001
6. Click the drop-down arrow next to the Log Message Processing Engine (MPE) Policy and
select LogRhythm Default

7. Click the Flat File Settings tab

Chapter 6 | Best Practices 213

 8. Enter the File Path to the file from which the new log messages will be gathered:
C:\LogRhythm\RuleGenerator\CustomIDSlog.txt
9. Click the button next to Date Parsing Format to make a selection
a. Scroll down to locate, then click to select LogRhythm Syslog File
i. NOTE: This Date Format does match the date format in our sample log message
b. Click OK

10. Click OK again to save your new Log Source Properties

11. You should see the new IDS001 Log Source Name in the list in the System Monitor Agent
Properties window

12. Click OK again to close the window

214 Chapter 6 | Best Practices

Part Two: Activate the Log Source to Start Gathering Logs
1. Click to open the Log Sources tab in the Deployment Manager
2. You should now see IDS001 in the Log Source tab in the Deployment Monitor; search the list to
verify it is there
3. Look at the Last Log Message field. It is empty because no logs have been gathered from the
IDS001 device yet

4. Click to put a checkmark the Action box for IDS001

5. Right-click and select Actions - Activate
6. Click to Refresh the screen

7. Review the Last Log Message field for IDS001 again

a. Do you see a date and time entry there?
i. If not, refresh the screen or wait a couple more minutes
ii. If you still do not see a Last Log Message date/time, verify your File Path was
entered correctly:
i. C:\LogRhythm\RuleGenerator\CustomIDSlog.txt

Chapter 6 | Best Practices 215

 Part Three: Review Logs in LogRhythm
1. Run an Investigation to search for logs in LogRhythm from your new IDS001 Log Source
2. Review the Investigation results in the Log Viewer tab

a. Scroll over to review the MPE Rule Name used to parse the logs
i. Notice it is a Catch All rule
b. Scroll over and review the metadata that was parsed from the logs
i. Classification
ii. Common Event
iii. MPE Rule Name
iv. Direction
v. Zone
vi. Entity
vii. Host (Origin) and Host (Impacted)
c. Notice the data that was NOT parsed from the logs
i. VMID
ii. Correct Host (Origin)
iii. Correct Host (Impacted)
iv. Recipient
d. Based on these results, the desired metadata is not being parsed correctly from your new
Log Source, IDS001
i. Since data is not being parsed correctly with the currently configured Log Source Type
and it's associated MPE Rules, you'll need to create a new custom Log Source Type
with associated MPE Rules

216 Chapter 6 | Best Practices

 3. Highlight all the logs in the Log Viewer, right-click and select Send All Logs
a. Click Save to File
i. Save the logs to the Desktop
ii. Name the file: IDSlogs.llx
iii. Click Save

b. Click OK to acknowledge the Export Complete message

c. Click Close on the Log Message Submission Tool window

Chapter 6 | Best Practices 217

 Part Four: Create a new Custom Log Source Type
The Log Source Type used previously, System : Flat File - Other, did not parse your IDS logs correctly.
Since you have to create a new MPE Rule to parse your logs correctly, you'll also need to create a new
Log Source Type.

1. Open the Deployment Manager

2. Click the Tools menu and select Knowledge - Log Source Type Manager
3. Click the New button
4. Enter the following information:
a. Name: Custom IDS
b. Abbreviation: IDS
c. Log Format: Text File
d. Click OK
e. Click Close

218 Chapter 6 | Best Practices

Part Five: Create a new MPE Rule
Now you need to create new rules to process logs from your new Log Source Type.

1. Open the MPE Rule Builder tool

2. Click to open the Test Center tab
3. Right-click in the bottom of the panel and select Import LogRhythm Log Export (llx) File
4. Select your IDSlogs.llx file from the Desktop and click Open
5. You should now see your list of log messages in the Test Center window

5. Create one new Base-rule that parses the following data from the all of the IDS logs
a. VMID
b. Host (Origin)
c. Host (Impacted)
d. Email Recipient (optional)

6. What did you name the new Base-Rule?

7. What Common Event did you assign?
8. What Log Message Source Type Association did you assign?

Chapter 6 | Best Practices 219

 The Rule Builder should look something like this:

220 Chapter 6 | Best Practices

Part Six: Create new Sub-rules
There are several different VMIDs found in the log messages. These VMIDs are associated to different
types of activity, as shown in the log message. You'll need to create new Sub-rules for the eight different
VMIDs.

Hint
All of the Sub-rules in this exercise will use a Common Event under the Classification of
Security : Attack. You may need to do some research (on the internet ) to learn more about the
message so you can assign an appropriate Common Event. See Example below.

Chapter 6 | Best Practices 221

 Example
The "DOS squid WWCP I_SEE_YOU message" describes an Overflow Attempt type of event.

The Sub-Rule for this VMID should be assigned an existing Common Event that describes this type
of event, such as Buffer Overflow/Underflow.

222 Chapter 6 | Best Practices

Your Sub-rule for this log message should look something like this:

These tend to be subjective. Use your best judgment when creating Names and assigning Common
Events to your new Custom MPE Rules. The new Sub-rules might look something like this:

Chapter 6 | Best Practices 223

 Your Test Results might look something like this, and each log should have matched one of your new
Sub-rules:

224 Chapter 6 | Best Practices

Part Six: Enable rules
Now that your new Custom MPE Rule has been tested and matched the logs successfully, you need to
create a new Log Processing Policy and Enable the Rules for use in LogRhythm.

1. Change the Rule Status to Production for your Base-rule and all Sub-rules and Save your new
MPE Rule
a. Use Synchronize Rule Status to easily change Status for all the Sub-rules
i. To change the status of the associated Sub-rules, right-click in the list of Sub-Rules
and select Synchronize With Base-rule
2. Open the Deployment Manager and click on the Log Processing Policies tab
3. Right-click inside the list of Log Processing Polices and select New
4. Locate and select the new Custom Log Source Type you created earlier in this exercise: Custom
IDS
a. In the MPE Policy Editor window, enter the following Name: Custom IDS
b. Place a checkmark in the boxes next to all of the Rules you created for this Log Source Type
5. Right-click and select Properties
a. Place a checkmark in the box next to Enabled
b. Leave the other settings at their default
c. Click OK, you should now see all the Rules listed in the MPE Policy Editor window have a
checkmark in the Enabled column
6. Click OK again
7. You should now see eight Enabled Rules associated with the Custom IDS Log Source Type
8. Click OK to close the MPE Policy Editor window
9. You need to assign this new Log Processing Policy, and the new Log Source Type you created
earlier, to your IDS001 device
a. Navigate to the Log Sources tab
b. Locate the Log Source Name: IDS001
c. Double-click to open the Log Message Source Properties window
i. Assign the Custom IDS Log Message Source Type
ii. Select the Custom IDS Log Message Processing Engine (MPE) Policy
iii. Click OK

Chapter 6 | Best Practices 225

226 Chapter 6 | Best Practices
Part Seven: Collect new Logs from IDS001
1. Because this is a Training environment and we are pushing fake logs into our environment, you'll
need to run a script now to generate some new IDS log messages
a. Navigate to the Desktop on your Training VM
b. Double-click on the RulesChallengeDataGenerator.py icon
c. Type 1 and press Enter to execute the script which will generate IDS logs into the
CustomIDSlog.txt file

2. Navigate back to the Log Sources tab in the Deployment Manager

a. Review the Last Log Message field for IDS001 again
i. Do you see a new date and time entry there?
i. If not, refresh the screen or wait a couple more minutes

Chapter 6 | Best Practices 227

 Part Eight: Verify your new Custom MPE Rules are working properly
Run an Investigation to verify your new Rules are parsing data from the Custom IDS logs.

1. Review the metadata fields

a. How many different Rules are matched? _________________________________________
b. Are all the required metadata fields populated? _____________________________________
c. What other metadata fields are now populated? ____________________________________
___________________________________________________________________________
d. Look at the Classification and Common Event fields for the old logs versus the new logs.
Why do you think they are different? _____________________________________________
___________________________________________________________________________
e. Look at the Location, Direction, and Entity fields for the old logs versus the new logs. Why
do you think they are different? _________________________________________________
___________________________________________________________________________

228 Chapter 6 | Best Practices

GLOSSARY
A settings, Expiration, and other
rule properties. A rule can have
up to 3 rule blocks.
AD
Active Directory. AI Engine Rule Block
A sub component of an AI Engine
Advanced Intelligence Engine (AI Rule that defines configuration
Engine) parameters, or criteria, which
logs must meet in order for the
The LogRhythm component,
log to be considered satisfied by
sometimes referred to as AI
the Rule Block. A rule can have
Engine or AIE, that performs
up to 3 Rule Blocks. There are 9
high-level, real-time analysis of
distinct variations of Rule
log messages forwarded by the
Blocks.
Log Managers. The AI Engine
can elevate logs using a complex
pattern matching rule-set, which Alarm
correlates and detects sophist-
icated intrusions, insider threats, A record indicating that an Event
operational issues, and audit or triggered an Alarm rule.
compliance issues.
Alarm Card
AI Engine Rule An Alarm record that shows
Advanced Intelligence Engine Alarm details.
rule. A Knowledge Base object
that analyzes if logs meet Alarm Notification
specific criteria to generate an
Event from the AI Engine. The A notification of an alarm via a
rule contains settings for notification policy: SMTP (email),
Common Events, Event SNMP, or Text Files. The
Suppression, Alarm/Notification person, role, or group to notify is
set on the Alarm Rule Properties

Custom MPE Rules Using Regular Expression │ Glossary 229

 Notify tab. The Alarm Notification rules identify log messages by matching
Policies are set via the Notification Policy the fields to a specific log format or
Manager which is a distribution tool option pattern.
within Deployment Manager.

C
Alarm Suppression
A setting to suppress identical alarms Case Card
based on the Events triggering the alarm
The most basic element of a case in the
within a given time span.
Web Console Cases tab; provides basic
case information.
Application
A web application or a protocol; a record Case Management
that defines an application and its ports
A forensic tool for tracking and
and protocols so that the MPE rules can
documenting suspicious logs and alarms
identify a log origin. Applications are
that are believed to be related to the same
managed via the Application Manager
threat.
accessed from the Knowledge Tools
menu within Deployment Manager.
CBDM
ARM Classification Based Data Management.
Available in the Global Data Management
Alarming, Reporting, and Response
Settings found in the Platform Manager
Manager. A Windows service installed on
tab in the Client Console.
the Platform Manager, which is
responsible for processing Reports and
Alarm rules and taking the appropriate Classification
response.
A second-tier group used to categorize all
identifiable logs and Events. There is one
B or more Classifications associated with a
Classification Type. Classification Types
include: Audit, Operations, and Security.
Base-Rule
A Classification can have one or more
Part of the MPE Rule that contains a Common Events, which are associated
tagged regular expression (regex) used to with an MPE Rule.
identify the pattern of a log and isolate
interesting pieces of metadata. Using a
tagging system, these metadata strings Classification Type
can be directed to special fields used by A first tier group used to categorize logs
LogRhythm to better interpret a log or and Events. There are three types: Audit,
specifically identify it. In general, base- Security, and Operations. Classifications

230 Custom MPE Rules Using Regular Expression │ Glossary

 are grouped into one of these three types.
There is one Classification type for one or Data Processor (DP)
more Classifications. The central processing engine for logs
sent from the System Monitor Agents.
Client The DP contains the Mediator service,
which is responsible for log identification
Initiator of a session, such as a and classification. A DP can be installed
workstation or laptop. on the Platform Manager appliance or it
can be a separate appliance in the SIEM
deployment. Large environments may
Client Console need more than one Data Processor in the
The LogRhythm Platform component that deployment.
provides deployment administration and
user interaction from a thick-client
Graphical User Interface (GUI).
Deduplication
A process that recognizes and consol-
idates duplicate Event data from log
Common Event sources into a single, aggregate record.
A short plain-language description for the All raw log data is captured and archived
activity described in a log message. for accuracy and compliance, while the
Common Events are associated with log deduplication process eliminates
Classifications. A Common Event is redundant online data and optimizes
created and managed through the Tools - forensic search capabilities and storage
Knowledge menu via the Common Event utilization. Deduplication is a Log
Manager. Common Events are Processing Setting that can be set at the
associated with MPE rules (base and log source or log processing policy and
sub) located within the Tools - Knowledge can be overridden using a Global Log
menu MPE Rule Builder option. There is a Processing Rule (GLPR).
one to one relationship between an MPE
rule and a Common Event.
Deep Packet Inspection (DPI)
Analyzes network data using a variety of
D methods, including pattern matching,
heuristic modeling, signatures for session
Dashboard identification, application identification,
and metadata extraction.
A layout in the Console that contains
graphs and charts in easy-to-read
formats, which allows you to view Events Deployment Manager
and drilldown on the data for further invest-
A utility window in the LogRhythm Client
igation. The Web Console includes an
Console. People with LogRhythm
Executive Dashboard, Security Analyst
administrator credentials use it to
Dashboard, and an IT Operations
Dashboard, by default.

Custom MPE Rules Using Regular Expression │ Glossary 231

 configure and manage LogRhythm sites require multiple Entity records.
components and functionality such as Entities are displayed in a tab within
alarming and reporting. Deployment Manager.

Deployment Monitor Event

A window in the Client Console that A log having more immediate operational,
provides administrators with a near-real- security, or compliance relevance.
time view of the performance of Events typically include logs that are
LogRhythm including host status, host classified as errors, failures, or attacks.
performance metrics, database
utilization, Data Processor metrics, and
Data Processor volume. F

DNS FIM
File Integrity Monitoring. An Endpoint
Domain Name Server.
Monitoring tool that monitors files and
directories for modifications. There are
E two modes: Standard and Realtime.

EMDB Flow
Event Manager Database. A database on A collection of activity by a single user on
the Platform Manager that contains the a single application.
LogRhythm configuration information.

G
Engine
NetMon's packet processing component Global Data Management Settings
that classifies data during Deep Packet
Administrators can enable global options
Inspection.
that override settings at the Data
Processor, Log Source, and MPE Policy
Entity levels. Global settings are applicable in
both Classification Based and Standard
A record that represents a logical Data Management configurations. Data
grouping. It organizes the deployment in Management profiles simplify config-
host records, network records, and the uration based on the deployment’s data
LogRhythm components. Small management model. Data Management
deployments may contain one Entity, settings have been pre-packaged into
while large deployments that span many configurations which support various
deployment models and uses of the
product: Collection Optimized, Search

232 Custom MPE Rules Using Regular Expression │ Glossary

 Optimized, Performance Optimized, or
Custom. You have the option to manage IDS
these settings at a more granular level. Intrusion Detection System.

Global System Settings Investigator

Global System Settings, available on the A search tool in the Client Console that is
Platform Manager tab in the Client used to query the Events Database on the
Console, include Global Maintenance Platform Manager,or the Data Indexer via
Settings and Identity Inference. Database the Data Processor for a given date range
backup paths and time-to-live (TTL) and with specified criteria for Log Source
values are configured here. and field filters, among other settings. The
results are displayed in a layout that can
GLPR be configured and saved and includes
various grids, charts, and graphs,
Global Log Processing Rule. A including Network Visualization. The data
Knowledge Base object used to provide a results can be drilled into to provide more
way to override settings defined in the detailed information.
Classification Based Data Management
(CBDM) or standard Data Management
settings (Log Message Source and Log IP
Processing Policy). It provides a way to Internet Protocol.
apply data management settings across
all Data Processors, Log Sources, and
Log Processing Policies to logs that meet K
specific criteria. The manager is
accessed via the administration tools
menu within Deployment Manager. KB Modules
Pre-packaged, customizable content
applicable to a specific regulation or need,
I
such as reports, investigations, alarms, or
AI Engine rules, to name a few. An
Icon example is a compliance module for PCI
which would include reports, invest-
A small graphic on the page, which you igations, and AI Engine rules that provide
can select to open a dialog. data relevant in meeting PCI
requirements. Modules are managed
Identified Log using the Knowledge Base Manager
accessed from the Knowledge tools menu
A log that has been sent through the within the Client Console. Modules are
Message Processing Engine (MPE) and imported with a Knowledge Base
processed against MPE Rules for parsing depending on their settings.
and forwarding as appropriate to the
Platform Manager.

Custom MPE Rules Using Regular Expression │ Glossary 233

 Receiver Manager and a Policy Manager
KB Objects both access via the distribution tools
Defined LogRhythm items such as Alarm menu within Deployment Manager.
Rules, AI Engine Rules, Lists, Report
Packages, Packaged Reports, Report List Manager
Templates, FIM Policies, Investigations,
Tails, and GLPRs that can be associated A tool in the Client Console used to
with a KB Module. manage Lists. A List is a record of data for
a given type to allow for grouping similar
values together in one location that can
Knowledge Base then be used with all LogRhythm tools
A LogRhythm package that includes both that allow filtering. For example, a List
required and optional content shared may include all PCI compliance related
across a LogRhythm deployment. It Log Sources or suspicious hosts or
consists of the core Knowledge Base as privileged users. These Lists can then be
well as modules. The core Knowledge used in filters within Investigator,
Base includes content applicable to all Personal Dashboard, and Reports (to
deployments, such as log processing name a few) while allowing it to be
rules, policies, and classifications. The created/modified from one location.
Knowledge Base is imported using the
Knowledge Base Import Wizard Log Message
accessed from the Tools - Knowledge
menu within the Client Console. A log in its original form as it was received
by the LogRhythm System Monitor
Agent. A raw log displayed in the Web
L Console's Analyzer grid. See also "Raw
Log."
Layout
The arrangement of charts and graphs Log Source
(widgets) on the Web Console A record that represents a single source
Dashboard. Layouts can be saved for of log data that is collected from a host. It
Private or Public use. is associated with a Log Source host,
collection agent (System Monitor), and
Log Message Source Type. It has
LDS
specific data management and log
Log Distribution Services. A policy based processing settings and can have a MPE
solution that allows users to forward policy applied.
specific syslog and non-syslog log
messages to an external syslog receiver
over TCP or UDP. It consists of a Logarithmic graphs
A graph that shows the rate of change or a
skew of multiplicative factors over time.

234 Custom MPE Rules Using Regular Expression │ Glossary

 Logger Metadata
NetMon's flow output component that Details of a log message in a simple
processes the metadata into "flows". format within the LogRhythm databases.
Metadata is parsed directly from a log
message (explicit) or can be inferred from
LogRhythm Console a log message (implicit). Includes the
See "Client Console" or "Web Console." fields that LogRhythm parses, derives,
and calculates from collected log data.

LogRhythm Platform
MPE
The fully integrated Security Information
and Event Management (SIEM) solution, Message Processing Engine. As part of
which includes the collection of the Data Processor's Mediator service,
LogRhythm components that work the Message Processing Engine is
together to bring log management, responsible for log identification and
advanced log analysis, Event classification, Event processing, and
management, monitoring, and reporting metadata processing.
into one integrated solution.
MPE Policy
Lucene search A collection of MPE Rules designed for a
An open source text retrieval library specific Log Source Type such as Cisco
released under the Apache Software PIX or Windows 2005 Security Event
License. Log. Only Logs that are generated from
Log Sources that have an assigned MPE
Policy are processed by the MPE. There
M can be multiple policies for a single Log
Source Type to allow for flexibility in
assigning the policy to various Log
MAC Address
Sources.
A Media Access Control address (MAC
address), which is a unique identifier
assigned to network interfaces for MPE Rule
communications on the physical network A record associated with a specific Log
segment. Message Source Type that includes a
Common Event, Base-rule regular
expression, Sub-rules, and other
Mediator service
processing and policy settings used for
A service running on the Data Processor processing logs.
responsible for log identification and
classification.

Custom MPE Rules Using Regular Expression │ Glossary 235

 N R

NAT Raw Log

Network Address Translation. See "Log Message."

Navigation bar RBP

The selections at the top of the page that Risk-Based Priority. A calculation that
allow you to move between Web Console results in a number between 1 and 100. It
pages. The currently active page is is used to determine how serious or
shown in blue. critical an Event is, based on a number of
other rating and probability factors.

P
Report Center
Packet An analysis tool in the Client Console that
provides users the ability to manage
A unit of data carried by a packet-
report templates, reports, and report
switched network.
packages.

PCAP File S
An industry-standard format for containing
packet capture data.
SIEM
Security Information and Event
Platform Manager (PM) Management. See also "LogRhythm
The central hub of Event and incident SIEM."
management, reporting, analysis, and
configuration across the entire
SMTP
LogRhythm SIEM deployment. The
Platform Manager includes the Alarming, Simple Mail Transfer Protocol.
Reporting, and Response Manager
(ARM), which is a Windows service
responsible for processing Alarm rules SNMP
and taking the appropriate response. The Simple Network Management Protocol.
LogRhythm SIEM includes only one
Platform Manager per deployment, which
is installed on its own appliance. SSL
Secure Sockets Layer.

236 Custom MPE Rules Using Regular Expression │ Glossary

 Sub-Rule URL
Part of the MPE Rule that differentiates Uniform Resource Locator.
log messages that match the same Base-
rule using values in the log. Sub-rule tags
can include a regex that only applies to User Activity Monitoring
the string in a specific field to identify An Endpoint Monitoring tool that is used in
information such as a log / event identi- conjunction with FIM, DLD, Process
fication number, a message string, or Monitor, and Network Connection Monitor
even a user or group name. to include the user information related to
the log activity. It is managed from the
System Monitor Agent properties within
SysMon
Deployment Manager.
LogRhythm SysMon, also called System
Monitor Agents, collect and forward raw
log data to the SIEM's Data Processor. W
SysMon can be installed on both
Windows and UNIX platforms. SysMon Web Console
are also integrated into LogRhythm's
components to collect diagnostic Web interface that runs on a browser or
information about the SIEM. tablet; provides easy access to analytic
tools, alarms, and customizable
dashboards to view Events and Alarms
T identified by LogRhythm.

TCP Widget
Transmission Control Protocol. A mini-application, such as a chart or
graph, that provides content for
Dashboards. Widgets can be re-sized and
U
repositioned in page Layouts to modify
existing Dashboards or create new ones.
UDP
User Datagram Protocol. Windows Host Wizard
A tool that allows users to configure
Unidentified Log LogRhythm to collect Windows Event
Logs. It has the ability to scan domains
A log that has been sent through the
and allows users to import computers. It
Message Processing Engine (MPE) that
is accessed via the Tools - Administration
was not identified against any of the MPE
menu within Deployment Manager.
Rules.

Custom MPE Rules Using Regular Expression │ Glossary 237

 X

XM Appliance
An all-in-one LogRhythm appliance that
includes the Platform Manager, Data
Processor, and Data Indexer. In small
deployments the AI Engine and Web
Console might also be installed on the
XM.

Zone
Used in an Entity Host Record or
Network Record to indicate whether it is
located Internal, External, or in the DMZ.

238 Custom MPE Rules Using Regular Expression │ Glossary

LogRhythm Headquarters
4780 Pearl East Circle
Boulder, CO 80301
+1 303 413 8745

For more information about our Global Offices visit: https://logrhythm.com/contact/