SPLUNK USER Manual 1.

SPLUNK USER Manual 1.3

Contents
1. 2. 3. 4. 5. 6. Starting Splunk web interface ........................................................................................................... 3 Splunk Search app ............................................................................................................................. 4 Searching data for specific host ........................................................................................................ 5 Splunk Indexes .................................................................................................................................. 6 Using time range option ................................................................................................................... 9 Searching erp logs, maxdb logs, webdisp logs, OS logs .................................................................... 9 6.1 6.2 6.3 6.4 7. 8. erp logs:..................................................................................................................................... 9 Database logs: ......................................................................................................................... 13 SAP Web dispatcher logs: ....................................................................................................... 15 OS logs:.................................................................................................................................... 17

Searching MaxDB KnlMsg trace files............................................................................................... 18 Installing Splunk universal forwarder ............................................................................................. 22 8.1 8.2 Steps to install universal forwarder ........................................................................................ 22 Steps to configure forwarding and receving ........................................................................... 23

9.

Configuring email alerts .................................................................................................................. 26 9.1 9.2 9.3 Configuration .......................................................................................................................... 26 Few examples to show how email alerting works .................................................................. 27 Managing scheduled searches and reports ............................................................................ 31 Setting up roles and users ........................................................................................................... 36 Setting up roles: ...................................................................................................................... 36 Setting up users....................................................................................................................... 39 Field tagging ................................................................................................................................ 41 Search using field tags............................................................................................................. 42

10. 10.2 10.3 11. 11.1

17-Nov-11

Page 1

SPLUNK USER Manual 1.3

Author
Anandagouda Gubbi, Rajashekhar (C5153390)

Change history table

Version 1.0(Draft) 1.1 1.2 1.3 Date 14-11-2011 17-11-2011 28-11-2011 Section 1, 7 8, 11 6,10,11 Page # 2, 13 15,25 9,36,41 Changes Added splunk indexer login details, web dispatcher logs search Added searching MaxDB KnlMsg files, configuring email alerts Added separate user login details for different logs, Added section Setting up roles and users, Added field tagging section

17-Nov-11

Page 2

SPLUNK USER Manual 1.3

1. Starting Splunk web interface

Login to SplunkWeb interface using the URL http://spwdfvm1490.wdf.sap.corp:8000 User Name: splunk Password : splunk Once logged in it will show below home screen.

To see the logs and traces Search application is used. Launch the Search application either from Splunk Home tab or from Welcome tab as shown in above screenshots. We have created separate users as below for Basis colleagues, DB colleagues, OS colleagues so that by default they can view corresponding logs.

17-Nov-11

Page 3

SPLUNK USER Manual 1.3

Logs erp logs Maxdb logs Webdispatcer logs OS logs

Username erpuser maxdbuser webdispuser oslogsuser

Password splunk splunk splunk splunk

Details of using each of these users will be explained in section 6.

2. Splunk Search app

1. Search app would look like as below, with different sections Sources, Hosts etc. Sources indicate the data inputs that are configured for monitoring; Hosts are the monitored systems which are connected to this Splunk indexer.

Next steps explain how to use different search functionalities of Splunk to view logs from different hosts and sources. 2. Click on Search link and click on search button, it will display all the events splunk indexer has recorded from all hosts and all sources. As shown in below screenshot.

17-Nov-11

Page 4

SPLUNK USER Manual 1.3

In the above screenshot we can find various options to filter the indexed data: Search field: where you can enter custom search strings to filter the data. Time range: to view the data of particular time range, you can select different time range from the dropdown. Fields section: We can see hosts(monitored hosts), source(monitors/input data) Detailed steps on how to use these options will be discussed later in this document.

3. Searching data for specific host

Click on host it will display all the monitored hosts and then from the list select the required host.

17-Nov-11

Page 5

SPLUNK USER Manual 1.3

Once the host is selected it will show all the data indexed from that host and you can see the selected host in search field.

4. Splunk Indexes
By default all the data consumed by splunk indexer is saved in default index called main. In Splunk we can create our own indexes and use them to push incoming data to specific index. In this setup we have created our own indexes named, erp, erp_maxdb, erp_webdisp, os_logs All the R/3 data coming from all hosts we are pushing into Index erp. All the database related logs/data is pushed into index erp_maxdb. Index os_logs is used for storing OS logs from all hosts. erp_webdisp is used for web dispatcher related logs. While searching we can make use of the index name to filter data. For example below screenshot shows erp index is being selected.

17-Nov-11

Page 6

SPLUNK USER Manual 1.3

Note: It will display matching terms as you type the search string.

To view the top10 sources, click on source link. And click on source you want to see the detailed data. Find the below screenshot.

17-Nov-11

Page 7

SPLUNK USER Manual 1.3

After selecting any source from above step you will see the data related to this source.

17-Nov-11

Page 8

SPLUNK USER Manual 1.3

5. Using time range option

Enter search string as explained in previous section and then select All time dropdown and depending on the requirement select the time range and then hit enter/click search button. As shown below.

6. Searching erp logs, maxdb logs, webdisp logs, OS logs

6.1 erp logs: We are saving all erp logs into index erp and have created separate user to view erp logs by default, with this user no need to mention index name in search string. Login to splunk using username/password as erpuser/splunk After logging in it will take you to Search page directly. There select any host as shown below.

17-Nov-11

Page 9

SPLUNK USER Manual 1.3

By default it will show only data from erp index as shown in below image, so that no need to specify index name in search string.

17-Nov-11

Page 10

SPLUNK USER Manual 1.3

Use the below search string: host="spwdfvml0247" source="/usr/sap/ix4/dvebmgs26/work*"

17-Nov-11

Page 11

SPLUNK USER Manual 1.3

Click on Show Source to view the logfile, it will display the logfile in separate window as shown in below screenshot.

17-Nov-11

Page 12

SPLUNK USER Manual 1.3

Similarly to view /usr/sap/ixv/dvebmgs00/work/available.log logs use the below search string. host="spwdfvml0247" source="*available.log*" OR host="spwdfvml0247" source=/usr/sap/ixv/dvebmgs00/work/available.log

6.2 Database logs: We are saving all maxdb logs into index erp_maxdb and have created separate user to view these logs by default, with this user no need to mention index name in search string. Login to splunk using username/password as maxdbuser/splunk After logging in it will take you to Search page directly. There select any host as shown below.

17-Nov-11

Page 13

SPLUNK USER Manual 1.3

17-Nov-11

Page 14

SPLUNK USER Manual 1.3

To access file /sapdb/data/wrk/ixv/dbm.ebp Use the search string: host="spwdfvml0249" source="/sapdb/data/wrk/ixv/dbm.ebp"

6.3

SAP Web dispatcher logs:

We are saving all web dispatcher logs into index erp_webdisp and have created separate user to view these logs by default, with this user no need to mention index name in search string. Login to splunk using username/password as webdispuser/splunk After logging in it will take you to Search page directly. There select any host as shown below.

17-Nov-11

Page 15

SPLUNK USER Manual 1.3

To search for web dispatcher available log use the below string. host="spwdfvml0063" source="/usr/sap/IXP/W00/work/available.log"

17-Nov-11

Page 16

SPLUNK USER Manual 1.3

Similarly we can search all the files that are configured to monitor with their paths or just by giving wildcard search (*) and with specific time range filters can also be applied as already explained in above section. 6.4 OS logs: Splunk can index all OS logs from different systems (windows, Linux, Unix..etc) We are saving all OS logs into index os_logs and have created separate user to view these logs by default, with this user no need to mention index name in search string. Login to splunk using username/password as oslogsuser/splunk After logging in it will take you to Search page directly. There select any host as shown below.

17-Nov-11

Page 17

SPLUNK USER Manual 1.3

Similarly we can search for any OS logs just by giving host name and any file path or wild search, below are few examples. host="spwdfvml0247" source="/var/log/warn" host="spwdfvml0247" error

7. Searching MaxDB KnlMsg trace files

In MaxDB database /sapdb/data/wrk/SID/KnlMsg* files will be in pseudoxml format which would be difficult to read and analyse. So to display these logs in easily readable table format we have to follow the below steps. Login to splunk using maxdbuser / splunk so that no need to enter index name in search string. Perform search with the string host="spwdfvml0249" source=*KnlMsg It will display the results as below. In this click on Events Table button which is highlighted below. It will display the results in tabular format.

17-Nov-11

Page 18

SPLUNK USER Manual 1.3

In the above screenshot you can find the results in tabular format, which is easily readable. This will list the fields (columns) that are already selected which are shown on the left hand side of the above screen. If we need further fields to be added in the result table we can click on Pick Fields and select the fields of interest as shown below. Just select the field and it will be added in the selected field. Then we can click on search button again to display the results in tabular format.

17-Nov-11

Page 19

SPLUNK USER Manual 1.3

There is another option to view the results in tabular format using .. | table field1, field2 command. Use the search host="spwdfvml0249" source=*KnlMsg

17-Nov-11

Page 20

SPLUNK USER Manual 1.3

And then refine the search by using search string host="spwdfvml0249" source=*KnlMsg |table TIME,TEXT where TIME and TABLE are field names which you can see on the left side. which will display the results in table as below.

In the above search string we can use * which will display all the fields/columns host="spwdfvml0249" source=*KnlMsg |table *

17-Nov-11

Page 21

SPLUNK USER Manual 1.3

8. Installing Splunk universal forwarder

Forwarder is used to collect the data and forward it to the Splunk Indexer for indexing and searching. In our setup we have installed universal forwarders in all monitored systems (ex. IX4, IXV etc). Full Splunk indexer is installed on spwdfvm1490 which will receive data from all the universal forwarders and index. So we can call all the monitored systems as forwarders and the indexer as receiver.

8.1

Steps to install universal forwarder

Download the universal forwarder from http://www.splunk.com/download/universalforwarder Depending on the OS flavour where we want to install. To install in Linux systems we downloaded rpm package. In Linux systems run the below command to install the Splunk RPM in the default directory /opt/splunk: rpm -i splunk_package_name.rpm To install on windows download the corresponding software from same location and install via GUI. 1) Starting and stopping universal forwarder Go to splunk home directory as shown below

17-Nov-11

Page 22

SPLUNK USER Manual 1.3

On windows servers Go to %SPLUNK_HOME%\bin and run this command: >splunk restart

8.2

Steps to configure forwarding and receving

To set up forwarding and receiving, you need to perform two basic actions, in this order: Set up one or more Splunk indexers as receivers. These will receive the data from the forwarders. Set up one or more Splunk forwarders. These will forward data to the receivers.

Use Splunk Web to enable receiving on the indexer designated as receivers. Open Splunk GUI and then click on Manager and then select Forwarding and receiving.

Click on Add new in configure receiving

17-Nov-11

Page 23

SPLUNK USER Manual 1.3

Below is our configuration:

Configure/add forwarder server:

The outputs.conf file is unique to forwarders. It defines how forwarders send data to receivers. We can specify some output configurations at installation time (universal forwarders only) or through Splunk Web (heavy/light forwarders only) or the CLI, but most advanced configuration settings require that you directly edit outputs.conf. The universal forwarder on Linux/Unix systems does not come with a outputs.conf file by default. When you use the CLI to configure the forwarder's outputs, Splunk creates outputs.conf in /opt/splunkforwarder/etc/system/local/ Below steps describe how to add a forward server. Navigate to /opt/splunkforwarder/bin Syntax is: ./splunk add forward-server server:port Where server is host name of the Splunk indexer and the port number (default port is 9997) For example: ./splunk add forward-server spwdfvm1490:9997 will connect the forwarder server to splunk index server spwdfvm1490 and outputs.conf file will be created in path /opt/splunkforwarder/etc/system/local

17-Nov-11

Page 24

SPLUNK USER Manual 1.3

From this configuration forwarder will understand to which receiver system it should forward the data. Next step is what all data needs to be sent or we need to define what all files needs to be monitored. This will be configured in 3 ways: using splunk web, Command Line interface(CLI) or edit inputs.conf file.

Adding monitors using splunk CLI:

To use Splunk's CLI, navigate to the $SPLUNK_HOME/bin/ directory and use the ./splunk command from the UNIX or Windows command prompt. Syntax is, splunkforwarder/bin> ./splunk add monitor <source> -<parameter> value source is the path of the file to be monitored, parameter can be index,host,sourcetype etc. This parameter should be preceded with symbol - For example,

In above screenshot OS log monitor (/var/log/messages) will be added. To push this data to specific index for example os_logs index we have to edit and mention the index name in file /opt/splunkforwarder/etc/apps/search/local/inputs.conf as shown below.

Save the file and do the restart of the splunk forwarder.

Note: If the data needs to be pushed to a specific index, do the restart only after adding index name by editing the monitor in inputs.conf file. Else the data will be saved in default main index.

Adding monitors by editing inputs.conf file: Navigate to inputs.conf file in the path /opt/splunkforwarder/etc/apps/search/local

17-Nov-11

Page 25

SPLUNK USER Manual 1.3

Below is the monitors added in this inputs.conf and after that restart the splunk forwarder.

If we want to add any new monitor, this file needs to be modified with new monitors.

9. Configuring email alerts

In Splunk we can send search results of any logs to our mailbox. This section explains the steps to configure email alerts and how this function works with few examples.

9.1

Configuration

Go to Manager, click on system settings--> Email alert settings and under Mail server settings Enter the mail host which is SMTP mail server as shown in below screenshots, in our case its mailwdf.sap.corp and other fields are optional and leave it as default.

17-Nov-11

Page 26

SPLUNK USER Manual 1.3

9.2

Few examples to show how email alerting works

Below search displays os_logs of host spwdfvml0249 for last 15 minutes time.

Enter the below string in search field. host="spwdfvml0249" | sendemail to=rajashekhar.anandagouda.gubbi@sap.com sendresults=true format=html server=mailwdf.sap.corp

17-Nov-11

Page 27

SPLUNK USER Manual 1.3

Optional parameters in above search string are format which will send the mail in desired format depending on need we can have format=html, format=text, or format=raw etc. Note: We have to mention our SMTP server name using the parameter server=mailserver.sap.corp Because by default splunk will assume splunk host as mailserver host and it tries to send mails which will not work. So we have to explicitly specify the mail server host.

With this sendemail functionality if some logs related to OS or DB or R/3 logs need to be send to responsible teams or an individual mailID, this can be achieved easily without logging into servers, which will reduce manual effort. Below is the mail which is received in our mailbox.

There is other option to send an alert which is described below. For example lets search with the string host="spwdfvml0249" db_offline for time range last 60 min

17-Nov-11

Page 28

SPLUNK USER Manual 1.3

Click on Create alert as shown in the above screenshot. And then enter the fields as shown below Search name, Search string (modify if required) and click on Next

In Next screen select the condition and schedule this search to run every 5 min and if there is any event in past 5 min it will trigger an alert mail.

If you enable throttling and mention number of minutes as 60 min, it will make sure that if the events are occurring every 5 min it will not trigger the alert mail every 5 min, only after 60 min it will trigger the the

17-Nov-11

Page 29

SPLUNK USER Manual 1.3

alert mail. And time range selected as last 60 min will get data of the last 60 min only so that it will prevent sending of the alerts for same event. Expiration time determines how long Splunk keeps a record of your triggered alerts. Severity can be set it Info,Low,Medium,High,Critical. As shown in the below screenshot.

Click on Next Enable Send email and then enter mail ID of the admins who should receive the alert mail. Include search results as inline to get as text in mail or select csv/pdf to get as attachment in mail. Enable tracking this will enable the trigger alerts to be viewed in Splunk Alert Manager and click on Finish as shown below.

17-Nov-11

Page 30

SPLUNK USER Manual 1.3

Success message will be displayed as shown below

Now this search will run every 5 minutes automatically and will trigger alert mail only if there are any events retrieved for db_offline in last 60 min.

9.3

Managing scheduled searches and reports

Scheduled search what we have created in previous section can be managed by using Searches & Reports->Manage Searches & Results as shown below.

17-Nov-11

Page 31

SPLUNK USER Manual 1.3

Here you can see the scheduled search IXV_db_offline we have created in last step.

From the above screenshot you can select View recent which will show the recent run details of this scheduled search as shown below.The search can be run by Clicking on Run or it can be disabled or deleted completely. Alerts that have been triggered can be seen by selecting Alerts on top right corner as shown below which will display the alerts that have been triggered so far.

Here you can view the results /edit search /you can delete the alert.

17-Nov-11

Page 32

SPLUNK USER Manual 1.3

10.3

Configure Alerting for missing forwarders:

Alerts can be configured for missing forwarders, for example if any forwarder is down we can receive the alert mail. Go to App->Deployment Monitor

Deployment Monitor will show Index Throughput, Forwarder Connections, Indexer Warnings, Forwarder Warnings etc.. In section Forwarder Warnings you can see Missing Forwarders link as shown below. In that Click on Configure Alerting.

Here we have configured the alerting settings as shown below.

17-Nov-11

Page 33

SPLUNK USER Manual 1.3

17-Nov-11

Page 34

SPLUNK USER Manual 1.3

This will trigger an alert email when any forwarder is down.

17-Nov-11

Page 35

SPLUNK USER Manual 1.3

10.

Setting up roles and users

10.2 Setting up roles: Login to splunk indexer using admin user(admin/tbntbitq). Goto Manager->Access Control and click on Roles ->Click on New Give name of the role as role_erp. Below screenshots shows the steps.

As role_erp is already created, below screenshots shows the properties of the same.

17-Nov-11

Page 36

SPLUNK USER Manual 1.3

17-Nov-11

Page 37

SPLUNK USER Manual 1.3

Similarly other roles have been created with the same properties as above. Click on any one of the roles highlighted in below screenshot to see its properties.

17-Nov-11

Page 38

SPLUNK USER Manual 1.3 10.3 Setting up users

Login to splunk indexer using admin user(admin/tbntbitq). Goto Manager->Access Control and click on Users ->Click on New Give name of the user as erpuser. Assign the role created(role_erp) in above step(section 10.1) to this user. Also set the initial password for this user. Below screenshots shows the steps.

As erpuser is already created, below screenshot just shows the properties and roles assigned to it

17-Nov-11

Page 39

SPLUNK USER Manual 1.3

Similarly we have created other users maxdbuser,webdispuser,oslogsuser Click on any of these users to view its properties as shown in the below screenshot.

17-Nov-11

Page 40

SPLUNK USER Manual 1.3

11.

Field tagging

In splunk we can tag fields for example host, source etc.. We have done host tagging to group the hosts belonging to particular landscape and SID. Below screenshots shows how to do tagging. Give the hostname in search string which will display the results, click on the down arrow next to host and click on Tag host as shown below

Then enter the tag names. We can enter multiple tags separated by comma as shown below. Here we have used tags LS_IXP and IXP LS: Stands for landscape, IXP: is SID of the SAP instance on this host.

For spwdfvml0247(host of IX4) we have used tags, LS_IXP and IX4 Similarly we have used tags for all IX* hosts.

17-Nov-11

Page 41

SPLUNK USER Manual 1.3 11.1 Search using field tags Now we can search using the tags created in previous section. Enter search string as index=erp tag::host=IX4. It will display the hosts tagged to IX4 which is only one host as shown in below screenshot.

Similarly use the search string as index=erp tag::host=LS_IXP, this will display all the hosts of IXP landscape.

17-Nov-11

Page 42