Preparation

1 Identification
2 Identification
2
■ A physical access to the suspicious system should be
offered to the forensic investigator. ■ Look for unusual files in /proc and /tmp. This last directory
- Huge number of authentication/login failures from local or
remote access tools (sshd,ftpd,etc.)
■ A physical copy of the hard-disk might be necessary for
is a place of choice for hackers to store data or malicious
binaries.
- Remote Procedure Call (RPC) programs with a log entry that
forensic and evidence purposes. If needed, a physical includes a large number of strange characters …)
access could be necessary to disconnect the suspected - A huge number of Apache logs mentioning “error”
machine from any network. Unusual Services - Reboots (Hardware reboot)
- Restart of applications (Software reboot)
■ A good knowledge of the usual network activity of the
(Linux only) Run chkconfig (if installed) to check for all enabled
services:
machine/server is needed. You should have a file on a # chkconfig --list Almost all log files are located under /var/log directory in most
secure place describing the usual port activity, to compare Linux distributions. Here are the main ones:
efficiently to the current state. Look at the running processes (remember: a rootkit might
■ A good knowledge of the usual services is needed. Don’t change your results for everything in this paper, especially /var/log/message: General message and system related stuff
/var/log/auth.log: Authenication logs
hesitate to ask a Unix/Linux Expert for his assistance, here!).
when applicable. # ps -aux /var/log/kern.log: Kernel logs
Use lsof –p [pid] on unknown processes /var/log/cron.log: Crond logs (cron job)
■ You should have a regularly updated list of all critical files, /var/log/maillog: Mail server logs
(especially SUID and GUID files) stored in a secure place You should know your usual running processes, and be able to /var/log/httpd/: Apache access and error logs directory
out of the network or even on paper. With this list, you can figure out which processes could have been added by a hacker. /var/log/boot.log: System boot log
easily separate usual SUID files and detect unusual ones. Pay a special attention to the processes running under UID 0. /var/log/mysqld.log: MySQL database server log file
■ Have a map of your usual port activity/traffic rules.
Unusual Network Activity
/var/log/secure: Authentication log
/var/log/utmp or /var/log/wtmp: Login records file
Try to detect sniffers on the network using several ways:
Look at your kernel log files for interfaces entering promiscuous To look through the log files, tools like cat and grep may be
mode such as :“kernel: device eth0 entered promiscuous mode” useful:
Identification
2 Use # ip link to detect the “PROMISC” flag. Prefer this method
cat /var/log/httpd/access.log | grep "GET /signup.jsp"

to ifconfig, since ifconfig does not work on all kernels. Unusual Kernel log Entries
Unusual Accounts
Look for any suspicious entry in /etc/passwd, especially with
■ Look for unusual port activity: # netstat –nap and # lsof –i
■ Look through the kernel log files on the system for
UID 0. Also check /etc/group and /etc/shadow. suspicious events.
to get more information about processes listening to ports.
Use :
Look for orphaned files, which could have been left by a deleted # dmesg
account used in the attack: ■ Look for unusual MAC entries in your LAN: List all important kernel and system information :
# find / \( -nouser –o –nogroup \) -print # arp -a # lsmod
# lspci
Unusual Files ■ Look for any unexpected IP address on the network. ■ Look for known rootkit (use rkhunter and such tools)
■ Look for all SUID and GUID files:
Unusual Automated Tasks File hashes
# find / -uid 0 \( –perm -4000 –o –perm 2000 \) –print
■ Look for unusual jobs scheduled by users mentioned in
■ Look for weird file names, starting with “. “ or “.. “ or “ “ : /etc/cron.allow. Pay a special attention to the cron jobs
Verify all MD5 hashes of your binaries in /bin, /sbin, /usr/bin,
/usr/sbin or any other related binary storing place. (use AIDE or
# find / -name “ *“ –print scheduled by UID 0 accounts (root): such tool)
# find / -name “. *“ –print # crontab –u root -l WARNING: this operation will probably change all file
# find / -name “.. *“ –print timestamps. This should only be done after all other
■ Look for unusual system-wide cron jobs: # cat /etc/crontab investigations are done and you feel like you can alter these
■ Look for large files (here: larger than 10MB) and # ls –la /etc/cron.* data.
# find / -size +10MB –print
On systems with RPM installed, use:
Unusual Log Entries # rpm –Va | sort
■ Look for processes running from or to files which have Look through the log files on the system for suspicious events, On some Linux, a script named check-packages can be used.
been unlinked : including the following: On Solaris: # pkg_chk –vn
# lsof +L1 On Debian: debsums –ac
On Openbsd (not really this but a way): pkg_delete -vnx
 Containment
3 Remediation
4
■ Backup all important data from the compromised machine, Temporary remove all accesses to the accounts involved in the
Incident Response Methodology
if possible using a bit-by-bit physical copy of the whole hard disk incident, and remove all fraudulent files.
on an external support. Also make a copy of the memory (RAM)
of the system, which will be investigated if necessary.

If the machine is not considered critical for the company and

can be disconnected, shut the machine down the hard way,
IRM #3
removing its power plug. If it is a laptop with a battery on, just
push the “off” button for some seconds until the computer
switches off.

Offline investigations should be started right away if the

Recovery
5 Unix/Linux Intrusion Detection
Live Analysis on a suspected system

identification step didn’t give any result, but the system is still No matter how far the hacker has gone into the system and the ___________________________________________________
suspected of being compromised. knowledge you might have about the compromission, as long as IRM Author: CERT SG / Cedric Pernet
the system has been penetrated, the best practice is to IRM version: 1.4
Try to find evidences of every action of the hacker: (using reinstall the system completely and apply all security fixes.
forensic tools like Sleuth Kit/Autopsy for example) In case this solution can’t be applied, you should: E-Mail: cert.sg@socgen.com

■ Find all files used by the attacker, including deleted files ■ Change all the system’s accounts passwords, and make
Web: https://cert.societegenerale.com
Twitter: @CertSG
and see what has been done with them or at least their your users do so in a secure way: they should use
functionality to evaluate the threat. passwords with upper/lower case, special characters,
■ Check all files accessed recently.
numbers, and at least be 7 characters long.

■ Check log files.

■ Check the integrity of the whole data stored on the Abstract
system, using MD5 hashes. This Incident Response Methodology is a cheat sheet dedicated
■ More generally, try to find how the attacker got into the ■ Restore all binaries which could have been changed to incident handlers investigating a precise security issue.
system. All leads should be considered. If no computer (Example: /bin/su) Who should use IRM sheets?
proof of the intrusion is found, never forget it could come  Administrators
from an insider.  Security Operation Center
■ Apply fixes when applicable, to prevent the same kind of


CISOs and deputies
CERTs (Computer Emergency Response Team)
intrusion, in case the attacker used a known fixed
vulnerability.

Report
Aftermath
6 Remember: If you face an incident, follow IRM, take notes
and do not panic. Contact your CERT immediately if
needed.

A crisis report should be written and made available to all of the

actors of the crisis management cell. The following themes
should be described:
■ Initial detection
Incident handling steps
■ Actions and timelines
6 steps are defined to handle security Incidents

■ What went right  Preparation: get ready to handle the incident

 Identification: detect the incident
■ What went wrong  Containment: limit the impact of the incident
■ Incident cost  Remediation: remove the threat
 Recovery: recover to a normal stage
Capitalize  Aftermath: draw up and improve the process

Actions to improve the Unix/Linux intrusion detection IRM provides detailed information for each step.
management processes should be defined to capitalize on this
experience.

This document is for public use