Scribd
Upload
90 views

Using Sysdig To Troubleshoot Like A Boss

This document provides an overview of how to use the sysdig tool to troubleshoot systems and applications. It demonstrates how to capture system calls and events, filter the output, and investigate specific processes, files, and network connections.

Uploaded by

IjazKhan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
90 views

Using Sysdig To Troubleshoot Like A Boss

This document provides an overview of how to use the sysdig tool to troubleshoot systems and applications. It demonstrates how to capture system calls and events, filter the output, and investigate specific processes, files, and network connections.

Uploaded by

IjazKhan
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 17

Using sysdig to Troubleshoot like a boss

1 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

sysdig
strace

tcpdump
sysdig

tcpdump

tcpdump
sysdig
sysdig
sysdig

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

2 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

sysdig
rpm

apt-get
yum

sysdig

sysdig
sysdig

curl

# curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add # curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list

/etc/apt/sources.list.d/

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

3 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

apt-get update

# apt-get update

sysdig

dpkg

# dpkg --list | grep header


ii linux-generic
Linux kernel and headers
ii linux-headers-3.11.0-12
ated to Linux kernel version 3.11.0
ii linux-headers-3.11.0-12-generic
ders for version 3.11.0 on 64 bit x86 SMP
ii linux-headers-generic
rnel headers

3.11.0.12.13

amd64

Complete Generic

3.11.0-12.19

all

Header files rel

3.11.0-12.19

amd64

Linux kernel hea

3.11.0.12.13

amd64

Generic Linux ke

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

4 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

linuxgeneric
uname

# uname -r
3.11.0-12-generic

apt-get
uname -r

# apt-get install linux-headers-<kernel version>

# apt-get install linux-headers-3.11.0-12-generic

sysdig

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

5 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

# apt-get install sysdig

sysdig

tcpdump

sysdig
tcpdump

-w

# sysdig -w <output file>

# sysdig -w tracefile.dump

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

6 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

tcpdump

sysdig

CTRL+C

sysdig
-r

# sysdig -r <output file>

# sysdig -r tracefile.dump
1 23:44:57.964150879 0 <NA> (7) > switch next=6200(sysdig)
2 23:44:57.966700100 0 rsyslogd (358) < read res=414 data=<6>[ 3785.473354] sysdig_probe: starting cap
ture.<6>[ 3785.473523] sysdig_probe:
3 23:44:57.966707800 0 rsyslogd (358) > gettimeofday
4 23:44:57.966708216 0 rsyslogd (358) < gettimeofday
5 23:44:57.966717424 0 rsyslogd (358) > futex addr=13892708 op=133(FUTEX_PRIVATE_FLAG|FUTEX_WAKE_OP) v
al=1
6 23:44:57.966721656 0 rsyslogd (358) < futex res=1
7 23:44:57.966724081 0 rsyslogd (358) > gettimeofday
8 23:44:57.966724305 0 rsyslogd (358) < gettimeofday
9 23:44:57.966726254 0 rsyslogd (358) > gettimeofday
10 23:44:57.966726456 0 rsyslogd (358) < gettimeofday

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

7 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

sysdig

-A

sysdig

# sysdig -A

# sysdig -A > /var/tmp/out.txt


# cat /var/tmp/out.txt
1 22:26:15.076829633 0 <NA> (7) > switch next=11920(sysdig)

sysdig

tcpdump
sysdig

sysdig

-l

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

8 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

# sysdig -l
---------------------Field Class: fd
fd.num
fd.type
fd.typechar

fd.name

the unique number identifying the file descriptor.


type of FD. Can be 'file', 'ipv4', 'ipv6', 'unix', 'pipe', 'e
vent', 'signalfd', 'eventpoll', 'inotify' or 'signalfd'.
type of FD as a single character. Can be 'f' for file, 4 for
IPv4 socket, 6 for IPv6 socket, 'u' for unix socket, p for pi
pe, 'e' for eventfd, 's' for signalfd, 'l' for eventpoll, 'i'
for inotify, 'o' for uknown.
FD full name. If the fd is a file, this field contains the fu
ll path. If the FD is a socket, this field contain the connec
tion tuple.

<truncated output>

sysdig

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

9 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

# sysdig -r tracefile.dump proc.name=sshd


530 23:45:02.804469114 0 sshd (917) < select res=1
531 23:45:02.804476093 0 sshd (917) > rt_sigprocmask
532 23:45:02.804478942 0 sshd (917) < rt_sigprocmask
533 23:45:02.804479542 0 sshd (917) > rt_sigprocmask
534 23:45:02.804479767 0 sshd (917) < rt_sigprocmask
535 23:45:02.804487255 0 sshd (917) > read fd=3(<4t>10.0.0.12:55993->162.0.0.80:22) size=16384

fd.name

# sysdig fd.name=/dev/log
14 11:13:30.982445884 0 rsyslogd (357) < read res=414 data=<6>[
e.<6>[ 582.136472] sysdig_probe:

582.136312] sysdig_probe: starting captur

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

10 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

# sysdig fd.name contains /etc


8675 11:16:18.424407754 0 apache2
passwd flags=1(O_RDONLY) mode=0
8678 11:16:18.424422599 0 apache2
8679 11:16:18.424423601 0 apache2
8680 11:16:18.424427497 0 apache2
8683 11:16:18.424606422 0 apache2
1.
8684 11:16:18.424623679 0 apache2
8685 11:16:18.424625424 0 apache2
9702 11:16:21.285934861 0 apache2
passwd flags=1(O_RDONLY) mode=0
9703 11:16:21.285936317 0 apache2
9704 11:16:21.285937024 0 apache2

(1287) < open fd=13(<f>/etc/apache2/.htpasswd) name=/etc/apache2/.ht


(1287)
(1287)
(1287)
(1287)

>
<
>
<

fstat fd=13(<f>/etc/apache2/.htpasswd)
fstat res=0
read fd=13(<f>/etc/apache2/.htpasswd) size=4096
read res=44 data=admin:$apr1$OXXed8Rc$rbXNhN/VqLCP.ojKu1aUN

(1287) > close fd=13(<f>/etc/apache2/.htpasswd)


(1287) < close res=0
(1287) < open fd=13(<f>/etc/apache2/.htpasswd) name=/etc/apache2/.ht
(1287) > fstat fd=13(<f>/etc/apache2/.htpasswd)
(1287) < fstat res=0

sysdig
sysdig
sysdig

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

11 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

-cl

sysdig

# sysdig -cl
Category: CPU Usage
------------------topprocs_cpu
Top processes by CPU usage
Category: I/O
------------echo_fds
Print the data read and written by processes.
fdbytes_by
I/O bytes, aggregated by an arbitrary filter field
fdcount_by
FD count, aggregated by an arbitrary filter field
iobytes
Sum of I/O bytes on any type of FD
iobytes_file
Sum of file I/O bytes
stderr
Print stderr of processes
stdin
Print stdin of processes
stdout
Print stdout of processes
<truncated output>

sysdig

sysdig

sysdig

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

12 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

-i

# sysdig -i bottlenecks
Category: Performance
--------------------bottlenecks
Slowest system calls
Use the -i flag to get detailed information about a specific chisel
Lists the 10 system calls that took the longest to return dur
ing the capture interval.
Args:
(None)

sysdig

-c

# sysdig -c topprocs_net
Bytes
Process
-----------------------------296B
sshd

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

13 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

echo_fds

# sysdig -A -c echo_fds proc.name=apache2


------ Read 444B from 127.0.0.1:57793->162.243.109.80:80
GET /wp-admin/install.php HTTP/1.1
Host: 162.243.109.80
Connection: keep-alive
Cache-Control: max-age=0
Authorization: Basic YWRtaW46ZUNCM3lyZmRRcg==
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/3
3.0.1750.152 Safari/537.36
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8

echo_fds
fd.cip

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

14 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

# sysdig -A -c echo_fds fd.cip=127.0.0.1


------ Write 1.92KB to 127.0.0.1:58896->162.243.109.80:80
HTTP/1.1 200 OK
Date: Thu, 17 Apr 2014 03:11:33 GMT
Server: Apache
X-Powered-By: PHP/5.5.3-1ubuntu2.3
Vary: Accept-Encoding
Content-Encoding: gzip
Content-Length: 1698
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=utf-8

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

15 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

1 Comment

sysdig

Benjamin, thanks for putting this together! This is an great guide for anyone getting started with sysdig. One note
- on your network traffic example, in order to capture network traffic specifically you can use the fd.type filter:
sysdig -A -c echo_fds proc.name=apache2 and fd.type=ipv4

[BENJAMIN CANE]

EMC PowerPath: superblock could not be read

Remote Command Execution with SaltStack

Loving your blog so much. Reading through


article upon article. Can you elaborate on one item here,
since you are specifically discussing the passno, can

Building Self-Healing Applications with Saltstack

Could't agree more :)

Awesome article!

Getting started with SaltStack by example:


Automatically Installing nginx

Hey kannan, it looks like either salt couldn't


find the top.sls or you don't have a top.sls file. Make
sure you perform the steps from this part

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

16 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

5/25/2015 12:21 PM

Using sysdig to Troubleshoot like a boss

17 of 17

http://bencane.com/2014/04/18/using-sysdig-to-troubleshoot-like-a-boss/

5/25/2015 12:21 PM

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy