CPNT217 – Introduction

to Network Systems
Module 4: Troubleshooting Methodology
What gets you to troubleshooting?
What is troubleshooting

• “Troubleshooting is a systematic process used to

locate the cause of a fault in a computer system and
correct the relevant hardware and software issues.
Approaching problem solving using a logical and
methodical approach is essential to successful
resolution.”
Cisco Network Academy.

• “Although experience is very useful to problem solving,

following a troubleshooting model will enhance
effectiveness and speed.”
What gets you to troubleshooting?

After building a solution/product and putting it in use, issues might

arise. These issue might come along as a result of:
- Poor design
- Software bug (OS bug)
- System environment
- Surge in network usage
How would you know about the issues?
- Communication from users/clients
- During preventive maintenance window.
- While testing
- During an audit
Bad design

• When designing a solution, thorough information need to be

collected before hand.
• Understand the solution requirements
• Talk to stakeholders
• Follow the standards
• Follow the recommended best practices, unless you are innovating a
new solution.
• Start with a proof of concept.
• Test, test and test again
• Have a pilot test of new software before proceeding with a full-scale
deployment.
• Document the solution and provide training to different level of
support teams.
Software bug (OS bug)

• Nothing is perfect. Devices might not have the best

software.
• Check the vendor documentation for any limitations.
• Check the vendor documentation for any known bugs
(Outstanding or Undergoing testing).
• Always use the most recent approved software release
(following a thorough test).
• Ensure hardware and software compatibility.
• Check vendor release notes and alert notifications.
System environment

• Check for any possibility of outside interference.

• Check for possible bad installation practices.
• Check for any possible electrical/cooling/humidity
issues.
Surge in the network demand

• More data is being pushed in the network from hungry

users and new applications (Gaming, IoT, Cloud
Computing, AI, 5G, etc). This will require additional
network resources. Meanwhile, until such resources
become available, the network performance might be
impacted.
Troubleshooting Methodologies
1. Identify the problem

• This is the most crucial step. Some problems are

evident and can be reproduced. Others are elusive or
intermittent and tough to be identified and resolved.
1.1 Gather information:
Collect information about the problem by facts and
observation, and document them so you can reproduce
the problem later. No conclusions to be made at this
point.

1.2 Duplicate the problem if possible:

Reproduce the problem, preferably in a test
environment so possible solutions can be tested. If the
problem can be reproduced, document how it was done.
Until you have a good grasp on what caused the issue, it
is difficult to proceed to repair.
1.3 Question users:
Collect information from end-users and read between
the lines, avoid bias. Analyze what could be part of the
problem and what should be factored out. When dealing
with intermittent problems get the user feedback when
testing the fix.

1.4 Identify symptoms :

Identify the symptoms that resulted from the problem
and under what conditions they were caused.
1.5 Determine if anything has changed:
Has there been any change in the network that might have contributed
to the problem?
Two types of documents are usually created to track changes in the
network. They outline methods and procedures to handle changes in the
organization.
The Change Control form has detailed information on:
• Item to be changed
• Reason for the change
• Change priority
• Change description/plan
• Change rollback plan
• Technical evaluation, and
Duration of work
The Change log has details such as what exactly has been changed,
when it was done, who did it, and why it was done.
1.6 Approach multiple problems individually:
When dealing with multiple problems don't assume that
there is a common root cause. Focus on one problem at a
time, collect information about each individual problem
and build a strategy to handle and solve each problem
individually.
2. Establish a theory of probable cause

• Based on the collected information and the network

usage, an initial theory of the problem’s cause can be
made. The suggested root cause might change as new
evidences emerge.
2.1 Question the obvious :
As you build the theory of probable cause don’t let the level
of the symptoms’ complexity lead you to over-complicate the
probable cause and overlook simple and obvious causes.

2.2 Consider multiple approaches :

When dealing with multiple problems don't assume that
there is a common root cause. Focus on one problem at a
time, collect information about each individual problem and
build a strategy to handle and solve each problem
individually.
• Top-to-bottom/bottom-to-top OSI model
• Divide and conquer
• Top-to-bottom/bottom-to-top OSI model :
Follow the flow of information from the source host, OSI layers,
then across the network, and then all the way up the
destination host’s OSI stack. The intention here is not to
overlook any possible problem root cause related to the upper
layers.

• Divide and conquer:

It is not only that each layer in OSI layer is investigated, but
also subprocesses of a complex process are looked at
separately as a possible root cause.
3. Test the theory to determine the cause

• Now you put the theory under test. (The moment of

truth). This is used to confirm the problem and validate
its root cause. The test is performed according to
specific methods and procedures and should have
expected outcomes which, if met, would validate the
supposition of the root cause.
3.1 If the theory is confirmed determine the next steps to
resolve the program.
More detailed testing might be required to narrow down
the root cause even more to a specific component of the
system or few settings and configurations.
3.2 If the theory is not confirmed, establish a new theory
or escalate.
Any failed test can also be considered as one step
toward the solution. A failed test might point you towards
a more appropriate direction of action. In some cases,
you may need to repeat this process multiple times to
come to a solid theory that meets the symptoms of the
issue.
4. Establish a plan of action to resolve the problem and
identify potential effects.

• As important fixing the problem in hand is, it is equally

important that implementing the chosen solution does
not lead to a new problem. Therefore, part of planning
for the solution is to identify potential problems that
might be triggered if the solution is implement. The plan
should be detailed and well documented.
5. Implement the solution or escalate as necessary.

• When implementing a solution, all processes and

procedures need to be followed regardless whether the
solution is simple or complex. Sometimes the
implementation might need to be handed over to in-
house experts or third-party experts for the actual
implementation or feedback.
5.1 Implement the solution
When you have a workable solution decided on and have a plan laid out
that takes all possible outcomes into account, it is time to put that plan
into action. Be sure to follow all procedures and best practices as you
implement the plan, to give the best chance for success.

5.2 Escalate the Problem

In some situations, it may not be possible to resolve the problem
immediately. A problem should be escalated when it requires a manager
decision, some specific expertise, or network access level unavailable to
the troubleshooting technician.
A company policy should clearly state when and how a technician
should escalate a problem.
6. Verify full system functionality and if applicable
implement preventive measures.
6.1 After implementing a solution, a full system check is
required, and end-to-end process/communication should
be verified. This is to make sure that all functions are
working as expected. Also, preventive measures need to
be implemented to help with collecting data if future
system failure occurs.
7. Document findings, actions and outcomes..
7.1 The final step is to document everything:
• The problem
• The symptoms
• The probable causes
• The results of testing
• The actions performed towards correction, and
• The expected outcomes
Documentation is a crucial step that helps in resolving a
similar issue in case it arises in the future, but with a
quicker service restoration time. Such documentation
should be made available to anyone that may have an
interaction with the system/service but the level of might
differ depending on the audience.
Available Troubleshooting Tools
Tools:
• Hardware (Details are skipped since these are • Software (These are used to detect root causes of
not relevant to software developer.) problems and troubleshooting networking issues.)

• Cable Crimps • Packet Sniffer/Protocol

• Cable Tester
•
Analyzer
Punchdown Tool
• OTDR (Optical Time Domain • Port Scanner
Reflectometer) to diagnose and test
the fiber-optic cable. • Spectrum Analyzer
• Optical power meter
• Bandwidth Speed Tester
• Tone generator
• Loopback Adaptor • Command Line
• Multimeter
• Spectrum Analyzer
Software Tools:

• Packet Sniffer/Protocol Analyzer:

Example: Wireshark
Microsoft Message Analyzer
A packet sniffer captures network traffic
(broadcast, multicast, unicast) on the host running
the software, NIC in the host should support
promiscuous mode.
A protocol analyzer takes those captured packets
and examines the data inside to determine as much
information as possible about protocols used inside
the network.
Wireshark has a protocol analyzer built in.
Software Tools:

• Port Scanner:
Example: Nmap
PortQry

Scans the remote server for open ports that

responses to the tool’s requests. Verify if a port is
open/close on the server/firewall. It helps reveals
misconfiguration and unauthorized access to
services..
Software Tools:

• Spectrum Analyzer:
Measures RF transmission information, providing
insight into Wi-Fi infrastructure and usage.
Example: Android Wifi Analyzer

A spectrum analyzer that provides information on

the power measurements of SSIDs, encryption type,
the RF channel being used, and the MAC address of
the Wireless Access Point announcing the SSID.
Software Tools:

• Bandwidth Speed Tester:

Example: speedtest.net
iPerf (Server and Client)
speedtest.net is an applet that communicates with
dedicated servers to determine the download and
upload speeds. Note that the server link is shared
by all tests performed by all users.

iPerf: saturates the entire line to report the actual

bandwidth speeds, also it can be used a part of a
stress test.
Software Tools:
• Command Line Tools:

• Ping (extended ping) • tcpdump

• Tracert/traceroute • pathping
(extended traceroute)
• nmap
• nslookup
• route
• ipconfig
• ifconfig • arp
• Iptables • dig
• netstat • debug
Software Tools:

• Ping: Windows/Linux/Unix

Example: ping 8.8.8.8

It use ICMP at L3. Is used to verify that a destination device

is reachable (# of packets Sent/ Received/ Percentage of
packet loss). Also, to check the (Max/Min/Average) round-trip
response time to a destination device.

If ICMP is blocked on the host, then it will not respond to

ping command.
Software Tools:

• tracert / traceroute: Windows/Linux/Unix

Example: tracert 8.8.8.8 (Windows)

traceroute 8.8.8.8 (Linux/Unix/IOS)
It use ICMP at L3. It provides details of the path the
packet takes and the number of hops across the path. It
is basically a ping on each hop to a destination IP
address. It displays the calculated round-trip times and
the name resolution of each hop .
Software Tools:

• nslookup: Windows/Linux/Unix

Example: nslookup 8.8.8.8

Name-server lookup: used to resolve Domain Name

System addresses by querying a specific DNS
server.
Software Tools:

• ipconfig: Windows

Example: ipconfig (/all, /release, /renew)

It is used to verify IP address configuration and its

associated options (subnet mask, default gateway).

/release and /renew will automatically release a DHCP

lease and request a new IP address from the DHCP
server.
 Software Tools:
ipconfig /all It is used to verify IP address configuration and
its associated options (subnet mask, default
gateway, MAC address, DHCP status, lease
times, DHPC server, DNS server and other info.),
and to renew/release IP addresses assigned by
a DHCP server, and controls the local DNP
cache.
ipconfig /release To renew/release IP addresses assigned by a DHCP
ipconfig /renew server and controls the local DNP cache.

ipconfig /displaydns Displays the local DNS cache

ipconfig /flushdns Dumps the DNS cache. It is used to enforce the update
of an IP address record on the local machine.
Usually a DNS record stays in the cache until the TTL
value expires.
ipconfig /registerdns It enforces the local machine to register its dynamic
DNS records with the DNS server.
Software Tools:

• ifconfig -a: Linux and Unix.

Example: ifconfig -a

Similar to ipconfig
 Software Tools:

ifconfig eth0 IP-ADDRESS netmask SUBNET- It is used to manually configure an IP address

MASK and subnet mask on an interface. This is not
persistent.
ifconfig It is used to verify IP address configuration and its
associated options (subnet mask, default gateway,
MAC address, DHCP status, lease times, DHPC server,
DNS server and other info.), and to renew/release IP
addresses assigned by a DHCP server, and controls
the local DNP cache.
Also, you can turn down or turn up an interface,
change the maximum transmission unit (MTU) on an
interface.
ifconfig eth0 promiscuous This is to turn on promiscuous mode for the interface.
 Software Tools:

• iptables: Linux

it is used to view and edit the firewall iptables.

Examples:

iptables -L It is used to list the current firewall rules

configured.
iptables -A INPUT -i eth0 -p tcp -- dport 80 -j This is to append the INPUT table for the interface of
ACCEPT eth0, for the protocol TCP with a destination port of
80, and an action of permit the traffic. This is not
persistent (on its own) through a reboot.
Software Tools:

• netstat: Windows/Linux/Unix
It displays all of the TCP connections and their
status, and UDP connections.
Example:
netstat -a To display all of the connections as well as any
ports that are listening. It is useful when
verifying an application is requesting a port to
listen for requests on.

netstat -ab (Windows) It is used to verify an application is communicating by

netstat -ap (Linux/Unix) displaying the program that has created the binding,
also to display applications that are listening.
Software Tools:

• tcpdump: Linux/Unix
Is used to dump network data to a file or the
console of the host. This is to see the network packets
entering or leaving a host.
Example:
tcpdump -s 0 port ssh -i eth0 This will output all of the packets that match
the port of the SSH on interface eth0.

tcpdump -s 0 port ssh -i eth0 -w capture.pcap This saves all the packets that match a port of SSH to
the file called capture.pcap
Software Tools:

• pathping: Windows

It is used to diagnose packet loss or suspected

packet loss to destination website. It traces the entire
path to a destination IP address or DNS host, then for
each hop it will be tested for packet loss and round-trip
time.
Software Tools:
• nmap: Linux/Unix
It is used to scan open and closed ports on a remote
system (or a range of devices) for auditing purpose.
It predicts the OS being used by the remote system.
Tries to discover the encryption key strength being
used. Help with discovering a host that might not be
known or protected with firewall rules.
Example:
nmap - A a.b.com It used to detect OS, version, scrip scanning and
remote tracing.
nmap -v It is used to see the progress of the scan.
 Software Tools:
• Route: Windows/Linux/Unix
Is used to view and change the routing table on a host.

Example: route change .. To change an existing route

route delete ….. To delete an existing route
route print (Windows) It is used to view the routing table.
route -n (Linux/Unix)
route add 10.0.0.0 mask 255.0.0.0 172.16.1.10 It is used to add a route in the routing table to
metric 5 if 11 destination network 10.0.0.0 subnet mask 255.0.0.0
reached through gateway 172.16.1.10 a metric of 5
and over interface 11. This is not persistent across a
reboot.
route -p add 10.0.0.0 mask 255.0.0.0 This addition is persistent across a reboot.
172.16.1.10 metric 5 if 11
 Software Tools:
• arp: Windows/Linux/Unix
Is used to observe, delete or change the Address
Resolution Table.

arp -g (Windows) It is used to view the ARP table

arp -a (Linux/Unix)
arp -d * It is used to flush the ARP table
arp -d 172.16.1.1 It is used to delete a specific entry from the ARP table

apr -s 172.16.1.1 00-15-5d-81-12-0b It is used to map an IP address to the MAC address

 Software Tools:

• dig: Linux/Unix

It is similar to nslookup but gives more details. It

queries records for a given host, and the output has
debugging turned on by default.

dig @8.8.8.8 mx sybex.com A query on the DNS server 8.8.8.8 for an MX

record of sybex.com
Software Tools:

• debug: IOS

It display OS process, protocol, mechanism and event messages in real-

time for analysis.
Use debug commands only to troubleshoot specific problems.
Use the debug ? To list a brief description of all the debugging command
options
Use undebug or add the no keyword in front of the debug command to
turn off debugging.
Be cautious using some debug commands, as they may generate a
substantial amount of output and use a large portion of system resources.
The router could get so busy displaying debug messages that it would
not have enough processing power to perform its network functions, or
even listen to commands to turn off.
Common Troubleshooting Tips
Common Troubleshooting Tips
• Common wired connectivity and performance issues:

• Attenuation • Transceiver mismatch

• Latency • Tx/Rx Reverse
• Jitter • Duplex/speed mismatch
• • Damaged cables
Crosstalk
• Bent pins
• EMI
• Bottlenecks
• Open/Short
• VLAN mismatch
• Incorrect pin-out • Network connection LED states
• Bad port indicators.
Common Troubleshooting Tips, cont.
• Common wireless connectivity and performance issues:

• Reflection • Channel overlap

• Refraction • Overcapacity
• Absorption • Distance limitations
• Latency • Frequency mismatch
• Jitter • Wrong SSID
• Attenuation • Wrong passphrase
• Incorrect antenna type • Security type mismatch
• Interference • Power levels
• Incorrect antenna placement • Signal-to-noise ratio
Common Troubleshooting Tips, cont.
• Common network service issues:

• Names not resolving • Incorrect time

• Incorrect gateway • Exhausted DHCP scope
• Incorrect netmask • Blocked TCP/UDP port
• Duplicate IP address • Incorrect host-based firewall
settings
• Duplicate MAC addresses • Incorrect ACL settings
• Expired DHCP server • Unresponsive service
• Untrusted SSL certificate • Hardware failure
Problem Location of Some of the Possible Causes Some of Corrective Action
Problem Taken
Interface Down Layer 1  Broken Connector/Cable  Check on the physical layer
 Improper Connection first and ensure all connections
 Interface is administratively down are in good condition and
properly connected. If
something’s broken, replace it.
 Basic Troubleshooting
techniques like checking the
status of the interfaces.
Example: “show ip interface
brief” If an interface is
administratively down, activate
it with no shutdown command

Two end points Layer 3  Misconfiguration  Ping or traceroute endpoints

from two different from different networks and
networks cannot investigate the possible cause
see each other of the problem based on the
despite of being result.
configured  Use show commands on routers
already. to check issues that exist on
the routing protocols
implemented, routing table,
and connectivity.
 Common Troubleshooting Tips, cont.
Duplex Operation and Mismatch Issues
• “Interconnecting Ethernet interfaces must operate in the same duplex mode
for best communication performance and to avoid inefficiency and latency
on the link.
• The Ethernet autonegotiation feature facilitates configuration, minimizes
problems and maximizes link performance between two interconnecting
Ethernet links. The connected devices first announce their supported
capabilities and then choose the highest performance mode supported by
both ends.
• If one of the two connected devices is operating in full-duplex and the other
is operating in half-duplex, a duplex mismatch occurs. While data
communication will occur through a link with a duplex mismatch, link
performance will be very poor.
• Duplex mismatches are typically caused by a misconfigured interface or in
rare instances by a failed auto-negotiation. Duplex mismatches may be
difficult to troubleshoot as the communication between devices still
[1] Cisco Networking
Common Troubleshooting Tips, cont.
IP Addressing Issues on IOS Devices
• “Two common causes of incorrect IPv4 assignment are manual assignment mistakes or DHCP-
related issues.
• Network administrators often have to manually assign IP addresses to devices such as servers
and routers. If a mistake is made during the assignment, then communications issues with the
device are very likely to occur.
• On an IOS device, use the show ip interface or show ip interface brief commands to verify
what IPv4 addresses are assigned to the network interfaces. For example, issuing the show ip
interface brief command as shown would validate the interface status on R1.”[1]

[1]Cisco Network Academ

 Common Troubleshooting Tips, cont.
IP Addressing Issues on End Devices
• “On Windows-based machines, when the device cannot contact a DHCP server,
Windows will automatically assign an address belonging to the 169.254.0.0/16
range. This feature is called Automatic Private IP Addressing (APIPA).
• A computer with an APIPA address will not be able to communicate with other
devices in the network because those devices will most likely not belong to the
169.254.0.0/16 network.
• Note: Other operating systems, such Linux and OS X, do not use
APIPA.
• If the device is unable to communicate with the DHCP server, then the server
cannot assign an IPv4 address for the specific network and the device will not be
able to communicate.
• To verify the IP addresses assigned to a Windows-based computer, use
the ipconfig command.”[1]

[1] Cisco Network

Common Troubleshooting Tips, cont.
Default Gateway Issues

• “The default gateway for an end device is the closest networking device,
belonging to the same network as the end device, that can forward traffic to
other networks. If a device has an incorrect or nonexistent default gateway
address, it will not be able to communicate with devices in remote networks.
• Similar to IPv4 addressing issues, default gateway problems can be related to
misconfiguration (in the case of manual assignment) or DHCP problems (if
automatic assignment is in use).
• To verify the default gateway on Windows-based computers, use the ipconfig
command.
• On a router, use the show ip route command to list the routing table and verify
that the default gateway, known as a default route, has been set. This route is
used when the destination address of the packet does not match any other
routes in its routing table.”[1]

[1] Cisco Network

Common Troubleshooting Tips, cont.
Troubleshooting DNS Issues
• “It is common for users to mistakenly relate the operation of an internet link to the
availability of the DNS.
• DNS server addresses can be manually or automatically assigned via DHCP.
• Although it is common for companies and organizations to manage their own DNS
servers, any reachable DNS server can be used to resolve names.
• Cisco offers OpenDNS which provides secure DNS service by filtering phishing and some
malware sites. OpenDNS addresses are 208.67.222.222 and 208.67.220.220. Advanced
features such as web content filtering and security are available to families and
businesses.
• Use the ipconfig /all as shown to verify which DNS server is in use by the Windows
computer.
• The nslookup command is another useful DNS troubleshooting tool for PCs. With
nslookup a user can manually place DNS queries and analyze the DNS response.” [1]

[1] Cisco Network

Academy
Summary
 Troubleshooting Methodologies
1. Identify the problem
2. Establish a theory of probable cause
3. Test the theory to determine the cause
4. Establish a plan of action to resolve the problem and
identify potential effects.
5. Implement the solution or escalate as necessary.
6. Verify full system functionality and if applicable
implement preventive measures.
7. Document findings, actions and outcomes.
Available Troubleshooting Tools

Hardware Software
• Cable Crimps • Packet Sniffer/Protocol
• Cable Tester
Analyzer
• Punchdown Tool
• OTDR (optical time domino • Port Scanner
reflectometer) to diagnose and test the
fiber-optic cable. • Spectrum Analyzer
• Optical power meter
• Bandwidth Speed Tester
• Tone generator
• Loopback Adaptor • Command Line Tools
• Multimeter
• Spectrum Analyzer
Command Line Tools

• Ping (extended ping) • tcpdump

• Tracert/traceroute (extended • pathping
traceroute)
• nmap
• nslookup
• ipconfig
• route
• ifconfig • arp
• Iptables • dig
• netstat • debug
Troubleshooting Tips
• Common wired connectivity and performance issues:

• Attenuation • Transceiver mismatch

• Latency • Tx/Rx Reverse
• Jitter • Duplex/speed mismatch
• • Damaged cables
Crosstalk
• Bent pins
• EMI
• Bottlenecks
• Open/Short
• VLAN mismatch
• Incorrect pin-out • Network connection LED states
• Bad port indicators.
Troubleshooting Tips
• Common wireless connectivity and performance issues:

• Reflection • Channel overlap

• Refraction • Overcapacity
• Absorption • Distance limitations
• Latency • Frequency mismatch
• Jitter • Wrong SSID
• Attenuation • Wrong passphrase
• Incorrect antenna type • Security type mismatch
• Interference • Power levels
• Incorrect antenna placement • Signal-to-noise ratio
Troubleshooting Tips
• Common network service issues:

• Names not resolving (DNS) • Incorrect time

• Incorrect gateway • Exhausted DHCP scope
• Incorrect netmask • Blocked TCP/UDP port
• Duplicate IP address • Incorrect host-based firewall
settings
• Duplicate MAC addresses • Incorrect ACL settings
• Expired DHCP server • Unresponsive service
• Untrusted SSL certificate • Hardware failure
References:
• CompTIA Network+ Review Guide, 4th Ed. By Jon Buhagiar
• Cisco Network Academy, Introduction to Networks, Module 17
• IT Essentials Companion Guide v7, by Cisco Network Academy.