D&T

INDEX:
1. Introduction
2. SSH problems
3. User Issues
4. Being Proactive
5. Troubleshooting Boot Issues
6. Identifying Hardware Issues
7. Troubleshooting Storage Issues
8. Troubleshooting RPM Issues
9. Troubleshooting Network Issues

Introduction
This course provides system admins with a tools and techniques they
need to be successfully diagnosis and fix for the issue.
Troubleshooting:
Troubleshooting is the art of taking of problem, gathering info
about it, analyzing it, and finally solving it.
Using Scientific Method:
1. Clearly define the issue
2. Collect Information
3. Form a hypothesis
4. Test the hypothesis
5. Fixing the problem
6. Rinse & Repeat

SSH
Scenario:
 Over the weekend, some of your colleagues have been running user
maintenance. This morning one of your users, student calls the desk to
complain about that his login is no longer working on servera.

Step: 1
#lab scientificmethod setup

Step 2:

#ssh student@servera
Connection to server closed, It wont login because we create setup no
to login

Step 3:
#cd /home/student
#vim .bashrc
We are seeing alias function and nothing is changed on the alias function

Step 4:
Next see the entries in the /etc/passwd file for the student login user

# cat /etc/passwd | grep student (or) #getent passwd srudent

Here the login bash for the student user is /bin/false

Student:x:1000:1000::/home/student:/bin/false
So we have to change the shell for that user

Step 5:
#usermod –s /bin/bash student (or) #chsh –s /bin/bash student

Step 6:
#ssh root@servera
It logged in

Step7:
Com back to workstation machine and check the grade of the lab setup
$logout
#lab scientificmethod grade
#lab scientificmethod reset
FOR CHECKING:
Do this command in server machine
#lastlog –u student{how many times you logging in student user)
If we have two machines servera and serverb we can login one machine
to another machine through the SSH protocol but we change to login to
their own machine.
We need to change in the settings the configuration file named as
/etc/ssh/ssh_config file

WORKSTATION MACHINE:
#vim /etc/ssh/ssh_config
go to last line write this line
hostname localhost
:wq!

#systemctl enable sshd.service

#systemctl restart sshd.service

Checking:
#ssh root@servera
It logged into the same machine not to login the servera machine.

USER ISSUES
 There is the directory /etc/skel which contain some important files
such as .bashrc, .bash_profile and .bash_logout.
 As the name say skel is a skeleton directory, which provides
structure for the user.
 When a user gets created these files will be copied to the default
home directory of the user.
 /etc/skel describes about all user behavior whereas these files(.bash
files) under the default home directory of the user determines the
property of the particular user.
 These files are symbolic, executable bash scripts.
 .bash_profile is the file which gets executed when a user switch to
his account.
 The user behaves based on the .bashrc file
 .bash_logout is the file which gets executed when the user logs out
from his account.
 Editing .bashrc file:

Adding parameter:-
The user behavior can be structured by editing the
corresponding .bashrc files

Step 1:
#vim /home/student/.bashrc

Step 2:
Go to the end of the file and add the parameter ‘exit’

Step 3:
#source .bashrc
reload the bashrc file

#su – student
It won’t login the student user …

Command Aliasing:
It means that the command can be alias into the single character. We
can reduce the command length and changed it into single or multiple
characters.
Step 1:
Edit the .bashrc file in the user’s home directory

Step 2:
Go top the last line and add the line
alias l=”ls –ltr’
save and quit

Step 3:
#source .bashrc
read the .bashrc file
Step 4:
# ls –ltr or # l
both are showing same output

Collecting Relevant Information

 In RHEL7, systemd place an important role in the starting services

and generating logs
 Systemd have concurrent and parallel processing at boot and also
take and manage logs.
 In RHEL6, init is the starting process and in RHEL7 it is replaced
by systemd .
 If we compare init and systemd, systemd does so many activities to
no more

#systemctl –t help
Available unit types (cmd to list all services controlled by systemd)
1. Service
2. Socket
3. Target
4. Snapshot
5. Device
6. Mount
7. Automount
8. Swap
9. Timer
10.Slice
11.Scope
 The important units are service, socket, device, target, swap etc.
 To understand more about systemd
#ls –lh /usr/lib/systemd/systemd

 When you try to open systemd file, we cannot open because it is

binary file.
 In RHEL 7 any command ending with ‘ctl’ or any service ending
with ‘d’ is directly controlled by “systemd”
 Uptime: Uptime gives a one line display of the f ollowing information
The current time, how long the system has been running, how many
users are currently login and system load average for the past 1,5 and
15 minutes.
SYSTEMD-ANALYZE:
Command to show how long system takes after booting
#system-analyze time

Cmd to show how long each service divides at booting time

#systemd-analyze blame

Command to show tree structure of the services at booting

#systemd-analyze critical-chain

#systemd-analyze plot or #systemd-analyze dot

Command to take backup the booting process and it saved in the file.

#systemd-analyze blame > /toot/20160227

#systemd-analyze blame > /root/20170227
Command to take difference between two booting process
#diff /root/21060227 /root/20170227

Various forms of Information

rsyslog Systemd-journal
Persistent Not-persistent
Stored in /var/log directory Stored in /run/log/journal file in
temporary filesystem
Summarized Information Detailed Information
Daemon is rsyslogd Daemon is journal

 Like systemctl, journalctl is also systemd utility

 Journalctl is the tool which is used to display the logs stored by
systemd-journal service
 The daemon responsible for the systemd-journal is journald
 This daemon will handle all the messages produced by the kernel
initrd, services etc
 Journald collects data from all the available and store them in the
binary format for the dynamic manipulation.
 By default RHEL-7 stored the systemd-journal logs stored in
/run/log/journal directory which is mount point to the temporary file
system(tmpfs).
 After the reboot, entire information stored will be deleted.
 If journal have persistent directory it is possible to make logs
permanent.

Making Journal Persistent:

#mkdir -p /var/log/joiurnal
#chgrp systemd-journal /var/log/journal
#chmod 2755 /var/log/journal
#killall –USR1 systemd-jourtnal

OWN NOTES:
o Initial create directory in the given path
o Check the permissions of the directory and change into the group
ownership to the systemd-journal
o For saving the files into that directory we have to give special
permissions to that directory
o It will not inherit into that directory, for that we are giving
special permissions (SGID)
o After kill the services and then reboot

Commands:
1. #journalctl –ef
command to view logs dynamically
-e - jump to the end in the pager
-f - follow the journal

2. #journalctl –u sshd.services
-u - units
command to view logs for the particular service

#journalctl _SYSTEMD_UNIT=sshd.service

3. #journalctl –b –l
 Only show the messages from the last system boot, this is
used for information about system crash.
-b -boot the system
-l - list the system

4. #journalctl –p err (or) #journalctl –p emerg..err

command to view priority logs occurred after booting
-p  priority
err - errors

5. #journalctl –-since recent

6. #journalctl –-since “2017-08-22 15:35:00” –-until

“2017-10-22 00:00”
Command to view logs periodic time

7. #journalctl –o verbose
Use verbose output mode, this will show all fields stored
in the journal with their field name and contents.
Scenario:
Your servera machine is running a webserver serving the file
http://servera.lab.exampole.com/test.html. A ticket just came from test
manager that this file is not accessible from the web browser. Investigate
this issue using the log files on the servera machine and then fix the issue.
For testing from the command line on workstation, if you do not want to
open a graphical browser, you can use the command line browser elinks.

#lab logread setup

Step 1:
There is no graphical browser so we need to install the command line
browser elinks.
#yum install –y elinks.x86_64

Step 2:

Check the webpage through the elinks

#elinks http://servera.lab.example.com/test.html
It shown error not open that webpage

Step 3:
 If it is shown error first check the firewall of the service
#firewall-cmd –-list-all
It shows that http service is running

Step 4:

It shows no error in the firewall and check the status of the service
#systemctl status httpd.service
Status: running

Step 5:

Now we have to check the log file of that service

#vim /var/log/httpd/error_log (or) #tail /var/log/httpd

It shows error it doesn’t have permission to log in

Step 6:

#cd /var/www/html
#ls -lZ
Shows no permission and SELINUX context is not default

Step 7:
First change the permission of the webpage
#chmod 644 /var/www/html/test.html
Second change the context of the file
#semanage fcontext –a –t http_sys_content_t “/var/www/html/test.html(/.*)?”

This command will reset to the default context

#restorecon –RFv /var/www/html/test.html

Step 8:
Enable & Restart the service
#systemctl enable httpd.service
#systemctl restart httpd.service

Step 9:
Open the web page now
 #elinks http://servera.lab.example.com/test.html

come to workstation machine

#lab logread grade

SOSREPORT:
 This command will collect diagnostic and configuration information
from the RedHat Enterpise Linux System and installed applications.
 An archive containing the collected information will be generated in
/var/tmp directory and may be provided to RedHat support
representative.
 The archive may be stored locally (or) remotely.
 The SOSREPORT doesnot make any changes to the system
configuration.
 It collects the information about the system.
 By default SOSREPORT using the ‘xz’ compression.
 MD5 file is generated for checking the integrity of the compressed
archive file.

Commands:
1. Command to create sosreport with non-interactively
#sosreport –-batch

2. Command to create sosreport in different location (or) diff directory

#sosreport –tmp-dir /new

3. Command to create sosreport with different compression method

#sosreport –Z gzip

These are piece of code, which is used to add a feature to the main
application
#sosreport –l

NOTE:
 Whenever we generate sosreport it collects the information from
the active plugin.
 Active plugin are the plugins, which is already installed and
running on the system.
 Installing and running the service can activate inactive plugins.
  Whenever we generate the sosreport the newly added plugin
information will be added to the sosreport.

The command to create a sosreport based on the services

#sosreport –o(only) rpm,samba,http
The command to create the sosreport on removing the particular
plug-ins
#sosreport –n(no plugin) ssh,selinux

AIDE: (Advanced Intruction Detection Environment)

How can a change to the directory detected?
This is how AIDE comes into the picture. Aide is used to track the
complete changes made from a directory, which is installed, by using side.
The changes include changing permissions, size, timestamp, user and
group ownerships.
First install a package
#yum install –y aide.x86_64

After install a package then enter into the configuration file of the aide,
then go to the directories that you want in database and add the new
directory. Now the new directory also enter into the database.
#vim /etc/aide.conf
Remove the other directories if you don’t want to check and make them
into comment in the configuration file.

Command to initialize the database

#aide –-init
It will be create the new database with new name aide.db.new.gz

After go to the configuration file there you can change the database file
name aide.db.gz . So we change the name of the previous creation one.
#cd /var/lib/aide
#mv aide.db.new.gz aide.db.gz

Command to give changes in the system and display the changes and
different between the before and after changes
#aide --check
*** if we get any error we have to check the database filename and
the change the name of the file to aide.db.gz
We need to edit the configuration file /etc/aide.conf
1.uncomment the 11th line
2.change the database output file name to aide.db.gz

Command to update the aide database

#aide -–update

Cockpit:
Cockpit is a free and an open source web based system admin interface
actively developed by Redhat. Using cockpit anyone can easily monitor
manage multiple servers at the same time in the web browsers.

Note:
It is perfectly for new system admin to perform simple task such as
storage admin, user admin, LVM operation, inspecting journals, start and
stop services etc..

Using cockpit we can also perform basic network operation, it also has
journal log viewer to perform troubleshooting and log analysis.

#yum install –y cockpit.x86_64

#systemctl enable cockpit
#systemctl retstart cockpit
#firewall-cmd –-permanent –add-service=cockpit
#firewall-cmd –-reload

System- information of all systems present in the cockpit

 The authentication to the cockpit is performing using system local OS

system database.
 The dashboard server in the cockpit interface provides an overview of
the system performance matrix.

PCP (performance co-pilot):

RHEL-7 introduce program called PCP. It provides the pcp package. It

allows admin to collect and query data from various sub systems.
#yum install –y pcp
#systemctl enable pmcd
 #systemctl restart pmcd

 PCP is available from version 6.6, the daemon which is responsible for
pcp is pmcd(performance matrix collection daemon)

Command to view system and process monitor

#pmatop
The program pmatop is an interactive monitor to view the load on a linux
OS. It shows the occupation of the most critical hardware resources (from
performance point of view) on system level i.e., CPU, memory, disk and
network.

Command to vie w the status of the system performance in interval time

#pmstat
pmstat provides dynamic real time view of virtual memory statistics.
#pmstat –s 4
after 4 samples it will be terminated

Resolving Boot Loader Issues

The primary responsibility of the GRUB is to identify the default
kernel.
In RHEL-7, we user GRUB2
In RHEL-6 or previous versions we use GRUB
GRUB identifies the default kernel by reading the configuration file.
If the boot loader is GRUB then the configuration file is /etc/grub.conf
which is soft link of /boot/grub2/grub.conf
In RHEL-7, the configuration file for grub is /etc/grub.conf which is
soft link of /boot/grub2/grub.cfg
/etc/default/grub ,this file describes how the grub menu looks like and
there is parameter called GRUB timeout which defines how long we
can stay before the default kernel is selected.

Scenario:
On server, restore the boot loader on a bios based machine that is
refusing to boot.
In workstation #lab biosbootbreak setup
Then open server machine
 If some reason, the MBR become damage and administrator will have to
reinstall grub into MBR.
The error showing is
Continuing boot loader
Booting from hard disk

Error is in MBR system cannot identify the grub, so that we are

reinstalling the grub file
Step 1:
Go to the send key configuration (ctrl+shift+f10)
Step 2:
Reboot your system with the network-boot
Step 3:
Boot into the recue environment
Step 4:
The rescue environment will now attempt to find your Linux installation
and mount it under the directory: /mnt/sysimage

Step 5:
Select option 1 to continue
Step 6:
Recue mount
Your system has been mount under /mnt/sysimage. If you would like to
make your system the root environment, run the command
#chroot /mnt/sysimage
#grub2-install /dev/vda (MBR is not present in the partition, It is
present in the first portion of the hard disk)
#exit
#exit

# lab biosbootbreak grade

#lab bioisbootbreak reset
Why should not give autorelabel?
Because we should not change (or) altered the system configuration file.

Configuration file is corrupted:

Scenario:
Someone decided to stream line the boot process a server machine while
doing this they decided accidently broke the ability of a system to boot.
You were assign to task make to server machine bootable.
#lab bootissues setup
Whenever we tries to load the default kernel it gives the error
Error: you need to load the kernel first
Press any key to continue
 Now, Check the rescue kernel is working or not
 If it is working login as the root user to perform the change

Command to recreate the configuration file

#grub2-mkconfig -o /boot/grub2/grub.cfg

Configuration file is generated by reading the contents in the

/etc/grub.d directory
#reboot
#lab bootissues grade
#lab bootissues reset

Setting password to GRUB:

 Anyone can have physical access to the system can easily change
the root user password, so we can avoid the setting the GRUB
password.
 In order to set GRUB password we have to set username and
password in GRUB configuration file.

#grub2-mkpasswd-pbkdf2
Enter password:
Re-enter password:
PBKDF2 hash your password is (password is in encryption format)
 It generates the password string suitable for use in grub
configuration file.
 Go to the directory grub.d in the etc directory
#cd /etc/grub.d
#ls
#vim 10_Linux
Go to the end of line
Write this entry
 cat << EOF
set superusers=”root”
password_pbkdf2 root paste the password before we created there.
EOF
#grub2-mkconfig –o /boot/grub2/grub.cfg
#reboot
Note:
1. Pbkdf2- password based key derivation function2
2. It is a part of RSA public key cryptography
3. After reboot your machine Press E button to edit the grub menu to
change the root user password and then it will ask the username and
password that we create before and we write the password in the
grub.d directory file

Identifying Hardware Issues

An important step in trouble shooting potential hardware issues knows
exactly which hardware is present in the system.

Command to determine the architecture of the system

#arch or #uname -m
x86_64

If we want to all other hardware requirement, check this command

#lscpu or #cat /proc/cpuinfo

If we want to list all about the hardware

#lshw
If the package is not there, we have to install by manually
#yum install –y lshw
If we want to know what OS and what version of OS installed in your
system
#cat /etc/redhat-release or #hostnamectl
Summarized information about hardware not detailed about the hardware
#dmidecode
If we want to know about the particular information about type of
hardware component
#dmidecode –t memory

If we want to know about what type of hard disk and which version is
using to list scsi devices
 #lsscsi –v
#lsscsi –V

By using this command it is possible to identify USB, SATA hard disk,

SCSI hard disk etc…
 Checking for IDE hard disk
#yum install –y hdparm.x86_64
#hdparm –I /dev/vda
Determine the USB details
#lsusb
If we want to see the peripheral component Interface
#lspci

Peripheral component Interconnect is local computer bus used for

attaching network devices a computer…

Hardware Error Reporting:

 Modern system can typically keen a watch on various hardware
failures.
 Altering and admin hardware occur faults RHEL-7 provides to
mechanism for logging hardware fault
1. Mcelog
2. Rasdaemon
Mcelog:
 Mcelog provides a framework for catching and logging machine
and check exceptions on x86 system.
 On supported system it can also mark bad sectors of RAM, so
they didn’t use again
#yum install –y mcelog.x86_64
#systemctl enable mcelog
#systemctl restart mcelog
#journalctl –u mcelog.service
mcelog stores the persistent log in /var/log/mcelog.

Rasdaemon:
Rasdaemon is more modern replacement for mcelogs that hooks into the
kernel trace sub system.
Ras stands for Reliability Awarability and Serveribility
#yum install –y rasdaemon.x86_64
 #systemctl enable rasdaemon
#systemctl restart rasdaemon
#journalctl –u rasdaemon.service

Memory Testing :--

 When a physical memory error is suspected an administrator might
want to run a complete memory test
 RHEL sheets a memory test tool called memtest86+
 It is a bootable utility that test physical memory by writing various
patterns to it and reading them back.
#yum install –y memtest86+.x86_64
Command to generate memory test template in /etc/grub.d directory
#memtest-setup
Template: Act as a shortcut but contains copy of all configurations.

Command to regenerate grub configuration file with memtest options.

#grub2-mkconfig -o /boot/grub2/grub.cfg

Scenario:
One of your jr.admin has provided you with a test dump of systemd-
journal file from a machine, there has been experiencing strange random
errors. He has ask you to inspect this journal file for any possible signs of
hardware failure.
Since machine kept experiencing, these issues even after a complete
wipe and reinstall the machine
The log dump location is /root/logdump
#lab hardwareissues setup
#vim /root/logdump
#grep –n mcelog /root/logdump
#yum install –y memtest*
#memtest-setup
#grub2-mkconfig -o /boot/grub2/grub.cfg
Identifying Storage Issues:
A fellow admin was ask to configure the server aid to use the iqn
iqn.2017-01.com.example.lab:ISCSI storage

Scenario:
The target is configured on workstation the admin has reported that he can
discover the target but he is unable to login successfully. Troubleshoot and
identify the issue so that the initiator can successfully login to the target

#lab iscsi setup

check the error in the iscsi storage
Step 1:
First check the firewall whether port is there or not
#firewall-cmd –-list-all
Step 2:
If it is correct go to the targetcli for checking the iqn in the ISCSI
#systemctl restart target.service
#targetcli
Step 3:
If the given portal and iqn are to be identified and go to the server side
configuration
#ssh root#servera
#vim /etc/iscsi/initiatorname.iscsi
Here is the error, Initiator name is wrong so we can edit the initiator
name and save again.
Step 4:
Then restart the service again
#systemctl restart iscsid
#systemctl enable iscsid

Now check the discovery and login of the ISCSI

If everything is correct, the issue is resolved..

Recovering from File System Corruption;

 File system is used to control how data is stored and retrieve; ext2 is not
widely used because it doesn’t provide file system journaling.
 Journaling is used to improve data integrity in the event of power loss,
system crash.
 After such an event journaled file can use its journal to quickly recover
and saved information in an attempt to avoid file system metadata
corruption but it doesn’t protect again the file system corruption.
Note:
1. A journaling file system is file system and that maintains special file
called journal.
2. There is used to repair any inconsistency that occurs due to improper
shutdown of the computer
3. Journaling file system writes metadata into the journal i.e., flushed to the
part of the hard disk each command turns.

 The journal is stored in the separate part in the particular, which is unable
to access by the user.
1. Command to check the filesystem
#e2fsck -n /dev/vdb1

2. Command to check filesystem forcefully

#e2fsck -f /dev/vdb1

3. Command to know where the alternate superblock is located

#dumpe2fs /dev/vdb1 | grep Backup superblock

4. Command to check the file system in alternate superblock with blockid

#e2fsck –b 32768 -n /dev/vdb1

5. Command to check filesystem forcefully in alternate superblock with

blockid
#e2fsck -b 163480 -f /dev/vdb1

6. Command to find the result of the filesystem check

#echo $?

7. Superblock is stored in the 16-32 sectors of the hard disk.

RHEL 4  ext2 file system
RHEL 5  ext3 files system
RHEL 6  ext4 files system
RHEL 7  xfs file system

EXIT CODE MEANING

0 NO error
 1 File system error

2 System should be rebooted

4 File system error corrected

8 Operational error

16 Usage error

32 Concede by user

128 Share library error

Checking the XFS file system:

 Unlike e2fsck, xfs_repair doesn’t require the administrator specifies, the
location of alternate superblock.
 It will automatically scan the xfs file system until it locates a secondary
superblock.
 Once located the secondary superblock, it will be used to recover the
file system.

Commands:
1. Check the file system
#xfs_repair -n /dev/vdb1

2. Repair the file system

#xfs_repiar -f /dev/vdb1

In XFS there are two exceed code

0  no file system correction
1  file system correction detected

lost+found directory:
  During file system check, it is possible to detect files and directories
but are un referred by their parent directories (corrupted file) so
called orphan file.
 These files are deposited in the ‘lost+found’ directory.
 If the files are missing after the file system check, check this
directory.
Scenario:
 /dev/vdb on servera contains XFS file system, which holds the content
of etc directory from another system.
 Check the integrity of XFS file system on /dev/vdb1
 Repair any file system inconsistency found and mount the file system
/mnt/etc_restore
 The file system repair deposit any filesto the lost+found directory
 Use the backup file /root/etc.tgz to determine the proper location and
name to restore orphan file once validated
 Restore the archive file back to its proper location

#lab xfs setup

Step 1:
First check the mount point
#df –h

Step 2:
The un mount the file system
#umount /mnt/etc_restore

Step 3:
Check the file system what type is mount and then repair
#xfs_repair –n /dev/vdb1
#xfs_repair –f /dev/vdb1
We are getting error at 144973(subscription-manager)

Step 4:
Them remount the file system
#mount /dev/vdb1 /mnt/etc_restore
#df -h

Step 5:
 Go to that directory
#cd /mnt/etc_restore
#ls
#cd lost+found
145282  the file is missing in the file system

Step 6:
We have to check the mount directory because it is the inode number

#find /mnt/etc_restore –inum 144973

/mnt/etc_restore/etc/security/console.apps

Step 7:
We have backup of etc directory in root directory in zip format
#tar –xf /root/etc.tgz -C /media
(we extract into the media directory)

Step 8:
Now we can check the files difference of the two directories what flies
are lost
#diff –s 145282 /media/etc/security/console.apps/subscription-
manager
Files both are identical..

Step 9:
Then rename the file to subscription-manager
#mv 145282 subscription-manager

Step 10:
#ls /mnt/etc_restore/lost+found
subscription manager

Step 11:
#mv subscription-manager /mnt/etc_restore/etc/security/console.apps
It moved to the error location

#ls
Step 12:
#lab xfs grade

Recovery from LVM Accidents

 LVM provides administrator with a powerful storage virtualization framework,
the behavior of LVM is configured in the file /etc/lvm/lvm.conf
 There is feature named as archiving
 LVM can keep archive copies of Volume Group (VG) metadata
 If the archive option is set to 1 (taking backup)
 The LVM tools we can create an archive copy of volume group metadata before
making the changes on disc
 The list of archive metadata for a volume group can be displaced and examine
the files under /etc/lvm/archive
 The list of backup metadata for a volume group can be displaced and examine
the files under /etc/lvm/backup

Scenario:
A request was recently receive for additional 20MB of storage to be
allocated for the use by directory /mnt/lvm on servera. After the request
was completed the user reported that the directory is no longer accessible.
The user thinks that the critical file was accidently deleted by the admin
who fulfill the request. The problem has been escalated and viewer asks to
investigate and identify the root cause of the problem. Once root cause is
completed attempt to restore the system to proper working condition.

In workstation
#lab lvm setup
It will show error it didn’t mount the file system after partition resizes the
data
Step 1:
#df –h
#mount –a
mount: /dev/mapper/vg00-lvolo : cant read superblock

Step 2:
#cat /etc/fstab or #vim /etc/fstab
/dev/vg00/lvolo /mnt/lvm xfs defaults 1 0
Now we can identify that the metadata of the lvm is missing, so we can
recover through the backup of metadata

Step 3:
After the metadata is missing, we can recover from the archive directory
#cd /etc/lvm/archive
#ls
 vg00_ -- same files is available but we don’t know which one is the
metadata file is actually missing…

Step 4:
#vgcfgrestore –l vg00
The above command list all the volume group metadata files

Step 5:
#vgcfgrestore –f /etc/lvm/archive/vg00_00002-400393107.vg vg00
Restore volume group vg00

Do all the vgcfgrestore for metadata files

Step 6:
To deactivate the lvm for the above changes
#lvchange –an /dev/vg00/lvolo
#lvchange -ay /dev/vg00/lvolo
#mount –a
#df –h
It shows mount point but didn’t add partition size

Step 7:
#lvextend -L +20M /dev/vg00/lvolo
#xfs_growfs /dev/vgoo/lvolo
#df –h

Step 8:
#lab lvm grade

LUKS (Linux Unify Key Setup): -

 LUKS is used to encrypt the partition on a system
 Keep in mind that LUKS works only on partition and logical volumes
only
 LUKS encrypt your partition and your data is secure in case you lose your
computer also because the LUKS encrypted partition requires pass phrase
(or) key file
 In order to work with LUKS encryption we need to know a dm_crypt
module
 If it is not loaded, load it manually
NOTE:
1. Modules are piece of code that can be loaded and unloaded into the
kernel upon demand
2. They extend the functionality of the kernel

First check the partition

#lsblk
#yum install –y cryptsetup-luks
Install this package for creating passphrase for partition
#modprobe dm_crypt
Tell to the kernel about dm_crypt
#lsmod | grep dm_crypt or #cat /proc/modules
Show the list of dm_crypt files

Create crypt setup:

#fdisk /dev/vdb
#partprobe
#cryptsetup -y –v luksFormat /dev/vdb1 (assigning luksID)
#lsblk
#cryptsetup luksOpen /dev/vdb1 backup (giving LUKS name)
#mkfs.ext4 /dev/mapper/backup
#mkdir /luks
#mount -a
#df –h
#touch /luks/f{1..10}

It will ask the passphrase when we rebooting you machine, backup is an

alias name.

Making alias name permanent:

#vim /etc/crypttab
backup /dev/vdb1
#cryptsetup status /dev/mapper/backup
#reboot
#df –h
Closing LUKS:
 We want to hand over your system that didn’t share confidential file then
close that partition
#umount /luks
#cryptsetup luksClose /dev/mapper/backup
#cd /luks
#ls
No data show

Password less login for LUKS:

First create a key file for saving passphrase
#vim /root/king
pragathi
#cryptsetup luksAddKey /dev/vdb1 /root/king
enter any passphrase
#vim /etc/crypttab
backup /dev/vdb1 /root/king

Creating LUKS header file:

 In a LUKS partition there is a portion called header. Header is the place
where metadata about the luks stored.
 Header consists of luks partition information as well as passphrase
information.
 If the header is corrupted, it is not possible to access the partition
 This issue can be resolved if we have previous header working backup
 #cryptsetup luksHeaderbackup /dev/vdb1 --header-backup-file
headerfile
 #cryptsetup luksDump /dev/vdb1

Adding passphrase to LUKS:

LUKS supports upto 8 passphrases. It is stored from slot 0 to slot 7.
The default passphrase will be allocated to 0
#cryptsetup luksAddkey /dev/vdb1 --key-slot 5
Enter any existing passphrase:
Enter the new passphrase for slot:
Verify successful
#cryptsetup luksDump /dev/vdb1

Removing a key in LUKS:

 It is possible to remove a key from keyslot. For removing a key we
must know the corresponding key or passphrase
#cryptsetup luksRemoveKey /dev/vdb1
enter passphrase to be deleted:
or
#cryptsetup luksRemoveKey /dev/vdb1 --key-slot 4

Restoring header file from backup:

#cryptsetup luksHeaderRestore /dev/vdb1 --header-backup-file
headerfile
It will ask YES/NO? YES
Header file is restored
#cryptsetup luksDump /dev/vdb1 (for checking whether it is
restored or not)
Backup Header we have to do it manually, it won’t be create
automatically

RESOLVING RPM ISSUES

RESOVING DEPENDENCIES ISSUES:
 In each package it includes the metadata of the package
 It has the dependency packages by defaults ‘RPM’ will not
resolve so that we are using the ‘YUM’ utility to resolve the
dependencies of the packages.
 In YUM we can use to install, erase and update packages
 The complete log related to the yum can be stored in the
/var/log/yum.log

The complete log can be seen for rpm is

#rpm –qa –-last

Command to view the dependencies of the package using rpm

#rpm –qR httpd

#yum deplist httpd
 Using YUM to lock package:

 Yum fails to install the specific version that requires a specific version of
another version and a different in compactable version of the required
package is already installed on the system.
 “YUM” has a plugin that allows an admin to lock down a package or
group of packages to a specific version.

Ex: HTTPD version (4.2)

D1 – (4.2)
D2 – (4.2)
D3 – (4.2)
D4 – (4.3) (D means dependency)

If you install this package it won’t install because dependency ‘D4’ is in

the same versions the above versions

SCENARIO:
The security team has updated the system requirements for host in the
data center part of the update requires a custom package “rht-main” on
servera machine without success you have been called to resolve the
problem and get the package install.

In workstation
#lab package-dependencies setup

In servera machine

Step 1:
#yum install -y rht-main

Package rht-main 0.1-1.el7 will be installed

Error: Package
Requires: rht-prereq=0.1
Installed: rht-prereq-0.2-1.el7
Rht-main = 0.2-1.el7
Available: rht-prereq – 0.1-1.el7
You could try using –skip broken to work problem
Step 2:
 #yum downgrade -y rht-prereq
First we have to downgrade the dependency package and there is plugin
named as yum-plugin-versionlock

Step 3:
#yum list yum-plugin*
Step 4:
#yum install –y yum-plugin-versionlock.noarch
we have to install that plugin package
Step 5:
#yum versionlock add rht-prereq
Now we have to add the latest plugin to the version lock for doing the
work, it will lock the latest version

Step 6:
#yum versionlock list
It will list all the versionlock packages

Step 7:
#yum update rht-prereq
To verify the update package

Step 8:
#yum install - y rht-main

Step 9:
Go to workstation run this command
#lab package-dependencies grade

Verifying Installing Packages:

One major advantage of rpm package management is a local rpm

database.
Each time a rpm package is install information about each of the file.
It provides to get recorded rpm database the information includes
filesize, content of the file, permission, user& group ownership,
timestamp etc

Command to see the path the commands

#which mkdir
 #which touch
#which chmod
#which chcon

Command to see which rpm is used for the existing files

#rpm –qf /usr/bin/mkdir
#rpm -qf /usr/bin/touch
#rpm -qf /etc/lvm

Command to verify the package

#rpm -V <package name>

Command to verify all packages

#rpm -Va

If there is any change of attribute then only the options will be displayed

S Changing file size

M Mode (permissions)

5 Changing context

L Symbolic link (changing

link path)
U Changing user
ownership
G Changing group
ownership
T Timestamp

c Configuration file

d Documentation file

r Readme file

l License file
In an rpm package, it consists of different type of file c, d, r, l

#yum list yum-plugin*

#yum install –y yum-plugin-verify.noarch
#yum verify <package name>

Not only by using rpm but also using ’yum’, it is possible to verify the
packages. Yum uses rpm database for performing these tasks.

Recovering Changed File:

#rpm --set
--setperms --setugids

#rpm –-setperms <package name>

Command to restore the default permissions of the package

#rpm –-setugids < package name>

Command to restore the default user & group ownerships

If there is any change in the size of the package and content of the
package, it is not possible to restore for using above two commands. So
we have to use these commands
#yum reinstall –y <package name>

#rpm –Uvh --replacepkgs <package name>

Scenario:
Reports are coming in that some of the command in servera is broken.
Users cannot list files with ls command and sudo command behaving
strangely. You have been called into diagnose and correct the problem.

Step 1:
#lab broken-commands setup

Step 2:
Open servera machine
#ls
 $: permission denied : ls

Step 3:
#which ls
/usr/bin/ls
Step 4:
#rpm -qf /usr/bin/ls
coreutils

Step 5:
#rpm –V coreutils

Step 6:
#rpm –setperms coreutils

Step 7:
#ls
files are shown

SUDO:
Step 1:
#su – student

Step 2:
#sudo fdisk –l
/bin/sudo:- permission denied

Step 3:
#logout
#rpm –qf /bin/sudo
sudo

Step 4:
#rpm -V sudo
two errors

Step 5:
#yum reinstall -y sudo

Step 6:
 #su – student
#sudo fdisk -l

Scenario:
Someone on the team install a webserver on servera machine
server.lab.example.com, instead of displaying the contents the following
error messages displaced when its access, failed to connect.

Step 1:
#lab package-issues setup

Step 2:
#ssh root@servera
#yum install –y elinks.x86_64

Step 3:
Now open the webpage using elinks
#elinks http://servera.lab.example.com
Network is unreachable

Step 4:
Since it is showing the network issue. First check the firewall settings.
#firewall-cmd --list-all

Step 5:
Since in the firewall list everything is perfect. So next option go and
check the status of the service
#systemctl status httpd.service

Step 6:
Now check the logs of the service
#journalctl -u httpd.service
It will show errors for the service in the permissions

Step 7:
Now check the binary file and what package is responsible for that
#rpm -qf /usr/bin/http
httpd (package name)
Step 8:
Check the errors using verify option in the rpm commands
#rpm –V httpd
Step 9:
Reinstall the package we can recover the permissions
#yum reinstall -y httpd

Step 10:
Restart and enable the service
#systemctl enable httpd.service
#systemctl restart httpd.service

Step 11;
Now open the webpage using elinks
#elinks http://servera.lab.example.com

Troubleshooting Network Issues

ICMP: (Internet control Message protocol)

It is a network layer used by network devices like routers to send error
messages indicating for example that a requested service is not available
(or) that host or router not to be reached
IPV4  ICMP4
IPV6  ICMP6
ICMP is a control protocol used not to carry application data.
It holds information about the status of the network itself. Ping is based on
this

1. One of the first step is to check the connectivity of sending ICMP

echo requests to the remote host
2. Host by default are configured to send an echo reply to indicate that
they are present and running.
3. This is accomplished by “ping” command

#ping <remote ip address>

#ping < remote hostname>
the above commands will work and check the connectivity to other
machines
 #ping –c 3 < hostname or ip address >

#ping –w 3 <hostname or ip address >

-c  count of the connectivity
-w  deadline to the connectivity upto three seconds
#ping 6 –I eth0 <ipv6 addresses or hostname>

-I  outer interface

The commonly used option in ping command

 b  broadcast
 n display host information numerically
 i  interval specific echo request in seconds
 I  Interface (echo request out interface)
 c  count (no. of echo request)
 w  deadline
 W  Time out before quitting

Scanning Network Host: -

Network Mapper(or)nmap:
 It is an open source port scanner there is provided by RHEL.
 It is a tool that administrator used to rapidly scan large network it can
also do more intensive scan on individual host
 nmap uses raw ip packets to determine what host are available on the
network. What services are host are offering.
 what OS they are running, what type of packet filters firewall are in used

#yum install –y nmap

#nmap workstation.lab.examle.com(or) 172.25.250.254
<hostname> <ip address>
Nmap scan : report for host (ip)
Not shown : 997 closed ports
Port: state service
Nmap done

#nmap ip/24
(172.25.250.254/24)
command to display nmap report of a network

#nmap -6 <host name >

Command to show ipv6 configuration

#nmap –sn <ip/24>

Command to display system present under network
-sn stands for no port scan

#nmap -sU -n < ip>

Command to display UDP ports in the network

Communicating Remote Services:(ncat)

 “ncat” is a troubleshooting tool that allows an administrator to
communicate with a service directly.
 ‘ncat’ uses either tcp or udp to communicate with a network service and
it also support SSL configuration
 ncat uses either nc or ncat command
 netcat can be used as a messenger, file grabber from other system and
get information about other system etc.
 netcat also perform port scanning and it is a heterogeneous tool.

#lab network-testing setup

Servera Serverb

1.yum install –y nmap-ncat 2. yum install –y nmap-ncat

3.check firewall (firewall-cmd – 5. nc –l –k 4231

list-all)
if the port is not added w have
to add manually 7. hi
4. nc serverb 4231 hello
6. hi
hello

Iptraf-ng:
 Iptraf is a open source network monitor software to monitor
the network traffic.
Iptraf-ng launches the application

#yum install –y iptraf-ng.x86_64

iptraf-ng can be launched by a root user and it must be a screen size of 80
columns and 24 lines
#iptraf-ng

 The filters main, menu, options allow an admin to create filters to

include and exclude specific type of network traffic.
 Each filter is a collection of rules that can select packets based on source
or destination address ports and ip protocol type

Resolving Connectivity Issues:

o Log files provide useful information and that helps troubleshoot network
device named issues.
o The ‘var’ log messages file has device detection information that is
performed at boot time
o It also records the message the network manager provides when it tries to
activate the device.

#grep –n eth0 /var/log/messages

Troubleshooting interface configuration problems:

» The log don’t provide the detailed information about he network

configuration
» #cat /etc/sysconfig/network-scripts/ifcfg-eth0
» If Dynamic Host Configuration Protocol (DHCP) is used to configure
in network interface, the network manager launches DH client to
handle that interface configuration.
» Interface n/w settings such as ip address, n/w mask, the default
gateway and DNS server are determine by the values provided by
DHCP server.
» When a n/w interface uses static address the system administrator must
correctly specify values static ipv4 and ipv6 n/w configuration settings
are define in the /etc/sysconfig/network-scripts/ifcfg-*
» When network manager is not used the values in this file determines
the persistent setting for n/w interface
» In this case, n/w configuration changes must be made to this files and
the n/w is restart.
» The BOOT PROTO variable determines what type of ipv4
configuration is performed on the n/w interface a value of none (or)
static indicates ipv4 n/w will be statically configures.
» DHCP (or) BOOTP value cause DH client to be used to manage the
interface configuration
» The ipv6init must have ‘YES’ for have ipv6 configuration to take
place on the n/w interface.

Resolving DNS Resolution Problem:

Client side DNS resolution can cause the service unavailable, the
main DNS resolution can be /etc/resolv.conf
Display the file identifies current DNS server

Troubleshooting Firewall Issues:

* Once basic n/w connectivity has been confirmed a n/w service
may be unavailable because of firewall problem.
* The firewall rules on the server hosting the n/w service may have
block the ports used to access the service
* #firewall-cmd --runtime-to-permanent
to make all services permanent
* A quick way to convert runtime rules into permanent rules is done
by this command
Scenario:
A second n/w interface has been added to servera but it doesn’t work
properly and it is connected to a private n/w and it should have following
assignments
Ipv4 address = 172.25.250.10/24
Ipv6 address = fd37:5265:6448:6174::0/64
DNS stop working when second interface was configure 172.25.250.254
is the ip address of working DNS. Your task is to correct the n/w
configuration issues on servera. The n/w settings is persist when the
machine is rebooted, when the n/w is functioning properly servera should be
able to ping the ipv6 address of server.lab.example.com using its host
name

Step 1:
#lab network-fix setup
Step 2:
#ifconfig
#cd /etc/sysconfig/network-scripts
#ls
It is showing ifcfg-enp2s0, it is not showing eth1
In that file, the ip address will be same as eth1 we are changing the name of
the file.

Step 3:
#mv ifcfg-enp2s0 ifcfg-eth1

Step 4:
#ls
#nmcli connection reload
#ifconfig
No error getting output perfectly

Second Error:
Step 1:
Check this command in servera
#ping serverb.lab.example.com (unknown host)

Step 2:
#cat /etc/resolv.conf
It is showing 4 name servers
This file won’t work more than 3 name servers

Step 3:
#nmcli connection show “System eth0”

Step 4:
#nmcli connection modify “System eth0” ipv4.dns “172.25.250.254”

Step 5:
#nmcli connection sown “System eth0” ; nmcli connection up “System
eth0”

Step 6:
#ping serverb.lab.example.com
#ping6 serverb.lab.example.com
 #exit
Step 7:
#lab network-fix grade

SELinux Logging

When SELinux blocks action from happening this action is logged using
“audit” daemon. The audit logs can be view /var/log/audit/audit.log

It is also used possible to search for exact messages that they are interested
in
#ausearch –m avc –ts recent
-m  message
avc  access vector control
-ts  timestamp
#ausearch –m avc –ts today

SELinux Troubleshooting Tools:

Simple SELinux denials like incorrectly label files in /var/www/html can

be designed and fixed by looking at /var/log/audit/audit.log and
restorecon under relevant files and directories
Some problems can be harder to fix
In this case we use setroubleshoot-server package which provides the
command “sealert” (simple log file)

#yum install –y setroubleshoot-server

sealert –a /var/log/audit/audit.log
-a  analyze
The above command is used for to see the file and what is the denials that
we can readable.

Common SELinux Issues:

Selinux issues broadly classified into 4 categories

1. Using Non-standard location for service data (context)

 Linux has a large set of default file context for the services even including
secondary location for most data.
Ex: Web server uses /var/www is a primary path and /srv/* will be
secondary path. Both paths obtain security context automatically. When a
non-standard location is used the system will need to be told about it.

Ex: /root/www/index.html has to be used as a document root that we are

changing this directory into the default one
#mkdir /root/www
#semanage fcontext -a –t httpd_sys_content_t “/root/www(/.*)?”
#restorecon –RFv /root/www
#ll –Zd /root/www

2. Switching from Disable Enforcing mode

Whenever a system is running with SELinux disable file context are

newly created file will not set according to the policy
Normally a file called /.autorelabel is created automatically when
switching to disable mode without the file present the system will not
perform as automatic relabel of all file system.
When switching back to enforcing mode causing the file to appear with
unlabeled_t type.
While process such as root shell running as unconfined can still access
this file confine services cannot, this will result in denial that can affect
the ability of a system boot.
#touch /.autorelabel  this will cause the system to perform a full file
system relabel on the next reboot

3. Booleans Not Set Correctly

Booleans acts as a switch that changes the behavior of the SELinux policy.
SELinux Booleans are the rules that can be enabled (or) disabled
#getsebool –a
It will show all the polices whether the switch is ON/OFF

#setsebool –P < policy name > ON/OFF

Command to show present state, default state and description

#semanage Boolean –l
4. SELinux Port Labeling

SELinux not commonly controls the files and directories but also it can
control the ports called port labeling.

Command to display the port contexts

#semanange port –l

Command to set the selinux context for the ports

#semanange port –a –t http_port_t -p tcp 5542
#semanage port –l | grep http_port_t

Scenario:
One of your co-worker recently perform some emergency maintenance and
troubleshooting on your servera machine while the original problem is
solved /var/log/audit/audit.log file on servera is now going rapidly. This
was spotted because the chronyd daemon now fails to start contrary to your
companies policy co-worker didn’t document any of this steps perform,
Investigate and fix the issues.

Step 1:
#lab selinuxts setup

Step 2:
See the selinux denials
#vim /var /log/audit/audit.log

Step 3:
#ausearch –m avc –ts recent
It will show the recent log files denials of selinux

Step 4:
Install the selinux troubleshoot package
#yum install –y setroubleshoot-server.x86_64

Step 5:
Show the readable denials of the log files, It will show the error named
unlabeled_t context
#sealert –a /var/log/audit/audit.log
Step 6:
#touch /.autorelabel
Relabel the entire files in the system

Step 7:
#reboot
#lab selinuxts grade

Recovering root password

Scenario:
One of your esteem colleague has recently left you company, one of the
final project include setting up servera machine. While your former
colleague did an admirable job in configuring the server the documentation
for servera is lacking. One of the places of information missing from the
documentation is the password for root account on servera . You have been
asked to reset the password to redhat.

Step 1:
Reboot you machine

Step 2:
Press ‘e’ button to edit the kernel

Step 3:
Go to end of the line starting with linux_16 and add rd.break console=tty1

Step 4:
Press ctrl+x to go to the next step

Step 5:
#mount –o rw,remount /sysroot

Step 6:
# chroot /sysroot

Step 7:
#echo “redhat” | passwd --stdin root
 Step 8:
#load-policy -i
It will load the policy

Step 9:
#restorecon -RFv /etc/
It will restore settings only that context file.
restorecon reset /etc/shadow context unlabeled_t to the shadow_t context

Step 10:
#exit
#exit

By using #touch /.autorelabel , it will force the system to relabel all

mounted file system during boot the reboot a quick way is to force the
system to load current selinux policy

Configuring Selinux Logging

 In standard log management configure log rotate in every week and retains
log keep for 4 weeks
 #vim /etc/logrotate.conf
 By sending log message to the remote host with the dedicated mass storage
administrators can maintain large achieves of system logs for the system
without changing the default log rotation configuration which is intended to
keep logs from over consuming disk storage
 It also provides a backup location for log messages in case a system suffers a
hard disk failure all other problem which cause the local logs to no longer be
available
 In this situation the copy of the log messages which resides on the central log
host can be used to help diagnose the issues that caused the problem.

Central log server has two configurarion

1. Central log host
2. Client

From server side:

Step 1:
#vim /etc/rsyslog.conf
go to the 19 and 20 lines and make them to uncomment
$ModLoad imtcp.so
$InputTCPserverRun 514

Enable the port 514 for tcp flowing from the clients
Next, go to the section rules

Step 2:
Write template for making central log host
$template DynamicFile,
“/var/log/loghost/%HOSTNAME%/%syslogfacilty-text%.log”

*.* -?DynamicFile

Step 3:
#vim /etc/logrotate.d/syslog
add
/var/log/loghost/*/*.log
It will rotate the logs each time whatever enters in that directory

Step 4:
#systemctl restart rsyslog
#systemctl enable rsyslog

Step 5:
#firewall-cmd --permanent --add-port=514/tcp
#firewall-cmd --reload
#cd /var/log/loghost

For client side:

#vim /etc/rsyslog.conf

go to rules section
*.* @@servera.lab.example.com:514

*.*  whatever file

@@  tcp
#systemctl restart rsyslog
#systemctl enable rsyslog

for checking
#logger –p user.info “text message from server”
To add logs to the system
#logger -p authpriv.crit “”message from server”

go to server side:

#cd /var/log/loghost
#ls
servera serverb
#cd serverb
#ls
authpriv.log daemon.log syslog.log user.log

D& T Completed
************************