Chapter 5

System Security

1/11/2025 1
 Outline
• System security problems
• Malicious software
• DoS attacks
• Firewalls and IDSs
• Software security
• OS security
• Database and cloud security

1/11/2025 2
 System security problems
• Malware:
• A pro gram that is inserted into a system, usually covertly, with the intent of
compromising the CIA of the victim’s data, applications, or operating system or
otherwise annoying or disrupting the victim.
• Types: adware, trapdoor, downloaders, exploits, spyware, TH, virus, worm, …
• Propagation mechanisms:
• Infection of existing exe or interpreted content that spread to other systems
• Exploit of SW vulnerabilities
• Social engineering attacks
• Countermeasures:
• Antiviruses - Do not allow malware to get into the system in the first place, or block the
ability of it to modify the system. [but nearly impossible to achieve]
• Main elements of prevention: policy, awareness, vulnerability mitigation, and threat
mitigation.
• If detection fails, use threat mitigation: detection, identification, removal
1/11/2025 3
 System security problems
• DoS Attacks:
• A denial-of-service (DoS) attack is an attempt to compromise availability by
hindering or blocking completely the provision of some service.
• E.g. flooding Web Server with so many spurious requests
• Focus on network services that are attacked over their network connection.
• Resources attacked include: network bandwidth, system and app resources
• E.g. resource attacks: SYN spoofing attacks – targets table of TCP connections on server,
poison packet – which crashes system by triggering a bug in system’s network SW
• Classic DoS attacks – overwhelm capacity of network
• Source address spoofing – use forged source addresses in attack packets
• SYN spoofing – overflows tables that manage TCP connections
• ICMP flooding – ping flood using ICMP echo request
• DDoS Attacks – use multiple systems to generate attacks
• Reading: Defenses against DoS attacks.
1/11/2025 4
 System security problems
• Intrusion Detection Systems (IDSs):
• One of the key threats to security is the use of some form of hacking by an
intruder, often referred to as a hacker or cracker.
• Classes of intruders – cybercriminals, activists, state-sponsored org.s, …
• Examples of intrusion:

1/11/2025 5
 System security problems
• Intrusion Detection Systems (IDSs) and IPSs:
• IDSs and IPSs counter the types of attacks listed above
• Security Intrusion: A security event, or a combination of multiple security
events, that constitutes a security incident in which an intruder gains, or
attempts to gain, access to a system (or system resource) without having
authorization to do so.
• Intrusion Detection: A security service that monitors and analyzes
system events for the purpose of finding, and providing real-time or near real-
time warning of, attempts to access system resources in an unauthorized
manner.
• IDS logical components:
• Sensors – collect data and forward it to analyzer
• Analyzers – determine if intrusion has occurred
• User interface – enables to view output from system or control system behavior
• Reading – Network-based IDS, Honeypots, example systems
1/11/2025 6
 System security problems
• Firewalls and IPSs:
• Firewalls protect a local system or network of systems from network-based
security threats while at the same time affording access to the outside world
via wide area networks and the Internet.
• Firewall is alternative or complement to host-based security service.
• Design goals:
• All traffic must pass through it
• Only authorized traffic is allowed to pass
• The firewall itself is immune to penetration
• Types of firewalls:
• Packet filtering
• Stateful inspection – creates directory of outbound TCP connections
• Application-level gateway/proxy – relays application traffic
• Location and configurations: Reading: VPNs, IPSec, IPS
• External, internal
1/11/2025 7
 System security problems
• Firewalls – uses, types:
• Firewalls isolate organization’s internal network from larger Internet, allowing
some packets to pass, blocking others

administered public
network Internet
trusted “good guys” untrusted “bad guys”
firewall

1/11/2025 8
 System security problems
• Firewalls – uses, types:
• prevent denial of service attacks:
 SYN flooding: attacker establishes many bogus TCP connections, no
resources left for “real” connections
• prevent illegal modification/access of internal data
 e.g., attacker trying to replace BDU’s homepage with something else
• allow only authorized access to inside network
 set of authenticated users/hosts
• three types of firewalls:
 stateless packet filters apply a set of rules to each incoming and outgoing IP
 stateful packet filters packet and then forwards or discards the packet
 application gateways

1/11/2025 9
 System security problems
• Firewalls – uses, types:
• Stateless packet filtering:
• internal network connected to Internet via router firewall
• filters packet-by-packet, decision to forward/drop packet based on:
• source IP address, destination IP address
• TCP/UDP source, destination port numbers
• ICMP message type
• TCP SYN, ACK bits
• example 1: block incoming and outgoing datagrams with IP protocol field = 17
and with either source or dest port = 23
• result: all incoming, outgoing UDP flows and telnet connections are blocked
• example 2: block inbound TCP segments with ACK=0
• result: prevents external clients from making TCP connections with internal
clients, but allows internal clients to connect to outside
1/11/2025 10
 System security problems
• Firewalls – uses, types:
• Stateless packet filtering – more examples:

Policy Firewall Setting

no outside Web access drop all outgoing packets to any IP address, port
80
no incoming TCP connections, except drop all incoming TCP SYN packets to any IP
those for institution’s public Web server except 130.207.244.203, port 80
only.
prevent Web-radios from eating up the drop all incoming UDP packets - except DNS and
available bandwidth. router broadcasts.
prevent your network from being used for drop all ICMP packets going to a “broadcast”
a smurf DoS attack. address (e.g. 130.207.255.255)

prevent your network from being drop all outgoing ICMP TTL expired traffic
tracerouted

1/11/2025 11
 System security problems
• Firewalls – uses, types:
• Packet filtering examples - ACLs:
• ACL: table of rules, applied top to bottom to incoming packets
source dest source dest flag
action protocol
address address port port bit
outside of any
allow 222.22/16 TCP > 1023 80
222.22/16
allow outside of 222.22/16
TCP 80 > 1023 ACK
222.22/16
outside of
allow 222.22/16 UDP > 1023 53 ---
222.22/16
allow outside of 222.22/16
UDP 53 > 1023 ----
222.22/16
deny all all all all all all

1/11/2025 12
 System security problems
• Firewalls – uses, types:
 stateful packet filter: track status of every TCP connection
• track connection setup (SYN), teardown (FIN): determine whether incoming, outgoing
packets “makes sense”
• timeout inactive connections at firewall: no longer admit packets

action source dest proto source dest flag check

address address port port bit connection
outside of any
allow 222.22/16 TCP > 1023 80
222.22/16

allow outside of 222.22/16

TCP 80 > 1023 ACK x
222.22/16

outside of
allow 222.22/16 UDP > 1023 53 ---
222.22/16

allow outside of 222.22/16

222.22/16
UDP 53 > 1023 ---- x
deny all all all all all all

1/11/2025 13
 System security problems
• Firewalls – uses, types:
 Application gateways:
• filter packets on application data as well as on IP/TCP/UDP fields.
• example: allow selected internal users to telnet outside

1. require all telnet users to telnet

through gateway. host-to-gateway
telnet session application
2. for authorized users, gateway sets gateway
up telnet connection to dest host router and filter
• gateway relays data between 2
connections
3. router filter blocks all telnet
connections not originating from gateway-to-remote
gateway host telnet session

1/11/2025 14
 System security problems
• Limitations of Firewalls:
• IP spoofing: router can’t know if data “really” comes from claimed source
• if multiple apps need special treatment, each has own app. gateway
• client software must know how to contact gateway
• e.g., must set IP address of proxy in Web browser
• filters often use all or nothing policy for UDP
• tradeoff: degree of communication with outside world, level of security
• many highly protected sites still suffer from attacks

1/11/2025 15
 Software Security
• Buffer/heap overflow attacks:
• Result from careless programming in applications/exploits to OS or applns/
• When more data is placed than capacity allocated; attackers can crash or gain
control of the system by inserting crafted code
• Detection – inspection of source program, tracing execution in oversized input
• Prevention: using HW/SW – protect address space, stack
• Software security:
• Many computer security vulnerabilities result from poor programming practices.
• Categories: insecure interaction between components, risky resource
management, porous defenses, insufficient checking and validation of data
• Solution: defensive/secure programming – attention to all details /no assumption/
• Incorrect handling of program input is one of the most common failings in
software security. All sources, value and type of data should be identified

1/11/2025 16
 OS Security
• OS Security layers:

• Each of these layers of code needs appropriate hardening measures in place

to provide appropriate security services.
• Strategies include:
• White-listing applications, patching, restrict privileges, defense-in-depth

1/11/2025 17
 OS Security
• Most organizations hold critical data; hence be aware of vulnerabilities
in OS and applications
• System may be compromised during:
• Installation process
• Before installing latest patches
• Implementing other hardening measures
• Hence, building and deploying a system should be a planned process
designed to counter such a threat, and to maintain security during its
operational lifetime.
• Installation planning – maximizes security, minimizes cost. Consider
things like:
• Purpose of system, user categories and authentication, what and how to access
stored data, administration (who, how)

1/11/2025 18
 OS Security
• OS hardening
• First step: secure the base operating system upon which all other applications
and services rely. Properly install, patch and configure OS. DON’T use default!
• Basic steps to secure an OS:
• Install and patch the operating system.
• Harden and configure the operating system to adequately address the identified
security needs of the system by:
• Removing unnecessary services, applications, and protocols.
• Configuring users, groups, and permissions.
• Configuring resource controls.
• Install and configure additional security controls, such as anti-virus, host-based
firewalls, and intrusion detection systems (IDS), if needed.
• Test the security of the basic operating system to ensure that the steps taken
adequately address its security needs.
1/11/2025 19
 OS Security
• Installation: Initial Setup and Patching
• System security begins with the installation of the OS (e.g. over network)
• Consider constructing new systems on a protected/isolated network, or severely
restrict Internet
• Use non-infected removable media to transfer OS plus patch
• Full installation and hardening process should occur before the system is
deployed to its intended, more accessible, and hence vulnerable, location.
• Initial installation should install the minimum necessary for the desired system,
with additional software packages included only if they are required for the
function of the system.
• Secure boot process – BIOS: adjust settings plus setting password
• Limit the media the system is permitted to boot from (avoids changing boot pro.
• Be careful with installing additional device driver programs
• Keep system up-to-date: install critical security related patches
1/11/2025 20
 OS Security
• Remove Unnecessary Services, Application, and Protocols
• SW packages may contain vulnerabilities, so minimize risk by reducing #packages
• Customize default installation – install only required packages
• Uninstalling may fail to remove all components of a package
• Configure Users, Groups, and Authentication
• All modern Oss implement discretionary access controls to data and resources
• Some systems may provide role-based or mandatory access control mechanisms as well.
• The system planning process should consider the categories of users on the
system, the privileges they have, the types of information they can access, and
how and where they are defined and authenticated. [e.g. admin, normal, guest]
• Allow users with elevated privileges access only when needed
• One key decision is whether the users, the groups they belong to, and their
authentication methods are specified locally on the system or will use a
centralized authentication server. Configure appropriate details
• Configure all authentication [passwords] and secure all default passowords
1/11/2025 21
 OS Security
• Configure Resource Controls
• Once the users and their associated groups are defined, appropriate permissions
can be set on data and resources to match the specified policy.
• Limits which users can execute some programs, or can r/w data to directories
• Install Additional Security Controls
• Install and configure tools such as anti-virus software, host-based firewalls, IDS
or IPS software, or application white-listing [3rd party or OS]
• Firewalls – limit access by port or protocol, from some or all external systems.
• IDSs and IPSs – for traffic monitoring, or file integrity checking to identify and
even respond to some types of attack.
• Tools for whitelisting – prevent attacker installing and running malware
• Test the System Security
• Goal – to ensure that security configurations are correctly implemented
• Use checklists or tools to review if system meets requirements

1/11/2025 22
 Application Security
• Similar steps used in the base OS may be followed
• Install software only for certain functionality
• Carefully select, configure and regularly update remote access software
• Install and patch services/applications to the most recent supported
secure version appropriate for the system.
• Application Configuration:
• This may include creating and specifying appropriate data storage areas for the
application, and making appropriate changes to the application or service default
configuration details.
• Configurations include default data, scripts, user accounts, access rights
• E.g. config. fault in web and file servers: using same user account as server –
results in attacks like web defacement – mitigation: ensure only read, but not write

1/11/2025 23
 Application Security
• Encryption:
• If used, must be configured; and appropriate cryptographic keys created, signed,
and secured. E.g. suitable public and private keys for ntk services: IPSec and TLS
• Suitable CA for creating and signing (X.509) certificates
• Security maintenance – includes the following additional steps:
• Monitoring and analyzing logging information
• Performing regular backups
• Recovering from security compromises
• Regularly testing system security
• Using appropriate software maintenance processes to patch and update all critical
software, and to monitor and revise configuration as needed

1/11/2025 24
 Database and Cloud Security
• Org. databases store sensitive info in a single logical system, including
financial data, phone records, customer and employee info, …
• Such data are targeted by internal/external threats during access; hence
database security is important component
• DBMSs:
• A database is a structured collection of data stored for use by one or more
applications.
• DBMS is a suite of programs for constructing and maintaining the database and for
offering ad hoc query facilities to multiple users and applications.
• A query language provides a uniform interface to the database for users and
applications.
• Database systems pose more security requirements than typical OSes; e.g. OS
limits read/write access to entire file, but not a record
• A DBMS more detailed access control to be specified, using commands: select,
insert, update, delete
1/11/2025 25
 Database and Cloud Security
• SQL injection attacks
• One of the most prevalent and dangerous network-based security threats.
• Designed to exploit the nature of Web application pages.
• An application server Web page will make SQL queries to databases to send and
receive information critical to making a positive user experience.
• In such an environment, an SQLi attack is designed to send malicious SQL
commands to the database server, e.g to extract bulk data (most common)
• Attackers can dump database tables with hundreds of thousands of customer
records.
• SQL injection can also be exploited to modify or delete data, execute arbitrary
operating system commands, or launch denial-of-service (DoS) attacks.
• The attack is viable when user input is either incorrectly filtered for string literal
escape characters embedded in SQL statements or user input is not strongly typed,
and thereby unexpectedly executed.
• Injection technique - by prematurely terminating a text string and appending a
new command.
1/11/2025 26
 Database and Cloud Security
• Reading: SQL injection attacks – how to prevent them:

1/11/2025 27
 Cloud Security
• Organizations are moving their IT operations to an Internet-connected
infrastructure known as enterprise cloud computing.
• The use of cloud computing raises a number of security issues.
• Definition [from NIST SP-800-145]:

• The definition refers to various essential characteristics and, service and

deployment models
1/11/2025 28
 Cloud Security
• Cloud
computing
elements:

1/11/2025 29
 Cloud Security – risks and countermeasures
• Security controls in cloud may be similar to any IT envt, but present risks
specific to cloud envt., because of ways to enable service.
• The enterprise loses a substantial amount of control over resources,
services, and applications but must maintain accountability for security
and privacy policies.
• Threats to cloud security:
• Abuse and nefarious use of cloud computing: ease of use enables users to get into
and attack cloud, such as malicious code, spamming and DoS attacks; e.g.
countermeasures – strict registration and validation, enhanced monitoring
• Insecure interfaces and APIs: due to exposed APIs used to manage and interact
with cloud services; e.g counter – ensuring strong authentication and AC
• Malicious insiders: due to necessity and trust in CP system admins, service
providers; counters – contracting HR reqts legally, transparency in security and mgt

1/11/2025 30
 Cloud Security – risks and countermeasures
• Threats to cloud security …
• Shared technology issues: shared IaaS, no strong isolation for multi-tenant
architecture and scaling; VMs are used, which are vulnerable; counters – best
practices for installation/configuration; monitoring for unauthorized activity, …
• Data loss or leakage: most devastating impact; counters – implement strong API
AC, encrypt and protect integrity of data in transit, strong key gen, …
• Account or service hijacking: using stolen credentials, to compromise CIA of
services; counters – prohibit sharing accounts, MFA, proactive monitoring
• Cloud Security as a Service (SecaaS)
• A package of security services offered by a service provider that offloads much of
the security responsibility from an enterprise to the security service provider.
• These include: authentication, anti-virus, antimalware/spyware, intrusion
detection, and security event management.

1/11/2025 31
 Cloud Security – risks and countermeasures
• Cloud Security as a Service (SecaaS)
• The Cloud Security Alliance defines SecaaS as the provision of security applications
and services via the cloud either to cloud-based infrastructure and software or
from the cloud to the customers’ on-premise systems.
• Categories:

1/11/2025 32
 Thank You,

Next: Network Security

1/11/2025 33