SCP Post-Mortem

FREEBSD POST MORTEM

Final Report and Post-Mortem Analysis on the Regional CCDC FreeBSD Server

J. Rzeznik

Waukesha County Technical College

 SCP Post-Mortem
2

Abstract

Every spring, the National Science Foundation, along with information technology sponsors and

participating colleges throughout the United States participate in the Collegiate Cyber Defense

Competition (CCDC). The base premise of this competition is very simple: provide college

students who are majoring in computer science, information technology, and/or network

engineering to gain hands-on experience in network security implementation and management in

scenarios that are meant to resemble enterprise networks of the types that can be found in

corporate, government, academic, or military environments.

This year marks the first time Waukesha County Technical College (WCTC) has assembled a

team and participated in this event. On 12 March 2011, this team won the first round of the

competition by beating out two other technical colleges located in Madison and Milwaukee,

Wisconsin and becoming the Wisconsin Collegiate Cyber Defense state champions. As a result,

WCTC was invited to represent Wisconsin at the Midwest Regional CCDC in Illinois on 25-26

March 2011. Unfortunately, WCTC did not win or rank within this round of the competition.
 SCP Post-Mortem
3

This report is primarily focused on one server in particular, and is meant to provide a basic report

as to what it was, what it was supposed to do, what happened to it, and perhaps what could have

been done to further improve its security.

 SCP Post-Mortem
4

Final Report and Post-Mortem Analysis on the CCDC FreeBSD Server

On 25 and 26 March, our team went and participated in the Collegiate Cyber Defense

Competition at Moraine Valley Community College in Palos Hills, Illinois. Our ultimate purpose

was to rehabilitate, restore, and defend a simulated network for an undisclosed health-care firm

after the previous Information Technologies staff was dismissed for failing a required audit under

the federal Sarbanes-Oxley Act.

I took primary responsibility for one of the primary file servers, which was designated for

FTP, SCP, and SSH services within this network. Upon entering the network lab, I found it was

running an out-of-date and unsupported version of the FreeBSD operating system.

Competition Particulars

Network Roles and Basic Information

Within this network were three separate virtual Local Area Network (VLAN) segments.

Two of these were assigned their own domain names: lookingglass.com and wonderland.com.

The server was named “tinkerbell” and located in the wonderland.com domain with an IP

address of 172.20.241.10, subnet mask 255.255.255.0. This server in particular was designated

as the “file” server for the wonderland.com domain and required secure copy (SCP) and secure

shell (SSH) services to be accessible at all times.

Upon accessing this server inside the network, I found that it was running FreeBSD (a

descendant of the old Berkeley Software Distribution of UNIX developed at the University of

California, Berkeley), version 7.1-RELEASE, which is no longer supported by the developers.

As such, there were no more current security or stability fixes being made available. Therefore, I

decided that after changing the weak passwords and putting additional backup users on the
 SCP Post-Mortem
5

system for access, I would upgrade it via the included utility to the current version, which as of

the competition date was 7.4-RELEASE.

Services Found

Surprisingly, there were few major services installed on the “tinkerbell” server. Included

was more or less the “default” install of FreeBSD 7.1 along with ProFTPd, OpenSSH version

5.1p1, portions of PHP 5.8, local system logging, and various configuration utilities. To my

knowledge, there was no known firewall running on the server.

Over the course of the competition, I made the following major system changes that

would have had a direct impact on the known services:

a.) Upgraded the FreeBSD version to 7.4-RELEASE, as that version will still be receiving

current updates, security fixes, and patches for approximately another two years.

Upgraded the entire ports tree to the current state as it existed on 26 March 2011, so as to

obtain the latest available software known to run safely on FreeBSD 7.4.

Removed ProFTPd and replaced it with vstftpd 2.3.2, which was the latest version on the

FreeBSD ports system.

Installed GNU Nano as another option for editors, as I knew I would have to be making

edits to various configuration files.

Removed as much of the PHP as I could, as I did not see a compelling need to continue

trying to run it.

Installed and enabled the FreeBSD port of pf, a very powerful-yet-flexible packet

filter/firewall program originally developed as part of OpenBSD.

 SCP Post-Mortem
6

Attempted to install a parallel SSH system manually in the hopes that doing so would

successfully restore the services back to the “white team” panel of judges and outside-

the-immediate-network corporate users. The failure of this attempt will be addressed later

in this report.

Known Issues

From the point-of-view of the server, the single most critical known issue was the

unreliability of the network given the penetration testers’ attempts at other network devices and

services. This severely hampered the ability to maintain the service even on an internal-domain

level, and made attempts to upgrade, patch, and further secure the server much more difficult, if

not clearly impossible to accomplish at certain junctures. I feel it is vital to note that this was not

a problem with the network connectivity on the FreeBSD server itself, but with either the

wonderland.com Domain Name System (DNS) server or the upstream Cisco networking

equipment—the failure of either (or both) usually meant widespread network outages and no

outbound TCP/IP traffic.

By no means was my system perfect, or even remotely worthy of being considered

impregnable. Though during the competition I did not recognize some of these errors, I am now

fully aware that there were plenty of overlooked items that I have found upon re-examination of

a similar system set up as a virtual machine. Notable examples of these include: finding that a

simple port scan using the commonly-available Nmap scanner was enough to reveal not just that

it was a FreeBSD server, but also the particular version of OpenSSH installed along with the

fingerprint for the server’s public key1, and that the fetch command for pulling critical updates

1
An example output from the Nmap scanner is attached to the end of this report as Figure 1.
 SCP Post-Mortem
7

often had severe issues if it managed to connect to public file mirrors established by the

FreeBSD development team.

Noted Direct Attacks

The primary means of direct attacks against the server that were observed were by means

of the SSH service, with the explicit attempt to flood or otherwise cripple the service on the

server. Most notably, the methods I observed over the two days were usage of port scanning tools

such as Nmap and brute-force/”hammering” attacks via programs such as THC-Hydra. I

frequently ran the “who” command on the box and did not see anyone but myself logged into the

system. Occasionally, the user “peter” would be logged in, particularly on Friday and very early

on Saturday, before the network services were severely disrupted.

From what I can remember noting off of the authentication logs stored at

/var/log/auth.log, these attempts were done at regular but infrequent intervals, leading me to

assume that whoever was logging in as the user “peter” was a member of the benign White team

for the purposes of verifying network service operations. 2

Noted Indirect Attacks

Likewise, there were plenty of indirect attacks against the server. Some of these could be

anticipated, such as through the Splunk server (running Ubuntu 8.04) on the lookingglass.com

VLAN or through connections into the server from the e-Commerce server running CentOS 5.5.

This CentOS server is of particular interest for the simple reason that it was literally next to the

FreeBSD server and sat on the same domain and VLAN. Such indirect attacks could have easily

been targeted to “my” SCP server by way of using these partially-compromised machines as an

2
A sample capture of the “re-constructed” /var/log/auth.log from 28 March 2011 can be found at the end of this
report as Figure 2.
 SCP Post-Mortem
8

intermediary—a pentester could conceivably use a compromised account or service on either the

Splunk or e-commerce servers instead of attempting to directly attempt to attack the SCP server

from his own remote machine.

Other indirect attacks weren't all that noticeable. During the Red team debriefing, it was

revealed that they had managed to get code that would enable PHP shells on most of the

machines across all of the competition networks, including that managed by the WCTC team. In

addition, the Linux and BSD boxes were mentioned to be potentially susceptible to odd or

malicious tasks being scheduled into the system's crontab. Unless one knew what they were

looking for exactly, such tools are very easy to overlook, and it is more than probable that there

was something of that sort on the SCP server. However, I did not see any obvious attempts by

remote users to execute it.

Service Reliability

With the exception of two brief scrubs on Friday due to failed upgrades and taking into

account the ability of the router to remain open for outgoing connections, the required SCP and

SSH2 services were up virtually all day Friday and for the first hour or so on Saturday. The

services went down in the second hour of scoring on Saturday and never came back up, despite

my repeated attempts to bring them back online. This was around the same time I completed the

user-land portion of the upgrade to 7.4-RELEASE, and thus I suspect this partially explains the

service outage to the White team.

However, I do wish to note that while the SCP and SSH2 services were rendered on the

scoring engine as “Failed,” the services themselves were running and accessible from all of the

VLANS, to which I submitted photographic proof in the form of a screen capture to the White
 SCP Post-Mortem
9

team in response to the “feedback” inject. To make matters worse, I did make inquiries via the

same inject system as to what kinds of errors they were seeing, but unfortunately I never

received a response other than a vague hint from the competition director that “it had something

to do with your authentication.” Without knowing what errors they were having when trying to

connect to the FreeBSD server, I do not know if I can even begin to assess what exactly was

going on, and why their attempts to connect were being kicked back even after thoroughly using

trial and error on both the SSH daemon configuration as well as the pertinent configuration files

in the /etc/pam.d directory.

Results

Ultimately, this means that it is difficult for me to objectively and effectively determine

any necessary results pertaining specifically to the FreeBSD server. However, I can say that the

team neither advanced forward to the national competition, nor placed in the top three teams out

of eight. As I do not have access to the scoring for our team, I further cannot judge what

vulnerabilities the Red team found and used without my finding out about them, nor can I

adequately determine whether such attacks did happen and whether they were directly from the

Red team or through a compromised device on the network. Thus, I am presently reduced to

what I have observed both during the competition and during the “re-construct” experiments

with the virtual machines along with my own semi-educated hypotheses.

Conclusions Drawn/Personal Evaluation

There are conclusions that can be drawn from this experience, but it is completely reliant

on the frame of references one applies. For the purposes of this report, I will go through three

such frames: educational, business, and reliability. I have chosen these three because I believe
 SCP Post-Mortem
10

they reflect most on what the competition is meant to be about, and meant to give some measure

of “real-world” applications.

Arguably, from a purely business perspective, my service as the FTP/SCP/SSH engineer

was an abject failure. Considering that this network was supposed to be for a healthcare

company, and that my predecessor at this hypothetical firm was dismissed along with the

remainder of the IT staff for failing audits related to compliance with federal laws, I cannot say

that I really did much to improve the company's situation. In the real world, I would most likely

be asked to tender my resignation from the company if I wasn't already fired. In any business,

and particularly in the healthcare industry where strong IT and security skills are necessary to

survive, I just didn't make the grade.

Sure, one could reasonably argue that “you're a networking student in a simulation filling

in the role of an FTP engineer, and that you've still got time left in your schooling before you'll

be out in the real-world.” In the “real world” though, there is a very thin tolerance for error, and

even the slightest mistake can have grave consequences. In this regard, I am reminded of the

sobering statistics regarding business continuity, such as that 93% of companies that suffer a

significant data loss go out of business within five years (Jensen, 2009). The consequences of

that don't just affect me as an individual, but affect co-workers, and even the general public.

Likewise, this weekend proved to be a failure on the reliability measure. In the

networking world, there is the “five nines” rule, which states that network services should be up

at least 99.999 percent of the time (Percy, 2003). Between the scrubbing and reboots, I know my

system and services were down for more than the roughly 6 seconds that would be allowed for in

a week, and much more if one includes the router not being able to send outbound network
 SCP Post-Mortem
11

traffic. This too has a major impact, particularly on companies and services that are mission-

critical. When services are not available, the resulting downtime can at best harm a company's

bottom line and detract from productivity. As this is a healthcare company, it is conceivable to

think that perhaps there are patients out there whose care is dependent on those services being

available at five-nines, the failure of which could very well mean life or death. Even

notwithstanding the business aspects, most service-level agreements provide for a high level of

availability and an FTP engineer who cannot provide that high level of reliability for his services

cannot really expect to be successful in his craft.

While the last two assessments are in many respects particularly damning against me and

specifically how successful I was in meeting the real-world demands that I can and probably

should expect when I go to practice my trade, there is the third element that proved to be a

success, and that is the educational element. As I have mentioned in my first frame of reference, I

am just a student filling in a simulated role as an FTP engineer. Though I do have my quibbles on

how realistic I think this seems to be, I also think that it is hard to set such an experience in an

academic setting. Further, do a lot of IT/networking students get the chance to try some of their

skills out in such a setting? Personally, I would imagine not.

The experience itself is invaluable, as it is a good way to showcase what one's networking

and security skill strengths are, while also providing a way to show where one needs more work

if they want to be successful in the industry. With respect to myself, this weekend showed that

while I am quite adaptable across Unix-like platforms and have some depth of knowledge as to

what needs to be done, I do not know all of the tricks of the trade when it comes to providing that

balance of security and reliability. As some of the other teammates can attest, the sheer stress of
 SCP Post-Mortem
12

trying to strike that balance under pressure can make me just a little more frazzled than normal—

which is also something I know I need to work on controlling more effectively. In other words,

like every student, there's plenty I know, and plenty I have yet to learn.
 SCP Post-Mortem
13

References

Jensen, S. (2009). Overview of business continuity planning. Informally published manuscript,

Extension Department of Agricultural Safety and Health, The Ohio State University,

Columbus, OH. Retrieved from http://fabe.osu.edu/empe/BCPOverview.pdf

Percy, K. (2003, April 14). Five nines, by the book. Network World, Retrieved from

http://www.networkworld.com/columnists/2003/0414testerschoice.html
 SCP Post-Mortem
14

Figure 1

Output of Nmap Scan run on “re-constructed” FreeBSD 7.1-RELEASE server:

Starting Nmap 5.35DC1 ( http://nmap.org ) at 2011-03-27 21:45 UTC

Nmap scan report for blofeld (192.168.1.90)

Host is up (0.0027s latency).

Not shown: 998 closed ports

PORT STATE SERVICE VERSION

21/tcp open ftp vsftpd 2.0.7

22/tcp open ssh OpenSSH 5.1p1 (FreeBSD 20080901; protocol 2.0)

|_ssh-hostkey: 1024 87:ed:d6:be:52:75:00:0a:c9:76:4d:51:09:e4:38:d2 (DSA)

MAC Address: 00:0C:29:BD:28:FF (VMware)

Device type: general purpose

Running: FreeBSD 7.X|8.X

OS details: FreeBSD 7.0-RELEASE-p1 - 8.0-CURRENT

Network Distance: 1 hop

Service Info: OS: Unix, FreeBSD

TRACEROUTE

HOP RTT ADDRESS

1 2.67 ms blofeld (192.168.1.90)

OS and Service detection performed. Please report any incorrect results at

http://nmap.org/submit/ .

Nmap done: 1 IP address (1 host up) scanned in 22.46 seconds

 SCP Post-Mortem
15

Figure 2

Sample capture of “re-constructed” /var/log/auth.log:

Mar 28 23:17:36 blofeld sshd[6641]: Did not receive identification string from 192.168.1.82

Mar 28 23:17:39 blofeld sshd[6644]: Protocol major versions differ for 192.168.1.82: SSH-

2.0-OpenSSH_5.1p1 FreeBSD-20080901 vs. SSH-1.5-Nmap-SSH1-Hostkey

Mar 28 23:17:39 blofeld sshd[6645]: Protocol major versions differ for 192.168.1.82: SSH-

2.0-OpenSSH_5.1p1 FreeBSD-20080901 vs. SSH-1.5-NmapNSE_1.0

Mar 28 23:30:52 blofeld sshd[6677]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6676]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6679]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6683]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6678]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6684]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6682]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6681]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6685]: error: PAM: authentication error for tom from linux-7mp1

Mar 28 23:30:52 blofeld sshd[6680]: error: PAM: authentication error for tom from linux-7mp1
 SCP Post-Mortem
16

Figure 3

Screen captures from “re-constructed” server:

This is an example of closed port RST response seen constantly while the actual server was in

operation.
 SCP Post-Mortem
17

(screen captures continued)

Example of the notifications displayed whenever the pentesters attempted a direct attack via SSH

hammering.
 Legend
Rough sketch of competition network Legend Subtitle
IP and Server Identities changed Symbol Count Description
Cisco 2811
(Integrated Services Directory
1
Router) server
IOS 12.4(24)T1
3 Ethernet
1 FTP server
Database
Windows 2000 1
Professional
server
Hostname: 1 Web server
ZORIN
IP: 172.16.3.35 E-Commerce
1
server
Cisco PIX 515E 1 File server
Firewall
Virtual LAN Segment C Firmware 7.2 2 PC

172.16.3.0/24

Windows
Server 2003
Hostname:
GOLDFINGER FreeBSD 7
IP: 172.16.2.21 Hostname:
SCARAMANGA Windows Server
IP: 172.16.1.101 2008 R2
Hostname:
STROMBERG
IP: 172.16.1.116
172.16.2.0/24
172.16.1.0/24
Virtual LAN Segment B Virtual LAN Segment A
Cisco Catalyst
2950
(24-port Main
Switch) Windows XP
Ubuntu 8 Professional
IOS 12.1(14)EA1a
Hostname: Hostname:
Windows Server 2003
KLEBB DRAX
Hostname:
IP: 172.16.2.22 CentOS 5 IP: via DHCP
LARGO
Hostname:
IP: 172.16.2.22
BLOFELD
IP: 172.16.1.124