See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261260567

Performance testing: Analyzing differences of response time between

performance testing tools

Conference Paper · June 2012

DOI: 10.1109/ICCISci.2012.6297157

CITATIONS READS

4 909

2 authors, including:

Dhiauddin Suffian
MIMOS Technology Solutions Sdn. Bhd.
15 PUBLICATIONS 77 CITATIONS

SEE PROFILE

All content following this page was uploaded by Dhiauddin Suffian on 01 January 2016.

The user has requested enhancement of the downloaded file.

 Performance Testing: Analyzing Differences of
Response Time between Performance Testing Tools

Muhammad Dhiauddin Mohamed Suffian Fairul Rizal Fahrurazi

Faculty of IT & Multimedia Communication Product Quality & Reliability Engineering
Open University Malaysia MIMOS Berhad
Kuala Lumpur, Malaysia Kuala Lumpur, Malaysia
dhiauddin@oum.edu.my fairul.fahrurazi@mimos.my

Abstract— This research focuses on the study and evaluation of performance test is very crucial to achieve this target. This is
response time differences given by three tools used for where the issues start to come into picture.
performance testing. The motivation for this research work is to
understand the behavior of various performance testing tools Several issues have been observed related to tools when
towards determining the accuracy of the response time result. It conducting performance testing such as tools compatibility
is conducted with the aim of demonstrating and proving that with the software under test, tools installation, tools setup, tools
differences of response time do exist between different tools when flexibility in doing test both for client and server side and also
conducting performance test for the same webpage as well as the one that becomes the focus of this research, which is
analyzing the reasons behind that situation. A static HTML response time generated by the tools.
webpage is put under load test for 1, 100, 200, 300, 400 and 500
concurrent users performed by the three tools. The findings The research problem falls into area on demonstrating and
clearly showed that different performance testing tool gave proving that the response time for conducting performance test
different response time when conducting load testing on same for the same website is different when using different
webpage. The findings are also supported with the justification performance testing tools. In addition, the research also
for these differences, which involve architecture and simulation suggests potential reasons or root cause behind these
mechanism of the respective tool. The summary and future work differences. This work tries to answer the question on “Why do
is presented at the end of the research different performance testing tools produce different response
time that are not even closed to each other?”
Keywords- performance testing; response time; load test; open
source tool; concurrent users The research discussion is organized into several sections.
Section II discusses the prior related works on webpage
response time and calculation of response time. Section III
I. INTRODUCTION outlines the overview of the common features of the testing
Performance testing is one of the test strategies performed tools used for the experiment. Section IV describes the
for the software under test, usually at system testing level. By environment setup while Section V drills down further on
conducting this test, we could assess the readiness of the discussing and elaborating the findings of the experiment.
software system in handling users’ loads thus ensure it could Section VI concludes the overall research together with
response within an acceptable time range as expected by the recommendation for future works
end users. This type of testing becomes more crucial if it
involves heavy integration with other external systems since II. RELATED WORKS
performance degradation might take place.
Ignoring performance test means that your system is not fully Most previous work on performance testing tools
tested, especially from the risk and operational profile comparison ignored on different result reported by each tools
perspectives. [1][2][4][5][7][8][9][10][11][12].VCAA[1] uses pricing and
user friendliness as a criteria to decide which tool to use while
The trigger point to embark on this research is driven by JDS[2] mentions about the ability to emulate a complex
real experiences as independent testing team in dealing with business process and support of unlimited number of
various issues, difficulties, challenges as well as successes in concurrent users.
doing performance test. These involve conducting performance
test for various type of software: standalone, web services, web Testingrefeclections.com [3] concludes that accuracy of
application, mobile application as well as grid and cloud load and response time is something we need to evaluate
applications. Suitable and correct approach or strategy of against our particular application and not something to
performance test need to be put in place for software so that compare when determining the tool to use or buy. But, there is
performance defects can be fixed and the test results are no work so far to understand why they are different against
accurate. Thus, the selection and use of right testing tool for each other at the first place. Shall we have a framework or
memorandum of understanding (MOU) about the uniformity of

978-1-4673-1938-6/12/$31.00 ©2012 IEEE

response time on all performance testing tools? Each tool IV. THE ENVIRONMENT SETUP
claims they are better than the others but none able to justify Correct test environment setup for this experiment is very
the performance testing results against the real world. important to ensure the accuracy of response time from the
Our research effort is proving if there is a difference performance testing tools. The setup involved two identical
response time between different performance testing tools and machines. One machine serves as the client with performance
potential reasons contribute to it. This is to enlighten the testing tools installed and a server which hosts the static web
performance testers that no tool in the world able to fully pages. These machines are connected via PC-to-PC LAN cable
replacing human when performing performance testing as the or commonly referred as cross cable connection. The reason for
result from each tools are not consistent, at least for now. this kind of environment setup is to minimize the network
factors from influencing the results of performance test. The
III. OVERVIEW OF PERFORMANCE TESTING TOOLS setup is represented in Figure 1 below:

Performance tools are used for several kinds

of performance testing including load test, stress test, volume
test and endurance test. Most tools are capable to do this either
open source or proprietary tools. For this research, three
performance testing tools was selected comprise of two open
source tools and one proprietary tool, which their names are not
mentioned and only represented as Tool A, Tool B and Tool C,
respectively. However, for the purpose of understanding the
tools, only their characteristics or features are explained below:

A. Tool A
Tool A is an open source tool that is purely developed on
Java platform. It sits as desktop based tool for performance
testing. It is designed to serve functional, load and stress Figure 1. Test setup for comparison of performance testing tools
testing, in which it is extensible to write own test to suit the
scenario. Tool A can simulate heavy load on the application, As presented in the diagram, strict rule for conducting the
server and even the network. Besides able to give instant visual test is to make sure both client and server machines must be
feedback, this tool is also capable in performing load and stress rebooted after completing each cycle of test for each tool used,
testing via distributed approach. It supports protocols such as so that the machine resources such as RAM and CPU are at
HTTP, JMS, JDBC, FTP, SOAP as well as LDAP. Other their fresh states. Another important element for this test is to
features of this tool are it can be used across platforms, use static webpage in order to maintain the page size for correct
supports full multithreading framework besides allows caching analysis of the results obtained.
and offline analysis with replaying of test results.
While the test was running for a particular tool, the services
for the other remaining tools as well as other unnecessary
B. Tool B
services were stopped or disabled. As for the server, it only
Tool B is an open source load testing tool that is developed hosted static HTML pages with no cache and cookies. Running
using C++ language. It can perform heavy load tests with processes on this machine were kept to a minimum since
performance measurements using scripted HTTP and HTTPS. unnecessary processes and server-side logging were disabled.
This feature-rich GUI-based web server benchmarking tool can Load testing was performed from 1 user access and gradually
only runs on Windows-based platform. Its performance scripts increased to 100, 200, 300, 400 and 500 concurrent users with
are recorded using own proprietary language that is simple and zero ramped-up time.
can support custom functions, variable scopes, and random or
sequential lists The hardware specifications for both machines are outlined
as below:
C. Tool C CPU/processor : Intel Pentium D 3.4 GHz
Tool C is a proprietary tool and is one of the established RAM/memory : 2 GB
performance testing tools in the market. It is built on Eclipse Hard disk storage : 80 GB
and Java. It offers automated performance testing for web and Network Card : Integrated 10/100/1000 Ethernet
served based application so that the scalability of those
applications can be validated. This tool can be used across As for the software specifications, the details are as
platforms such as Windows, UNIX and Linux. System follows:
performance bottlenecks and their root cause can be identified
by using this tool. Besides the capability to create code free Client machine
tests, this tool is also able to automate test data variation and Operating system : Windows XP SP2
enables insertion of custom Java code for flexible test Java JDK : JDK 1.6.0 update 21
customization. In terms of operating systems, it supports Tool : Tool A (open source); Tool B
operating systems such as Windows, Linux, and z/OS. (open source); Tool C (proprietary)
 Server machine C. Result of Tool C
Operating system : Windows Server 2003 Enterprise
Edition SP1 The result of load test for Tool C is presented below in
Java JDK : JDK 1.6.0 update 21 Table III:
Web server : Internet Information Services 6
TABLE III. PERFORMANCE TEST RESULT OF TOOL C
HTML page size : 65.8 KB (Page: 7 KB; Image 1:
25.2 KB; Image 2: 33.6 KB)

V. FINDINGS AND DISCUSSION

The test began by simulating one user accessing the
HTML page at the server machine using Tool A. At the same
time, the CPU, memory and disk I/O utilization for both
client and server machines were recorded. This was repeated
another two times for the same virtual user and also for other
virtual users set until the number reached 500 virtual users.
The same kinds of load test were performed using Tool B
and Tool C. Response time, average CPU utilization,
memory utilization and disk I/O utilization for each set of
virtual users of every test are the metrics recorded and
collected for further analysis.

A. Result of Tool A D. Comparison of Performance Test Results between Tools

The result of load test for Tool A is presented in Table I Response time results obtained from the three tools were
below: grouped and plotted into the graph. Three graphs were produced
TABLE I. PERFORMANCE TEST RESULT OF TOOL A
to represent results for each round of test. The graphs are as
below in Figure 2, Figure 3 and Figure 4:

B. Result of Tool B
The result of load test for Tool B is presented below in Figure 2. Performance (load) test result for round 1
Table II:

TABLE II. PERFORMANCE TEST RESULT OF TOOL B

Figure 3. Performance (load) test result for round 2

 identical which is based on last byte sent and last byte
received. However, Tool C for an example introduces Inter-
Request-Delay, where some requests may have delays
associated to them. Tool A does not automatically
implement a delay and it is up to the user to manually
configure it based on their preferences. To note, Inter-
Request-Delay or wasted time is a separate entity from Think
Time .All these 3 tools are identical when excluding Think
Time from total response time.

Architecture for each tool differs greatly. For an example,

Tool A and C developed by using Java and they require JVM
to run , the value setting for Java Heap Size plays a role here
to generate the best user load without putting extra burden to
the client and by default after the installation, both tools have
their own settings. As for Tool B, its architecture relies on web
Figure 4. Performance (load) test result for round 3
relay daemon facility allowing CORBA-based communication
to be transmitted between machines during executing the
From the graphs plotted, it is clear that different performance test.
performance testing tools produced different response time
results. The experiment of running the test using different
tools via load test for the same HTML web page has achieved VI. CONCLUSION AND RECOMMENDATION
the objective on proving that when using different
performance test tools, the response times obtained are not This paper has shown that different performance testing tools
similar or even close to each other, especially as the give a different response time and leading us in future for an
number of concurrent users increases. At the same time, it opportunity to quantify and prove further for each of the
also triggers another question “Why do different performance potential reasons contribute to that. The next critical research is
testing tools give different response times when testing the around capturing and simulating the load by each tools, where
same web page?”. Subsequent sub-section tries to provide we may need to continue analyzing each of the HTTP request
possible reasons on this scenario. and response through tool available in the market for an example
Wireshark. With this, we can fully understand at the packet level
what are being transferred and received at each tools and why
E. Potential Reasons for Response Time Differences between they are being included or excluded. Currently, there is no tool
Performance Testing Tools able to tell us if our application is fast enough in term of user
Some fundamental understandings are already in place to experience in a reality or in other words, response time reported
explain potential reasons behind the differences of response by testing tools versus one done by naked eye. Hence it is crucial
time between performance testing tools. Among others for performance testers to understand that there is no tool able to
are capturing and simulating the load used for the automate and tell us the full picture of the application's
performance test, the method of calculating metrics performance going to be in a real world. It is back to human
gathered by each tool, the language to develop the tools, the brain to analyze the information given and performance testing
architecture of the respective tools. Subsequent tools are just one of the tool can be used to achieve that.
paragraphs try t o compare potential reasons between Tool
A and Tool C.
REFERENCES
Capturing and simulating the load plays the biggest role
so far that contribute to the variation of response time. In
[1] VCAA (2005). Stress, Load, Volume, Performance, Benchmark,
comparison between Tool A and C reveals there are several Baseline Testing Tool Evaluation and Comparison
extra items being recorded and simulated when generating [Online].Available:http://www.vcaa.com/tools/
user loads. Tool C by default uses Internet Explorer browser loadtesttoolevaluationchart-023.pdf
when recording and it's observed ASHX files recorded in [2] JDS Australia (2009), A comparison of Open Source Load Testing
the list as an additional items compared to Tool A. ASHX tools [Online]. Available:http://www.jds.net.au/tech-tips/load- testing-
is a web handler file . Tool A however does not do the tool-comparison.
same. Apart from that, JavaScript and CSS files seem have [3] Barber, S. (2006). Choosing Performance Testing [Online].
higher response time in Tool C compared to Tool A Testingreflections.com. Available: http://www.testingreflections.com/
node/view/3939.
around 18 percent. It is observed the remaining file types
[4] Vinod, P. (2008). Open Source & Commercial Performance Testing
consists of images (GIF, JPG) and HTML do have a small Tools [Online]. Accenture. Available: http://www.scribd.com/
variation between 1 to 4 percent. doc/27765939/Performance-Test-Tools-Comparison.
The method for calculating metrics gathered by each tool [5] Nageswaran, S. (2005). Performance Tools Comparison [Online].
Testingreflections.com. Available:http://www.testingreflections.com/
contributes to the variation of the response time. The node/view/1756.
fundamental formula to calculate for Tool A and Tool C is
[6] IBM Support Portal (2011). RPT HTTP Response Time [9] Subraya, B. M. (2006). Selection of Tools - in Test Preparation Phase
Breakdown [Online]. Available:http://www-01.ibm.com/support/ II:Test Design. IGI Global. United States of America.
docview.wss?uid=swg21391178. [10] Barber, S. (2007). How Fast Does a Website Need To Be. PerfTestPlus
[7] Molyneaux, I. (2009). Choosing the Right Performance Testing Tools - Inc. Florida, United States of America.
in The Art of Application Performance Testing. O’Reilly Media. United [11] Savoia, A. (2001). Web Load Test Planning: Predicting How Your Web
States of America. Site Will Respond to Stress. Google Inc. Mountain View, United States
[8] Microsoft Corporation. (2007). Quantifying End-User Response Time of America.
Goals - in Performance Testing Guidance for Web Applications. [12] Barber, S. (2007). Beyond Performance Testing., Rational Developer
Microsoft Press. United States of America. Network. PerfTestPlus Inc. Florida, United States of America.

View publication stats