API Testing
API Testing
API Testing
Asha KR
Asha_kr@mcafee.com
Shwetha D J
Shwetha_naik@mcafee.com
Abstract
A right testing strategy for Application Programming Interface (API) is crucial for developing a
successful product. However, many businesses fail to see a complete picture during project management
phase. Changes in product and project requirements are inevitable, which can potentially disrupt the
timelines or quality of the product. Eventually, more time and resources are required to accommodate the
change requests. It is challenging to deliver a good quality product to the end users in such situations.
Testing of API after it integrated with the UI is not an easy task especially when we deliver a quality
software as a service to our customers and partners.
API driven testing is a testing framework that uses a programming interface and the application to
validate the behavior under test. Typically API driven testing overcomes application user interface
altogether. API driven testing is also being widely used by software testers as it serves additional benefits
as compared to other testing strategies. Programmers or testers write scripts using a programming or
scripting language that calls interface exposed by the application under test. These interfaces are custom
built or commonly available interfaces like COM, HTTP. Present development world changes every day
with new technologies and this is directly proportional to UI changes. In this era we cannot rely only on UI
testing for functional verification. The test scripts created are executed using an automation framework to
compare test results with expected behavior of the API. The journey was more challenging while bringing
up a startup project to a stabilized API automation framework with fair number of test cases automated in
it. There were many initiatives and ideas put forth for converting manual testing to stabilized automation
testing.
Outcome of API automation is increased test coverage by 100% with quality deliverables. 20% of early
detection of defects and scope for manual testers increased to become Test Automation Engineers.
There are nightly builds running with automation for Build Validation and Functional Validation, which has
helped testers pitching in early defect analyses with in less time.
Biography
Asha KR is a Senior Software QA Engineer at Intel Security, currently working in the Intel Security India
Center in Bangalore. She has been working for the past 5+ years in different QA roles SaaS products.
Asha holds Bachelor of Engineering in E&C from VTU, Karnataka, India.
Shwetha D J is a Software Development Engineer for Test, currently working in the Intel Security India
Center in Bangalore. She has been working for the past 1+ years on SaaS products. Shwetha holds
Bachelor of Engineering in E&C from VTU, Karnataka, India.
Web services extend the World Wide Web infrastructure to facilitate a software to connect to other
software applications. Web services (sometimes called as application services) are services usually
including a combination of the software program and data, but possibly including human resources as
well that are made available from a business's Web server for Web users or other Web-connected
programs. Applications access Web services via Web protocols and data formats such as HTTP, XML,
and SOAP. Web services combine the best aspects of component-based development and the Web.
They are a cornerstone of the Microsoft .NET programming model.
In this era of agile development, we are moving to a model where UI is ever changing and the
consumption of web services is taking the center stage and hence giving this layer the attention it truly
deserves by shifting the onus from UI testing to Web Services based validation.
API stands for Application Programming Interface, which specifies how one component should interact
with the other. It consists of a set of routines, protocols and tools for building the software applications. An
API is similar to user interface, the difference is that, instead of a user-friendly collection of windows,
dialog boxes, buttons, and menus, the API consists of a set of direct software links, or calls, to lower-level
functions and operations. APIs can look formidable, but theyre designed to be accessible to trained,
knowledgeable programmers.
API testing in many respects is like testing software at the user-interface level, however instead of testing
by means of standard user inputs and outputs, the testers use software to send calls to the API, get
output, and log the systems response. General steps involved while performing API testing are
mentioned below:
Details of API information is found in Specification Document which lists the signatures of each
API function (the input parameters, the function or method name, and the return type)
Identify the software to be used for API testing
Add the web service call to be tested
API call will have request and response parameters
Prepare the inputs for the request
Invoke the method with all provided inputs
Analyze the output response
The above diagram Fig (1.2) represents architecture of where the Web services, API come into the big
picture of the solution under test. Our team is responsible for development of Web Services. There is a
Web Services layer that hosts all the web services. External tools block is one in which the methods
are present and they can be accessed from Web Services layer. Web Service layer consists of two sub
layers:
1. Data Service Layer: This layer consists of all Data related APIs which provide the business logic for
the application. This layer also in turn communicates to the data source or the data layer.
2. Management Service Layer: This layer consists of Management related APIs which provide the
management solutions to the applications that consumes the corresponding services in this layer.
User requests reaches management service layer, which communicates to data service layer, in turn it
fetches data from data source. Each layer has multiple web services and each web service has multiple
methods called APIs. Different systems access these APIs for their business purpose. Web portal is the
UI layer which consumes all the web services and presents the relevant data to the end user. Partners
can make use of external tools to directly invoke the APIs.
2. Problem Statement
Testing of an application which integrates multiple systems is not an easy task. When we promised to
deliver flawless product to customer, detailed testing of the individual components, systems and
subsystems that integrates to main product is very crucial. If we are planning to test the end to end
scenarios that spans across all the systems and subsystems in consideration it may lead to product which
is not impeccable. If some functionality fails at UI then analysis of failure at different system is more time
consuming. Having API testing one step before the UI can help in managing different systems without a
glitch. However when we start to test multiple different systems as one, lot of manual effort and time is
required and also maintaining the product will eventually become tiresome.
Main challenges in API testing are Parameter Combination, Parameter Selection, and Call
Sequencing
There is no GUI available to test the API which makes it difficult to provide input values
Validating and Verifying the output in different system is not very easy for testers
Parameters selection and categorization required to be known to the testers
Exception handling function needs to be tested
Coding knowledge is necessary for testers
3. Approach
Considering the problems that arise during API testing, choosing an approach for an effective and
efficient API testing depends on a lot of things. Will the API be a public API that will be consumed by
some external users/systems, or is it a part of a larger product's infrastructure? API is a general term that
is sometimes used to describe anything from a COM interface, to a DLL or JAR you can reference, to a
REST web service. Different approaches can be applied to testing these different categories.
Below flowchart represents the steps to consider while performing API testing.
When we understand and realize the importance of Web Services Automation, the next challenge is
to select the right Test approach and Test Type before jumping into automation design and
implementation.
It is important to answer the basic question Why do we want to do Web Services Testing? and
that will help us to decide the right kind of Test we should choose to automate them. Lets begin
with few possible answers
Once the objective of the test is clear, we need to look at the technologies that can make a difference
to the way we test the Web Services.
Manual API testing is one where in which testers needs to give inputs and invoke the method for
response, analyze the results and report them.
Inputs for AuthenticateSession method are: SessionId, Username and Password. Output for this
call will be Success or Fail [With fail reason as message]
There are many steps involved in execution of API methods manually, which also consumes lot of
effort and time. Identifying the tools to suit the testing is also one of the challenge. There are many
tools available for executing manual testing such as
WCF is the tool which comes with Microsoft .net framework and user friendly. Soap UI is the open
source cross platform functional testing solution with an easy to use graphical interface. Soap UI
allows easy and rapid creation and execution of test cases.
The Fig (3.2.1) shows a WCF client window, where left pane represents the Web service
requests and the API methods included. Right top part represents the request with input
parameters and down part is the response of the API invoked.
3.3.1.1 Advantages
The Service Host instance is not shut down gracefully thus any pending requests
are aborted when the WCF Service Host is closed.
Cannot customize Service Host initialization and it is common to programmatically
initialize fixed behaviors and provide exception handling.
The WCF Test Client does not provide a way to save and reload inputs to service
operations thus must retype those inputs each time you run the test client.
The user interface for supplying test values is automatically generated and
cumbersome to use.
3.3.2 SOAP UI
SOAP stands for Simple Object Access Protocol. A better way to communicate between
applications is over HTTP, because HTTP is supported by all Internet browsers and servers.
SOAP was created to accomplish this. SOAP provides a way to communicate between
applications running on different operating systems, with different technologies and
programming languages.
3.3.2.1 Advantages
Automation approach for API Testing: When we need to test a feature multiple times because of
frequent modification in the dependent subsystems, then automation is the best of testing.
Regression testing is made easy with automation. Automation can be done with some tools or if
someone is very good in programming can do API automation without any tool. This is explained in
below sections
3.4.1 Soap UI
3.4.2 Microsoft Web Test
3.4.3 Using Programming Language
3.4.1 SOAP UI
Automation to execute test suites developed using Soap UI, without using the user interface
of SoapUI, and instead trigger it from some kind of automation build tool. There are many
different tools that could trigger the execution. Most of them have in common that they are
good at triggering anything from a command line prompt. This could be a bat script in
Windows, a shell script in UNIX, or it could be Maven project in a Java build environment.
Command line: Command line tools are good if you want to be able to execute something
from a script that you have created yourself. SoapUI has support for executing from a
command line. Execute it like this:
Testrunner project_file
Where project_file is a SoapUI project file that could look like this c:\ projects\my-soapui-
project.xml if you are on Windows. If you want help with running the exact same execution
from a command line that you executed from the TestRunner, then try this:
Automate the execution after creating your SoapUI functional tests with test suites and
connecting SoapUI tests to your build tool, test suite will get executed, as soon as a change
has occurred.
We will save a lot of time and be able to focus on the real problem, testing, instead
of the problem of creating your test tool.
Delivery of high quality APIs is assured, compared to when testing the services with
manual effort.
3.4.1.2 Disadvantages
Web tests allow simulating a user performing a set of operations typically a defined use
case on ASP.NET Web application, and validating the responses to see if the application is
working as expected.
Web test is a better when we have relatively simpler web services where our objective is to
test simpler / straightforward functionality and primary focus is to test their performance
down the line. We are working on legacy app that is based on ASMX kind of services which
have a better support in Web Test.
Steps to write Web Test: Web test can be created from Microsoft visual studio.
If we dont want add web test by record and playback, we can add service requests manually and
write test cases.
Performance testing: We can use test results obtained by running webtest for performance
testing. This can be done by analyzing metrics on the server and on the client, explore the HTTP
traffic requests and performance counter data. Double click the test name in test result pane to
see the details.
3.4.2.1 Advantages
Creating web tests without writing code using the Web Performance Test Recorder
is good for beginners.
We can edit recorded tests to tailor to specific needs.
Simplify JavaScript interaction by automatically promoting dynamic parameters to
editable web test parameters
Load tests can be created by aggregation of existing web tests
3.4.2.2 Disadvantages
Not much validation logic can be put into the tests: Validation can be put only on
validation rules available in webtest.
Maintenance is difficult: A small change in contract, would impact in changing all
affected test cases
Data Cleanup was not done properly
High Volume tests were not at all executed in the test suite
Limitations of Microsoft webtest and dependency of third party tool like, SoapUI led us to
write an automation framework on Microsoft .Net platform by using programming language
C#.
Framework: When we talk about test framework, there is lot more things comes in picture.
Before starting to develop framework we had to think about below aspects
Choosing a programing language is the key thing to start developing test framework.
Because all the team members should be educated in that language or proper training should
be given to educate them. If there is any tool available for automation it can be adopted. With
all these boundaries C# can be used as programming language to start with framework. A
framework should be an integrated system which integrates different libraries, test data
sources, object details, reusable components and different utility modules. Once framework
is developed integrate the framework with testing environment where we can run all the API
tests.
API testing is often compared with unit testing because the purpose served by both API
testing and unit testing is same in most of the scenarios. Unit testing is covers some block of
Excerpt from PNSQC 2015 Proceedings PNSQC.ORG
Copies may not be made or distributed for commercial use Page 13
application code and the test cases can only verify its behavior with respect to boundary,
valid input data and incorrect input values. Whereas API testing works for the same kind of
testing but API testing can be taken to test end to end flow where we have to testing APIs
which internally call one or more APIs to get the response. Unit testing has limited in scope
whereas API testing has broader in scope as it can run for end to end testing.
Test Case: Any test case written for API testing will have below 3As (as shown in Fig 3.4.3b
below) as mandatory steps:
Arrange: Arrange all requirements. Create test data required for API request.
Act: Act on the API. With the test data created in step 1 invoke any API or invoke
required APIs and get the response.
Assert: Assert on the response obtained from the step 2 Validate the response from
the API request with the local or remote data source.
Writing appropriate helpers, utility libraries, reusable methods one can design a good
automation framework which can be used by different systems as platform of writing test
cases of their own.
When we are dealing with the APIs more than one in one test case we have to make sure
first API is not blocking the invocation of the second or rest APIs. When we call multiple
APIs in one call or in multiple call it is good practice to have meaningful error messages
at all steps so that tester can easily analyze error message or result message and can
log defect according to that.
C# framework for End to end testing: API testing framework can be used for end to end
testing before going to UI. To do this end to end testing user should be aware of
sequence of APIs to be called and amount of time wait required between each API. For
end to end Arrange part should be kept as simple as much so that it should not lead to
overburden for any person who joins later.
Test Result: Reporting test execution results is very important part of testing, whenever
test execution cycle is complete, tester should make a complete test results report which
includes the Test Pass/Fail status of the test cycle. If manual testing is done then the test
pass/fail result should be captured in an excel sheet and if automation testing is done
Excerpt from PNSQC 2015 Proceedings PNSQC.ORG
Copies may not be made or distributed for commercial use Page 15
using automation tool then the HTML or XML reports should be provided to stakeholders
as test deliverable.
By writing some scripts for running all the automated test cases at some cycle and
generate proper test result. Test result of the run should include,
3.4.3.1 Advantages
3.4.3.2 Disadvantages
C# mainly depends on .NET framework. Any library which is not found in .net will be
difficult to implement
C# programs are largely for Microsoft windows environment but C# doesnt come up
with open source technologies and operating systems like Linux.
4. Result
Moving from manual testing to automation framework for API testing, increased lot of scope in testing.
The journey from manual to automation and stabilization of code gave good results in managing the web
services.
100%
80%
60%
Test Coverage
40%
20%
0%
BVT FVT End to End
Test Cases
5. Summary
API consists of set of classes / functions / procedures which represent the business logic layer. If API is
not tested properly, it may cause problems not only the API application but also in the calling application.
5.1 Advantages
More effort into API testing leads to much healthier final product, ensuring the data access
through the API only.
API testing simplifies security and compliance testing and thereby certification, since there is only
one interface.
Ensuring that all the required business rules are being enforced at the API tier allows time for
much more complete user-experience tests once the UI is released, and not having to
concentrate on testing every single business rule and path through the application near the end of
the project.
Ensuring that the API offers complete functionality allows for easy future expansion of the
application as new business needs arise.
As API testing is structured way to testing, it makes automation more feasible.
5.2 Disadvantages
No encryption - All requests and responses are visible to anyone between the requesting server
and the API server.
Increasing Tooling needs: API testing can be done only using different tools available. Cannot be
tested without the tool like in GUI.
Straightforward reason for failing API is not known, until and unless debugged into the code,
unlike in GUI where it gives user-friendly errors.
Excerpt from PNSQC 2015 Proceedings PNSQC.ORG
Copies may not be made or distributed for commercial use Page 18
Though automation is 100% possible, still there remains a question whether automating 100% test
cases is really profitable? Because of few reasons like priority of test cases, frequency of those test
cases run and effort calculated in automating them.
5.3 Future
With all the merits and de-merits kept in mind, we have strategy planned to automate End to End
tests and Integration tests.
End to End Testing: Plan to extend all BVT scenarios to cover end to end testing. This is one
step ahead from normal API testing where we just test one functionality. End To End testing
make hold of the scenario which user performs from the UI. Calling different APIs in single
method is little exasperating because failing of some API will mark whole test case to fail but with
good coding style and capturing exact result at all points we can develop a robust test suite.
Integration Testing: Integration of different systems in testing is little difficult. We have a
roadmap of covering different systems from the BVT perspective. Capturing behavior of each
system for one single functionality and automation of this will lead us to have non-defective
system.
James D. McCaffrey. 2012. .NET Test Automation Recipes: A Problem-Solution Approach. California.
The Experts Voice
Web Sites:
Marc vant Veer. 2013. Testing the API behind a mobile app, https://www.polteq.com
Michael-churchman, 2014. API Testing: Why It Matters, and How to Do It. https://blog.udemy.com on
2014. https://blog.udemy.com/api-testing/,
Joe Colantonio, 2013, UFT What is an API test type? www.joecolantonio.com on February 19, 2013
http://www.joecolantonio.com/2013/02/19/uft-what-is-an-api-test-type/
http://sqa.stackexchange.com/questions/6745/how-do-you-test-a-backend-api
http://sqa.fyicenter.com/FAQ/Testing-Techniques/What_is_API_Testing_.html
http://www.encodedna.com/wcf/tutorial/advantages-of-wcf.htm
http://www.soapui.org/
http://www.codeproject.com/
http://en.wikipedia.org/
http://www.guru99.com/api-testing.html
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: