SaaS Testing Overview

Ram Garg CEO

Software Performance Assurance (SPA)

Software Performance Assurance Confidential

Outline

SaaS Introduction

What is it? Why SaaS?

SaaS Challenges

Key Challenges

SaaS Testing

How to approach SaaS testing? How SaaS Testing is different? SaaS Test Tools

Software Performance Assurance Confidential

What is SaaS?

SaaS Introduction

Software Performance Assurance Confidential

SaaS: What is it?

Software-as-a-Service (SaaS) is the application layer of the cloud computing model.
Application

Software as a Service (SaaS)

Platform

Platform as a Service (PaaS)

Infrastructure

Infrastructure as a Service (IaaS)

Software Performance Assurance Confidential

SaaS Definition
Software as a service (SaaS), sometimes referred to as "software on demand," is software that is 100% deployed over the internet . With SaaS, a provider licenses an application to customers as a service on demand, through a subscription or a "pay-as-you-go" model.
Wikipedia definition

Software Performance Assurance Confidential

Architecture: Three Key Differentiators

Multi-tenant Efficient

Scalable

Configurable

Key Architecture Differentiators

Software Performance Assurance Confidential

Why SaaS?

SaaS Introduction

Software Performance Assurance Confidential

The Software Business Model Changes

non -SaaS
Software Ownership

SaaS
Provider

Customer

Infrastructure management Responsibility

Customer

Provider

Installation

Client - Server

Central

Upgrades

Larger, less frequent

Shorter, frequent

Platform

Multi-version

Single-version

Software Performance Assurance Confidential

Changing the Business Model cont

non -SaaS
Configurability Requirement

SaaS
High, Critical

Low

Multi-tenancy

No

Yes

Backward compatibility

No

Yes

Maintenance

Customer

Provider

Software Performance Assurance Confidential

Changing the Business Model cont

non -SaaS SaaS

Cost, Allocation

High, CapEx Close the deal, License Revenue Initial sale + maintenance

Low, OpEx Provide value, Lack of Churn Ongoing Subscription

Sales focus / Success

Profits

Software Performance Assurance Confidential

Ecosystem

SaaS (Software)
PaaS (Platform) IaaS
(Infrastructure)

Starting point to every service within cloud computing ecosystem Highest on the maturity level and growth curve amongst XaaS Highest revenue generating service within cloud model for next several years

SaaS Catalysts
Intrigue of access to unlimited Computing and storage resources, made available and paid for only as needed (Pay as you go) Decreasing cost of bandwidth Customer frustration with traditional software buying cycle (license, maintenance contract, upgrades) Perceived increase in control over the vendor relationship Early SaaS success stories Salesforce.com, WebEx, etc. Demonstrate broader value proposition You focus on your application and not on infrastructure

Software Performance Assurance Confidential

SaaS Early Adoption

SaaS Introduction

Software Performance Assurance Confidential

SaaS Adoption: Verticals

AMI-Partners report "Software-as-Service Hot Spots : Education, Healthcare, and Financial"

Adoption rate of SaaS in SMB is higher than large enterprises

In-Stat research on vertical market adoption shows the healthcare market is currently most satisfied with hosted applications, with education identified as an important potential target market also.

All research points out that SaaS has been adopted heavily in :

HR applications (ERP,
Salesforce automation, eRecruiting)

Online Collaboration (Web

conferencing)

CRM Healthcare

Software Performance Assurance Confidential

SaaS Adoption Factors

(source: Forrester) Availability & Reliability (71%) Fast Deployment (45%)

Reduced dependencies on internal IT (24%)

Flexible consumption (35%)

Simplified pricing model (32%)

Software Performance Assurance Confidential

The SaaS Market is and will Continue to Grow

Software Performance Assurance Confidential

Today and tomorrow

Software Performance Assurance Confidential

Today and tomorrow

Software Performance Assurance Confidential

Today and tomorrow

Software Performance Assurance Confidential

Challenges

SaaS Challenges

Software Performance Assurance Confidential

Primary Concerns with the SaaS Model

Security concerns Concerns over compliance with standards Concerns over integration with legacy applications Concerns over reliability and robustness of service Not comfortable with putting mission-critical application on shared infrastructure and have access over internet SaaS can not be configured to meet my corporations all specific needs

Software Performance Assurance Confidential

Numerous Challenges

Security

Performance

Availability

SLA

Data Integrity

Privacy

Compliance with Standards

Regulatory

Integration

Multi-tenancy

Lack of confidence

Software Performance Assurance Confidential

SaaS Testing

Software Performance Assurance Confidential

Why SaaS Testing?

SaaS Testing

Software Performance Assurance Confidential

The average enterprise IT department devotes up to 50 percent of its entire technology infrastructure to development and test, with up to 90 percent of the available test infrastructure remaining idle. IBM Research

Why SaaS Testing?

A delay in website page loading by even one second can impact conversions by 7% and customer satisfaction by 16%. Aberdeen Group

Cloud Testing
Those using cloud facilities to run applications, 49.8% said they were doing so experimentally or for prototyping; 28.6% for noncritical business applications and 21.6% for business critical applications Evans Data Survey

The average online business had 8 days of downtime a year due to performance issues leading to lost revenues Aberdeen Group

Software Performance Assurance Confidential

Trend leads to trend

Overall growth in Cloud Computing (investment, revenue, ROI, market cap, etc.) is leading to a marked increase in adoption of SaaS Testing

What is SaaS Testing?

SaaS Testing

Software Performance Assurance Confidential

Classifying SaaS Testing

Testing The Cloud SaaS Testing applications which are in cloud or migrating to cloud.

Testing the SaaS

Test the performance, reliability & security of applications

Cloud / SaaS Testing

Testing SaaS Using Cloud

Testing Testing SaaS Using Using Cloud Cloud Testing applications using the cloud-based hardware, software, infrastructure and computing resources. Tests performed like performance, load & stress, security and compatibility

Software Performance Assurance Confidential

How SaaS Testing is Different

Easier
No client-server installation
No multiplatform support No multiple version upgrades No backward compatibility

Challenges
Frequent live upgrades Too many pages Multiple privileges

Immature SaaS-specific testing tools

Access partitions critical

Software Performance Assurance Confidential

Approach

SaaS Testing

Software Performance Assurance Confidential

SaaS Testing Preparation

Assess Testing Needs

Understand Usage Patterns

Prepare Test Plan and Test Suite

Prepare Test Data

Prepare Test Environment

Software Performance Assurance Confidential

SaaS Testing Areas

Infrastructure Testing Network testing Application testing

Secured SaaS Application

Ensure SaaS testing does not damage the shared infrastructure

Software Performance Assurance Confidential

SaaS Functional Testing

Perform Manual and Exploratory test (similar to traditional applications) Check Browser compatibility Perform regression
SaaS regression must cover significant more functionality as single defect will impact all the customers

Functional and regression test automation.

This is important to comprehensive testing cost-effectively as above point indicates that, in case of SaaS, regression should cover more functionality.

Test SaaS on cloud-based staging environment

Testing on in-house environment and then directly deploying on cloudbased production environment may result in chaos

Test in a multi-user environment where different users are performing different activities.
Software Performance Assurance Confidential

SaaS Security Testing

Test application security
Cross Site scripting, SQL Injection, HTTP header injection

Test network security

Secured communication, Backup and storage policies, Controlled access to sites and servers

Test for possible attacks Test roles and privileges in multi-tenant environment
Many roles and privilege levels

Test all the upgrades to detect broken security, Ensure security patches are not missed Test data security / integrity among multiple tenants
Software Performance Assurance Confidential

SaaS Security Testing cont

Identify scenarios where application might be vulnerable Test PCI compliance Test cookie setting, SSL configuration, information leak, etc. Log errors, warnings and suspicious requests

Tests tampering with the licensing mechanism; gaining unauthorized access to application data, editing registry entities or configuration files

Software Performance Assurance Confidential

SaaS Performance Testing

Test performance extensively first time and for each upgrade
SaaS is revenue generating and business impacting applications. Single performance issue will impact all your customers Test performance extensively as SaaS need to handle large number of users and we dont have luxury of re-booting or going down every once in a while

Test load characteristics, ramp up rate, protocols, load origination, Establish ability to test performance of your SaaS isolated fashion so that network related issues can be identified and passed to network infrastructure and internet providers

Test key metrics and SLAs

Test performance for mobile users

Software Performance Assurance Confidential

In-Service Upgrade Testing

Ability to upgrade (full release, patches) system with zeroto-minimum down time is essential

Global users make this complicated as somewhere in the world someone might be accessing your application
Thoroughly test in staging environment before perform upgrade ass this is one of the most complex features

Remote Accessibility and Usage

Global access with acceptable response time 24 hours availability

Software Performance Assurance Confidential

Disaster Recovery
Shutdown to bring up application quickly: server configuration; install and deploy software ; restore data Roll back to last known stable data

Localization
Make sure it supports international characters as SaaS application can be accessed from anywhere in the world Optimize by supporting target market language

Software Performance Assurance Confidential

Test Tools

SaaS Testing

Software Performance Assurance Confidential

Traditional and Cloud-based Test Tools

Traditional Test Tools
QTP, WinRunner Rational Robot Rational Functional Tester SilkTest Selenium, STAFF, Fitness, Watir

Cloud-based Test Tools

Functional

Performance

LoadRunner, Rational Robot, Rational Performance Tester Silk Performer Jmeter, Grinder, WebLoad, OpenSTA

Gomez PushToTest, SOASTA CloudTesting uTest Many others

Software Performance Assurance Confidential

Traditional Vs. Cloud-based Tools

Testing SaaS Using Traditional Tools
Many providers who provide SaaS testing using traditional testing tools Many open source matured tools Many options for professional services by independent service providers Professional service options allow cheaper cost

Testing SaaS Using Cloud-based Tools

A few providers who provide SaaS testing using their own cloud-based test tools Limited open source test tools options Captive professional service by test tools provider which is not even core for their business Captive profession service is sold at very high rate

Software Performance Assurance Confidential

Conclusion

Software Performance Assurance Confidential

Conclusion
Customers buy each month keep them happy
SaaS is only a delivering model, customers will not compromise on functionality, stability, security, and performance. They will not switch to SaaS it if not better than traditional systems.

Save time on support, maintenance, and upgrade, etc.

You need to pay more attention to stuff like accessibility, security, deployments. The way you earn reputation Dont buy cloud-based test solution just because you are testing SaaS. In most cases, traditional test solutions work just fine and save your 30% 50% cost.

Software Performance Assurance Confidential