COMPASS EXPRESS –

PHASE 1
SYSTEM TEST
PLAN/STRATEGY DOCUMENT

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 1 of 26

 Name Org/Group Role Date
Prepared by: Sameer Aggarwal TV Systems Offshore QA Lead 09/22/2015
Reviewed by: Sanjeev Rohilla Birlasoft Delivery Manager 09/30/2015

Reviewed by: Manisha Golatkar TV Systems NBCU QA Lead

Approved by: Monica Malhotra TV Systems NBCU Program Manager

All document control issues pertaining to this document or for information about its access and
location please contact the following resource(s):

CONTACT INFORMATION

Manisha Golatkar
Name:
NBC QA Lead
Role:
TV Systems- QA
Org/Group:
212-887-5934
Phone:
manisha.golatkar@nbcuni.com
E-mail

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 2 of 26

REVISION HISTORY

Version # Date Section/ Page # Description of Changes Reviewed by Author

Sanjeev Sameer
1.0 09/22/2015 Initial Draft Version
Rohilla Aggarwal
Changes made in the below
sections:
Ritesh Sameer
2.0 10/08/2015 - Test Scope
Srivastav Aggarwal
- Assumption and Risk
- QA Approach & Strategy

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 3 of 26

 COMPASS Express – Phase 1 - System Test Plan

Table of Contents

1. Business Objectives..............................................................................................................................7

2. Test Objectives.....................................................................................................................................7

3. Test Scope.............................................................................................................................................7

A. Covered during testing................................................................................................8

B. Not covered during testing.........................................................................................8

C. Assumptions and Risks.................................................................................................9

4. Strategy & QA Schedule....................................................................................................................10

A. QA Approach/Strategy...............................................................................................10

B. Test Schedule..............................................................................................................13

5. Test Approach.....................................................................................................................................15

A. Test Design..................................................................................................................15

B. Test Automation Strategy.........................................................................................18

C. Test Preparation.........................................................................................................18

6. Release Information...........................................................................................................................18

A. Cycle 1 (Sprint plan yet to be finalized by project team)...................................18

7. Test Environment...............................................................................................................................18

A. Testing Front End.......................................................................................................18

B. Testing Database........................................................................................................19

C. Platform Compatibility..............................................................................................19

8. Test Management Process.................................................................................................................19

A. Test Management - Phase wise guideline...............................................................19

B. Test Methodologies....................................................................................................20
Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 4 of 26

9. Resources.............................................................................................................................................23

A. Roles & Responsibilities.............................................................................................23

B. Contact Information...................................................................................................24

C. Hardware.....................................................................................................................24

D. Software.......................................................................................................................24

E. Facilities......................................................................................................................25

F. Data..............................................................................................................................25

10. Success Factors...................................................................................................................................25

A. Test Exit Deliverables................................................................................................25

B. Control Documentation.............................................................................................25

C. Deployment Decision..................................................................................................25

11. Communication...................................................................................................................................25

A. Review and Status Meetings......................................................................................25

B. Defect Management...................................................................................................26

12. Appendices..........................................................................................................................................27

A. Test Artifacts..............................................................................................................27

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 5 of 26

 1. Business Objectives
Currently, network schedulers are meeting with excel documents to rearrange their schedule to get the
best ratings. Compass Express is a system that optimizes the viewing of schedules for networks and its
competitors. This product will reduce errors and potential problems caused by managing a multitude of
documents and emails. Users will also be able to see the metadata for each program and competitive
data will be automatically added from Rovi. Compass Express will have all the needed information in
one place

2. Test Objectives
The objective of Compass Express Phase 1 release will be to verify new system “Compass Express” that
optimizes the viewing of schedules for Networks and its Competitors. It will be web based application
that would be accessible on Desktop, Mac, Tablet and Mobile devices. QA team will verify the
application by performing UI, functional, integration, end to end and exploratory testing on various
platform/devices and browser combinations.

3. Test Scope
QA team would be doing Functional Testing, UI, Exploratory, Integration, Security testing (refer Test
Execution Strategy section for more details on security testing) and End to End testing. Team will
provide UAT support as needed. Testing team will be responsible for entire Test Life cycle including-
Test Planning, Preparation, Execution, closure and deliverables for functional testing of Compass
Express project.

 Scope for Compass Express Phase 1 is given below:-

o Features Screens:
- Login/Logout - Authorized/Unauthorized Access
- Navigation Bar - Settings Menu, Network Settings, Date Picker, Time Picker, Navigation
- Schedule - Grid View, List View
- Block View
- Block Detail View
- Event Tracking - Event View
- Rovi Data comparison testing will be done on sampling basis

 Coverage of Test Execution on multiple Platform/Browser and Device combinations are as

mentioned below:
Cycle 1
Phase Activity Details
Compass Express Testing - IE11 & Win 7 (100% of test cases)
Functional Testing (Cycle 1)
Cross Browser Testing Firefox & Win 7 (50% test cases)
  Chrome & Win 7 (50% test cases)
  Safari & OS X 10.9 MAC Machine (50% test cases)

Cycle 1   Firefox & OS X 10.9 MAC Machine (50% test cases)

Sprint 1 to   Chrome & OS X 10.9 MAC Machine (50% test cases)
4
Platform Testing/Device Android 4.1+ & Samsung Galaxy S5+ (50% test
Defect Re-
Testing cases)
testing
  iOS 7+ & iPad Air 2+ (50% test cases)
  iOS 7+ & iPhone 5S+ (50% test cases)
  iOS 7+ & iPad Mini 3+ (50% test cases)
  Data Sampling (Comparison only)
Defect Retesting Cycle 1 Defect Testing (Spread between Sprint 1 & Sprint 4)

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 6 of 26

 Cycle 2
Phase Activity Details
Integrated Sprint Testing 100% Test cases (Chrome & win 7)
Compass Express Firefox & Win 7 (100% test cases)
Functional Testing (Cycle
2)
  IE11 & Win 7 (100% test cases)
  Safari & OS X 10.9 MAC Machine (100% test
Cycle 2 cases)
Integrated Build   Firefox & OS X 10.9 MAC Machine (100% test
Multiple cases)
Browsers   Chrome & OS X 10.9 MAC Machine (100% test
Multiple Devices cases)
Defect Re-   Android 4.1+ & Samsung Galaxy S5+ (100% test
Testing cases)
  iOS 7+ & iPad Air 2+ (100% test cases)
  iOS 7+ & iPhone 5S+ (100% test cases)
  iOS 7+ & Pad Mini 3+ (100% test cases)
  Data Sampling (Comparison only)
Defect Retesting Cycle 2 Defect Testing

 NBCU Libraries will be used to store and update Documents including use cases, use case-user story
mappings.
http://libraries.inbcu.com/foldersIndex.do?entity_id=4635000101&sid=101&SF=1#18217959101

 All project delivery related documents will be stored in the below mentioned location on the NBCU
Shared Drive:
\\nbcuni.ge.com\shared\IT\us-nyc\Share\SolutionDelivery\QA\Testing & Verification\TV-
SYSTEMS\TV-East Coast\Affiliate & Enterprise (Research +
Compass)\SCHEDULING\COMPASS\Compass-Projects\2015\COMPASS Express

A. Covered during testing

The objective is to verify the correct implementation of functionalities implemented as a
part of the “Compass Express” project mentioned in the scope section.

B. Not covered during testing

 Search Function, used to search the schedule database
 Settings menu – Network Authorization
 Editing of the metadata within the Compass Express site
 Scrubber (used to rearrange the schedule)
 Creating Custom Events for Event Tracking
 Local Networks are out of scope (i.e. only show National Networks/Broadcast)
 Touch Screen Gestures (i.e. double tap, pinch)
 Web Service Testing & Interface testing with other systems
 Database Testing and backend Data validation
 Events Following
 Regression Testing of Compass Application is Out of Scope
 Any other module/feature not defined in the scope would be considered out of scope

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 7 of 26

 C. Assumptions and Risks
Assumptions
# Assumption Confirmed by
1 The project team shall communicate to QA team for any changes to Project Team
the existing system. The changes will also be updated in the
document updated in the NBCU Libraries and communicated
accordingly.

2 In case of testing required on an additional new functionality/Existing Project Team

of the system, the project team shall give advance notice to the QA
team to prepare test sets & environment. In general, additional
hardware will require a longer notice period than additional software.
New Estimation for the increase in scope will be done by the QA
team.

3 Application / Build to be tested shall be available to the QA team on Project Team

time

4 Development team shall provide a stable build to the QA team for test Project Team
preparation and execution.

5 Any changes on the approved functionality or instant changes Project Team

approved for the development like component or Web Services should
get updated to QA team in advance or immediately for planning the
test schedule. New Estimation for the increase in scope will be done
by the QA team, if any.

6 Any delay in the QA build deployment including delay in the defect Project Team
fixes will impact the timelines

Risk Assessment
Approach to Manage or
# Risk Probability Impact Responsibility
Mitigate
1 Changes in High Rework on Project team needs to BA/PM
requirements or test cases , be informed and work
changes not reflected impacting with them how to
in documentation Execution accommodate the
(during or close to test completion changes within that
execution beginning) date sprint/release.
2 Application Downtime Low Execution Availability issues to be Dev
delays reported to Dev team
3 Availability of testing Low Delay in Availability issues to be QA
tools reporting reported and downtimes
maintained
4 Change in scope Med/High Rework our Do the “Impact Analysis’ PM
(Overall or Sprint) Strategy and and rework on the scope
execution prior to execution.
plan

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 8 of 26

 5 Loss of test data due Low Execution Uploading of data Dev
to database cleanups delay through backend or
or any other updates creation of new test
data by QA team
6 Moving Defect to Med Delay in Following Defect Dev
Correct State and Defect lifecycle and movement
Bucket Retesting of fixed defects to
correct state and
Bucket.

4. Strategy & QA Schedule

A. QA Approach/Strategy
The test strategy consists of a series of different tests with the primary purpose to uncover the systems
limitations and measure its full capabilities. A list of the various planned tests and a brief explanation
follows below.

 Test Preparation Strategy

o QA team will get project overview & requirements understanding from the business analyst
and project team
o Based on BRD/UX documents and walkthrough with the Business Analyst and Technical Project
Manager, QA team will create Test Scenarios for these new features
o Test Scenarios will be reviewed and Signed off by Business Analyst
o QA Team will follow the approach mentioned below for Test preparation and requirements
mapping in ALM:
 Project team will create User stories taking BRD as the base.
 QA team will upload these user stories as functional requirements in ALM
under respective Sprints.
 Test Scenarios will be then uploaded and tagged against corresponding user
stories in ALM
 For each Test Scenario subsequent Test Cases will be prepared that would
cover all positive and negative flows
 For traceability Test cases created will be mapped against their
corresponding test scenarios in ALM Requirements tab.

o Functional, UI, End to End and Smoke Suite will be prepared in ALM in the Test Plan tab.
Device specific test cases which would mostly be UI test Cases would be created separately
under device specific folders.
o Test cases will be prioritized High, Medium, and Low on basis of criticality. ALM Details tab will
be used to define the priority of each test case.
o Test Cases to be reviewed and Signoff by Business Analyst and Technical Project Manager

 User stories – Acceptance Criteria

o All the user stories would be available in Jira and would be tagged to respective sprints by the
Project Team
o QA team will analyze the user stories and provide sign off based on acceptance criteria below:
- Independent - user stories should have reduced dependency
- Should be detailed and Granular
- Testable
- Should be in sync with the requirements
- Sprint wise loading should be consistent
- Entry and Exit criteria for the User Story is properly defined

 Test Execution Strategy

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 9 of 26

 o QA Team will be following an Agile Sprint methodology for the Test Case execution which will
consist of 4 sprints.
o The QA Team would be receiving the build for respective Sprints for Test execution as per the
below mentioned plan:
- Sprint 1 – 2nd November’15 IST Morning
- Sprint 2 – 30th November’15 IST Morning
- Sprint 3 – 28th December’15 IST Morning
- Sprint 4 – 18th January’16 IST Morning
- Cycle 2 – 25th January’15 IST Morning
Any delay in the deployment of the above mentioned builds in the QA environment may have
an impact on the overall Test Execution schedule.
o Smoke Testing – QA Team will execute the smoke test cases as soon as the sprint build is
deployed in the QA environment to check the overall health of the build. Functional Test
Execution will depend on the results of this smoke test as per below mentioned criteria
o In case Critical or High Priority defects are found in smoke test, QA team will reject
the build for Functional Test Execution and will report the defects in TFS. Parallel to
this, QA will also send in a mail to the Project Team informing them about the issues
found in the build.
o In case Medium or Low priority defects are found, QA team will report them in TFS. In
parallel, QA team will also begin the Functional Test Execution on the build.
o Functional Testing – Based on the results of Smoke Testing, QA team will initiate the functional
testing on the build. Approach that would be followed by the QA team for functional testing is:
o QA Team would touch base with the BA in order to set the priority of the feature from
the business perspective.
o Team will begin the Test Execution on the basis of the priority of functionality i.e.
Critical and high priority features would be tested first followed by medium & low
priority features.
o All the Test Execution will be performed by the QA team in Test Lab tab in ALM
o QA Team will capture all the required screenshots and actual results while executing
the test cases and will record the same in ALM
o UI Testing – UI testing will be performed as per requirements/Wireframes as baseline for
Compass Express application on various Platform/devices & browser combinations
o Security Testing – QA team will perform security testing to the extent of validating the access
control of the users on the applications using various user roles as configured in Admin module
o Integration testing
o QA Team will perform integration testing once the build for Cycle 2 is deployed in the
QA environment
o High Level check of all the features developed in all the sprints under cycle 1 would
be validated in this phase
o Key points that would be taken care of in this are:
 Data flow between modules
 Verify the application as a whole to validate weather it is behaving as
expected
 Validate the exception handling for exceptions that might arise in integrated
environment
o End To End Testing – End to End testing will be performed by QA to verify the overall
application & transactional level flow
o Execution strategy on various Platform/Device is as mentioned below:
i. Windows Platform Test Execution Approach
 Test Execution on Windows Platform will include both UI and Functional Testing
 The sequence in which the browsers would be picked for execution is:
- Chrome 41+
- IE 11+
- Firefox 36+

ii. MAC Test Execution Approach

 Test Execution on Mac will include both UI and Functional Testing
Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 10 of 26

  Test Execution will take place on actual devices and not on any emulator
 The sequence in which the browsers would be picked for execution is:
- Safari
- Chrome 41+
- Firefox 36+

iii. Mobile Test Execution Approach

 NBCU mobile lab will be used for testing at onshore location. Onsite QA Lead will
place the booking for the use of mobile lab devices in mobile lab at least 1 week in
advance prior to test execution kick off. Un-availability of the devices in the mobile
lab for the onsite resource for Test Execution will have an impact on the execution
timelines.
 8 concurrent users will be able to access the same device through mobile lab
 In case of any issues arise in accessing the mobile lab devices, mobile lab support
group can be contacted for immediate support by registering a ticket at
zendesk.mobilelabs.com. In case of any delay in the resolution of issues, there may
be impact on the Test Execution timelines.
 Actual physical devices will be used by offshore team for testing
 Test Execution will take place on actual devices and not on emulators
 Key considerations while performing mobile specific testing are:
- UX Testing: Special consideration to be kept for various screen resolutions, UI
layout text alignment, and scroll features
- Mobile Devices: Wireframes will be referred for UI testing on mobile devices
- Interrupt Testing: An application while functioning may face several interruptions
like incoming calls or network coverage outage and recovery. Common interrupts
include Voice Calls, SMS, GPS, Low memory, low Battery reminders etc.
 Below is the list of devices and the sequence in which the test execution would take
place :
- iOS 7+ & iPad Air 2+
- iOS 7+ & iPad Mini 3+
- iOS 7+ & iPhone 5S+
- Android 4.1+ & Samsung Galaxy S5

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 11 of 26

  Defect Review Strategy
o Defect logging and tracking will be done in TFS.
o QA team will retest any resolved defects deployed in QA environment in parallel to Test
Execution
o Defect Triage calls will be arranged by QA Team after end of each functional sprint
o QA team will be involved in all Scrum calls & Defect calls as scheduled
o Review of defects marked as Not a Defect will be done with the Business Analyst and Technical
Project Manager

Based on defects resurfaced during QA cycle, execution of further round will be determined by the
project team. User stories and other documents related to this release are uploaded in shared
drive and library for reference.

 Resource Ramp up Strategy

o KT Strategy for New Resources
- Resource Ramp Up would be done as per the below mentioned schedule:

Resource Name Location Onboarding Date

Vinay Thakur Onsite 14th September 2015
Sameer Aggarwal Offshore 5th October 2015
Resource 3 Offshore 14th December 2015
Resource 4 Offshore 18th January 2015
Resource 5 Offshore 18th January 2015
Resource 6 Offshore 18th January 2015
Resource 7 Offshore 18th January 2015
Resource 8 Offshore 25th January 2015

- Well formulated KT plan would be created and maintained for the resources ramped up
during the course of the project
- KT for the resources would start once the resource is on boarded as per the Ramp Up Plan
and will go in parallel with regular project task
- KT progress would be shared with the NBCU QA Program Manager at regular interval

o Task Allocation Strategy

- New resources in the project would initially be allocated Medium/Low priority features for
execution. Critical/High priority features would be taken by existing team member having
complete functional knowledge of Compass Express
- The allocation of the tasks to each resource would be planned in advance in the QA
tracker sheet. This sheet would be shared with the NBCU QA Program Manager at regular
intervals.

B. Test Schedule
Based on the Project Scope below is the QA schedule COMPASS Express – Phase 1 project release
(09/14/2015– 03/10/2016):

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 12 of 26

 QA Task From Date To Date Duration Resources

KT/Test strategy/ALM/Mobile lab set

9/14/2015 9/29/2015 12 Vinay
up
Test Preparation (Sprint 1 - 4) 9/28/2015 10/30/2015 25 Vinay, Sameer

Cycle 1:
Sprint 1 Test Execution (Cross 11/2/2015 11/27/2015 20 Vinay, Sameer
Browser + Platform/Device + Defect
Retesting)
Cycle 1:
Sprint 2 Test Execution (Cross 11/30/2015 12/24/2015 19 Vinay,Sameer,R3
Browser + Platform/Device testing +
Defect Retesting)
Cycle 1:
Sprint 3 Test Execution (Cross 12/28/2015 1/15/2016 14 Vinay,Sameer,R3
Browser + Platform/Device testing +
Defect Retesting
Cycle1:
Sprint 4 Test Execution (Cross 1/18/2016 1/22/2016 5 Vinay,Sameer,R3,R4,R5,R6,R7
Browser + Platform/Device testing +
Defect Retesting
Cycle 2: 
Testing on Integrated build for Cycle Vinay,Sameer,R3,R4,R5,R6,R7,R
1/25/2016 3/10/2016 34
1 (multiple browser + 8
Platform/Devices + Defect Retesting
Test Closure & Sign Off 3/10/2016 3/10/2016    
Total     129  

 KT/Test strategy/ALM/Mobile lab set up (09/14/2015 – 09/29/2015)

 Test Preparation (Sprint 1-4) (09/28/2015 – 10/30/2015)
 Cycle 1
o Sprint 1 (11/02/2015 – 11/27/2015)
- Functional Test Execution
- Cross Browser Testing
- Platform/Device Testing
- Defect Retesting

o Sprint 2 (11/30/2015 – 12/24/2015)

- Functional Test Execution
- Cross Browser Testing
- Platform/Device Testing
- Defect Retesting

o Sprint 3 (12/28/2015 – 01/15/2016)

- Functional Test Execution
- Cross Browser Testing
- Platform/Device Testing
- Defect Retesting

o Sprint 4 (01/18/2016 – 01/22/2015)

- Functional Test Execution
Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 13 of 26

 - Cross Browser Testing
- Platform/Device Testing
- Defect Retesting

 Cycle 2 (01/25/2016 – 03/10/2016)

o Integrated Sprint Testing on all Browser/Platform/Device combinations
o Complete set of Functional Test Cases would be executed on all Browser/Platform/Device
combinations
o Defect Retesting
o Exploratory Testing

 Functional QA Signoff
o COMPASS Express – Phase 1 Sign Off - 03/10/2016

During the completion of execution cycle, if there would be any defect left out for retesting which is not
fixed:
a. Based on the severity of the defects either they need to be fixed within QA timelines or QA timeline
need to be increased to facilitate the retesting of the defects
b. Defects rose during UAT Phase need to be supported – Need further discussion with Project team.

After the completion of Iteration Execution, if there are any open defects left out for retesting which has
not been fixed yet then:
a. Based on the severity of the defects either they need to be fixed or timeline need to be increased to
facilitate the retesting of the defects
b. Based on the discussion with the Project Team, defect may be put on hold (to be fixed in future
releases).

5. Test Approach
A. Test Design
Test Design for the Test cases will be based on the “Test Strategy /Test approach”. There will be one
rounds of testing- to test the user stories and second round will be to test defect fixes for that
“Iteration”. The team will execute each Functional Test Case designed as a part of COMPASS Express –
Phase 1 project as well as verify all defects that have been fixed for the same. ALM will be updated
accordingly.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 14 of 26

  Test Cases will be prepared in ALM. Following is the path:
Subject\Affiliates Enterprise & Scheduling Systems\COMPASS\Compass_Project\Compass Express
Phase1

 Test execution will be done in Quality Center. Following is the path:

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 15 of 26

 Root\COMPASS\ Compass_Project\2015_Project\Compass Express

 Defects will be logged in TFS.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 16 of 26

 B. Test Automation Strategy
N/A

C. Test Preparation

a. Test Entry Deliverables

Artifact Description Location

Use Case and UI Depicts the business usage, requirements and UI of NBCU Library
the application

Test Strategy Depicts Scope and Strategy Shared Drive

QA build QA Build deployed in QA environment. NBCU QA environment

Build Note Detail of user stories/functionalities delivered for NBCU Library

QA and any outstanding defects.

b. Traceability
 Requirements mapping will be done in ALM>>Requirements tab.

c. Training
 The business analyst will conduct release walkthrough for QA team to make aware of the
release functionalities.
 There is no other training required for the current team at this point.

6. Release Information
A. Cycle 1 (Sprint plan yet to be finalized by project team)
 Sprint 1
 Sprint 2
 Sprint 3
 Sprint 4

7. Test Environment
A. Testing Front End

QA URL- To be Shared by Dev after configuration

STG URL- To be Shared by Dev after configuration

Test URL
Training URL - To be Shared by Dev after configuration

Role Name Description of Role Username Password

Privileges
Admin Access to everything To be Shared To be Shared
Admin Stg. Access to everything To be Shared To be Shared
Admin Training Access to everything To be Shared To be Shared
B. Testing Database
N/A

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 17 of 26

 C. Platform Compatibility
N/A

8. Test Management Process

A. Test Management - Phase wise guideline
Major Minor Maintenance Maintenance: Maintenance:
Types of Projects
Enhancement Enhancements : Break Fix SW upgrades HW upgrades

SW upgrades,
Major Project/RTS RTS Releases Server or
data
Examples Enhancements, Releases (Minor (Break fixes, Infrastructure
upgrades, DB
New Projects Enhancements) Bug Fixes) Upgrades
Upgrades

Unit Testing N/A N/A N/A N/A N/A

String Testing N/A N/A N/A N/A N/A

Integration
R R R O N/A
Testing

Data Conversion /
Validation Testing- R N/A N/A N/A N/A
For all SPRINTS

Functional Testing
R R R O O
– for all SPRINTS

Regression Testing
*-
R R R R N/A
Final SPRINT
(Integrated
/Hardening Sprint)

Security Testing O O O O N/A

Performance
N/A N/A N/A N/A N/A
Testing

UAT N/A N/A N/A N/A N/A

** Note that all the above activities occur for a given RTS or a Project Release.
* Regression testing – Regression Testing of Compass application is not in scope.
In the above: R = Required, O = Optional (on case-by-case basis)

Team Responsibilities

Dev Team Unit Test, String Testing

QA Team Functional Testing

Performance Team Performance Testing

Business UAT (User Acceptance Testing)

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 18 of 26

 B. Test Methodologies
 Test Management Process - Agile Test Process

o Process Overview

Agile test processes which are much more diffuse can be aligned to the Waterfall Test Management
Process where test activities are performed in parallel rather than sequential. An agile project
contains one or more releases, where each release comprises of multiple iterations and each
iteration will have multiple stories. Stories are addressed based on value and priority, utilizing a
risk based approach. Each story is started and completed within the same iteration.

o Control & Planning

Since we will be testing continuously updated material, it is important to continue planning as
the project progresses and involve testing discussions in daily scrum meetings. Since goals
change based on lessons from prior iterations, feedback and business opportunities would be
continuously incorporated into the plan. Each day, the QA and Test Teams will report on the
status of the executed tests and quality of the product.
Deliverables
I. Master Test Plan (high level long term plan)
II. Test Plans pertaining to each Iterations
III. Risk Analysis (if applicable, will be part of this document)
IV. Test Estimation
V. Test Organization Definition (if applicable, part of Test Plan document)

o Preparation & Specification

The test organization will be included in Sprint Planning meetings and test scenarios will be
formulated based on user story discussions.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 19 of 26

 Deliverables
I. Test Scenarios
II. Manual test scripts
III. Automated test scripts (if applicable)

o Execution
Continuous testing from one story to another and one iteration to another incorporating
lessons learned from the prior tests is important to the success of Agile. The following tests
need to be carried out to help deliver quality services:

Unit Testing
The primary objective of these tests is to test the internal behavior of the system. Following
are the characteristics of these tests
 These tests are code facing, developed by programmers in support of programming
features
 Developed using the unit testing framework, where the objective is to test the
functional behavior of the unit of code
 To be executed with every Sprint and as often as possible

Functional/ Regression/Story testing

These tests monitor the functional behavior of the system as well confirm that the new
development has not affected the earlier functions which were working as per the
requirements.
 These tests are developed primarily by the testing team along with other members of
the project team
 Manually written and can be tracked using a test management tool
 Developed with an objective to test the functional behavior of the system as per the
requirements
 Automated regression tests are recommended to deliver better coverage. These tests
can be executed as part of the continuous integration process

UAT
UAT tests have characteristics similar to functional tests but are carried out by business users.
The aim of the acceptance test is monitor that the delivered products are a reflection of the
original user stories.

Security and Performance Testing

Security and Performance tests are related to the nonfunctional aspect of application and may
need code optimization, and therefore these are conducted in collaboration with the
development teams. These tests are automated and are executed as part of the continuous
integration process.

Deliverables
I. Test Execution Report
II. Defect Tracking Report

o Completion
In an Agile environment, we focus continuously on Adapt and Evolve with each delivery and
test cycle so the leading practices can be incorporated in the next cycle.
Deliverables
I. Lessons Learned

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 20 of 26

 II. Reusable test assets

 Deliverable Matrix - Agile (RTS Release)

 Major Release > 10 Business Days

 Minor release < 10 Business Days

Testing Deliverables Matrix – RTS

Small
Large
Test Phases Deliverables (Based on
Deliverables (Based on Applicability)
Applicability)
 Master Test Plan (high level long term  Master Test Plan (high level long term plan)
plan)  Test Plans pertaining to each Iterations
 Risk Analysis (part of Test Plan  Risk Analysis (part of Test Plan document)
Test Planning document)
 Test Estimation
 Test Estimation

 Test Scenarios  Test Scenarios

 Manual test scripts (System  Manual test scripts [Based on RQSDs/ Other
Integration) Specification Documents]
Test Preparation  Test Preparation Query Log (if  Test Preparation Query Log (if applicable)
applicable)

 Test Execution Reports  Test Execution Reports

Test Execution  Defect Tracking Reports  Defect Tracking Reports
 Phase End Test Summary Report  Phase End Test Summary Report
 Reusable test assets  Reusable test assets
Test Completion
 Lesson Learnt

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 21 of 26

  Deliverable Matrix - Agile (Project Release)

 Major Release > 20 Business Days

 Minor release < 20 Business Days

Testing Deliverables Matrix – Project

Small Large
Test Phases
Deliverables(Based on Applicability) Deliverables(Based on Applicability)
 Master Test Plan (high level long term  Master Test Plan (high level long term plan)
plan)
 Test Plans pertaining to each Iterations
 Risk Analysis (part of Test Plan
Test Planning document)  Risk Analysis (part of Test Plan document)
 Test Estimation  Test Estimation
 Test Organization Definition

 Test Scenarios
 Test Scenarios
 Manual test scripts [Based on Specification
 Manual test scripts (System Documents/Wireframes/User Stories]
Integration/UAT) [Based on
Test Preparation  Automated test scripts (if applicable)
Specification Documents/Wireframes]
 High Level test data needs report
 Test Preparation Query Log (if
applicable)  Test environment needs report
 Test Preparation Query Log (if applicable)

 Test Execution Reports  Test Execution Reports

Test Execution  Defect Tracking Reports  Defect Tracking Reports
 Phase End Test Summary Report  Phase End Test Summary Report

 Reusable test assets  Reusable test assets

Test Completion
 Lesson Learnt

9. Resources
A. Roles & Responsibilities

Role Responsibilities
Project Manager Overall project management

Communicate information across team; resolve large issues due to architecture or

Technical Lead
infrastructure

Business Analyst Gather business rules; Maintain design specification; create Use Case

Assign resources to complete test execution. Gather status and issues. Report and
QA Lead
follow up on results and defects

Tester Create, Modify and Execute test cases

Automation Automate manual test cases

Development Lead Communicate Development status, build status and any changes to the design spec

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 22 of 26

 B. Contact Information

Project or Contact Name Role Phone # Email

System
Project Monica Malhotra QA Program Manager +1 212 887 5984 Monica.malhotra@nbcuni.com

Project Manisha Golatkar Onsite QA Lead +1 212 887-5934 Manisha.golatkar@nbcuni.com

QA Kiran Chirra ODM +1 818-777-3605 Kiran.Chirra@birlasoft.com

Offshore Delivery
QA Sanjeev Rohilla +91 958 294 8725 Sanjeev.rohilla@nbcuni.com
Manager

Offshore Delivery
QA Ritesh Srivastav +91-880073 3636 Ritesh.Srivastav@birlasoft.com
Manager

QA Sameer Aggarwal Offshore QA Lead +91 981-102-0391 sameer.aggarwal2@nbcuni.com

QA Vinay Thakur Onsite QA Lead +1 212 887-5861 Vinay.thakur@nbcuni.com

C. Hardware
We currently have all the hardware necessary to begin testing for COMPASS Express – Phase 1. This
includes:
 Windows 7 machine(s)
 Mac OS X 10.9+
 iPhone 5S+
 iPad Mini 3+
 iPad Air 2+
 Samsung Galaxy S5+

D. Software
We currently have all the software needed to begin testing with the exception of the actual
software to be tested. This includes:

For Actual Testing:

 Software in test
 Microsoft Windows 7 machines
 Mac OS 10.9+
 Android 4.1+
 iOS 7+

For Testing Support:

 HP Quality Center will be used for test cases preparation and execution.
 Defects will be logged in TFS
 NBCU Libraries will be used to store and update Documents including use cases, use case-
user story mappings.
 All project delivery related documents will be stored in -
\\nbcuni.ge.com\shared\IT\us-nyc\Share\SolutionDelivery\QA\Testing & Verification\TV-
SYSTEMS\TV-East Coast\Affiliate & Enterprise (Research +
Compass)\SCHEDULING\COMPASS\Compass-Projects\2015\COMPASS Express

E. Facilities
For COMPASS Express – Phase 1 testing will take place at Birlasoft offshore (India) office.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 23 of 26

 F. Data
If required, the test data will be provided and generated by Business analysts. However, Dev team
will populate the same test data in the application before QA team receives the build

10. Success Factors

A. Test Exit Deliverables
Artifact Description Location
Test Cases / Test Test cases along with test steps Quality Center(ALM)
Artifacts
Test Results Pass/Fail of tests executed Quality Center (ALM)
Defect List Final States of remaining defects TFS
Test Report Overall report of testing Shared Drive and Email

B. Control Documentation
History of an artifact is automatically captured in the following tools:
 Quality Center (ALM)

C. Deployment Decision
Exit criteria will include:

 100% of the test cases that are in scope must be documented and executed
 100% of “High” priority test cases must pass.
 100% of “Critical” priority defects must be resolved and closed
 100% of “High” priority defects must be resolved and closed
 All defects should be reviewed with development team by QA for resolution.
 No Outstanding defects in “Open” state prior to release date. For any open defect (Not
Showstopper / Critical) project team will take decision.
 If any open defect should get reviewed with Triage team to defer it to next release if
possible with release notes.
 Sign Off from NBCU QA & Project Team for production deployment

11. Communication
A. Review and Status Meetings
 The Scrum Master will hold a regular scrum meeting where entire project status is shared
(including QA / testing status).
 Regular internal QA Meetings to discuss review defects or any issue encountered with Dev /project
team.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 24 of 26

 B. Defect Management

Defect Life Cycle

The Defects in TFS are also categorized into 2 different Request Types. They are as mentioned below:

 Bug – All the observations that have been agreed upon as a defect after the discussion with the
project team, the request type of such QC id’s will be marked as a “Defect”.

 Enhancement – All the suggestions that the QA team provides that can be taken care as an
enhancement in the future releases.

Defect Flow:

 When tester finds a defect, the defect needs to be logged in TFS and assigned to the Project/Dev
Team and will be in the ACTIVE state.
 The Project/Dev Team will analyze the defect
 If accept as a valid defect, will enter the 'Root Cause Code' and ‘Reason e.g. Fixed’ to the
defect and will change the states as 'RESOLVED' and fix available state as ‘QA’ and send it to
QA.
 If accept as invalid defect, will enter the 'Root Cause Code' and ‘Reason e.g. Deferred/ Not able
to Reproduce/ Obsolete etc.’ to the defect and will change the states as 'RESOLVED' and send it
to QA.
 QA Team will retest the defect and will mark the state as 'Closed' if the defect is closed or will
mark the state as 'Open' in case the defect is not fixed and reopened.

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 25 of 26

  QA Team will raise Service Now tickets for all the Active Issues in TFS logged by QA team which are
not fixed at the end of the project

Measurements:

During and/ or after test execution following metrics to measure the application health as well as test
productivity:

 A matrix that reveals number of test cases executed, how many passed, how many failed
 How many defects logged during test cycle

12. Appendices
A. Test Artifacts
All the below artifacts can be referred from Quality Center except the Test Summary Report. Summary
report will be published by QA team at the end of test phase.

 Test Cases
 Test Case Execution Result
 Defects List
 Test Summary Report / Operational Graphs
 Working Spreadsheets of Defects/Status/Resolution/Action Items

Classification: NBCUniversal Internal

COMPASS EXPRESS – Phase 1 – System Test Plan Page 26 of 26