C H A P T E R

Implement
Introduction to Implementation
The goal of implementation is to introduce the new system into the network with the least amount of disruption and the highest level of interoperability with the existing network. An essential component of this process is the implementation plan.

Tip

You can navigate to any topic on this tab by using the tab navigation pane at the left of the content pane. This navigation pane contains the table of contents (TOC) for the active tab.
Before You Begin

You should understand how to implement Cisco Unified Communications. For more information, see Cisco Unified Communications Implementation. You should have a completed implementation plan from the detailed design. Use the equipment list and site specification from the detailed design to do the following:

Order and stage equipment Perform a detailed site survey Create site-specific installation guidelines Deployment strategy Network maps and topology diagrams Installation and commissioning tests Site survey results List of all devices to be installed Installation guidelines Configuration worksheets Test and turn-up plan

Your implementation plan should include:

When You Are Done

All components are installed and ready to configure.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-1

Chapter 5 Order Equipment

Implement

Major Tasks in This Process

Order Equipment Install and Configure System Components Preparing Your Network for Troubleshooting and Recovery Conduct User Acceptance Test

Order Equipment
This topic includes links to ordering guides and descriptions of tools that you need to choose your ordering options.

Solution Expert Tool

Solution Expert is a web-based tool that assists in the design, configuration, quoting, and ordering of Cisco Unified Communications products. Solution Expert is available for Cisco sales and partner systems engineers with Cisco Unified Communications specializations. With the Solution Expert tool, users can generate a recommended solution based on their requirements. Users can modify the recommended configuration if desired. Solution Expert validates any changes when it presents the new solution. Solution Expert also generates a bill of materials with list pricing, a Visio diagram, and other design documentation. For an overview of how to use the tool, see the introductory PDF on the home page.

Ordering Guides
Ordering guides for most Cisco Unified Communications products are available for Cisco partners, Cisco sales staff, and Cisco service providers.

Install and Configure System Components

When implementing a new Cisco Unified Communications system, create a site-specific installation plan for your team. Describe what needs to be installed and configured. Your plan should reference product-specific installation and configuration guides that can be used during the implementation process. Your plan will help you manage timelines for implementing equipment and scheduling outages. Include an installation schedule, as well as a test plan that will verify that the operation conforms to the design objectives. Performing Your System Installation provides guidance for the installation order of components for a Cisco Unified Communications System Release 8.5(1) contact center deployment. It does not describe installation procedures for individual components. For links to the complete documentation set for each contact center system component, see the Configuration Command Files topic on the Resource Library tab. The following topics provide additional information for installing and configuring individual contact center products:

Component Installation and Configuration Guides provides a complete list of components and links to related installation and configuration documents.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-2

Chapter 5

Implement Install and Configure System Components

Installation and Configuration Checklists provides checklists for installing and configuring some of your contact center components. Component Compatibility and Interoperability provides links to information about compatibility between Cisco products and with third-party systems or a hardware platform.

For system configuration examples, as well as other system implementation topics, see the documentation wiki (DocWiki) at http://docwiki.cisco.com/wiki/Unified_Communications_System_Implementation.
When You Are Done

Components are configured and ready to test:

For basic standalone operation For interoperability in your application(s) Acceptance Test Plan is completed

Performing Your System Installation

Before You Begin

See Plan and Prepare for Your System Installation to plan your overall strategy.

Install Contact Center Software Components

Once you have your installation plan and preparations in place, perform your system installation by following the guidelines and sequence in Performing Your System Installation:

See Deployment Models for the general installation sequence for the various components in the different generic deployment models, including Single Site, Multisite Centralized, Multisite Distributed, and Clustering over IP WAN. For a description of the deployment models tested in this system release, see the following:
Unified IP IVR Test Sites Parent and Child Test Sites Unified CVP Test Sites

Test Deployment Models and Sites.) See Installing Components for more detailed installation information for each major installation strategy: single-stage system installation, multistage system installation, and multisite phased installation. See Related Documentation for links to component installation and upgrade documentation.

Component Installation and Configuration Guides

This topic provides references to installation and configuration documents for the software and hardware components that are part of the Cisco Unified Communications System Release 8.5(1) family of products. This information includes component names and related documentation.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-3

Chapter 5 Install and Configure System Components

Implement

Table 5-1

Component Installation and Configuration Documentation

Product Category Call Control

Product Technology Cisco Unified Communications Manager

Components

Documentation Title

Cisco Unified Communications Manager Cisco Real-Time Monitoring Tool Music on Hold Cisco Security Agent (only as a standalone agent) Third-party Antivirus

Installing Cisco Unified Communications Manager Replacing a Single Server or Cluster for Cisco Unified Communications Manager Upgrading to Cisco Communications Manager Release 8.0(1) from Release 6.1(2) and Later Cisco Unified Communications Manager Administration Guide Cisco Unified Serviceability Administration Guide Cisco Unified Communications Manager System Guide Cisco Unified Communications Manager Bulk Administration Guide CLI Configuration Guide for Cisco Unified SIP Proxy Release 8.5 GUI Configuration Guide for Cisco Unified SIP Proxy Release 8.5 Cisco Unified SIP Proxy Command Reference

Cisco Unified SIP Proxy

Cisco Unified SIP Proxy

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-4

Chapter 5

Implement Install and Configure System Components

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category Contact Center

Product Technology Cisco Unified Intelligent Contact Management Enterprise

Components

Documentation Title

Cisco Unified Intelligent Contact Management Enterprise Cisco Unified Contact Center Enterprise Cisco Unified Expert Advisor Progger (Peripheral Gateway, CallRouter, and Logger) Rogger (CallRouter and Logger) Generic Peripheral Gateway Cisco Unified Contact Center Gateway Enterprise Cisco Unified Contact Center System Gateway Media Routing PG CTI Server Real-Time Admin Workstation (RTAW) Historical Data Server WebView Cisco Outbound Option Cisco Unified Mobile Agent Cisco Security Agent Windows 2003 Active Directory Third-party Antivirus

Pre-installation Planning Guide for Cisco Unified ICM Enterprise and Hosted Hardware and System Software Specification (Bill of Materials) for Cisco Unified ICM/Contact Center Enterprise & Hosted Installation Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted Installation and Configuration Guide Cisco Unified Contact Center Enterprise Upgrade Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted Cisco Contact Center Gateway Deployment Guide for Cisco Unified ICME/CCE/SCCE/CCX Configuration Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted Administration Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted Administration Guide for Cisco Unified Contact Center Enterprise & Hosted WebView Installation and Administration Guide Mobile Agent Guide for Cisco Unified Contact Center Enterprise & Hosted Cisco Security Agent Installation/Deployment Guide for Cisco Unified Intelligent Contact Management Enterprise/Hosted Outbound Option Guide for Cisco Unified Contact Center Enterprise Installation Guide for Cisco Unified Expert Advisor Administration and Configuration Guide for Cisco Unified Expert Advisor

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-5

Chapter 5 Install and Configure System Components

Implement

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category

Product Technology Cisco Unified IP IVR

Components

Documentation Title

Cisco Unified IP IVR Cisco Security Agent Third-party Antivirus

Installation Guide for Cisco Unified CCX and Cisco Unified IP IVR Upgrade Guide for Cisco Unified CCX and Cisco Unified IP IVR Administration Guide for Cisco Unified CCX and Cisco Unified IP IVR Installation and Upgrade Guide for Cisco Unified Customer Voice Portal Getting Started with Unified CVP Configuration and Administration Guide for Unified CVP Cisco Security Agent Installation/Deployment Guide for Cisco Unified Customer Voice Portal Reporting Guide for Unified CVP

Cisco Unified Customer Voice Portal

Cisco Unified Customer Voice Portal Unified CVP Call Servers Cisco Unified CVP Reporting Server Cisco Unified CVP Operations Console HTTP Media Server/Web Server Cisco Unified Call Studio Nuance OSR Nuance OSMS Cisco Security Agent Third-party video components Third-party Antivirus Cisco Agent Desktop Server Cisco Supervisor Desktop CAD Agent Desktop Cisco Security Agent (on desktop) Voice over IP (VoIP) Monitor Server

Cisco Agent Desktop

Cisco CAD Installation Guide for CAD 8.0 for Unified CM Cisco Desktop Administrator User Guide Cisco Supervisor Desktop User Guide Cisco Agent Desktop User Guide

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-6

Chapter 5

Implement Install and Configure System Components

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category

Product Technology Cisco Telephony Integration Object Server

Components

Documentation Title

Cisco Telephony Integration Object Server Cisco Supervisor Desktop CTI OS Agent Desktop Cisco Security Agent (on desktop)

CTI OS System Manager's Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted CTI OS Supervisor Desktop User Guide for Cisco Unified Contact Center Enterprise & Hosted CTI OS Agent Desktop User Guide for Cisco Unified ICM/Contact Center Enterprise & Hosted Hardware & System Software Specification (Bill of Materials) for Cisco Unified Contact Center Enterprise Installation Guide for Cisco Unified CCX and Cisco Unified IP IVR Upgrade Guide for Cisco Unified CCX and Cisco Unified IP IVR Getting Started with Cisco Unified Contact Center Express Cisco CAD Installation Guide for CAD 8.0 for Unified CM Administration Guide for Cisco Unified CCX and Cisco Unified IP IVR Installation Guide for Cisco Unified Intelligence Center User Guide for the Cisco Unified Intelligence Center Reporting Application Hardware and System Software Specification (Bill of Materials) Installation and Upgrade Guide for Cisco Unified Presence Deployment Guide for Cisco Unified Presence User Guide for Cisco Unified Presence Serviceability Configuration and Maintenance Guide for Cisco Unified Presence

Cisco Unified Contact Center Express

Cisco Unified Contact Center Express Cisco Agent Desktop Cisco Agent DesktopBrowser Edition Cisco Supervisor Desktop Cisco Desktop Administrator

Cisco Unified Intelligence Suite

Cisco Unified Intelligence Center

Applications

Cisco Unified Presence

Cisco Unified Presence

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-7

Chapter 5 Install and Configure System Components

Implement

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category End Points and Clients

Product Technology Cisco Unified Personal Communicator Cisco Unified IP Phones 6900 Series

Components

Documentation Title

Cisco Unified Personal Communicator Cisco Unified IP Phones (SCCP) 6911, 6921, 6941, 6961 Cisco Unified IP Phones (SIP) 7941G, 7961G, 7970, 7971G Cisco Unified IP Phones (SCCP) 7940, 7941G, 7960, 7961G, 7970, 7971G Cisco Unified IP Phone 7921G (Wireless) Cisco Unified IP Video Phone 7985G Cisco Unified IP Phone (SIP) 8961

Deployment Guide for Cisco Unified Presence Cisco Unified IP Phone 6900 Series End-User Guides Cisco Unified IP Phone 6900 Series Maintain and Operate Guides Cisco Unified IP Phone 7900 Series Install and Upgrade Guides Cisco Unified IP Phone 7900 Series End-User Guides Cisco Unified IP Phone 7900 Series Maintain and Operate Guides

Cisco Unified IP Phones 7900 Series

Cisco Unified IP Phones 8900 Series

Cisco Unified IP Phone 8900 Series Install and Upgrade Guides Cisco Unified IP Phone 8900 Series End-User Guides Cisco Unified IP Phone 8900 Series Maintain and Operate Guides Cisco Unified IP Phone 9900 Series Install and Upgrade Guides Cisco Unified IP Phone 9900 Series End-User Guides Cisco Unified IP Phone 9900 Series Maintain and Operate Guides Cisco Aironet 1240 AG Series Installation and Configuration Guides

Cisco Unified IP Phones 9900 Series

Cisco Unified IP Phones (SIP) 9951, 9971

Wireless

Cisco Aironet 1240 AG Series

Cisco Aironet Access Point 1240AG

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-8

Chapter 5

Implement Install and Configure System Components

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category Security

Product Technology Cisco ASA 5500 Series Adaptive Security Appliances

Components

Documentation Title

Cisco ASA 5520, ASA 5540, ASA 5580 Cisco ASA Advanced Inspection and Prevention Security Services Module (AIP-SSM)

Cisco ASA 5500 Series Adaptive Security Appliance Install and Upgrade Guides Cisco ASA 5500 Series Adaptive Security Appliance Configuration Guides Cisco Security Modules for Security Appliances Configuration Guides Installing Cisco Unified Communications Manager

Cisco Security Agent for Unified Communications Manager Cisco Security Agent for Unified IP IVR Cisco Security Agent for Unified Intelligent Contact Management Enterprise Cisco Security Agent for Unified Customer Voice Portal Network Management Cisco Unified Operations Manager

Cisco Security Agent for Unified Communications Manager Cisco Security Agent for Unified IP IVR Cisco Security Agent for Unified Intelligent Contact Management Enterprise Cisco Security Agent for Unified Customer Voice Portal Cisco Unified Operations Manager

Installation Guide for Cisco Unified CCX and Cisco Unified IP IVR Cisco Security Agent Deployment Guide

Cisco Security Agent Deployment Guide Installation Guide for Cisco Unified Operations Manager User Guide for Cisco Unified Operations Manager

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-9

Chapter 5 Install and Configure System Components

Implement

Table 5-1

Component Installation and Configuration Documentation (continued)

Product Category Communications Infrastructure

Product Technology Cisco 7800 Series Media Convergence Servers Catalyst 3700 Series

Components

Documentation Title

Cisco 7800 Series Media Convergence Servers Cisco Catalyst 3750 multilayer switch

Cisco 7800 Series Media Convergence Servers Install and Upgrade Guides Cisco Catalyst 3750 Series Switches Install and Upgrade Guides Cisco Catalyst 3750 Series Switches Configuration Guides Cisco Catalyst 6500 Series Switches Install and Upgrade Guides Cisco Catalyst 6500 Series Switches Configuration Guides Cisco AS5400 Series Universal Gateways Install and Upgrade Guides Cisco AS5400 Series Universal Gateways Configuration Guides Cisco 3800 Series Integrated Services Routers Install and Upgrade Guides Cisco 3800 Series Software Configuration Guide Cisco VGD Series Voice Gateways Install and Upgrade Guides Cisco VGD 1T3 Voice Gateway Software Configuration Guide Cisco IOS Software Releases 15.1 Configuration Guides Cisco 7206 Installation and Configuration Guide Cisco 860 Series, Cisco 880 Series, and Cisco 890 Series Integrated Services Routers Hardware Installation Guide Cisco 860, Cisco 880, and Cisco 890 Series Integrated Services Routers Software Configuration Guide Cisco CSS 11000 Series Content Services Switches Install and Upgrade Guides Cisco CSS 11000 Series Content Services Switches Configuration Guides Cisco CSS 11000 Series Content Services Switches Maintain and Operate Guides

Catalyst 6500 Series

Cisco Catalyst 6506, 6509 Cisco Catalyst 6506, 6509 (MSFC) Unified CVP, VXML Voice/data, H.323, SIP, and MGCP Gateways: Cisco 3825 and 3845 Unified CVP, VXML voice, H.323, and PSTN Gateways: AS5400, HPX/AS5400XM Cisco VGD 1T3 Voice Gateway RSVP Agent: Cisco 38xx Platforms

Gateways/Gatekeepers

Routers

Cisco 7206VXR Cisco 881 router (teleworker access router)

Cisco Content Services Switch

Cisco 11500 Content Services Switch

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-10

Chapter 5

Implement Install and Configure System Components

Installation and Configuration Checklists

Use the checklists from the following documents to install and configure the required components for your applications.

Note

The order in which you install components depends on your site. Follow the order recommended in your site-specific implementation plan.

Installing Cisco Unified Communications Manager Installation and Configuration Guide Cisco Unified Contact Center Enterprise Cisco Contact Center Gateway Deployment Guide for Cisco Unified ICME/CCE/SCCE/CCX Getting Started with Cisco Unified Contact Center Express

Component Compatibility and Interoperability

Use the following links to access commonly used interoperability and compatibility information for Cisco products with each other, with a third-party system, or with a computer hardware platform.

Cisco Unified Communications Compatibility Tool Cisco Unified Communications Manager Compatibility Information Cisco Unified Contact Center Enterprise Software Compatibility Guide Cisco Unified Contact Center Express Software and Hardware Compatibility Guide Cisco Interoperability Portal Hardware and System Software Specification (Bill of Materials) for Cisco Unified ICM/Contact Center Enterprise & Hosted Cisco Unified Communications Manager Server Support Matrix Cisco Unified Communications System Release Summary Matrix for Contact Center

Software Versions and System Caveats

For specific information on the product software versions used, including system limitations and known caveats, see the System Release Notes for Contact Center: Cisco Unified Communications System, Release 8.5(1).

Call Flow Configuration Examples

Sample configuration commands for infrastructure components that are involved in the call flows are in downloadable zip files for all test bed components in Configuration Command Files in the Resource Library. System-level configuration examples are available on the Cisco DocWiki at http://docwiki.cisco.com/wiki/Category:Contact_Center_System_Configurations.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-11

Chapter 5 Introduction to Troubleshooting

Implement

Introduction to Troubleshooting
This topic describes how to develop a system-level troubleshooting methodology as you install and configure a Cisco Unified Communications network for the first time. It also provides recommendations for preparing and documenting the network that may assist you in diagnosing and isolating problems when they occur. This topic contains the following sections:

System Troubleshooting Methodology Preparing Your Network for Troubleshooting and Recovery

For more information about system-level troubleshooting, see the Cisco Unified Communications category on the documentation wiki (DocWiki) at http://docwiki.cisco.com/wiki/Unified_Communications_System_Troubleshooting.

System Troubleshooting Methodology

The Implementation phase of your network deployment is an excellent time to develop a methodology for troubleshooting the network as a whole. Troubleshooting networking equipment at a system level requires solid detective skills. When a problem occurs, the list of potential suspects is long. You must collect detailed information and systematically narrow the list of potential causes to determine the root problem. This topic does not provide step-by-instructions for resolving problems that occur during network installation. Instead, this topic describes sound methods for troubleshooting your network using the following general steps:
1. 2. 3.

Gather Information on the Problem. Isolate Point(s) of Failure. Apply Tools to Determine the Problems Root Cause.

Gather Information on the Problem

Problems are typically discovered and reported by one of the following types of users:

External customers dialing into a call center to order products, obtain customer service, and so forth. Internal agents receiving incoming calls from a call queue or initiating outbound collection calls to customers. Internal users using administrative phones to call employees in other company locations or PSTN destinations, and perform basic actions such as call transfers and dialing into conferences.

As the network administrator, you must collect sufficient information from these users to allow you to isolate the problem. Detailed, accurate information will make this task easier. Table 5-2 lists recommended questions to ask users when they report a problem. As you turn up your network, you may consider putting these questions in an on-line form. A form will encourage users to provide more details about the problem and also put them into the habit of looking for particular error messages and indicators. Capturing the information electronically will also permit you to retrieve and re-examine this information in the future, should the problem repeat itself.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-12

Chapter 5

Implement Introduction to Troubleshooting

Table 5-2

Questions to Ask Users When They Report Problems

Ask this Question...

Did something fail or did it simply perform poorly?

To Determine... Whether the issue relates to system degradation or a connectivity failure. An example of a failure is when a user dials a phone number and hears fast busy tone. An example of a performance problem is when a user dials into a conference call and hears choppy audio when other parties speak. Quality of service or performance issues require a different approach than connectivity or operational problems. You must still isolate the potential sources of the problem, but you will typically use performance management tools instead of log files. The device type, model and version of software installed. It is also critical to capture the IP address assigned to the device, as well as its MAC address. If the case of IP phones, determining the phones active Cisco Unified Communications Manager server is also important. On Cisco Unified IP phones, these important network values can be displayed by pressing the Settings button and choosing the Network Configuration option from the menu. If a device was recently installed and the problem occurred while making it work for the first time, or if the device was operating normally before the problem occurred. If the device was newly installed, the problem is most likely due to improper configuration or wiring of that particular device. Problems with devices that are already up and running can typically be traced back to one of two causes: (a) the user modifying their device, such as changing their configuration or upgrading software, or (b) a change or failure elsewhere in the network. The steps that led up to the problem, including which buttons were pressed and in which order. Capturing this information in detail is important so that you can consistently reproduce the problem. The visual and audio indicators of the problem. Ask users to provide the exact text that appears and any error codes in either an E-mail or on-line form. If the error indication was audible, ask the user to write down the announcement they heard, the last menu option they were able to successfully choose or the tone they heard when the call failed.

What device were you trying to use?

Did it ever work?

Exactly what action(s) did you perform?

What error message(s) appeared or announcements did you hear?

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-13

Chapter 5 Introduction to Troubleshooting

Implement

Table 5-2

Questions to Ask Users When They Report Problems (continued)

Ask this Question...

What time did the problem occur?

To Determine... The date and time to compare against entries in log files. If the problem occurred on a Cisco Unified IP phone, make certain the user provides the timestamp that appears on their phones display. Several Cisco components in a network may capture the same problem event in separate log files, with different ID values. In order to correlate log entries written by different components, you must compare the timestamps to find messages for the same event. Cisco Unified IP phones synchronize their date and time with their active Cisco Unified Communications Manager server. If all Cisco components in the network use Network Time Protocol (NTP) to synchronize with the same source, then the timestamps for the same problem messages will match in every log file. If the problem relates to a WAN or PTSN link, or a Cisco Unified Communications Manager dial plan issue. Ask the user the phone number he or she dialed (called number) and determine if the destination was within his or her site, another site within the corporate network, or a PSTN destination. Because the calling number (the number of the phone used) also affects call routing in some cases, capture this number as well. If the problem is not directly related to the calling number or called number but rather to the supplementary service setup on Unified Communications Manager or the problem is at the destination phone the user tried to reach by transferring or forwarding the call. If the problem is isolated to that users device or represents a more widespread network problem. If the user cannot make a call from his or her phone, ask the user to place a call to the same destination using a phone in a nearby office.

What is the number of the phone you used and what was the phone number you called?

Did you try to perform any special actions, such as a transfer, forward, call park, call pickup, or meet-me conference? Is the phone set up to automatically perform any of these actions? Did you attempt the same action on another device?

Isolate Point(s) of Failure

After collecting information on the symptoms and behavior of the problem, to narrow the focus of your efforts you should:

Identify the specific devices involved in the problem. Check the version of software running on each device. Determine if something has changed in the network. Verify the integrity of the IP network.

Identify Devices Involved in the Problem

In large- to medium-sized networks, it is crucial to identify the specific phones, routers, switches, servers and other devices that were involved in a reported problem. Isolating these devices allows you to rule out the vast majority of equipment within the network and focus your time and energy on suspect devices. To help you isolate which devices were involved in a problem, two types of information can prove invaluable:

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-14

Chapter 5

Implement Introduction to Troubleshooting

Network topology diagrams: It is strongly recommended that you have one or more diagrams that show the arrangement of all Cisco Unified Communications products in your network. These diagrams illustrate how these devices are connected and also capture each devices IP address and name (you may want to also have a spreadsheet or database of the latter information). This information can help you visualize the situation and focus on the devices that may be contributing to the reported problem. See Network Topology Diagrams for recommendations on how to prepare these diagrams. Call flow diagrams: Cisco equipment, including Unified Communications Manager servers, typically provide detailed debug and call trace log files. To interpret these log files, however, it is useful to understand the signaling that occurs between devices as calls are set up and disconnected. Using the network topology and call flow diagrams in conjunction with the log files, you can trace how far a call progressed before it failed and identify which device reported the problem. Examples of using call flow diagrams for problem isolation are shown in Troubleshooting Daily Operations.

Check Software Release Versions for Compatibility

After you have identified which devices may be involved in the problem, verify that the version of software running on each device is compatible with the software running on every other device. As part of Cisco Unified Communications Release 8.5(1) verification, Cisco Systems has performed interoperability and load testing on simulated network environments running specific software versions. The System Release Notes for Contact Center: Cisco Unified Communications System, Release 8.5(1) lists the combination of software releases that were tested. However, if the combination of releases installed in your network does not match the values in the System Release Notes, it does not necessarily mean the combination is invalid. To check interoperability for a specific device and software release, locate and review its individual Release Notes. Release Notes contain up-to-date information on compatibility between the product and various releases of other products. This document also describes open caveats, known issues that may cause unexpected behavior. Before beginning extensive troubleshooting work, examine the Release Notes to determine if you are experiencing a known problem that has an available workaround.

Tip

The open caveat information in the Release Notes contains links to the Bug Toolkit. The Bug Toolkit requires that you are a Cisco partner or a registered Cisco.com user with a Cisco service contract. To access the Bug Toolkit, go to the http://tools.cisco.com/Support/BugToolKit/ .

Determine if Network Changes Have Occurred

Before focusing on the particular device or site where the problem occurred, it may be useful to determine if a change was made to surrounding devices. If something has been added, reconfigured or removed from elsewhere in the network, that change may be the source of the problem. It is recommended that you track changes to the network such as:

New agent phones added Modifications to Cisco Unified Communications Manager call routing settings, such as new directory numbers, route patterns and dial rules to support new sites or devices Changes to port configurations on switches, routers or gateways (new equipment, wiring changes or new port activation) Changes to IP addressing schemes (such as adding new subnets) that may have affected route tables

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-15

Chapter 5 Introduction to Troubleshooting

Implement

Verify the IP Network Integrity

Always remember that Cisco Unified Communications equipment relies on a backbone IP network. Many connectivity problems are not caused by configuration errors or operational failures on Cisco devices, but rather by the IP network that interconnects them. Problems such as poor voice quality are typically due to IP network congestion, while call failures between locations may be the result of network outages due to disconnected cables or improperly configured IP route tables. Before assuming that call processing problems result from Cisco Unified Communications devices themselves, check the integrity of the backbone IP network. Keep the OSI model in mind as you perform these checks. Start from the bottom, at the physical layer, by checking that end-to-end cabling. Then verify the status of Layer 2 switches, looking for any port errors. Move from there to confirm that the Layer 3 routers are running and contain correct routing tables. Continue up the OSI stack to Layer 7, the application layer. To resolve problems occurring at the top levels of the stack, a protocol analyzer (or sniffer) may be useful. You can use sniffer to examine the IP traffic passing between devices and also decode the packets. Sniffers are particularly useful for troubleshooting errors between devices that communicate using Media Gateway Control Protocol (MGCP) or Session Initiation Protocol (SIP).

Apply Tools to Determine the Problems Root Cause

After you have eliminated the IP network as the source of the problem and you have isolated the specific Cisco Unified Communications components involved, you can start applying the many diagnostic tools provided by Cisco components. Table 5-3 lists the diagnostic tools and supporting troubleshooting documentation available for most components in a contact center network. Note that this summary table is provided for reference only. The procedures in Troubleshooting Daily Operations specify when to use each tool and provide links to the troubleshooting instructions where appropriate.
Table 5-3 Contact Center Component Troubleshooting Tools and Documentation

Category Call Control

Component Cisco Unified Communications Manager

Diagnostic Tools Available Serviceability System tools:

Information Available In...

Alarms Real-Time Monitoring Tool window Communications Manager trace log SDL trace log (under TAC direction)

Troubleshooting Guide for Cisco Unified Communications Manager Cisco Unified Communications Manager Real-Time Monitoring Tool Administration Guide Cisco Unified Serviceability Administration Guide Cisco Unified CDR Analysis and Reporting Administration Guide Disaster Recovery System Administration Guide for Cisco Unified Communications Manager Troubleshooting TechNotes Configuring Cisco Unified SIP Proxy Version 1.1.4 for an Enterprise Network, Troubleshoot section

Trace log files

Cisco Unified SIP IOS command line tools (such as Proxy Show commands)

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-16

Chapter 5

Implement Introduction to Troubleshooting

Table 5-3

Contact Center Component Troubleshooting Tools and Documentation (continued)

Category Contact Center

Component Cisco Unified Intelligent Contact Management Enterprise

Diagnostic Tools Available Distributed Diagnostics and Services Network (DDSN) Support Tools Dashboard (requires additional software)

Information Available In...

Administration Guide for Cisco Unified ICM/Contact Center Enterprise Scripting and Media Routing Guide Cisco Unified ICM/Contact Center Enterprise & Hosted, Troubleshooting Tools for Internet Script Editor section Cisco Support Tools User Guide for Cisco Unified Software Troubleshooting TechNotes Mobile Agent Guide for Cisco Unified CC Enterprise & Hosted, Configuration and Troubleshooting Appendix for Remote Agent appendix Troubleshooting Guide for Cisco Unified Contact Center Management Portal Cisco CAD Troubleshooting Guide Cisco Unified Contact Center Enterprise and Hosted Troubleshooting TechNotes

Cisco Unified Contact Center Enterprise

Log files:

Error/event logs Agent desktop activity logs Debugging logs Chat Service Enterprise Service IP Phone Agent Service Packet Capture Driver Recording and Statistics Service Voice Over IP Monitor service

Test programs:

Sniffing Adapter Update Utility Cisco Unified IP Interactive Voice Response (Unified IP IVR) Log files Alarms

Cisco CAD Troubleshooting Guide for Cisco Unified Contact Center Express, Cisco Unified Communications Manager Edition Cisco CAD Troubleshooting Guide for Cisco Unified Contact Center Express, Cisco Unified Communications Manager Express Edition Troubleshooting TechNotes Troubleshooting Guides for Cisco Unified Customer Voice Portal Cisco Support Tools User Guide for Cisco Unified Software Troubleshooting TechNotes

Cisco Unified Customer Voice Portal

Error messages Alarms Support Tools Dashboard (requires additional software)

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-17

Chapter 5 Introduction to Troubleshooting

Implement

Table 5-3

Contact Center Component Troubleshooting Tools and Documentation (continued)

Category Contact Center (continued)

Component CTI Object Server (CTIOS)

Diagnostic Tools Available Log files:

Information Available In...

CTI OS Server logs CTI Toolkit logs

Cisco Support Tools User Guide for Cisco Unified Software Troubleshooting TechNotes

Error messages in the CTI OS Server console window Support Tools Dashboard (requires additional software) Cisco Agent Desktop (CAD) Log files:

Error/event logs Agent desktop activity logs Debugging logs

Cisco Support Tools User Guide for Cisco Unified Software Cisco CAD Troubleshooting Guide for Cisco Unified Contact Center Express, Cisco Unified Communications Manager Edition Cisco CAD Troubleshooting Guide for Cisco Unified Contact Center Express, Cisco Unified Communications Manager Express Edition Troubleshooting TechNotes

Test programs:

IP Phone Agent Service test program Voice Over IP Monitor service test program

Support Tools Dashboard (requires additional software) Cisco Unified Intelligence Center Cisco Unified Contact Center Express Log files Alarms

Installation Guide for Cisco Unified Intelligence Center, Frequently Asked Questions about the Installation chapter Cisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified CCX and Cisco Unified IP IVR Cisco Unified Serviceability Administration Guide Disaster Recovery System Administration Guide for Cisco Unified CCX and Cisco Unified IP IVR Serviceability Administration Guide for Cisco Unified CCX and Cisco Unified IP IVR Cisco CAD Troubleshooting Guide for Cisco Unified Communications Manager Troubleshooting TechNotes

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-18

Chapter 5

Implement Introduction to Troubleshooting

Table 5-3

Contact Center Component Troubleshooting Tools and Documentation (continued)

Category Applications

Component Cisco Unified Presence

Diagnostic Tools Available Configuration Troubleshooter Trace log files Alarms

Information Available In...

Serviceability Configuration and Maintenance Guide for Cisco Unified Presence Instant Messaging Compliance Guide for Cisco Unified Presence, IM Compliance Serviceability and Troubleshooting chapter Integration Guide for Configuring Cisco Unified Presence for Interdomain Federation, Troubleshooting chapters Integration Note for Configuring Cisco Unified Presence with Microsoft Exchange, Troubleshooting Tips Integration Note for Configuring Cisco Unified Presence with Microsoft OCS for Microsoft Office Communicator Call Control, Troubleshooting Tips Disaster Recovery System Administration Guide for Cisco Unified Presence Troubleshooting TechNotes

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-19

Chapter 5 Introduction to Troubleshooting

Implement

Table 5-3

Contact Center Component Troubleshooting Tools and Documentation (continued)

Category Endpoints and Clients

Component Cisco Unified IP phones

Diagnostic Tools Available Network configuration, status and phone model information on Settings menu

Information Available In...

End-User Guides
Cisco Unified IP Phone 6900 Series Cisco Unified IP Phone 7900 Series Cisco Unified IP Phone 8900 Series Cisco Unified IP Phone 9900 Series

Cisco Unified IP Phone Administration Guides for Cisco Unified Communications Manager, Troubleshooting and Maintenance chapters
Cisco Unified IP Phone 6900 Series Cisco Unified IP Phone 7900 Series Cisco Unified IP Phone 8900 Series Cisco Unified IP Phone 9900 Series

Error Message Decoder Output Interpreter Troubleshooting TechNotes

Cisco Unified IP Phone 7900 Series

Cisco Unified Personal Communicator

Problem Reporting Tool/Assistant Trace logs

Troubleshooting Guide for Cisco Unified Personal Communicator User Guide for Cisco Unified Personal Communicator for Windows, Troubleshooting Cisco Unified Personal Communicator chapter for Windows version User Guide for Cisco Unified Personal Communicator for Macintosh, Troubleshooting Cisco Unified Personal Communicator chapter for Macintosh version Troubleshooting TechNotes Cisco Aironet 1200 Series Access Point Hardware Installation Guide, Troubleshooting chapters Cisco IOS Software Configuration Guide for Cisco Aironet Access Points, Troubleshooting chapter Troubleshooting TechNotes

Wireless

Cisco Aironet 1200 Access Point

LEDs Carrier busy test Ping/link test

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-20

Chapter 5

Implement Introduction to Troubleshooting

Table 5-3

Contact Center Component Troubleshooting Tools and Documentation (continued)

Category Network Management

Component Cisco Unified Operations Manager

Diagnostic Tools Available Alarms and events appearing in Dashboard displays Phone status tests Synthetic test Node-to-node tests

Information Available In...

User Guide for Cisco Unified Operations Manager, Administering Operations Manager chapter

Communications Infrastructure

Cisco Catalyst 3750 Access Switch

IOS command line tools (such as Show commands and Debug trace utilities)

Catalyst 3750 Switch Software Configuration Guide, Troubleshooting chapter Catalyst 3750 Switch System Message Guide, Catalyst 3750 Switch Debug Commands chapter Error Message Decoder Output Interpreter Troubleshooting Tech Notes Catalyst 6500 Series Switch Installation Guide, Troubleshooting chapter Catalyst 6500 Series Error and System Message Guides Error Message Decoder Output Interpreter Troubleshooting TechNotes Cisco VGD 1T3 Voice Gateway Card Installation Guide, Troubleshooting chapter Cisco VGD 1T3 Voice Gateway Chassis Installation Guide, Troubleshooting chapter Cisco VGD 1T3 Voice Gateway Software Configuration Guide, Using the ROM Monitor chapter

Cisco Catalyst 6506, 6509 Switches

IOS command line tools (such as Show commands and Debug trace utilities)

Cisco VGD-1T3 Voice Gateway

LEDs IOS command line tools (such as Show commands and Debug trace utilities) ROM Monitor

Preparing Your Network for Troubleshooting and Recovery

Before your network becomes operational, you can take several proactive steps to make troubleshooting easier, including:

Produce network topology diagrams to help you isolate potential sources of problems. Synchronize the date and time on all servers. Set trace/logging levels on key devices so that diagnostic information is available when problems occur. Create IVR flowcharts that illustrate how calls are routed between agents and sites.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-21

Chapter 5 Introduction to Troubleshooting

Implement

Network Topology Diagrams

One of the first lines of defense is possessing current topology information. One of the most important pieces of topology information is a detailed network diagram (usually created using Microsoft Visio or a similar application). At a minimum, your network topology diagrams should include the following information:

The name assigned to each major device (typically the DNS name) IP addresses for all devices in the network
Addresses for each router, core and access switch Addresses for all telephony and application servers, including the IP address for each server in

a Cisco Unified Communications Manager cluster

DHCP address range for addresses assigned to endpoints such as IP phones and agent

workstations

Phone extension number ranges assigned to sets of agents or users, as well as the main inbound dial-up numbers for each location. This information is useful in resolving dial plan configuration errors. WAN IP and PSTN links between sites.

This information is critical for isolating which components are involved in a particular problem. For medium- to large-sized networks, you may want to take a layered approach in your diagrams. Create a high-level diagram that illustrates the overall physical layout of your network, including all sites and the links between them. Then for each site create additional diagrams that show detailed addressing information, port numbers and dial plan configurations.

Tip

Frequent adds, changes and upgrades to your network can quickly make these diagrams out-of-date. Inaccurate diagrams slow down the troubleshooting process and may lead to misdiagnosing the problem. Remember to keep these diagrams as current as possible. Figure 5-1 shows a typical high-level topology diagram for a two sites in a contact center network. Agents accepting calls from customers are located in a central distribution site in Kansas, while the equipment supporting interactive voice is located in a data center in San Francisco.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-22

Chapter 5

Implement Introduction to Troubleshooting

Figure 5-1

Contact Center Network Topology Diagram Example

DATA CENTER San Francisco PSTN

PSTN PSTN

Frame Relay WAN PSTN PSTN

CVP CC Server CC1 10.12.15.50

CVP CC Server CC2 10.12.15.51

CVP Web Server WS1 10.12.15.54

CVP Media Server MS1 10.12.15.55

Cisco Content Switch CCS2 10.12.15.101

8 ISDN PRIs

8 ISDN PRIs Roggers RGRB 10.12.15.26 RGRBH 10.12.15.27

AS5400 H.323 Gateways

HGW1 10.12.13.101

HGW2 10.12.13.102

WAN Router 2 10.12.13.98

WAN Router 1 10.12.15.98

Peripheral Gateways PG1B 10.12.15.34 PG1BH 10.12.15.35

Skill Group #301 308 agents GW1 10.12.13.99 GW2 10.12.13.100 IPC Operations Manager IPM1 10.12.15.56 VXML/H.323 Gateways

Access Switch 10.12.15.115

Peripheral Gateways PG2B 10.12.15.36 PG2BH 10.12.15.37

VoIP Monitor VM1 10.12.13.82 Unified Contact Center Peripheral Gateways PG1 10.12.13.34 PG2 10.12.13.35 Cisco Agent Desktop Servers CAD1 10.12.13.210 CAD2 10.12.13.211

Unified CM First Node 4 10.12.13.2

Peripheral Gateways PG3B 10.12.15.38 PG3BH 10.12.15.39

Skill Group #302 47 agents Access Switch 10.12.13.114

Unified CM Sub Node 5 10.12.15.3 Gatekeeper GK1 10.12.15.100 Unified CM Sub Node 6 10.12.15.4 Gatekeeper GK2 10.12.15.101

Music On Hold Server MOH1 10.12.15.5 Active Directory Server AD1 10.12.15.58

Skill Group #303 145 agents

Unified CM Sub Node 1 10.12.13.2

Unified CM Sub Node 2 10.12.13.4

Unified CM TFTP Server 3 10.12.13.3

CENTRAL DISTRIBUTION SITE Kansas

Synchronizing Server Date and Time

The best resources for diagnosing problems within your network are the debug and trace log files produced by individual Cisco devices. Tracing can be enabled on multiple devices and the log file output compared to isolate problems. In order to correlate messages for the same activity in different log files, you must compare the message timestamps and the source device MAC and IP addresses (there is no universal call ID value shared between Cisco devices). You should synchronize every device to the same date and time source so that the timestamps match. To accomplish this synchronization, set each device to obtain its date and time from the same Network Time Protocol (NTP) source. For Cisco IOS-based devices (switches, routers or voice gateways), you can configure each device to act as a NTP client and periodically poll a master NTP source using the following command: ntp server ip-address [version number] [key keyid] [source interface] [prefer] Additional IOS commands are available to establish a device as a NTP peer (operating as the master source for other devices), as well as setting up NTP broadcasting instead of polling. See the Cisco IOS Configuration Fundamentals Command Reference for details on these IOS commands.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-23

231730

Chapter 5 Introduction to Troubleshooting

Implement

Recommended Trace/Logging Settings

In order to have diagnostic information available when you begin to research problems, you must configure devices in your network to capture signaling, processing and other activity in log files.

Cisco Unified Communications Manager Trace Settings

Trace settings for Cisco Unified Communications Manager servers are maintained using the Cisco Communications Manager Serviceability graphical interface. There are two ways to set trace logging levels for Unified Communications Manager services:

Customize trace levels for individual parameters: This approach offers a high-degree of control and flexibility over the trace output. However, in order to use this approach you should understand not only the significance of each parameter, but also the impact of tracing on Unified Communications Manager server performance. For example, setting trace levels to Error has a minimal impact to CPU cycles while leaving the Detail level set for long periods of time may impact call processing. For instructions on setting individual trace levels, see the Cisco Unified Serviceability Administration Guide, Configuring Trace chapter. Apply predefined trace levels: This approach allows you to quickly enable and disable tracing for each Unified Communications Manager service based on predefined levels. You can also use these default troubleshooting trace settings in combination with customized settings to temporarily override the your custom settings. For instructions on using the Troubleshooting Trace Settings option in the Cisco Unified Communications Manager Serviceability interface, see the Cisco Unified Serviceability Administration Guide, Configuring Troubleshooting Trace Setting Configuration chapter.

Debug Trace Settings for Unified IP IVR System

If you encounter any problems with the Unified CCX platform and Unified IP IVR system, activate the following debug trace settings to generate debug logs:

For Unified CCX platform issues: SS_TEL, SS_ICM, and LIB_ICM. For JTAPI Client issues: Enable all Trace Levels and select all debug levels except MISC_DEBUGGING.

However, deactivate the above trace settings if you experience any degradation in performance during heavy load situations.

IVR Flowcharts
In a contact center environment, another tool that can help you troubleshoot call processing problems is a flowchart that traces the call routing process based on the interactive voice response (IVR) menu choices that callers make. Figure 5-2 traces the processing of an incoming call received by a central distribution site. The call receives a voice treatment prompting the call to select between three menu options or hold for an agent. The flowchart indicates which set of agents receives the call based on the menu option selected, as well as describing the capacity (number of agents) in the particular skill set group. For calls received during high traffic periods, with more than 20 calls queued up for agents, the flowchart indicates the announcements played to the callers and how the calls are routed. This type of flowchart is useful for troubleshooting problems reported by external users (customers). While Figure 5-2 shows a fairly simple example, where calls stay within a particular site, some contact center applications may

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-24

Chapter 5

Implement Conduct User Acceptance Test

overflow calls to other sites. In those cases, the overflow calls may traverse an IP WAN to a secondary site and may be handled by additional devices. In situations like that, you need to also view a network topology diagram for the secondary site to trace the call processing.
Figure 5-2 Interactive Voice Response Flowchart Example
Main Menu "Thank you for calling Siren Supplies & Stores' Distribution Center. Please select from the following services: Press 1 if you wish to place a new order from our catalog Press 2 if you want to check on the status of an order Press 3 for our Speciality Items Catalog Or stay on the line and someone will assist you shortly " High Call Volume Announcement "We are experiencing heavier than normal call volumes. There are" << insert number of calls from real time feed >> "ahead of you in this queue. You may wish to call back later for faster service."

Incoming call to 785-555-0135 or 888-555-0195 routed to Kansas central distribution site

Caller hears Main Menu

Caller presses 1? No

Yes

Call routed to queue for agents with skill set #301 in Kansas site 308 Agents

Agent available?

Yes

Call answered by agent

No No More than 20 calls in queue?

Yes

Caller hears High Call Volume Announcement

Yes Caller presses 2? No

Call routed to queue for agents with skill set #302 in Kansas site 47 Agents

Agent available?

Yes

Call answered by agent

No No More than 20 calls in queue? Yes Caller hears High Call Volume Announcement

Caller presses 3?

Yes

Call routed to queue for agents with skill set #303 in Kansas site 145 Agents

Agent available?

Yes

Call answered by agent

No No

No More than 20 calls in queue? Yes Caller hears High Call Volume Announcement

Caller does not choose an option and 5 second timer expires

Conduct User Acceptance Test

After the components are configured and integrated with other Cisco contact center applications, the field engineer prepares the system for the user acceptance test. Test scripts are run and compared against expected results. Any variability in network performance is noted and addressed before the user acceptance test. Testing the customer solution involves the following tasks:

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-25

Chapter 5 Additional Sites and Services

Implement

Determine the user acceptance test parameters and deliverables and record these in the user acceptance test plan. Conduct a prelaunch testUsing an incremental approach, test the solution against the system design in a low-risk environment with limited users. If the system is stable, the rollout pace is increased until the full implementation is operational. The customer signs the Ready-for-Use Acceptance Letter acknowledging that the acceptance test yielded satisfactory results.

Train End Users

The final stage of the Implement phase is to help ensure that the customers system administration team and end users are trained to take over management of the new system. Cisco Systems offers several training and certification programs for customers to maximize the usage of their newly adopted systems. See the Training Library for more information on Cisco training websites and videos on demand (VODs).

Additional Sites and Services

Steps to Success is a Cisco methodology that outlines the tasks required to complete a successful customer engagement. Registered users can visit the Steps to Success resource site for Cisco Unified Communications process flows. Cisco Unified Communications Services is a Cisco service offering that provides engineering expertise and best practices.

Registered users can visit the Cisco Unified Communications Services Nonregistered users can visit the Cisco IP Communications Services site.

partner site.

Cisco Unified Communications System for Contact Center Release 8.5(1)

5-26