Day 5 - MSP Bootcamp Training 201
Day 5 - MSP Bootcamp Training 201
Day 5 - MSP Bootcamp Training 201
2 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Traffic Forwarding::Troubleshooting
Common Issues
Cause:
● No Connectivity: ISP Down, Client not connected to the network, Specific Zscaler DC unavailable
● DNS Resolver: Unable to resolve zscaler gateway or PAC server. Incorrect PAC file Or GRE/IPsec Tunnel Interface status and connectivity
● Zscaler Client Connector in Error State Or Incorrect firewall rules: Internal firewall blocking client outbound connections.
Solution:
Step 1: Basic Network Connectivity Check
First, make sure that the client is connected to the network. Perform several tests for example, Ipconfig on client machine, Ping default gateway & Ping external IP address (8.8.8.8)
Step 4: Verify GRE/IPsec Interface status and connectivity (Only if Tunnel Forwarding)
Ping the Zscaler internal tunnel IP address to validate if the tunnel is up and routing is correct. Verify IP SLA functionality and track uptime to validate the tunnel stability and interface flapping.
In case of IPsec VPN Tunnels, check if the VPN tunnel status is active & To validate that VPN is passing traffic in both directions, Check the SA is active or up.
Cause:
● Network ACLs blocking access to specific destination
● Destination Webmaster blacklist
Solution:
Workaround:
In almost all cases of destination web server blacklisting, routing traffic away from the impacted Zscaler node or DC through PAC will be the quickest and most effective solution .
5 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case3: Routing to Incorrect ZEN
Problem:
Zscaler routing users to far off Zscaler node than the closest Zscaler DC. For instance: "Why do I get sent to LAX1 when I'm in Atlanta?"
Cause:
● Zscaler uses MaxMind's GeoIP database, MaxMind coordinates are wrong
● Primary ZEN connection timeout, kicking it to the secondary ZEN.
● Customer's geographically correct ZEN is within the subcloud that customer is trying to use.
Solution:
6 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case4: Slowness accessing websites through ZIA
Problem:
User is connected to Zscaler Service but the website are loading slow through Zscaler.
Cause:
● If your users are reporting performance issues and slow downloads when using the Zscaler Internet Access (ZIA) there could be more than one reason to slowness, perform the steps to
identify the cause of the issue and troubleshoot it.
Troubleshooting Steps:
Step 1: Identify the Scope
1.) Identify if the problem is reported by a single user or multiple users. A single user incident could be specific to local user network and you may need to work with that user.
2.) Also, Validate if the issue is seen with some specific destination, regional websites or all web pages.
7 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
ZCC::Troubleshooting Common Issues
By the end of the module, you will be able to use the Zscaler Admin
Portal GUI to perform related support and administrative tasks.
● Use the ZIA Admin Graphical User Interface.
● Describe the different functionalities of Admin portal.
● Explain how Zscaler protects user traffic using ZURLDB, Threat Prevention,
PageRisk, and Sandbox etc.
● Locate and create SSL/URL/Cloud-App/File-type policies.
● Locate and create DLP policies.
● Locate and create Firewall/DNS-Control policies.
● Locate and create various security policies.
● Recognize the various location settings like IP-surrogacy and implement it.
● Describe authentication bypass and apply it
9 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case1: Captive Portal Error
Problem:
Zscaler Client Connector shows “Captive Portal Detected Error”
Cause:
● Captive portal feature is a software implementation that blocks clients from accessing the network unless user verification is completed. A very common example is accessing internet at airport,
coffee house or Hotel where necessary user input is needed before granting access to internet.
● Zscaler detects captive portal in two ways, 1.) Reaches out to internet resource (http://gateway.zscloud.net/generate_204) and expects a HTTP response code of 204. If it gets a response code of
anything else apart from 204 it will error out with captive portal error. 2.) Download the default PAC file and parses the content of the PAC file (http://pac.zscloud.net/proxy.pac) and parses the
content of the file. If the contents are not equal to the Zscaler PAC file, then there is a captive portal.
Solution:
Step 1: Check the ZCC logs
1.) Collect Z-App logs and look for ZSATunnel.log or Navigate to C:/Program Data/Zscaler and find the latest ZSATunnel.log
2.) Search for keyword “detectCaptive” something like below should show up
3.) As we can see in the above screenshot the response is 302 instead of therefore the captive portal is detected.
10 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case2: Network Error
Problem:
Zscaler Client Connector shows “Network Error”
Cause:
This error occurs when the Zscaler Client Connector is unable to connect to the Zscaler cloud. Below are some of the possible reasons for connectivity issues between user machine and Zscaler mobile
server that can cause this error on Zscaler Client Connector:
● Host not found (i.e. DNS failure), When the DNS resolution to mobile.<cloudname>.net fails
Sample Tray Logs: 2019-06-24 08:31:42.667324 #NORMAL #ERROR : Error checking updates: {"error":-8,"errorMessage":"Host not found. mobile.zscloud.net","response":"","success":"false"}
● Connection reset by peer, Connectivity from the Client PC and Mobile Server has been intercepted.
Sample Tray Logs: 2019-06-21 13:57:57.271950 #NORMAL #ERROR : Error checking updates: {"error":-8,"errorMessage":"Connection reset by peer. ","response":"1.4.3.1","success":"false"}
● No route to host, Zscaler Couldn’t find a route to mobile.<cloudname>.net in the routing table .
Sample Tray Logs: 2019-07-03 01:19:54.568124 #NORMAL #ERROR : Error checking updates: {"error":-8,"errorMessage":"Net Exception. No route to host","response":"","success":"false"}
● Network is unreachable, If Zscaler Client Connector is unable to reach mobile.<cloudname>.net, you get this error.
Sample Tray Logs: 2019-06-27 06:38:30.554731 #NORMAL #INFO : Keep Alive Response: {"error":-8,"errorMessage":"Net Exception. Network is unreachable","success":"false"}
● Certificate validation error,Traffic to mobile.<cloudname>.net, shouldn’t be intercepted. You get this error if an intermediate device is performing SSL Decryption.
Sample Tray Logs: 2019-06-27 06:38:30.554731 #NORMAL #INFO : Keep Alive Response: {"error":-8,"errorMessage":"Net Exception. Network is unreachable","success":"false"}
Solution:
1.) Zscaler strongly recommends that the Zscaler Client Connector have unrestricted outbound access to the Internet on port 443, to ensure access to all Zscaler nodes as our infrastructure evolves and
expands.
2.) Click Retry to fix the issue, if retry doesn’t fix the issue. Please contact Zscaler Support.
11 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case3: Driver Error
Problem:
Zscaler User sees “Driver error” on Zscaler Client Connector, repair option does not help
Cause:
Driver Error issue occurs when the files are corrupted. Uninstalling and reinstalling the Zscaler Client Connector, without rebooting the machine after uninstallation may result in Driver Error on the
Zscaler Client Connector.
Solution:
1.) In the More window, click Repair. This option is available under the Troubleshoot menu.
2.) If repairing the driver does not fix the issue, administrators can reinstall the driver. There are two ways to do it:
● Using the ZCC MSI package - Reinstall Zscaler Client Connector and force the driver re-installation using the command line option REINSTALLDRIVER=1. For more information, see
Customizing Zscaler Client Connector with Install Options for EXE.
● Perform a fresh install manually
○ Uninstall the Zscaler Client Connector from the user device. For more information, Manually uninstall Zscaler Client Connector on Windows.
○ Delete the mentioned folders at the following location:
C:\Windows\System32\DriverStore\FileRepository
zapprd.inf_xxxxxxx
ztap.inf_xxxxxxx
○ Reinstall Zscaler Client Connector.
12 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case4: FW/AV Error
Problem:
Zscaler Client Connector on shows “Endpoint FW/AV Error”.
Cause:
In most cases, we get this error when the Zscaler Client Connector traffic is blocked a firewall or Antivirus. Zscaler Client Connector also uses carrier-grade NAT range 100.64.0.0/16 as part of internal
health checking and for the ZPA service. Zscaler sends probes on the default NIC on IP address 100.64.0.6 to check Firewall/AV. If the probe is not received by Zscaler Client Connector on the default NIC
it will display “Endpoint FW/AV Error”
Solution:
1.) Investigate if the health check traffic is routed to VPN Adapter . To achieve this, exclude the IP from the VPN range or have a specific route for 100.64.0.6 traffic to the physical interface.
● A command like "Find-NetRoute -RemoteIPAddress 100.64.0.6" should be used to check which interface will be used for the ZApp health check traffic. Make sure that it is Wi-Fi or Ethernet
and not the VPN adapter.
● Note: Use PowerShell to run this command and check for “InterfaceAlias” field:
13 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case5: Update Issue
Problem:
Zscaler Client Connector update not working, trying to update manually might give the error.
Cause:
Resolution:
5. Check ZCC logs, you should see a message with new version that you are expecting to be pushed:
6. Check is “ZSAUpdater” service is not disabled (services.msc) and the service file (ZSAUpdater.exe) exists at: C:\Program Files (x86)\Zscaler\ZSAUpdater:
7. Check the group membership for particular user, make sure that the user is part of the group that this version is enabled for.
14 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Auth: Troubleshooting Common Issues
By the end of the module, you will be able to use the Zscaler Admin
Portal GUI to perform related support and administrative tasks.
● Use the ZIA Admin Graphical User Interface.
● Describe the different functionalities of Admin portal.
● Explain how Zscaler protects user traffic using ZURLDB, Threat Prevention,
PageRisk, and Sandbox etc.
● Locate and create SSL/URL/Cloud-App/File-type policies.
● Locate and create DLP policies.
● Locate and create Firewall/DNS-Control policies.
● Locate and create various security policies.
● Recognize the various location settings like IP-surrogacy and implement it.
● Describe authentication bypass and apply it
16 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case1: Internal Error
Problem:
After Submitting credentials to IdP, User authentication fails with an internal error. For instance, “Why did my authentication fail with the below error, despite providing valid credentials?"
Cause:
● The domain is not provisioned on the Zscaler instance. For example, User provisioned in OKTA with domain hnaseer1.zscloud.net is not provisioned in Zscaler tenant.
Solution:
17 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case2: Authentication Failure – Connection to IDP
Problem:
Users getting Authentication failure as connection to IdP is failing
Cause:
● One of the reasons could be that the IdP URL/s is going through Zscaler.
Solution:
18 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case3: Authentication Error – 42000
Problem:
Users getting Authentication Error – 42000 after providing valid credentials on the Zscaler Private Access re-authentication prompt.
Cause:
● The user is entering a different username instead of the one provided during initial enrollment.
● The IdP SAML response has a different NameID instead of the one sent during initial enrollment. For instance, If the user has used < Username@huma.com initially to enroll to the Zscaler
Client Connector and if the user uses <Username>@huma1.com to re-authenticate to ZPA (Private Access on Zscaler Client Connector), the user will get this error message.
Solution:
19 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case4: ZCC Authentication Error
Problem:
Users getting Authentication Error, user authentication and Restart Service option does not work. The Authentication Error is also displayed in cases where the authentication timeout has been configured
to 'NEVER' for Zscaler Private Access. Even if users try authenticating with their credentials, authentication fails.
Cause:
● Zscaler Client Connector collects device information and sends it to Zscaler which enables fingerprinting of the device for security and reporting purposes. The fingerprint contains key unique
data from the device, to prevent any possibility of cloning the machine for unauthorized access. Any update in the user's device attributes triggers Zscaler to re-enforce authentication for that
user.
Solution:
1.) First, Look for the following error in the "ZSATunnel_YYYY-MM-DD-hh-mm-ss.xxxxx.log" file, which is located at %programdata%/Zscaler/ location.
ERR zpn_client_authenticate error: BRK_MT_AUTH_SAML_FINGER_PRINT_FAIL
2.) Request the user to log out of the Zscaler Client Connector and log in again to validate the new device fingerprint.
20 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Case5: No Authentication Enforced
Problem:
Authentication not getting enforced when users are browsing HTTPS/SSL websites, Enforce Authentication is enabled on the location. Logs do not display any authenticated user and instead state,
example - special unauthenticated users.
Cause:
● This may be due to the fact that Zscaler is unable to decrypt the transactions in question. For example, Either the SSL Inspection is not enabled for the location in question or Zscaler does not
support the cipher suite that is used by the destination.
Solution:
1.) Enabling SSL Inspection for the affected locations. This ensures that authentication will be enforced for all transactions that can be decrypted by Zscaler, if 'Enforce Authentication' is enabled in
parallel.
2.) Enabling 'Enable IP Surrogate' in the Location Management for the affected locations. This enables mapping between private IP addresses to known users if they have previously authenticated. The
'Idle Time to Disassociation' is configurable in the same options menu.
3.) Using Zscaler Client Connector in order to authenticate all transactions without the need for cookie-based authentication and therefore a required SSL decryption.
21 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Policies::Troubleshooting Common Issues
Diagnosis : Go to Analytics>Web Insights and search for the logs using filters for interesting traffic.
Cause:
1. Another URL Filtering rule matching the traffic: Verify the weblogs to check URL Filtering Policy Name this will confirm
if you are matching the current rule
2. Cloud app matching that traffic: In logs verify if Cloud Application Policy Name is matching this traffic with another
CloudApp rule. (since CloudApp take preference over URL filtering)
3. SSL Policy reason: This will show weather the traffic is SSL inspected or not. A lot of polices depend on traffic being SSL
inspected.
Solution:
23 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
CloudApp Control
Problem : CloudApp policy not working
Diagnosis : Go to Analytics>Web Insights and search for the logs using filters for interesting traffic.
Cause :
1. Cloud app matching that traffic: In logs verify if Cloud Application Policy Name is matching this traffic with another
CloudApp rule.
2. SSL Policy reason: This will show weather the traffic is SSL inspected or not. A lot of policies depend on traffic being SSL
inspected.
3. Target URL part of CloudApp: Work with Zscaler Support to verify if the URL in question belongs to the CloudApp.
Solution:
4. Check the rule order and correct cloud app rule configured.
5. Check for the user group, if there is a group/dept configured.
6. Check SSL inspection (if SSL bypassed the rule will not match)
24 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
SSL Policy
Problem : policy not matching/traffic not inspected
Diagnosis : Go to Analytics>Web Insights and search for the logs using filters for interesting traffic.
Cause :
1. Check SSL Bypass list: URLs added to SSL bypass would be exempted from any policy (unless specified) and would
work as a pass through from Zscaler Security Stack.
2. Check Wildcard Domains: Wildcard domains in SSL bypass may affect your traffic and could be hard to diagnose.
3. Check CDNs in SSL bypass: Some CDNs are added to SSL bypass which may affect another URL as hosting sites
sometimes use common CDNs
Solution:
25 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Break
26 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
HAR traces
27 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
HAR traces
• This is a very important utility to understand the flow of http request response when opening a webpage
or when replicating an issue.
28 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Trust posts
29 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Zscaler Trust Post
• We would recommend you subscribe to https://trust.zscaler.com/ to get real time notification of cloud
incidents.
• Once you receive Zscaler Trust post and your customer is impacted, you can open a support case with
Zscaler to get more details.
30 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
ZIA Quiz
31 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
ZIA Quiz
20 Question Quiz
32 ©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Thank You
©2021 Zscaler, Inc. All rights reserved. ZSCALER CONFIDENTIAL INFORMATION SECURING YOUR DIGITAL TRANSFORMATION
Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.
Alternative Proxies: