This document discusses an issue with Daylight Saving Time (DST) in Citrix Provisioning Server environments that use Standard Image Mode for virtual disks (vDisks). When vDisks are created during DST and then used after DST ends, there is a 1 hour time difference that prevents Group Policies and Kerberos authentication from functioning properly. This can negatively impact the configuration and security of servers booting from those vDisks until the time difference is resolved. The document examines the root causes and provides potential workarounds and solutions to address the problem.
This document discusses an issue with Daylight Saving Time (DST) in Citrix Provisioning Server environments that use Standard Image Mode for virtual disks (vDisks). When vDisks are created during DST and then used after DST ends, there is a 1 hour time difference that prevents Group Policies and Kerberos authentication from functioning properly. This can negatively impact the configuration and security of servers booting from those vDisks until the time difference is resolved. The document examines the root causes and provides potential workarounds and solutions to address the problem.
This document discusses an issue with Daylight Saving Time (DST) in Citrix Provisioning Server environments that use Standard Image Mode for virtual disks (vDisks). When vDisks are created during DST and then used after DST ends, there is a 1 hour time difference that prevents Group Policies and Kerberos authentication from functioning properly. This can negatively impact the configuration and security of servers booting from those vDisks until the time difference is resolved. The document examines the root causes and provides potential workarounds and solutions to address the problem.
This document discusses an issue with Daylight Saving Time (DST) in Citrix Provisioning Server environments that use Standard Image Mode for virtual disks (vDisks). When vDisks are created during DST and then used after DST ends, there is a 1 hour time difference that prevents Group Policies and Kerberos authentication from functioning properly. This can negatively impact the configuration and security of servers booting from those vDisks until the time difference is resolved. The document examines the root causes and provides potential workarounds and solutions to address the problem.
Fixing the Daylight Saving Time Issue in Citrix Provisioning Server Environments v11.docx
Fixing the Daylight Saving Time Issue in Citrix Provisioning Server Environments
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 2 of 14 04.02.2009
Version 0.9 Authors Benjamin Haberkorn Date of Change 27.01.2009 Changes Initial Version
Version 1.0 Authors Andreas Huther Date of Change 27.01.2009 Changes Quality Assurance
Version 1.1 Authors Benjamin Haberkorn Date of Change 31.01.2009 Changes Additional Information
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 3 of 14 04.02.2009 1 Table of Contents 1 Table of Contents ...................................................................................................................... 3 2 Introduction .............................................................................................................................. 4 2.1 What exactly is the problem with Daylight Saving Time (DST) in Citrix Provisioning Server- Environments? ........................................................................................................................... 4 2.2 How is your Citrix PVS environment affected by this? .................................................................. 8 2.3 Why havent I observed this in my PVS environment? .................................................................. 9 2.4 Who's to blame? ....................................................................................................................... 9 3 Solutions to the Problem .......................................................................................................... 10 3.1 Possible Workarounds .............................................................................................................. 10 3.1.1 Boot the vDisk in Private Mode ................................................................................... 10 3.1.2 Schedule a script to change vDisks automatically ......................................................... 10 3.2 Our recommended solution...................................................................................................... 11 4 Disclaimer ................................................................................................................................ 12 5 About the authors ................................................................................................................... 13 6 Resources ................................................................................................................................ 14
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 4 of 14 04.02.2009 2 Introduction This whitepaper discusses a particular issue which occurs in Citrix Provisioning Server-environments where Standard Image Mode is used for vDisks. The target audience of this document should be familiar with the basic concepts of Citrix Provisioning Server as well common Microsoft Technologies e.g. Active Directory. 2.1 What exactly is the problem with Daylight Saving Time (DST) in Citrix Provisioning Server-Environments? Before going into details on what trouble the timezone issue might cause, let's have a look at the characterics of the components involved. For the sake of this example, we're using servers which are located in the "W. Europe Standard Time"-timezone, although the issue should be exactly the same for any other timezone that uses Daylight Saving Time (DST). If the country you live in doesn't use DST (Daylight Saving Time), the issue described in this whitepaper doesn't apply to you.
Fact #1: TimeZone Settings are saved in the Windows registry. As you can see on the following diagram, Timezone information is maintained in the Windows Registry. To be spefific, these settings are located under "HKLM\System\CurrentControlSet\ Control\TimeZoneInformation". The "ActiveTimeBias"-value represents the current (active) timezone bias. In our example (W. Europe Standard Time), the Value "0xffffff88" means that Windows thinks it's currently Daylight Saving Time. The other value "0xffffffc4" is basically the baseline (Wintertime).
Diagram 1 Registry Timezone - Summer Time is active
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 5 of 14 04.02.2009
Diagram 2 Registry Timezone - Winter Time is active
Fact #2: Standard Image Mode makes data on vDisks non-persistent Citrix Provisioning Server helps making sure that server builds are 100% consistent by leveraging the "Standard Image Mode". This basically means that all changes that are being made to a Target Device which is booted off a vDisk, are dismissed when the Target Device gets rebooted. We all love Citrix PVS for this, especially in Citrix XenApp-Environments where it helps customers to: Finally achieve 100% server consistency Deliver workloads faster & more flexible Shorten Rollout cycles Roll back to a previous vDisk in just minutes ... and much more.
But, as we all know, nobody's perfect. So, in some cases the lack of persistant data becomes a problem, so you have to find a workaround. Most of the time, these problems that result from the non-persistent vDisks are related to hard-coded Computernames etc. So to work around that, you basically have to generate personalized configuration files & registry entries that reflect the unique identiy of the Target Device. For the most part, this isn't necessary or Citrix Provisioning Server takes care of it (e.g. UUID, Active Directory computer password). One popular example of software which needs some adjustments to run in Citrix Provisioning Server vDisks are Citrix XenApp configuration files which contain hard-coded computer names. For this particular issue, the Citrix XenApp PrepTool or Wilco van Bragt's script-based solution can be used. Personally, I prefer Wilco's script framework, because it does everything that XenApp PrepTool does plus I easily integrate further sub-scripts to fix other configurations that need to be generated during startup of the vDisk.
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 6 of 14 04.02.2009 Fact #3: Group Policies can only be applied "when the time is right"... Picture the following scenario: - You're using a vDisk which was created during Daylight Saving Time (see Diagram 1) within your XenApp Server Farm. - Now, Wintertime has arrived and you boot your Target Devices from the "old" vDisk which was last modified during Daylight Saving. - You have implemented Group Policies to lock down and configure your XenApp Servers. Maybe you're even using "Group Policy Computer Startup Script"-functionality to launch "XenApp PrepTool" (or any custom script) whenever the XenApp Servers boot.
When the target devices boot from the vDisk, there will be a clock skew of 1 hour between Domain Controller and Client (Target Device). This is due to the wrong Timezone information which is saved in the vDisks's registry. The Target Device's Eventlog will report that it tried to establish a kerberos session but failed to do due to a different time on the server (domain controller).
Diagram 3 Kerberos Error due to Time Sync Error
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 7 of 14 04.02.2009 Just a moment later, the Target Devices now tries to apply the Group Policies from the Domain Controller. This also fails, because the time difference is still too big (1 hour).
Diagram 4 Group Policies cannot be applied due to Time Sync Error After the Group Policy Processing has failed, the Target Device finally synchronized it's time to the domain controller. The Target Device and the Domain Controller can now authenticate again, but it's too late for the Group Policies, they won't get processed until the next cycle (typically: 90 minutes).
Diagram 5 Target Device has finally synchronized Time with Domain Controller
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 8 of 14 04.02.2009 2.2 How is your Citrix PVS environment affected by this? Example 1: You have configured TS Lockdown policies as well as e.g. a central TS Profile Path using Group Policies. Another example would be if you use Citrix Portable Profiles in conjunction with the .ADM files, which also heavily relies on Active Directory Group Policies. What happens if your servers don't have these policies applied and users log on? Users will be logged on with a new/temporary user profile because the TS Profile Path isn't configured. Settings will not be available etc. So basically, your XenApp Server is not properly configured until the gpupdate-process has succeeded. Like previously mentioned, Group Policies are applied every 90 minutes (by default). So if you either wait for 90 minutes or do a "gpupdate /force", the group policies WILL be applied. .. and many more potential pitfalls.
Example 2: You use XenApp PrepTool or a custom program/script to generate unique config files for XenApp and subsequently start the Citrix Services. You have chosen to invoke this script using Group Policies "Computer Startup Script". What happens when your servers can't apply the Group Policies during boot-time and therefore can't run the script/program? The XenApp Servers will be unreachable for users because XenApp PrepTool or your custom script (and therefore e.g. Citrix IMA) did not launch. In a worst-case-scenario, when your servers do their typical reboot cycle, this could take a significant part of your server farm down (depening on how many servers reboot at the same time) and you wouldn't know why. Nothing would have changed, except that the clock would been adjusted. Group Policies Computer Startup Scripts are ONLY launched during the initial gpupdate (when the system boots). If the initial gpupdate failed and the regular 90 minute intervall kicks in, it will refresh the group policy settings but NOT the startup script. As a result, you need to manually trigger your startup script. Which Platforms are affected? The Timezone Issue occurs with every Citrix Provisioning Server Production Version. As for the Operating System on the Target Devices, we have tested Windows Server 2003 and Windows Server 2008 and they behaved exactly the same. Also, it doesn't matter wether you have a 2003 or 2008-based AD.
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 9 of 14 04.02.2009 2.3 Why havent I observed this in my PVS environment? There are various reasons why you haven't yet observed the issue which is descibed in this paper. Which ever applies to do, the issue is present in ALL Citrix Provisioning Environments. Reason #1: Your GPOs are cached within the vDisk Symptoms: Your vDisk will boot with wrong timezone bias data, but will still process GPOs because they are locally cached. This is very tricky, because it's a fortunate coincidence that the GPOs were cached for your particular environment. Reason #2: Your Target Device is a Virtual Machine Symptoms: Although the vDisk contains the wrong timezone setting, the Target Device gets the current time from the Hypervisor. I've seen this e.g. in HyperV-environments. You basically have two sources for the Target Device to get its time: the Hypervisor (using Integration Tools) and the standard NTP-source (Domain Controller). The difference between the two is that the Hypervisor's clock synchronization takes place earlier in the boot process and therefore group policies can be successfully applied. Reason #3: You don't use Group Policies. Well, if you don't use Group Policies at all, you can forget about the timezone issue for now. 2.4 Who's to blame? As outlined before, Citrix Provisioning Server Standard Image-Mode and Windows Operating Systems do work very well together. But in some cases, you can see that the Operating System existed before Citrix Provisioning Server was introduced. In this particular case, Windows requires that a computer can maintain and remember its current Timezone information in the registry. But this alone wouldn't be a "deal breaker": one could still rely on the automatic NTP time synchronization that is taking place when a Domain-based Windows computer boots up. The real problem is rather the order in which things are done: Windows first tries to get group policies (knowing that it will fail if the time is not synchronized) and THEN synchronize the time with the domain controller. I think this basic behaviour of group policy processing should be changed. Wouldn't it make sense to first get the correct time before trying to get Group Policies? ;-) Citrix Provisioning Server does exactly what you expect the product to do, so it's a typical case of "works as designed". Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 10 of 14 04.02.2009 3 Solutions to the Problem Before we get to the solution that we chose, I'd like to point out several other workarounds/solutions which we didn't look into. 3.1 Possible Workarounds As previously noted, the basic issue is that the vDisk has a different timezone setting than the Domain Controller. This could theoretically be fixed using one of the following methods. 3.1.1 Boot the vDisk in Private Mode If you would boot the vDisk in Private Mode after the Daylight Saving Time has taken place, the correct timezone would be rememberd within the vDisk. Disadvantages: You need to act with every future Daylight Saving Time. Even if you make a note of this task in your calendar, chances are that you'll forget it. You'd need to create a copy of the existing vDisk prior to the Daylight Saving Time, because the vDisk used by the Target Devices is locked. You would then need to prepare the new vDisk with the new Timezone, then assign the new vDisk and finally reboot the Target Devices after the Daylight Saving Time. Not very Admin-friendly. 3.1.2 Schedule a script to change vDisks automatically I am certainly no programing expert, but it should be possible to automate the following using Citrix Provisioning Server Management Interface (SDK): Create a copy of vDisk that is being used Map the vDisk as a local drive Open the registry hive on the mapped vDisk and alter the Timezone Registry-Key appropriately Unmap the vDisk Assign the new vDisk to the appropriate Target Devices
These tasks, of course, would then have to run each time a Daylight Saving Time occurs. Like I said, it seems possible but it has also many variables and therefore I chose a simpler, less complex path.
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 11 of 14 04.02.2009 3.2 Our recommended solution The solution we chose is pretty simple: We created a small Windows Service which executes "gpupdate /force" at boot time. This Service runs in the LocalSystem security context. Gpupdate /force uses the computer machine account to authenticate against Active Directory. Instead of launching e.g. XenApp PrepTool (or any other comparable script) directly via Group Policies, we launch it using the Windows Service. This is to make sure that XenApp PrepTool (or any other comparable script) is launched after gpupdate /force has run and that its successful execution is independent from Group Policies. So the Windows Service would basically do this: o Execute gpupdate /force o Execute XenApp PrepTool (or any other comparable script) The Windows Service needs to be dependent on "W32Time" as well as "Netlogon". This makes sure that the time has been synchronized (W32Time is started) and group policies can be retrieved (Netlogon is started).
Tools required: You need to have a program or script which executes "gpupdate /force" as well as e.g. "XenApp Preptool". o You can either program a "real" executable or try compiling a CMD-Script as an .EXE using Freeware Tools (see [3.]). Using a native CMD-script should also work, so compiling it to an .EXE is an optional step. You need "Srvany.exe" & "Instsrv.exe" to register the executable or script you created as a Windows Service. Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 12 of 14 04.02.2009 4 Disclaimer The information in this Whitepaper is provided "as is" without warranty of any kind. net on Netzwerktechnologien Online GmbH hereby disclaims all warranties and conditions with regards to this information. Use the information on your own risk.
Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 13 of 14 04.02.2009 5 About the authors Andreas Huther, Sascha Zimmer and Benjamin Haberkorn have been working in the Citrix industry, designing and implementing Citrix Infrastructures since the year 2000. In early 2008, they joined "net on Netzwerktechnologien Online GmbH", a Citrix Partner which is based in Mainz, Germany. Fixing the Daylight Saving Time issue in Citrix PVS Environments
Version 1.2 Page 14 of 14 04.02.2009 6 Resources [1.] Citrix XenApp Prep http://support.citrix.com/article/CTX113939 [2.] Running Citrix Presentation Server with Ardence OS Streaming http://sbc.vanbragt.net/mambo/index.php?option=com_content&task=view&id=406&Item id=155 [3.] Bat to Exe Converter http://www.f2ko.de/Deutsch/b2e/download.php [4.] Windows 2003 Resource Kit Tools http://www.microsoft.com/downloads/details.aspx?familyid=9d467a69-57ff-4ae7-96ee- b18c4790cffd&displaylang=en