One of those lesser known Terminal Server (Citrix) concepts which is often overlooked.
What is it and what does it do?! I learned about the shadow key in more detail about 6 years ago and since then I noticed that not all, or very few to be honest, system admins know about its existence, or that they do know, but don’t know how to treat it… does that make any sense? First I’ll briefly try and explain the use, advantages and disadvantages. Also, there have been made a few small changes to the overall architecture with the introduction of 64 bit systems and Server 2008R2 which I’ll address later on
We probably all know, or should know, about the ‘change user install’ command or the ‘addremove programs’ feature used on Terminal Server systems to install applications for multiple user access. There are basically two modes a Terminal Server can be in: Install mode, as explained above, or execute mode in which users connect to the Terminal Server to launch applications, desktops and so on… This is the mode the Terminal Server is in 99% of the time. Both these modes have a few underlying processes which not everyone is aware of.
When an application gets installed, in most cases it needs to create and run certain registry keys during the installation process. These ‘keys’ hold initial application configuration values and are written to the HKCUSoftware hive of the user currently logged on installing the application, which will be the Administrator configuring the Terminal Server for production use. At a later stage this could create a problem, as the users who connect to the Terminal Server to start these applications will also need this initial configuration information, and if that info is only written to the HKCUSoftware hive of the Administrator who installed the application, how does this info get copied to the HKCUSoftware hive of the users logging on? This is where the shadow key comes in. let’s start with a short general overview first.
Application installation:
As mentioned earlier, when an application gets installed on a Terminal server, one way or the other, install mode kicks in, this is what happens:
During the installation process, when an application writes registry information to the HKCUSoftware hive, these writes also get copied to the HKEY_LOCAL_MACHINESoftwareMicrosoftWindows NTCurrentVersionTerminal ServerInstallSoftware key, the so called shadow key! It does this separately for every application that gets installed and every key that gets modified. This is normal behavior in 32 bit environments like we were used to a few years ago. I’m using this as an example just to make clear how the process works, more on 64 bit later on (some small changes).
Each time a write to the shadow key takes place, as mentioned above, the system updates a timestamp on the server called ‘LatestRegistryKey’ which can be found under: HKLMSOFTWAREMicrosoftWindows NTCurrentVersionTerminal ServerInstallIniFile Times. The value this record holds are the seconds since January 1st 1970.
In fact, every key has its own time stamp that shows the last time it was updated and or modified. So for every write on every key to the shadow key location a separate time stamp on each individual key gets updated.
When the installation is done, and assuming all the other configuration stuff is finished as well, the server is put back into execute mode, manually or automatically, depends on the installation method used. Next, users can start connection to the Terminal Server and launch the application. There now needs to be some kind of way to get the shadow key information into the users HKCUSoftware registry key, here goes…
User logon process:
What happens during logon: A process called userinit.exe runs, in Terminal Server environments this executable compares the hidden timestamp of the servers ‘IniFile Times’ key to a timestamp in the users HKCUSoftware hive that holds the last synchronization time: HKCUSoftwareMicrosoftWindows NTCurrentVersionTerminal ServerLastUserIniSyncTime.
This way the system can figure out which one is newer, the settings in the users HKCUSoftware hive or the settings on in servers HKLM IniFile Times file. If the timestamp in IniFile Times key is newer the systems knows that new software must be installed or that an update has taken place. If the users HKCUSoftware hive information is newer then nothing happens.
If the system detects that the IniFile Times key is newer then userinit.exe will enumerate all the keys from the shadow key area and compare each time stamp to the corresponding key in the users HKCUSoftware hive. Remember that every registry key has its own time stamp. Every time the system finds that a shadow key is newer it will delete the corresponding users HKCUSoftware key and replace it with the most recent one. If it can’t find an corresponding key in the users HKCUSoftware hive, for example the first time you log on and your current profile is still fresh, or an new application is installed, it will copy over all the key’s it can’t find and will compare them the next time you log on, the way it all starts to begin with.
This is how all the registry information that gets written into the Administrator HKCUSoftware hive during installation of one or more applications (in this example) when setting up the Terminal Server, will eventually end up in de user profile logging on to the system. I won’t go into any further detail regarding user profiles, the different types, the way they are loaded etc… I assume we all know the basics ;-)
But what if…
So, no worries right? Hold on… What happens if you add a few new servers to your farm for extra capacity with the same applications installed and published? Let’s say the original farm was set up and configured a year ago. These extra servers are all ‘new’ so the applications installed will also be ‘new’ and will generate ‘new’ timestamps. Now imagine the same process as explained above repeating itself with users logging on to those ‘new’ servers accessing the same applications as installed on the ‘old’ servers. Timestamps will again be compared and since these servers, and thus the applications installed on them, are new(er) so will be the timestamps. This will result in deletion of the users application specific configuration key’s which were obtained a year ago when the original farm was set up.
Something needs to be done. This isn’t a process that can be disabled in any way. When adding new servers, you need to make sure that the shadow key timestamps on these servers are older than the timestamps in the users HKCUSoftware hive, which they got from the ‘old(er)’ farm servers a year ago (in this example). There are three ways this can done:
1. Use Sysprep and “image” new servers. Inherit the registry timestamps from original build.
2. Write to HKCUSoftware in Install mode with the system clock set in the past.
3. Remove / delete all shadow keys that could potentially overwrite user preferences
Each one has its own drawbacks. I won’t go into any details regarding the first two, let’s just say that number three is the preferred one to work with. This does mean that you’ll have to find an alternative way to get the application specific configuration information into the users HKCUSoftware hive when they log on. So monitor what happens to the HKCUSoftware hive when installing the application and use some sort of login script, user preferences, app sense or whatever, to load the configuration information into the users HKCUSoftware hive during log on. Ok, that the general idea on how the shadow key works.
As mentioned earlier there are a few exceptions regarding 64 bit systems and windows Server 2008R2. The idea and infrastructure behind the shadow key is almost identical, the process itself is just slightly off. Let’s first start with 64 bit systems.
64 bit
It is possible to install and run 32 bit applications on 64 bit systems, as we all know. This is made possible by the WoW64, or, Windows-on-Windows 64 layer. When a 32 bit application gets installed on a 64 bit system the WoW64 layer redirects the shadow key location from: HKLMSOFTWAREMicrosoftWindows NTCurrentVersionTerminal ServerInstall to: HKLMSOFTWAREWow6432NodeMicrosoftWindows NTCurrentVersionTerminal ServerInstall. This means that when the Administrator installs an 32 bit application on an 64 bit system the writes to the Administrators HKCUSoftware hive are echoed copied to the HKLMSOFTWAREWow6432NodeMicrosoftWindows NTCurrentVersionTerminal ServerInstall shadow key location. Same process as before, different (shadow key) location.
When the Administrator installs a 64 bit application on the same system the HKCUSoftware writes are echoed to the 64 bit shadow key location, which is: HKLMSOFTWAREMicrosoftWindows NTCurrentVersionTerminal ServerInstall. In fact the whole shadow key principle is exactly the same like I explained on a 32 bit environment, only now there are two different shadow key locations, provided by the WoW64 layer.
When an application gets installed all the modifications taking place in the HKCUsoftware hive (of the logged on user performing the installation) are copied to the 32 or 64 bit shadow key location within the registry. When users log on to the system (the normal execute mode) the content of both keys gets copied to the users HKCUSoftware hive. When the system encounters a conflict, because both the 32 and 64 bit shadow keys are copied and could contain the same settings, the 32 bit shadow key overrides any conflicting 64 bit shadow key settings. Same here, the overall idea is still the same the process is just a bit different.
Server 2008R2
Since the introduction of Server 2008R2 applications have become TSAWARE. As the name implies, the application is ‘aware’ that it is being installed on a Terminal Server enabled system. This doesn’t mean that all applications installed on an R2 system have this feature, they just started developing applications differently around the release of 2008R2. So, when installing a TSAWARE app nothing gets written to the shadow key right? Right! Newer applications have a special bit enabled (flag) in the header of the executable which tells the system that the application is terminal server aware and doesn’t need to copy anything to the shadow key location, this goes for 32 and 64 bit applications as well as 32 bit applications on a 64 bit system. Shadow key echoing is only enabled if the executable being installed is NOT marked as TSAWARE. Same rules apply for Windows 8 and 2012.
With all the application and operating system virtualization going on these days the whole shadow key concept isn’t always applicable or might function slightly different… it all depends. Fact is, these are the basics you need to know!
Well… That’s about it, my first Blog done!
Please let me know what you think, all comments are welcome!
Bas van Kaam ©
Reference materials used: Citrix.com, Brian Madden.com and Google.com
[twitter-follow screen_name='BasvanKaam']
10 responses to “The Shadow key… what’s it all about?!”
Not bad. Application streaming is also becoming more popular with TA/Citrix XA and also changes this process removing the shadow key.
Totally agree! Although baseline applications like Word, Excel, Acrobat Reader and a few others are often excluded from App-V like solutions or similar. I think it’s important to understand the basics before moving on, that’s why I thought I’d share :-) I did mention application virtualization by the way, just near the bottom. Thank you for your reply.
Good article, I wish I had had it in 2002 when installing a medical pharmacy app, which was suppose to be a single user PC app, I had to find out the hard way, and used a mixture of option 1 and 3 plus additional scrip to restore user app settings, I never did find out where the time stamps were, so if I hadn’t retired in 2010 I would have been able to refine the scripts to make life easier!
Hi Gordon, I can imagine how frustrating that must have been, hopefully I’m still in time to help a few others :-) Thanks for leaving a reply! Gr, Bas.
Excellent explanation on a subject usually quite cryptic ! I read your “about me” and I agree with you : writing down technical knowledge is an excellent way to consolidate it and keep it longer in memory. Thanks for sharing your expertise.
You’re very welcome :-) Thank you for the kind words!
Really liked what you had to say in your post, The Shadow key… whats it all about?! Thanks for the good read!
Wilson.
Nice one!
Nice article, it helps me to understand how the register key worked for applications.
very very helpful.
That’s good to hear, glad I could help!