Troubleshooting Server Application Virtualization 1.0
In this post, I’m going to walk through most of Server Application Virtualization (SAV) with the intention of calling out all the different resources and techniques you can use to troubleshoot problems. I’ll go through installing the sequencer, doing a simple sequencing, installing the agent, and deploying a package and talk about what you can do in each of those phases. I won’t dive deeply into any particular technique but instead will try to give you an overview of all the options available.
I’m doing all of this on Windows 2008 R2 Server virtual machines with SAV v1 (4.7.27.1491) from the SystemCenter 2012 release. I’ll use WS_FTP from Ipswitch as a sample app. For details on how to sequence and deploy it, see our recipe at http://blogs.technet.com/b/serverappv/archive/2012/05/09/server-app-v-recipe-ws-ftp-server.aspx.
Installing the Sequencer
The Sequencer installer is actually a chainer (an executable that orchestrates the install process) wrapped around a few MSI files. To get the most verbose logs from the MSI installers, you can turn on MSI logging by setting the registry value Logging to ‘voicewarmup’ under HKLM\Software\Policies\Microsoft\Windows\Installer before you install. For more details on that technique, see http://support.microsoft.com/kb/314852.
I set this registry value and ran the Sequencer installer with no command line parameters, just clicking through and taking all the default options. I didn’t run into any errors, but if I had, I would have first looked at the chainer’s log, since this summarizes overall setup progress. This is a file called SavSetupChainerLog.txt in your temp directory. (Easy ways to find the temp directory are to open an Explorer window and type ‘%temp%’ in the address bar, or open a command prompt and type ‘cd %temp%’.)
When the chainer runs, it installs an MSI with the SAV Sequencer components, pre-requisite MSIs, and the desktop App-V Sequencer (because the SAV Sequencer is built on top of the desktop App-V one). In this example, the 2005 VC redistributable was the only pre-requisite that needed to be installed. Since the installation was pretty straightforward, the log was, also. Here’s the whole thing:
6/15/2012 11:43:41:753 Setup will perform an Install on the product.
6/15/2012 11:43:55:721 Performing install of product Packages\SavSequencer.msi with command-line INSTALLDIR="C:\Program Files (x86)\Server Application Virtualization Sequencer" REBOOT="ReallySuppress" ARPSYSTEMCOMPONENT="1"
6/15/2012 11:44:4:110 Performing install of product Packages\vcredist2005sp1_x86.exe with command-line Packages\vcredist2005sp1_x86.exe /Q
6/15/2012 11:44:18:218 Performing install of product Packages\Setup.msi with command-line INSTALLDIR="C:\Program Files (x86)\Server Application Virtualization Sequencer" REBOOT="ReallySuppress" ARPSYSTEMCOMPONENT="1"
You can see the command lines passed to each phase of installation, which can help you figure out what might have gone wrong if anything did. Also, if the MSIs return any errors, they’ll show up here so you can tell which phase of the install failed. In the event of a failure, the next step is to look at the MSI logs. The link about verbose MSI logging above has more details on how these files are named, but in short they are the files in the temp directory that start with MSI and end with .txt. They tend to have a lot of information and as a result can be hard to interpret. A good technique is to search for the first instance of the word ‘error’ to get to the approximate area of the log file where you might learn something about what went wrong.
Sequencing the Application
I started the Sequencer, chose Create a New Virtual Application Package, and got to the Prepare Computer page. This page automates some of our best practices and troubleshooting tips.
It’s a great idea to pay attention to warnings and errors you see here. In my example, both System Center Configuration Manager (“SMS Agent Host”) and an antivirus program were running, and either of those could kick off tasks during sequencing that would invalidate the package I produce. Double-clicking on each warning provides a pop-up dialog with more info on what is wrong and how to fix it.
I continued through the Sequencer, specifying the WSFTP installer and a name for the package, to the point where the Sequencer started the installer for me. If I saw any errors up to this point, my best option would have been to look in the Sequencer’s log file, sft-seq-log.txt in the Logs folder under the Sequencer install path. By default on a US English machine that’s C:\Program Files (x86)\Server Application Virtualization Sequencer\Logs. This is a simple text file with the date, time, and a couple other pieces of information in square brackets at the start of each line. For example, here’s a section of the log file where virtualization subsystems are being loaded. Note the warnings about IIS and WebDeploy not being installed—I had to go back and fix these before I could sequence WSFTP).
...
[06/15/2012 14:46:04 VRB ] The 'Performance Counters' subsystem factory was loaded from module 'c:\Program Files (x86)\Server Application Virtualization Sequencer\sav_perf32.dll'.
[06/15/2012 14:46:05 WRN ] The Virtual IIS Subsystem could not be loaded because the IIS WMI Provider is not installed. IIS applications cannot be sequenced (error 4960113B-8004100E).
Corrective actions:
Enable the IIS role.
Enable the IIS Management Scripts and Tools role service (on IIS 7).
Restart the Server Application Virtualization Sequencer.
[06/15/2012 14:46:07 WRN ] The Virtual IIS Subsystem could not be loaded because the Web Deployment Tool is not installed. IIS applications cannot be sequenced.
Corrective actions:
Install Web Deploy.
Restart the Server Application Virtualization Sequencer.
[06/15/2012 14:46:07 VRB ] The 'SSRS' subsystem factory was loaded from module 'c:\Program Files (x86)\Server Application Virtualization Sequencer\sav_ssrs32.dll'.
[06/15/2012 14:46:07 VRB ] The 'Windows Roles and Features' subsystem factory was loaded from module 'c:\Program Files (x86)\Server Application Virtualization Sequencer\sav_roles32.dll'.
...
Once you’ve reached the point where your app’s installer is running, you’re in the same situation you would be in if you were trying to install without SAV. Any techniques you’d use to debug a failed installation of a native application, like looking at MSI log files, are equally valid for debugging failures here.
After I finished with the installer, I went back to the Sequencer and checked the I am finished installing check box, then clicked Next. After the Sequencer collected system changes and generated a package, I got to the package report page:
This is another useful page to look through to proactively find problems with the package you’ve created. For example, I double-clicked Files excluded from package and saw all the files that were modified during packaging time but not included in the package (because of heuristics we use to try to exclude things like temp files from your packages). The Sequencer also saves all of the information on this screen into a file called report.xml that ends up on disk with the rest of the package you create. If you notice a package failure at deployment time, it can be useful to go back and look at this report.xml file to see if you missed anything that would point you at a solution.
After I finished and saved the package, the package artifacts like the SFT file were available on disk for examination. Sometimes looking at these artifacts helps to diagnose other potential problems. A tool like Application Virtualization Explorer is valuable here. AVE allows you to open up the SFT file and see what files and registry settings ended up as part of the package. AVE was originally for desktop App-V packages but works just as well with Server App-V packages, except that the virtual services tab is not functional.
Package Upgrade Notes
If you’re doing a package upgrade in the Sequencer, there are a couple of extra things to consider.
First, if your package had deployment configuration, we have to apply it during upgrade on the Sequencer. If your deployment configuration file has errors in it (like missing credentials for services), you’ll see a failure. We put information about deployment configuration failures during package upgrade in the Application Event Log, not in sft-seq-log.txt, so you will have to open up the Event Viewer to see this set of errors.
Second, a common source of errors on package upgrade is extension point conflicts. As part of setting up the package to be upgraded, we have to deploy it to the machine, which includes registering all of its extension points. If there are any conflicts, e.g. a COM object that’s part of your package but is also present on the native system, this registration fails. You should be able to find more details about these failures in the Sequencer log, sft-seq-log.txt. You can also use AVE to look at the application’s manifest (the appv_manifext.xml file in the root of the package) to see what COM objects, services, WMI providers, etc. we register on the local system.
Installing the Agent
The SAV Agent setup is structured much like the Sequencer setup, with a chainer that orchestrates several MSI installations. The agent chainer log is also called SavSetupChainerLog.txt and will also show up in the temp directory. You can enable verbose MSI logging with the same ‘voicewarmup’ registry value and see the MSI logs in the temp directory.
For the purposes of this blog post, I am going to use the SAV PowerShell cmdlets rather than managing the SAV Agent through System Center Virtual Machine Manager (VMM), so I installed those separately. The same error codes and messages that I talk about PowerShell returning below are also surfaced through VMM when you use it. The cmdlet installer uses the same chainer and the logs end up in the same place (the temp directory).
Deploying the Package
Before I started deployment, I took a look at the package. This can help you get your target machine set up correctly, or can help you debug issues after you see them. The manifest (called <package name>_manifext.xml) includes, among other things, the list of Windows Roles & Features that were enabled on the Sequencing station. Here you can see a bit of that section of my manifest:
Note that if any roles and features were enabled on the Sequencer, you need to enable them on the machine you deploy the package to.
If you use AVE to get into the SFT file (as described in the sequencing section above), there’s a second manifest in the root of the package, called appv_manifest.xml. Near the top of this manifest, you can find a list of applications that were installed on the sequencing machine, like this:
This can help you figure out things like what version of .NET you had on the Sequencing station, since the Agent you deploy to should be configured the same way.
For the actual deployment, I used the Add-ServerAppVPackage PowerShell cmdlet. I deliberately typed one of the paths wrong to get an example error message:
The first lines of that include an error code (0FD02560-00000002 in this case), whether it’s a general Windows error code or an App-V-specific error code (Windows, here), and the text associated with that error (“The system cannot find the file specified”). For Windows error codes, the last eight digits are the hexadecimal representation of a standard Windows system error codes. For App-V errors, you can often find help by searching online for the last 10 digits (60-00000002 in this example). In general, answers about desktop App-V error codes will also apply to Server App-V.
To get more information about what was wrong, I looked in the Application Event Log. I opened up Event Viewer, navigated to the Application log under Windows Logs, and looked for errors from the source App-V Services. Here’s what I see:
The text of the message in the screenshot is:
AddPackage could not find or open the Manifest file ‘C:\packages\WSFTPmanifest.xml’ due to error 0FD02520-00000002: The system cannot find the file specified.
I left out the underscore in the middle of the manifest file name, hence this error. In most cases, you’ll see more than one message from SAV in the event viewer so be sure to check through all the ones around the error you saw—sometimes you’ll be able to put together a more detailed picture of the problem by assembling pieces of data that we logged separately.
If you want to crank up the verbosity of the logs we generate, you can do that in the registry. Under HKLM\Software\Wow6432Node\Microsoft\SoftGrid\4.5\Client\Configuration, there are values called LogMinSeverity and SystemEventLogLevel. You can set them independently from 1 (least logging) to 5 (most). SystemEventLogLevel controls how much gets written to the event log we looked at here. LogMinSeverity controls how much gets dumped to a corresponding text file log on disk. LogFileName under the same key shows you the path to the log file on disk. You do need to reboot (or at least restart all of the App-V services) to apply log level changes.
The event logs are useful for errors from Server App-V, but you may also see errors directly from your sequenced application. We’ve tried to make the process of debugging application-specific errors as close as possible to the native experience. For example, if a service isn’t starting, you can open the Windows Services console (services.msc) and try to start it there, look at its properties from there, etc. You can use tools like dcomcnfg or wbemtest to interact with COM objects or WMI providers that don’t seem to behave correctly. If you need to launch a tool or process inside a virtual environment, so it has full access to the application’s registry etc., you can do this by appending /RunInVE:<package name> to any command line anywhere on the system. For example, to get regedit running inside the virtual environment for my WSFTP package, I ran “regedit.exe /RunInVE:WSFTP”.
If you run into a situation where a deployment fails and you can’t find the answer in a log message, you may want to examine the state of the system to see if you can find any clues. This can be difficult, since Server App-V rolls back any changes it makes as part of package deployment if the deployment fails. You can prevent this rollback by setting the DWORD value DisableSavUndo to 1 under HKEY_LOCAL_MACHINE\Software\Microsoft\SoftGrid\4.5\Client. After this is set, if you deploy a package and deployment fails, the Server App-V agent will leave the system in whatever partially-installed state it ends up in when the failure occurs.
The ultimate troubleshooting tool for misbehaving virtual applications is Process Monitor. This tool lets you see almost every resource request an application makes, so you can look for errors or compare transcripts of virtual and native instances of an application. Depending on your platform, you may need to add the /HookRegistry flag to catch virtual registry transactions (see ProcMon’s documentation for more details). For many more details about using Process Monitor, see http://blogs.technet.com/b/appv/archive/2008/01/24/process-monitor-hands-on-labs-and-examples.aspx.
Conclusion
I hope this post provides a good reference for the set of techniques and logs that can help you troubleshoot Server App-V problems. If you have follow-up questions or ideas of your own, please feel free to post here.
Eric Jewart
Principal Development Lead, Microsoft Server Application Virtualization
