Methodology in a Box™ For Citrix® Presentation Server 4.0
Virtual Memory Optimization Enterprise Edition
May 26, 2006
Written by: Douglas A. Brown
Copyright © 2002 - 2006 DABCC, Inc., All Rights Reserved
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 0
Table of Contents 1
CITRIX VIRTUAL MEMORY OPTIMIZATION FEATURE ............................. 2 1.1
Let’s Take a Deeper Look at the Problem ................................................................................... 2
1.2
How Does Citrix Memory Optimization Work? ......................................................................... 3
2 HOW TO INSTALL AND CONFGIURE THE CITRIX MEMORY OPTIMIZATION FEATURE ............................................................................... 5
3
4
2.1
How to Enable Citrix Memory Optimization Citrix Farm Wide............................................... 6
2.2
How to Enable Citrix Memory Optimization Per-Citrix Server................................................ 7
2.3
How to Set the Citrix Memory Optimization Schedule for Running Optmizations................. 8
HOW TO TROUBLESHOOT PROBLEMATIC APPLICATIONS ................ 12 3.1
How to Exclude Applications from Being Optimized ............................................................... 13
3.2
How to Remove Existing Optimization from DLLs .................................................................. 16
3.3
How to View the Status DLL Rebasing ...................................................................................... 17
3.4
How to Verify DLLs Are Being Rebased ................................................................................... 18
HOW TO MONITOR VIRTUAL MEMORY OPTIMIZATION SAVINGS ....... 22 4.1
How to Create a Virtual Memory Optimization Report........................................................... 23
5
BONUS – RTO SOFTWARE’S ADVANCED CONFIGURATION TOOL .... 30
6
CONCLUSION............................................................................................. 33
7
ADDITIONAL RESOURCES ....................................................................... 34
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 1
1 Citrix Virtual Memory Optimization Feature In Citrix MetaFrame Presentation Server 4.0, Citrix added numerous new features that enhance the ability to add more users per server while improving user response time and experience. In this section we will discuss the Citrix Memory Optimization feature of Citrix Presentation Server 4.0.
The Citrix Memory Optimization feature is designed to decrease Virtual Memory consumption of applications, improve application initialization time and reduce page file usage. It does this by rebasing DLLs that cause collisions with its virtual address to avoid relocation of these DLLs. Rebasing changes the base address of the DLL so that it loads at an optimal base memory address to avoid the collision and hence the relocation.
Note: This feature is only available in the Enterprise Edition version of Citrix Presentation Server 4.0 and Citrix Access Essentials 1.x.
1.1 Let’s Take a Deeper Look at the Problem Microsoft Windows and 3rd party applications contain Dynamic Link Libraries (DLLs). DLLs contain common code that is shared by all applications so that only one instance of the code is required in to memory (RAM) at any time. When an application initializes and loads its DLL into memory, there is a chance that the base memory address where it is attempting to load that DLL may already be occupied by another DLL. If this is the case then a DLL collision occurs. When a collision occurs the DLL has to be relocated to an available base memory address. The loader then has to update all the hard-coded pointers in the DLL to reflect the new base address, which slows application load-time. At this point, the DLL can no longer be shared by other applications, and hence this takes up more actual memory. This may increase the amount of paging that the system must do and as a result it slows down the overall performance.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 2
1.2 How Does Citrix Memory Optimization Work? The first thing you must know when trying to understand how the Citrix Memory Optimization works is its components. This feature consists of four main components: •
The Citrix Virtual Memory Optimization service (CtxSFOsvc.exe) - This executable monitors all system and application DLLs on a server in-order to find when and where collisions are occurring.
•
The Rebaser program (CtxBace.exe) This executable does the actual work of rebasing the DLLs.
•
The File system driver (ctxaltstr.sys) This system driver is responsible for loading the rebased DLL.
•
The tester program (ctxTestDll.exe). This Citrix DLL is responsible for testing all DLLs that have been modified to make sure they are working as they did before relocation. Note: In the 64-bit version of Citrix Presentation Server 4.0 the tester program is actually ctxtestdll64.exe.
Note: In addition to the above four main components, the Repair.SFO file is a log file that shows what will be rebased and the Bind*.SFO files are a log of what has been rebased. Both will be discussed later on.
All the above files are located in the following folder and/or subdirectory: C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management\Data C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management\Program C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management\ System32\Drivers
Now that we understand the files associated with the Memory Optimization feature and their purpose we now look at how exactly the Memory Optimization works and in what order.
1.
Once enabled, the Citrix Virtual Memory Optimization service is tied to the CtxSFOSvc.exe process and monitors all DLLs on the system to find out where DLL collisions and relocation are occurring. It writes the name of these DLLs to a file called Repair.sfo located in C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management\Data.
Note: In the 64-bit version of Citrix Presentation Server 4.0 there is a ctxsfosvc64.exe service that writes to repair64.sfo – this service is automatically started by the ctxsfosvc service.
2.
At the administrator configured scheduled intervals and during the Virtual Memory Optimization service startup, CtxBace.exe reads the Repair.sfo file to optimize the DLLs found.
This actually
occurs through a Windows Scheduled Task which you can see if you go to|: Start Æ Control Panel Æ Scheduled Tasks. Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 3
Note: The command line for this is: ctxbace –optimize.
However, there is an undocumented
command: ctxbace –optimizeshow. When this command is run from a command prompt it will execute an ad-hoc optimization. The –optimizeshow parameter will display a progress dialog.
3.
After CtxBace.exe optimizes the DLL, CtxTestDLL.exe loads the DLL to ensure that the DLL is still functional.
4.
Other than rebasing DLLs, the optimization process also involves binding executables to make application initialization time faster.
Now that we understand what the problem is and how the Citrix Memory Optimization feature works, we are ready to install and configure it.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 4
2 How to Install and Confgiure the Citrix Memory Optimization Feature In this section you will learn how to install and configure the Citrix Memory Optimization Feature. It consists of the following sections: •
How to enable the Citrix Memory Optimization feature on a Citrix farm wide basis
•
How to enable the Citrix Memory Optimization feature on a per-Citrix server basis
•
How to Set the Citrix Memory Optimization Schedule for Running Optimizations
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 5
2.1 How to Enable Citrix Memory Optimization Citrix Farm Wide The following details how to enable the Citrix Memory Optimization feature on a farm wide basis. 1.
Login to the Management Console for MetaFrame Presentation Server 4.0, right click the farm node, and click Properties.
2.
Click the Memory/CPU Utilization Management node.
Click to check the Enable memory
optimization checkbox. This enables the Memory Optimization feature on a farm wide basis.
Important! It takes time to fully rebase all the DLLs on a server. The system is informed about application DLLs once the application is executed the first time. It is considered a best practice to execute every user application at once in order for the system to relocate any DLL collisions that might occur before placing the server in to production. You can then trigger the optimization from the command line by typing: ctxbase –optimize. At this point you should see the DLLs requiring rebasing in the Repair.sfo or on a 64-bit OS the Repair32.sfo file. Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 6
2.2 How to Enable Citrix Memory Optimization Per-Citrix Server The following details how to enable the Citrix Memory Optimization feature on a per-Citrix server basis. 1.
Login to the Management Console for MetaFrame Presentation Server 4.0. Click to expand the Servers node then click to select the server you wish to enable the Citrix Memory Optimization feature on and click on Properties.
2.
Click the Memory/CPU Utilization Management node. Click to uncheck the Use farm settings for memory optimization checkbox. Click to check the Enable memory optimization checkbox. This enables the Memory Optimization feature only on the specified server.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 7
2.3 How to Set the Citrix Memory Optimization Schedule for Running Optmizations The following details how to schedule when the system will run the virtual memory optimization program. To ensure full use of system resources while the program runs, schedule the program to start when the fewest users are on the system. 1.
Login to the Management Console for MetaFrame Presentation Server 4.0. Click to expand the Servers node, click to select the server you wish to install the Memory Optimization feature on and then click Properties.
2. Click the Memory/CPU Utilization Management node.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 8
3.
You are now prompted to configure the Optimization interval. This setting specifies when the Citrix server will run the optimization program. You have the choice to configure it on a daily, weekly, or monthly basis. If you select weekly or monthly you will also be required to select the day. Select the desired setting in the Optimization interval drop down box.
Note: It is a best practice to run the optimization when the system has the lowest amount of overhead. The reason this is a best practice is that when the least number of users are on is when the DLLs aren’t in use (otherwise it can’t rebase them).
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 9
4.
Once you defined the optimization interval you will be required to specify the Optimization time. This setting specifies the time the optimization will be kicked off. Select the desired time in the Optimization time box.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 10
5.
You also have the ability to define the account that will be used to run the optimization program. The Use local system account checkbox is defined as the default and it allows the memory optimization program to run automatically using the built-in local system account. But if you wish to run the optimization program from a user account, for example if the program needs to access files on a remote computer, you will need to uncheck the Use Local system account checkbox and supply a valid user name and password in the Account text box. When finished, click OK to save the settings and continue.
Note: The username must be entered using the DOMIAN\USERNAME format.
Once completed, you need to restart every server for the new schedule to take effect.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 11
3 How to Troubleshoot Problematic Applications In case you run in to problems where an application worked prior to enabling the Citrix Memory Optimization feature but fails to function as designed after optimizing the applications DLLs then you will need to do a bit of troubleshooting.
This is actually a very simple process as this feature tends to work or not work. The issue is that some applications are just not designed to have their DLLs rebased and hence work with the Citrix Memory Optimization feature. The following types of applications will require to be excluded from the list of applications optimized. •
Applications that have digitally signed components
•
An application who’s DLLs is protected by Microsoft Windows Rights Management (WRM). Applications such as Office 2003.
Do note that the digitally signed files and the components
protected by WRM are detected by the Citrix server automatically and are added to each server’s registry exclusion list, as shown below. •
Applications whose executable programmatically checks the DLL after it has been loaded.
In order to troubleshoot troublesome applications you can do two simple tasks: 1.
Exclude the application from being optimized
2.
If the application still does not function as designed then you can remove the optimization through RTO Software’s DeleteOptimizations tool.
Below I will discuss these two methods of troubleshooting issues with applications and the Citrix Memory Optimization feature.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 12
3.1 How to Exclude Applications from Being Optimized By default, Presentation Server 4.0 ships with a pre-configured exclusion list. These lists can be viewed at the following registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\SFO\ComponentExclusionList
HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\SFO\ProcessExclusionList
If any applications fails to load or function after you enabled Memory Optimization you will need to exclude it from the list of application optimized as defined below.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 13
The following steps define how to add an application executable to the memory optimization exclusion list.
1. Login to the Management Console for Citrix MetaFrame Presentation Server 4.0, click to expand the Servers node, click to select the server you wish to enable the Citrix Memory Optimization feature on and then click Properties.
2. Click the Memory/CPU Utilization Management node and click the Add button.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 14
3. Browse to the location of the application’s executable in question and click OK to add it to the exclusion list.
4. Repeat steps 2 and 3 for each application that experience troubles with when running under in a memory optimized environment.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 15
3.2 How to Remove Existing Optimization from DLLs If after excluding the application in question through the Citrix Presentation Server Console it still does not fuction as designed then you will be required to remove the optimizations from the applicaitons DLLs.
The following details how to remove exisitng optimizations from the application is question:
1.
The first thing you will need to do is download the Delete Optimizations utility from RTO Software’s web site. It will be found at: http://www.rtosoft.com/demos/deleteoptimizations.zip
2.
From the server you are experiencing troubles on run the deleteoptimization.exe utility.
3.
Browse to the folder where the application DLLs are stored and click OK.
4.
Once this path is specified, DeleteOptimizations will enumerate all files under that directory and each subdirectory and delete optimization streams from each file.
5.
The DeleteOptimizations tool will the clean all the applications DLLs until it runs in to any of the two possible errors:
6.
1.
The user does not have access to read the optimization streams from a file in the path. In this case, DeleteOptimizations will skip that file.
2.
The user does not have permissions to delete an optimization stream. In this case, DeleteOptimizations will skip that stream.
Once finished the DeleteOptimizations utility will close.
You have now successfully removed the optimizations from the desired sub directory. If you are still experiencing issues you will need to use a program like SysInternal’s Process Explorer to find the locations of all the applications DLLs and rerun the DeleteOptimizations utility.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 16
3.3 How to View the Status DLL Rebasing Another thing you can do when troubleshooting virtual memory optimization is to view the Citrix Repair.sfo XLM. This file shows the DLLs that have experience collisions and are pending rebasing.
The file is created by the Citrix Virtual Memory Optimization Service (CtxSFOSvc.exe) and is located in the C:\Program Files\Citrix\Server Resource Management\Memory Optimization Management\Data folder.
Important! It takes time to fully rebase all the DLLs on a server. The system is informed about application DLLs once the application is executed the first time. It is considered a best practice to execute every user application at once in order for the system to relocate any DLL collisions that might occur before placing the server in to production. You can then trigger the optimization from the command line by typing: ctxbase –optimize. At this point you should see the DLLs requiring rebasing in the Repair.sfo or on a 64-bit OS the Repair32.sfo file.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 17
3.4 How to Verify DLLs Are Being Rebased From time to time it might be necessary to verify if the Citrix Memory Optimization feature is truly rebasing DLLs. To verify that DLL rebasing is taking place, you can use SysInternal’s Process Explorer. It will show the DLLs being relocated and hence if the Memory Optimization feature is working properly.
To learn more about Process Explorer and to download it please visit: http://www.sysinternals.com/utilities/processexplorer.html
The following defines how to use SysInternal’s Process Explorer to verify that DLLs are being rebased. 1.
The first thing you will need to do is to download and install SysInternal’s Process Explorer application and then execute it.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 18
2.
Press Ctrl+D to view all the DLLs in a selected process
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 19
3.
From the File menu bar click Option Æ then click Configure Highlighting.
4.
Click to check the Relocated DLLs checkbox and change the highlight color to one that can be easily spotted. Click OK when finished.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 20
5.
Select the executable you would like insure that the DLLs are being rebased in the Process panel. In the bottom of the Process Explorer you will see all the associated DLLs of the selected executable and the DLLs being rebased as highlighted with the color specified above.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 21
4 How to Monitor Virtual Memory Optimization Savings From time to time it might be necessary for an administrator to generate a report to view a list of all the DLLs being rebased along with the amount of memory being saved using the Citrix Virtual Memory Optimization feature. An administrator can use the ‘Report Center’ feature of the Citrix Access Suite Console to create a Virtual Memory Optimization report to do just that.
The Citrix Access Suite Console generates this report by analyzing the Bind ????.sfo file that was created by the CtxBace.exe file during optimization.
Do note that there is not just one Bind ????.sfo file. There will be numerous log files created which are not deleted and/or archived by the system. You can however delete and/or archive them to another location to save disk space.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 22
4.1 How to Create a Virtual Memory Optimization Report The following defines how to create a Memory Optimization Report.
Note: This feature does NOT require the use of the Citrix Resource Manager Summary Database to be configured to generate this report.
1.
Click the Start button Æ click Programs Æ click Citrix Æ click MetaFrame Presentation Server Æ click Access Suite Console for Presentation Server.
2.
Click to select the Report Center node and then click Generate Report from the Common Tasks box.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 23
3.
Click Virtual Memory Optimization Report from the Report type drop down box. Click Next to continue.
4.
Click the Create and use a new specification radio button and then click Next to continue.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 24
5.
Click the desired application farm in the Selected Farm drop down box and click Next to continue.
6.
You are now prompted to add the servers you wish to include in the report. Click the Add button.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 25
7.
Click to select the servers you wish to run the Optimization Report on and click the Add button to add them to the Selected items list box. Click OK when finished to continue.
8.
Click Next to continue.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 26
9.
You are now prompted to store your report for later viewing or Publishing to a web site. For this example we have chosen to store the report for later viewing. Click the desired result and click Next to continue.
10. You are now prompted to save the report for future use with a name and description or to not save the report. For this example, we have chosen not to save the report. Click your desired result and click Next to continue.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 27
11. You are prompted to review your configuration settings. If you are happy with everything then click Finish to run the report. If not, click the Back button and make the appropriate changes.
12. You are brought to the Jobs view of the Report Center where you will see your report and the status of it. Once it reads, “Completed” you will want to click to highlight it and then click the View report in the default format link in the tasks window.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 28
13. Your report is opened in the default Internet browser. This report will list all the servers you selected during the above configuration wizard. Click a server in the list of Selected Server to view the Optimization Report.
You will then be presented with the Virtual Memory Optimization Report for your desired server. In this view you will see the savings of each application optimized.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 29
5 Bonus – RTO Software’s Advanced Configuration Tool The Citrix Virtual Memory Optimization feature is technology developed by and then acquired from RTO Software. This technology is found in RTO’s TScale product but not all of TScale’s features where included in to Citrix’s implementation.
This being the case, RTO Software created a product that they call, the RTO Optimization Console. The RTO Optimization Console is a free application for managing one Citrix server at a time. It is required to be installed on each Citrix server and is used to monitor and configure such items as: •
View and report application optimization results
•
Exclude optimizations by application for the server
•
Exclude optimizations by DLL for the server
•
Report optimization history
•
Turn on / off optimizations
•
Trigger an optimization on the fly.
•
Diagnose applications to determine the specific resources for exclusion
To manage multiple servers a customer can purchase the Professional Edition. The RTO Optimization Console, Professional Edition allows you to easily handle memory optimization for your entire Citrix Presentation Server farm from one location.
The following screen shot shows the RTO Optimization Console’s view of a particular server. On the screen you can view such items as the amount of virtual memory saved, the optimization schedule events and results, and any optimization alerts.
For example, you will notice that in this screen shot the
pdfshell.dll optimization will take effect on the next instance of the application. You will also notice a list of all the applications and their optimization results in the left “tree view” display.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 30
In this screen shot you are presented with a graph of each application executable that is currently being optimized with the amount of per user virtual memory saved due to the application optimizations.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 31
You can even drill down in to an application itself to view the DLLs the application uses and their respective virtual memory saving.
The RTO Software Optimization Console allows you to perform tons of other tasks along with presenting you with even more granular details on your systems virtual memory utilization and optimizations.
For additional information or to download the Optimization Console please visit the following link: http://www.rtosoft.com/enter.asp?ID=437
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 32
6 Conclusion In conclusion, the Citrix Virtual Memory Optimization feature will save RAM, cut page file usage and speed up application and end-user response time.
If you run into issues with this feature you are not required to turn it off, instead Citrix has given you easyto-use tools to exclude the problem applications while at the same time continuing to take advantage of the savings from other applications.
Ultimately, in memory-bound situations you can expect to see a 20 to 30 percent increase in the number of end-users who can log into a particular server.
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 33
7 Additional Resources Refer to the following Citrix knowledge base article for additional information on the Citrix Memory Optimization feature:
•
Citrix Support Article: CTX108995 - JDE OneWorld ERP, Finance One, and other Oracle Client Applications Fail to Launch with Virtual Memory Optimization Enabled
•
Citrix Support Article: CTX109101 - Configuring Virtual Memory Optimization Technologies within Citrix Access Essentials Version 1.x for Application Incompatibility
•
Citrix Support Article: CTX106023 - Virtual Memory Optimization Feature
•
DABCC | Methodology in a Box Memory Optimization Section http://www.dabcc.com/dabcc/webapplication/aspx/virtual-memory-optimization.aspx?first=dabcc
Written by: Douglas A. Brown - May 26, 2006 - www.dabcc.com
Page 34
