InterSystems: The power behind what matters   
Class Reference   

Calculating System Parameters for UNIX®, Linux, and macOS
This section explains how you can calculate the best parameters for your system in these sections:
For optimal Caché performance, you need to calculate proper values for certain Caché system parameters. These values allow you to determine whether you need to adjust certain system level parameters. The values you choose should minimize swapping and paging that require disk accesses, and thus improve system performance.
Review this section carefully and calculate the proper values for both your operating system and Caché before proceeding. Use the tables provided here to record the current and calculated values for your system level parameters. You can then refer to these tables when you install Caché. After your system is running, you may need to adjust these values to gain optimal performance.
If you are not already familiar with the memory organization at your operating system level, consult the appropriate system documentation.
Determining Memory and Disk Requirements
This section outlines the basic memory and disk requirements for most systems. Because these requirements vary by platform, consult your platform documentation for additional information. The specific requirements include the following:
See the section Managing Caché Memory for information on the two primary ways that you can manage memory in Caché.
Calculating Memory Requirements
Use the breakdown of memory usage shown in the following table to calculate the memory your system needs for Caché.
UNIX® Memory Requirements
Components Memory Requirements
Operating system 1800 KB (operating system dependent)
Caché 842 KB
Global database cache 8 KB per buffer
Routine cache 32 KB per routine buffer
User overhead 1024 KB per process
Network (if present) 300 KB per port for each network system process (DMNNET and RECEIVE). Caché ports have two DMNNET system processes per port. In addition, there is a network shared memory requirement, which depends on the number of ports and the number of remote hosts configured. For a basic system, this requirement is about 304 KB.
By default, the system automatically allocates shared memory, including routine buffers and global buffers, to a total of one-eighth of the system-available shared memory space. If you plan to run large applications or support large numbers of users, tune the system according to the following formula:
            (number of routine buffers)*32 KB
            + (number of global buffers)*(block size)
            + 4 MB
            = Shared memory needed
For applications where load growth is reflected in the number of simultaneous direct Caché sessions, the memory demand to accommodate the processes increases as the computing power increases. For example, a system that is upgraded from 4 to 8 cores would be capable of supporting a much larger number of sessions (that is, processes). Since each process consumes memory, it might be necessary to increase physical memory.
The amount of memory per process may vary depending on the application and can be larger than the default value recommended in the UNIX® Memory Requirements table.
For configurations dedicated to servers with a limited number of processes (for example, ECP Data Server or Ensemble), an increase in the load does not necessarily involve a greater number of processes. Therefore, a larger load on a more powerful system may not require more memory for processes.
Support for Huge Memory Pages for Linux
The default memory page size on Linux systems is 4 KB. Most current Linux distributions include an option for Huge Pages, that is, a memory page size of 2 MB or 1 GB depending on system configuration. Use of Huge Pages saves memory by saving space in page tables. When Huge Pages are configured, the system automatically uses them in memory allocation. InterSystems recommends the use of Huge Pages on systems hosting Caché under most circumstances.
With the 2.6.38 kernel, some Linux distributions have introduced Transparent Huge Pages (THP) to automate the creation, management, and use of HugePages. However, THP does not handle the shared memory segments that make up the majority of Caché’s memory allocated, and can cause memory allocation delays at runtime that may affect performance, especially for applications that have a high rate of job or process creation. For these reasons, InterSystems recommends that THP be disabled on all systems hosting Caché. For more detailed information on this topic, see Linux Transparent Huge Pages and the impact to Caché on InterSystems Developer Community.
To configure Huge Pages on Linux, do the following:
  1. Check the status.
    /proc/meminfo contains Huge Pages information. By default, no Huge Pages are allocated. Default Huge Page size is 2 MB. For example:
    HugePages_Total:     0
    HugePages_Free:      0
    HugePages_Rsvd:      0
    Hugepagesize:     2048 KB
  2. Change the number of Huge Pages.
    You can change the system parameter directly: For example, to allocate 2056 Huge Pages, execute:
    # echo 2056 > /proc/sys/vm/nr_hugepages
    Alternatively, you can use sysctl(8) to change it:
    # sysctl -w vm.nr_hugepages=2056  
    Huge pages must be allocated contiguously, which may require a reboot. Therefore, to guarantee the allocation, as well as to make the change permanent, do the following:
    1. Enter a line in /etc/sysctl.conf file:
      echo "vm.nr_hugepages=2056" >> /etc/sysctl.conf  
    2. Reboot the system.
    3. Verify meminfo after reboot; for example:
      [root woodcrest grub]# tail -4 /proc/meminfo
      HugePages_Total:  2056
      HugePages_Free:   2056
      HugePages_Rsvd:      0
      Hugepagesize:     2048 KB
  3. Verify the use of Huge Pages by Caché.
    When Caché is started, it reports how much shared memory was allocated; for example, a message similar to the following is displayed (and included in the cconsole.log file):
    Allocated 3580MB shared memory: 3000MB global buffers, 226MB routine buffers
    The amount of memory available in Huge Pages should be greater than the total amount of shared memory to be allocated; if it is not greater, Huge Pages are not used.
    To prevent Caché from starting if Huge Pages are configured but cannot be allocated, set the memlock parameter to 7 (see memlock in the Caché Parameter File Reference).
    Huge Pages are allocated from physical memory. Only applications and processes using Huge Pages can access this memory.
    Physical memory not allocated for Huge Pages is the only memory available to all other applications and processes.
    It is not advisable to specify HugePages_Total much higher than the shared memory amount because the unused memory will not be available to other components.
    If Caché fails to allocate Huge Pages on start-up and switches to standard pages, Caché will be allocating shared memory from the same memory pool as all other jobs.
    When Caché is configured to lock the shared memory segment in memory to prevent paging, Huge Pages can provide the required increase in the maximum size that may be locked into memory, as described in the Locked-in Memory section of the Red Hat Linux Platform Notes in this chapter.
Support for Large (16 MB) Pages on IBM AIX®
AIX® supports multiple page sizes: 4 KB, 64 KB, 16 MB, and 16 GB. Use of 4 KB and 64 KB pages is transparent to Caché. In order for Caché to use 16 MB large pages, you must configure them within AIX®. AIX® does not automatically change the number of configured large or huge pages based on demand. Currently, Caché does not use 16 GB huge pages.
Large pages should be configured only in high-performance environments because memory allocated to large pages can be used only for large pages.
To allocate large pages, users must have the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities or have root authority unless memlock=64.
By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages then it is allocated in standard (small) pages. For finer grain control over large pages, see memlock in the Caché Parameter File Reference.
Configuring Large Pages for AIX®
Configure large pages using the vmo command as follows:
vmo -r -o lgpg_regions=<LargePages> -o lgpg_size=<LargePageSize>
where <LargePages> specifies the number of large pages to reserve, and <LargePageSize> specifies the size, in bytes, of the hardware-supported large pages.
On systems that support dynamic Logical PARtitioning (LPAR), you can omit the -r option to dynamically configure large pages without a system reboot.
For example, the following command configures 1 GB of large pages:
# vmo -r -o lgpg_regions=64 -o lgpg_size=16777216
Once you have configured large pages, run the bosboot command to save the configuration in the boot image. After the system comes up, enable it for pinned memory using the following vmo command:
vmo -o v_pinshm=1
However, if memlock=64, vmo -o v_pinshm=1 is not required. For more information on memlock, see memlock in the Caché Parameter File Reference.
Calculating Swap Space
The amount of swap space available on your system should never be less than the amount of real memory plus 256 KB.
With this minimum in mind, InterSystems recommends the following value as the minimum amount of swap space needed for Caché:
                       ((# of processes + 4)† * (1024 KB))‡  
                     +           total global buffer space  
                     +          total routine buffer space  
                     =                  Minimum swap space
† Add 4 to the # of processes for the Caché Control Process, the Write daemon, the Garbage Collector, and the Journal daemon. Also add 1 for each slave Write daemon. The # of processes must include all user and jobbed processes which might run concurrently. If you are running networking, add 1 for the RECEIVE system process plus the number of DMNNET daemons you have running (2 per port).
‡ The 1024 KB number is approximate. It is based on the current size of the Caché executable and grows with the partition size you allocate to each Caché process. On most systems, provide only as much swap space as necessary. However, some systems require you to provide swap space for the worst case. Under these conditions, you need to increase this number to as high as 1.5 MB, depending on the partition size you specify.
Be sure to confirm that your UNIX® system permits the amount of swap space you require. For specific information about swap space on your system, consult your UNIX® operating system manual.
Solaris Swap Space
To calculate swap space for the Solaris platform:
swap –l
>swap –l
swapfile                   dev     swaplo blocks   free
/dev/dsk/c0t2d0s0         136,0      16    526304   526304
/dev/dsk/c0t2d0s1         136,1      16    2101184 2101184
AIX® Swap Space
To display swap space for AIX®:
lsps –a
Page Space  Physical Volume   Volume Group    Size %Used  
Active  Auto  Type
hd6         hdisk2            rootvg                   512 MB      72     
yes   yes    lv
HP-UX Swap Space
To display swap space for HP-UX:
swapinfo (3M)
# /usr/sbin/swapinfo
            KB      KB      KB   PCT    START/         KB
dev      524288  138260  386028   26%       0       -    1  /dev/vg00/lvol2
reserve       -   78472  -78472
memory   195132  191668    3464   98%
Calculating Disk Requirements
In addition to the swap space you just calculated, you need disk space for the following items:
Although you do not need to remove any installation files after completing the installation procedure, you can do so if you are short on disk space. The installation program tells you how much space can be saved, and asks if you want to delete the installation files.
Determining Number of Global Buffers
Caché supports the following maximum values for the number of global buffers:
Set your values to less than the maximum number of buffers.
For more information, see globals in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.
Determining Number of Routine Buffers
Caché supports the following maximum value for the number of routine buffers:
Set your values to less than this maximum number of buffers.
For more information, see routines in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.
Determining Maximum Number of Users
The maximum users allowed by Caché is the lowest of the following values:
For more information, see Determining License Capacity and Usage in the “Managing Caché Licensing” chapter of the Caché System Administration Guide.
Determining Maximum Database Size
The ulimit parameter in UNIX® determines the maximum file size available to a process. For the Caché Manager group, the value of ulimit should either be unlimited or as large as the largest database you may have.
For more information, see Configuring Databases in the “Configuring Caché” chapter of the Caché System Administration Guide.
Configuring UNIX® Kernel Parameters
The following sections describe issues related to tuning and performance on various UNIX® platforms:
Setting Values for Tunable UNIX® Parameters
Caché uses a configurable number of semaphores, in sets whose size you define. The parameters SEMMNI, SEMMNS, and SEMMSL reflect the number of semaphores per set and the total number of semaphores Caché uses. The UNIX®/Linux parameters that govern shared memory allocation are SHMMAX, SHMMNI, SHMSEG, and SHMALL. Caché uses shared memory and allocates one segment of shared memory; the size of this segment depends on the area set aside for global buffers and routine buffers. It uses the following formula to determine the segment's minimum size:
                       space required for routine buffers
                     +  space required for global buffers
                     +                               4 MB
                     =         Shared memory segment size
If you are distributing your data across multiple computers, Caché allocates a second segment; by default, there is no memory allocated for the second segment. (If you plan to use distributed data, contact your vendor or InterSystems support for configuration guidelines.) You can alter NBUF and NHBUF according to other system requirements. Because Caché does all its own disk buffering, you should keep NBUF and NHBUF small. The following table lists the most common names of the UNIX® parameters that you may need to change, the minimum value InterSystems recommends for each parameter, and a brief description of each. Verify that your parameter values are set to at least the minimum value. Certain parameters may not be implemented on all platforms or may be referred to differently. Refer to platform-specific tuning notes for more information.
Tunable UNIX® Parameters
Kernel Parameter Recommended Minimum Value Definition
CDLIMIT Number of bytes in largest virtual volume Maximum size of a file.
MSGMAX 2 KB Maximum message size, in bytes.
MSGMNI Number of Caché instances x 3; each Caché instance uses three message queues Maximum number of uniquely identifiable message queues that may exist simultaneously.
NOFILES 35 Number of open files per process.
SEMMNI Product of SEMMNI and SEMMSL must be greater than the # of user processes + 4 Number of semaphore identifiers in the kernel; this is the number of unique semaphore sets that can be active at any one time.
SEMMNS 128 or ... Total number of semaphores in the system. User processes include jobbed processes and all other semaphores required by other software.
Number of processes expected to run. If the process table might expand, use a larger number to provide for expansion.
SEMMSL See SEMMNI Maximum number of semaphores per identifier list.
SHMALL 60 KB or ... Maximum total shared memory system-wide. Units should be in KB. 1000 represents the MCOMMON shared region.
1000 + total global buffer space+ total routine buffer space *
SHMMNI 3 Maximum number of shared memory identifiers system-wide.
SHMSEG 3 Number of attached shared memory segments per process.
SHMMAX 60 KB or ... Maximum shared memory segment size in KB.
1000 + total global buffer space+ total routine buffer space
* This is the minimum value for SHMALL required for Caché UNIX®. You must also take into account any other applications that use shared memory. If you are unsure of other shared memory use, calculate SHMALL as SHMSEG multiplied by SHMMAX, in pages; this larger value suffices in all cases.
Enough swap space must be created to support the memory allocated, unless the operating system documentation explicitly states otherwise. On certain operating systems (Solaris, for example) Caché creates locked shared memory segments, which are not pageable but still may need swap space.
Adjusting Maximum File Size
The hard limit for the maximum file size (RLIMIT_FSIZE) on any system running Caché must be unlimited. Set the value to unlimited on the operating system before installing. Make sure that the limit is set to unlimited for both the root user and the user who will run Caché. Caché also sets the process soft limit to RLIMIT_FSIZE in its daemons to prevent I/O errors.
Caché will not install or start up if RLIMIT_FSIZE is not set to unlimited.
See the operating system documentation for your platform for instructions on how to set the system hard limit for the maximum file size, RLIMIT_FSIZE.