Calculating System Parameters for UNIX®, Linux, and macOS
This section explains how you can calculate the best parameters for your system in these sections:
For optimal Caché performance, you need to calculate proper values for certain Caché system parameters. These values allow you to determine whether you need to adjust certain system level parameters. The values you choose should minimize swapping and paging that require disk accesses, and thus improve system performance.
Review this section carefully and calculate the proper values for both your operating system and Caché before proceeding. Use the tables provided here to record the current and calculated values for your system level parameters. You can then refer to these tables when you install Caché. After your system is running, you may need to adjust these values to gain optimal performance.
If you are not already familiar with the memory organization at your operating system level, consult the appropriate system documentation.
Determining Memory and Disk Requirements
This section outlines the basic memory and disk requirements for most systems. Because these requirements vary by platform, consult your platform documentation for additional information. The specific requirements include the following:
See the section Managing Caché Memory
for information on the two primary ways that you can manage memory in Caché.
Calculating Memory Requirements
Use the breakdown of memory usage shown in the following table to calculate the memory your system needs for Caché.
UNIX® Memory Requirements
||1800 KB (operating system dependent)
|Global database cache
||8 KB per buffer
||32 KB per routine buffer
||1024 KB per process
|Network (if present)
||300 KB per port for each network system process (DMNNET and RECEIVE). Caché ports have two DMNNET system processes per port. In addition, there is a network shared memory requirement, which depends on the number of ports and the number of remote hosts configured. For a basic system, this requirement is about 304 KB.
By default, the system automatically allocates shared memory, including routine buffers and global buffers, to a total of one-eighth of the system-available shared memory space. If you plan to run large applications or support large numbers of users, tune the system according to the following formula:
(number of routine buffers)*32 KB
+ (number of global buffers)*(block size)
+ 4 MB
= Shared memory needed
For applications where load growth is reflected in the number of simultaneous direct Caché sessions, the memory demand to accommodate the processes increases as the computing power increases. For example, a system that is upgraded from 4 to 8 cores would be capable of supporting a much larger number of sessions (that is, processes). Since each process consumes memory, it might be necessary to increase physical memory.
The amount of memory per process may vary depending on the application and can be larger than the default value recommended in the UNIX® Memory Requirements
For configurations dedicated to servers with a limited number of processes (for example, ECP Data Server or Ensemble), an increase in the load does not necessarily involve a greater number of processes. Therefore, a larger load on a more powerful system may not require more memory for processes.
Support for Huge Memory Pages for Linux
The default memory page size on Linux systems is 4 KB. Most current Linux distributions include an option for Huge Pages, that is, a memory page size of 2 MB or 1 GB depending on system configuration. Use of Huge Pages saves memory by saving space in page tables. When Huge Pages are configured, the system automatically uses them in memory allocation. InterSystems recommends the use of Huge Pages on systems hosting Caché under most circumstances.
With the 2.6.38 kernel, some Linux distributions have introduced Transparent Huge Pages (THP) to automate the creation, management, and use of HugePages. However, THP does not handle the shared memory segments that make up the majority of Caché’s memory allocated, and can cause memory allocation delays at runtime that may affect performance, especially for applications that have a high rate of job or process creation. For these reasons, InterSystems recommends that THP be disabled
on all systems hosting Caché. For more detailed information on this topic, see Linux Transparent Huge Pages and the impact to Caché
on InterSystems Developer Community.
To configure Huge Pages on Linux, do the following:
contains Huge Pages information. By default, no Huge Pages are allocated. Default Huge Page size is 2 MB. For example:
Hugepagesize: 2048 KB
Change the number of Huge Pages.
You can change the system parameter directly: For example, to allocate 2056 Huge Pages, execute:
# echo 2056 > /proc/sys/vm/nr_hugepages
Alternatively, you can use sysctl(8)
to change it:
# sysctl -w vm.nr_hugepages=2056
Huge pages must be allocated contiguously, which may require a reboot. Therefore, to guarantee the allocation, as well as to make the change permanent, do the following:
echo "vm.nr_hugepages=2056" >> /etc/sysctl.conf
after reboot; for example:
[root woodcrest grub]# tail -4 /proc/meminfo
Hugepagesize: 2048 KB
Verify the use of Huge Pages by Caché.
When Caché is started, it reports how much shared memory was allocated; for example, a message similar to the following is displayed (and included in the cconsole.log
Allocated 3580MB shared memory: 3000MB global buffers, 226MB routine buffers
The amount of memory available in Huge Pages should be greater than the total amount of shared memory to be allocated; if it is not greater, Huge Pages are not used.
Huge Pages are allocated from physical memory. Only applications and processes using Huge Pages can access this memory.
Physical memory not allocated for Huge Pages is the only memory available to all other applications and processes.
It is not advisable to specify HugePages_Total
much higher than the shared memory amount because the unused memory will not be available to other components.
If Caché fails to allocate Huge Pages on start-up and switches to standard pages, Caché will be allocating shared memory from the same memory pool as all other jobs.
When Caché is configured to lock the shared memory segment in memory to prevent paging, Huge Pages can provide the required increase in the maximum size that may be locked into memory, as described in the Locked-in Memory
section of the Red Hat Linux Platform Notes
in this chapter.
Support for Large (16 MB) Pages on IBM AIX®
AIX® supports multiple page sizes: 4 KB, 64 KB, 16 MB, and 16 GB. Use of 4 KB and 64 KB pages is transparent to Caché. In order for Caché to use 16 MB large pages, you must configure them within AIX®. AIX® does not automatically change the number of configured large or huge pages based on demand. Currently, Caché does not use 16 GB huge pages.
Large pages should be configured only in high-performance environments because memory allocated to large pages can be used only for large pages.
By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages then it is allocated in standard (small) pages. For finer grain control over large pages, see memlock
in the Caché Parameter File Reference
Configuring Large Pages for AIX®
Configure large pages using the vmo
command as follows:
vmo -r -o lgpg_regions=<LargePages> -o lgpg_size=<LargePageSize>
specifies the number of large pages to reserve, and <LargePageSize>
specifies the size, in bytes, of the hardware-supported large pages.
On systems that support dynamic Logical PARtitioning (LPAR), you can omit the -r
option to dynamically configure large pages without a system reboot.
For example, the following command configures 1 GB of large pages:
# vmo -r -o lgpg_regions=64 -o lgpg_size=16777216
Once you have configured large pages, run the bosboot
command to save the configuration in the boot image. After the system comes up, enable it for pinned memory using the following vmo
The amount of swap space available on your system should never be less than the amount of real memory plus 256 KB.
With this minimum in mind, InterSystems recommends the following value as the minimum amount of swap space needed for Caché:
((# of processes + 4)† * (1024 KB))‡
+ total global buffer space
+ total routine buffer space
= Minimum swap space
† Add 4 to the # of processes for the Caché Control Process, the Write daemon, the Garbage Collector, and the Journal daemon. Also add 1 for each slave Write daemon. The # of processes must include all user and jobbed processes which might run concurrently. If you are running networking, add 1 for the RECEIVE system process plus the number of DMNNET daemons you have running (2 per port).
‡ The 1024 KB number is approximate. It is based on the current size of the Caché executable and grows with the partition size you allocate to each Caché process. On most systems, provide only as much swap space as necessary. However, some systems require you to provide swap space for the worst case. Under these conditions, you need to increase this number to as high as 1.5 MB, depending on the partition size you specify.
Be sure to confirm that your UNIX® system permits the amount of swap space you require. For specific information about swap space on your system, consult your UNIX® operating system manual.
Solaris Swap Space
To calculate swap space for the Solaris platform:
swapfile dev swaplo blocks free
/dev/dsk/c0t2d0s0 136,0 16 526304 526304
/dev/dsk/c0t2d0s1 136,1 16 2101184 2101184
AIX® Swap Space
To display swap space for AIX®:
Page Space Physical Volume Volume Group Size %Used
Active Auto Type
hd6 hdisk2 rootvg 512 MB 72
yes yes lv
HP-UX Swap Space
To display swap space for HP-UX:
KB KB KB PCT START/ KB
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 138260 386028 26% 0 - 1 /dev/vg00/lvol2
reserve - 78472 -78472
memory 195132 191668 3464 98%
Calculating Disk Requirements
In addition to the swap space you just calculated, you need disk space for the following items:
3 MB for the Caché Server Pages (CSP).
3.5 MB for Caché ODBC support.
2.5 MB for the Caché manager sources.
6.6 MB for the Caché engine link libraries.
Space for your Caché application database.
Approximately 12.5% of the buffer pool size for the initial size of the write image journal file. If your disk does not have enough space for the write image journal file, when you start Caché it displays a message indicating that the system did not start.
Desired space for journal files.
Although you do not need to remove any installation files after completing the installation procedure, you can do so if you are short on disk space. The installation program tells you how much space can be saved, and asks if you want to delete the installation files.
Determining Number of Global Buffers
Caché supports the following maximum values for the number of global buffers:
For 32-bit platforms, any 8-KB buffers that are:
Less than 2GB for other 32-bit platforms
The 2-GB value is the total address space the operation system allocates for the process data, which includes not only shared memory, but other Caché and operating system data as well. Therefore, it represents an upper limit that is not achievable in practice.
The number of global buffers is limited only by the operating system and the available memory.
Set your values to less than the maximum number of buffers.
Determining Number of Routine Buffers
Caché supports the following maximum value for the number of routine buffers:
Set your values to less than this maximum number of buffers.
Determining Maximum Number of Users
The maximum users allowed by Caché is the lowest
of the following values:
Determining Maximum Database Size
parameter in UNIX® determines the maximum file size available to a process. For the Caché Manager group, the value of ulimit
should either be unlimited or as large as the largest database you may have.
Configuring UNIX® Kernel Parameters
The following sections describe issues related to tuning and performance on various UNIX® platforms:
Setting Values for Tunable UNIX® Parameters
Caché uses a configurable number of semaphores, in sets whose size you define. The parameters SEMMNI
, and SEMMSL
reflect the number of semaphores per set and the total number of semaphores Caché uses. The UNIX®/Linux parameters that govern shared memory allocation are SHMMAX
, and SHMALL
. Caché uses shared memory and allocates one segment of shared memory; the size of this segment depends on the area set aside for global buffers and routine buffers. It uses the following formula to determine the segment's minimum size:
space required for routine buffers
+ space required for global buffers
+ 4 MB
= Shared memory segment size
If you are distributing your data across multiple computers, Caché allocates a second segment; by default, there is no memory allocated for the second segment. (If you plan to use distributed data, contact your vendor or InterSystems support for configuration guidelines.) You can alter NBUF
according to other system requirements. Because Caché does all its own disk buffering, you should keep NBUF
small. The following table lists the most common names of the UNIX® parameters that you may need to change, the minimum value InterSystems recommends for each parameter, and a brief description of each. Verify that your parameter values are set to at least the minimum value. Certain parameters may not be implemented on all platforms or may be referred to differently. Refer to platform-specific tuning notes for more information.
Tunable UNIX® Parameters
||Recommended Minimum Value
||Number of bytes in largest virtual volume
||Maximum size of a file.
||Maximum message size, in bytes.
||Number of Caché instances x 3; each Caché instance uses three message queues
||Maximum number of uniquely identifiable message queues that may exist simultaneously.
||Number of open files per process.
||Product of SEMMNI and SEMMSL must be greater than the # of user processes + 4
||Number of semaphore identifiers in the kernel; this is the number of unique semaphore sets that can be active at any one time.
||128 or ...
||Total number of semaphores in the system. User processes include jobbed processes and all other semaphores required by other software.
|Number of processes expected to run. If the process table might expand, use a larger number to provide for expansion.
||Maximum number of semaphores per identifier list.
||60 KB or ...
||Maximum total shared memory system-wide. Units should be in KB. 1000 represents the MCOMMON shared region.
|1000 + total global buffer space+ total routine buffer space *
||Maximum number of shared memory identifiers system-wide.
||Number of attached shared memory segments per process.
||60 KB or ...
||Maximum shared memory segment size in KB.
|1000 + total global buffer space+ total routine buffer space
* This is the minimum value for SHMALL
required for Caché UNIX®. You must also take into account any other applications that use shared memory. If you are unsure of other shared memory use, calculate SHMALL
multiplied by SHMMAX
, in pages; this larger value suffices in all cases.
Enough swap space must be created to support the memory allocated, unless the operating system documentation explicitly states otherwise. On certain operating systems (Solaris, for example) Caché creates locked shared memory segments
, which are not pageable but still may need swap space.
Adjusting Maximum File Size
The hard limit for the maximum file size (RLIMIT_FSIZE
) on any system running Caché must be unlimited
. Set the value to unlimited
on the operating system before installing. Make sure that the limit is set to unlimited
for both the root user and the user who will run Caché. Caché also sets the process soft limit to RLIMIT_FSIZE
in its daemons to prevent I/O errors.
Caché will not install or start up if RLIMIT_FSIZE
is not set to unlimited.
See the operating system documentation for your platform for instructions on how to set the system hard limit for the maximum file size, RLIMIT_FSIZE