Calculating System Parameters for UNIX®, Linux, and macOS

Configuring Large and Huge Pages

As noted in Large and Huge Pages, the use of large and huge memory pages where supported can be of significant performance benefit and is highly recommended.

The default memory page size on Linux systems is 4 KB. Most current Linux distributions include an option for Huge Pages, that is, a memory page size of 2 MB or 1 GB depending on system configuration. Use of Huge Pages saves memory by saving space in page tables. When Huge Pages are configured, the system automatically uses them in memory allocation. InterSystems recommends the use of Huge Pages on systems hosting Caché under most circumstances.

On Linux platforms, if shared memory is allocated in Huge Pages, they are automatically locked in memory and no further action is required. You can configure Caché to lock the shared memory segment in memory to prevent paging as described in the memlock entry of the Configuration Parameter File Reference.

To configure Huge Pages on Linux, do the following:

1. Check the status.

   /proc/meminfo contains Huge Pages information. By default, no Huge Pages are allocated. Default Huge Page size is 2 MB. For example:

   HugePages_Total:     0
   HugePages_Free:      0
   HugePages_Rsvd:      0
   Hugepagesize:     2048 KB
   

   

2. Change the number of Huge Pages.

   You can change the system parameter directly: For example, to allocate 2056 Huge Pages, execute:

   # echo 2056 > /proc/sys/vm/nr_hugepages
   

   
   Note:

   Alternatively, you can use sysctl(8) to change it:

   # sysctl -w vm.nr_hugepages=2056  
   

   

   Huge pages must be allocated contiguously, which may require a reboot. Therefore, to guarantee the allocation, as well as to make the change permanent, do the following:

   1. Enter a line in /etc/sysctl.conf file:

      echo "vm.nr_hugepages=2056" >> /etc/sysctl.conf  
      

      

   2. Reboot the system.

   3. Verify meminfo after reboot; for example:

      [root woodcrest grub]# tail -4 /proc/meminfo
      HugePages_Total:  2056
      HugePages_Free:   2056
      HugePages_Rsvd:      0
      Hugepagesize:     2048 KB
      

      

3. Verify the use of Huge Pages by Caché.

   When Caché is started, it reports how much shared memory was allocated; for example, a message similar to the following is displayed (and included in the cconsole.log file):

   Allocated 3580MB shared memory: 3000MB global buffers, 226MB routine buffers
   

   

   The amount of memory available in Huge Pages should be greater than the total amount of shared memory to be allocated; if it is not greater, Huge Pages are not used.

   Note:

   Huge Pages are allocated from physical memory. Only applications and processes using Huge Pages can access this memory.

   Physical memory not allocated for Huge Pages is the only memory available to all other applications and processes.

   It is not advisable to specify HugePages_Total much higher than the shared memory amount because the unused memory will not be available to other components.

   If Caché fails to allocate Huge Pages on start-up and switches to standard pages, Caché will be allocating shared memory from the same memory pool as all other jobs.

   When Caché is configured to lock the shared memory segment in memory to prevent paging, Huge Pages can provide the required increase in the maximum size that may be locked into memory, as described in the Locked-in Memory section of the Red Hat Linux Platform Notes in this chapter.

AIX® supports multiple page sizes: 4 KB, 64 KB, 16 MB, and 16 GB. Use of 4 KB and 64 KB pages is transparent to Caché. In order for Caché to use 16 MB large pages, you must configure them within AIX®. AIX® does not automatically change the number of configured large or huge pages based on demand. Currently, Caché does not use 16 GB huge pages.

Large pages should be configured only in high-performance environments because memory allocated to large pages can be used only for large pages.

To allocate large pages, users must have the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities or have root authority unless memlock=64.

By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages then it is allocated in standard (small) pages. For finer grain control over large pages, see memlock in the Caché Parameter File Reference.

Configure large pages using the vmo command as follows:

vmo -r -o lgpg_regions=<LargePages> -o lgpg_size=<LargePageSize>

where <LargePages> specifies the number of large pages to reserve, and <LargePageSize> specifies the size, in bytes, of the hardware-supported large pages.

For example, the following command configures 1 GB of large pages:

# vmo -r -o lgpg_regions=64 -o lgpg_size=16777216

Once you have configured large pages, run the bosboot command to save the configuration in the boot image. After the system comes up, enable it for pinned memory using the following vmo command:

vmo -o v_pinshm=1

However, if memlock=64, vmo -o v_pinshm=1 is not required. For more information on memlock, see memlock in the Caché Parameter File Reference.

Calculating Disk Requirements

Your Caché instance needs disk space for the following items:

• 67 MB for Caché.

• 3 MB for the Caché Server Pages (CSP).

• 3.5 MB for Caché ODBC support.

• 2.5 MB for the Caché manager sources.

• 6.6 MB for the Caché engine link libraries.

• Space for your Caché application database.

• Approximately 12.5% of the buffer pool size for the initial size of the write image journal file. If your disk does not have enough space for the write image journal file, when you start Caché it displays a message indicating that the system did not start.

• Desired space for journal files.

Although you do not need to remove any installation files after completing the installation procedure, you can do so if you are short on disk space. The installation program tells you how much space can be saved, and asks if you want to delete the installation files.

Determining the Number of Global Buffers

Caché supports the following maximum values for the number of global buffers:

• For 32-bit platforms, any 8-KB buffers that are:

  • Less than 2GB for 32-bit platforms

  The 2-GB value is the total address space the operation system allocates for the process data, which includes not only shared memory, but other Caché and operating system data as well. Therefore, it represents an upper limit that is not achievable in practice.

• For 64-bit platforms:

  The number of global buffers is limited only by the operating system and the available memory.

For guidelines for your initial allocation of memory to the database cache (the global buffer pool), see Calculating Initial Memory Requirements; for the procedure for this allocation, see Allocating Memory Within Caché. For further information about the database cache, see globals in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.

Determining Number of Routine Buffers

Caché supports the following maximum value for the number of routine buffers:

65,535

Set your values to less than this maximum number of buffers.

For guidelines for your initial allocation of memory to the database cache (the global buffer pool), see Calculating Initial Memory Requirements; for the procedure for this allocation, see Allocating Memory Within Caché. For further information about the database cache, see routines in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.

Determining Maximum Number of Users

The maximum users allowed by Caché is the lowest of the following values:

• License limit

• # of semaphores - 4

For more information, see Determining License Capacity and Usage in the “Managing Caché Licensing” chapter of the Caché System Administration Guide.

Determining Maximum Database Size

The ulimit parameter in UNIX® determines the maximum file size available to a process. For the Caché Manager group, the value of ulimit should either be unlimited or as large as the largest database you may have.

For more information, see Configuring Databases in the “Configuring Caché” chapter of the Caché System Administration Guide.

Setting Values for Tunable UNIX® Parameters

Caché uses a configurable number of semaphores, in sets whose size you define. The parameters SEMMNI, SEMMNS, and SEMMSL reflect the number of semaphores per set and the total number of semaphores Caché uses. The UNIX®/Linux parameters that govern shared memory allocation are SHMMAX, SHMMNI, SHMSEG, and SHMALL. Caché uses shared memory and allocates one segment of shared memory; the size of this segment depends on the area set aside for global buffers and routine buffers. It uses the following formula to determine the segment's minimum size:

                       space required for routine buffers
                     +  space required for global buffers
                     +                               4 MB
                    _____________________________________
                     =         Shared memory segment size

If you are distributing your data across multiple computers, Caché allocates a second segment; by default, there is no memory allocated for the second segment. (If you plan to use distributed data, contact your vendor or InterSystems support for configuration guidelines.) You can alter NBUF and NHBUF according to other system requirements. Because Caché does all its own disk buffering, you should keep NBUF and NHBUF small. The following table lists the most common names of the UNIX® parameters that you may need to change, the minimum value InterSystems recommends for each parameter, and a brief description of each. Verify that your parameter values are set to at least the minimum value. Certain parameters may not be implemented on all platforms or may be referred to differently. Refer to platform-specific tuning notes for more information.

* This is the minimum value for SHMALL required for Caché UNIX®. You must also take into account any other applications that use shared memory. If you are unsure of other shared memory use, calculate SHMALL as SHMSEG multiplied by SHMMAX, in pages; this larger value suffices in all cases.

Adjusting Maximum File Size

The hard limit for the maximum file size (RLIMIT_FSIZE) on any system running Caché must be unlimited. Set the value to unlimited on the operating system before installing. Make sure that the limit is set to unlimited for both the root user and the user who will run Caché. Caché also sets the process soft limit to RLIMIT_FSIZE in its daemons to prevent I/O errors.

See the operating system documentation for your platform for instructions on how to set the system hard limit for the maximum file size, RLIMIT_FSIZE.