Preparing to Install Caché on UNIX®, Linux, and macOS

Configuring Large and Huge Pages

As noted in Large and Huge Pages, the use of large and huge memory pages where supported can be of significant performance benefit and is highly recommended.

The default memory page size on Linux systems is 4 KB. Most current Linux distributions include an option for Huge Pages, that is, a memory page size of 2 MB or 1 GB depending on system configuration. Use of Huge Pages saves memory by saving space in page tables. When Huge Pages are configured, the system automatically uses them in memory allocation. InterSystems recommends the use of Huge Pages on systems hosting Caché under most circumstances.

On Linux platforms, if shared memory is allocated in Huge Pages, they are automatically locked in memory and no further action is required. You can configure Caché to lock the shared memory segment in memory to prevent paging as described in the memlock entry of the Configuration Parameter File Reference.

To configure Huge Pages on Linux, do the following:

1. Check the status.

   /proc/meminfo contains Huge Pages information. By default, no Huge Pages are allocated. Default Huge Page size is 2 MB. For example:

   HugePages_Total:     0
   HugePages_Free:      0
   HugePages_Rsvd:      0
   Hugepagesize:     2048 KB
   

   

2. Change the number of Huge Pages.

   You can change the system parameter directly: For example, to allocate 2056 Huge Pages, execute:

   # echo 2056 > /proc/sys/vm/nr_hugepages
   

   
   Note:

   Alternatively, you can use sysctl(8) to change it:

   # sysctl -w vm.nr_hugepages=2056  
   

   

   Huge pages must be allocated contiguously, which may require a reboot. Therefore, to guarantee the allocation, as well as to make the change permanent, do the following:

   1. Enter a line in /etc/sysctl.conf file:

      echo "vm.nr_hugepages=2056" >> /etc/sysctl.conf  
      

      

   2. Reboot the system.

   3. Verify meminfo after reboot; for example:

      [root woodcrest grub]# tail -4 /proc/meminfo
      HugePages_Total:  2056
      HugePages_Free:   2056
      HugePages_Rsvd:      0
      Hugepagesize:     2048 KB
      

      

3. Verify the use of Huge Pages by Caché.

   When Caché is started, it reports how much shared memory was allocated; for example, a message similar to the following is displayed (and included in the cconsole.log file):

   Allocated 3580MB shared memory: 3000MB global buffers, 226MB routine buffers
   

   

   The amount of memory available in Huge Pages should be greater than the total amount of shared memory to be allocated; if it is not greater, Huge Pages are not used.

   Note:

   Huge Pages are allocated from physical memory. Only applications and processes using Huge Pages can access this memory.

   Physical memory not allocated for Huge Pages is the only memory available to all other applications and processes.

   It is not advisable to specify HugePages_Total much higher than the shared memory amount because the unused memory will not be available to other components.

   If Caché fails to allocate Huge Pages on start-up and switches to standard pages, Caché will be allocating shared memory from the same memory pool as all other jobs.

   When Caché is configured to lock the shared memory segment in memory to prevent paging, Huge Pages can provide the required increase in the maximum size that may be locked into memory, as described in the Locked-in Memory section of the Red Hat Linux Platform Notes in this chapter.

AIX® supports multiple page sizes: 4 KB, 64 KB, 16 MB, and 16 GB. Use of 4 KB and 64 KB pages is transparent to Caché. In order for Caché to use 16 MB large pages, you must configure them within AIX®. AIX® does not automatically change the number of configured large or huge pages based on demand. Currently, Caché does not use 16 GB huge pages.

Large pages should be configured only in high-performance environments because memory allocated to large pages can be used only for large pages.

To allocate large pages, users must have the CAP_BYPASS_RAC_VMM and CAP_PROPAGATE capabilities or have root authority unless memlock=64.

By default, when large pages are configured, the system automatically uses them in memory allocation. If shared memory cannot be allocated in large pages then it is allocated in standard (small) pages. For finer grain control over large pages, see memlock in the Caché Parameter File Reference.

Configure large pages using the vmo command as follows:

vmo -r -o lgpg_regions=<LargePages> -o lgpg_size=<LargePageSize>

where <LargePages> specifies the number of large pages to reserve, and <LargePageSize> specifies the size, in bytes, of the hardware-supported large pages.

For example, the following command configures 1 GB of large pages:

# vmo -r -o lgpg_regions=64 -o lgpg_size=16777216

Once you have configured large pages, run the bosboot command to save the configuration in the boot image. After the system comes up, enable it for pinned memory using the following vmo command:

vmo -o v_pinshm=1

However, if memlock=64, vmo -o v_pinshm=1 is not required. For more information on memlock, see memlock in the Caché Parameter File Reference.

Calculating Disk Requirements

Your Caché instance needs disk space for the following items:

• 67 MB for Caché.

• 3 MB for the Caché Server Pages (CSP).

• 3.5 MB for Caché ODBC support.

• 2.5 MB for the Caché manager sources.

• 6.6 MB for the Caché engine link libraries.

• Space for your Caché application database.

• Approximately 12.5% of the buffer pool size for the initial size of the write image journal file. If your disk does not have enough space for the write image journal file, when you start Caché it displays a message indicating that the system did not start.

• Desired space for journal files.

Although you do not need to remove any installation files after completing the installation procedure, you can do so if you are short on disk space. The installation program tells you how much space can be saved, and asks if you want to delete the installation files.

Determining the Number of Global Buffers

Caché supports the following maximum values for the number of global buffers:

• For 32-bit platforms, any 8-KB buffers that are:

  • Less than 2GB for 32-bit platforms

  The 2-GB value is the total address space the operation system allocates for the process data, which includes not only shared memory, but other Caché and operating system data as well. Therefore, it represents an upper limit that is not achievable in practice.

• For 64-bit platforms:

  The number of global buffers is limited only by the operating system and the available memory.

For guidelines for your initial allocation of memory to the database cache (the global buffer pool), see Calculating Initial Memory Requirements; for the procedure for this allocation, see Allocating Memory Within Caché. For further information about the database cache, see globals in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.

Determining Number of Routine Buffers

Caché supports the following maximum value for the number of routine buffers:

65,535

Set your values to less than this maximum number of buffers.

For guidelines for your initial allocation of memory to the database cache (the global buffer pool), see Calculating Initial Memory Requirements; for the procedure for this allocation, see Allocating Memory Within Caché. For further information about the database cache, see routines in the “config” section of the Caché Parameter File Reference and Memory and Startup Settings in the “Configuring Caché” chapter of the Caché System Administration Guide.

Determining Maximum Number of Users

The maximum users allowed by Caché is the lowest of the following values:

• License limit

• # of semaphores - 4

For more information, see Determining License Capacity and Usage in the “Managing Caché Licensing” chapter of the Caché System Administration Guide.

Determining Maximum Database Size

The ulimit parameter in UNIX® determines the maximum file size available to a process. For the Caché Manager group, the value of ulimit should either be unlimited or as large as the largest database you may have.

For more information, see Configuring Databases in the “Configuring Caché” chapter of the Caché System Administration Guide.

Setting Values for Tunable UNIX® Parameters

Caché uses a configurable number of semaphores, in sets whose size you define. The parameters SEMMNI, SEMMNS, and SEMMSL reflect the number of semaphores per set and the total number of semaphores Caché uses. The UNIX®/Linux parameters that govern shared memory allocation are SHMMAX, SHMMNI, SHMSEG, and SHMALL. Caché uses shared memory and allocates one segment of shared memory; the size of this segment depends on the area set aside for global buffers and routine buffers. It uses the following formula to determine the segment's minimum size:

                       space required for routine buffers
                     +  space required for global buffers
                     +                               4 MB
                    _____________________________________
                     =         Shared memory segment size

If you are distributing your data across multiple computers, Caché allocates a second segment; by default, there is no memory allocated for the second segment. (If you plan to use distributed data, contact your vendor or InterSystems support for configuration guidelines.) You can alter NBUF and NHBUF according to other system requirements. Because Caché does all its own disk buffering, you should keep NBUF and NHBUF small. The following table lists the most common names of the UNIX® parameters that you may need to change, the minimum value InterSystems recommends for each parameter, and a brief description of each. Verify that your parameter values are set to at least the minimum value. Certain parameters may not be implemented on all platforms or may be referred to differently. Refer to platform-specific tuning notes for more information.

* This is the minimum value for SHMALL required for Caché UNIX®. You must also take into account any other applications that use shared memory. If you are unsure of other shared memory use, calculate SHMALL as SHMSEG multiplied by SHMMAX, in pages; this larger value suffices in all cases.

Adjusting Maximum File Size

The hard limit for the maximum file size (RLIMIT_FSIZE) on any system running Caché must be unlimited. Set the value to unlimited on the operating system before installing. Make sure that the limit is set to unlimited for both the root user and the user who will run Caché. Caché also sets the process soft limit to RLIMIT_FSIZE in its daemons to prevent I/O errors.

See the operating system documentation for your platform for instructions on how to set the system hard limit for the maximum file size, RLIMIT_FSIZE.

I/O Pacing Parameters

AIX® implements an I/O pacing algorithm that may hinder Caché write daemons. In AIX® 5.2 and AIX® 5.3, I/O pacing is automatically enabled when using HACMP clustering; beginning in AIX® 6.1, however, I/O pacing is enabled on all systems and the default high-water mark is set higher than in earlier releases.

If write daemons are slowing or stalling, you may have to adjust the high-water mark; for information, see the “Using Disk-I/O Pacing” section of the AIX® Performance Management Guide at the following IBM web page:http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.prftungd/doc/prftungd/disk_io_pacing.htmOpens in a new tab.

File System Mount Option

Different mount options may improve performance for some workloads.

For more information on JFS2 file systems that contain only journal files, see UNIX® File System Recommendations in the “Journaling” chapter of the Caché Data Integrity Guide.

To improve recovery speed using the CACHE.WIJ file after a hard shutdown or system crash, InterSystems recommends a mount option that includes file system buffering (for example, rw) for the file system that contains the CACHE.WIJ file.

For information about mount options, see the AIX® Commands Reference at the following IBM web page: http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.cmds/doc/aixcmds3/mount.htmOpens in a new tab.

Memory Management Parameters

The number of file systems and the amount of activity on them can limit the number of memory structures available to JFS or JFS2, and delay I/O operations waiting for those memory structures.

To monitor these metrics, issue a vmstat -vs command, wait two minutes, and issue another vmstat -vs command. The output looks similar to the following:

# vmstat -vs
              1310720 memory pages
              1217707 lruable pages
               144217 free pages
                    1 memory pools
               106158 pinned pages
                 80.0 maxpin percentage
                 20.0 minperm percentage
                 80.0 maxperm percentage
                 62.8 numperm percentage
               764830 file pages
                  0.0 compressed percentage
                    0 compressed pages
                 32.1 numclient percentage
                 80.0 maxclient percentage
               392036 client pages
                    0 remote pageouts scheduled
                    0 pending disk I/Os blocked with no pbuf
                 5060 paging space I/Os blocked with no psbuf
              5512714 filesystem I/Os blocked with no fsbuf
               194775 client filesystem I/Os blocked with no fsbuf
                    0 external pager filesystem I/Os blocked with no fsbuf

If you see an increase in the following parameters, increase the values for better Caché performance:

• pending disk I/Os blocked with no pbuf

• paging space I/Os blocked with no psbuf

• filesystem I/Os blocked with no fsbuf

• client filesystem I/Os blocked with no fsbuf

• external pager filesystem I/Os blocked with no fsbuf

When increasing these parameters from the default values:

1. Increase the current value by 50%.

2. Check the vmstat output.

3. Run vmstat twice, two minutes apart.

4. If the field is still increasing, increase again by the same amount; continue this step until the field stops increasing between vmstat reports.

See the following IBM web pages for more detailed information:

• For a complete description of each of the fields reported by vmstat, see the vmstat Command page of AIX® Commands Reference, Volume 6, v - z at:

  http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.cmds/doc/aixcmds6/vmstat.htmOpens in a new tab

• For instructions on how to increase these parameters, see the VMM page replacement tuning section of the AIX® Performance Management Guide at:

  http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.prftungd/doc/prftungd/vmm_page_replace_tuning.htmOpens in a new tab

• For a complete description of managing I/O tunable parameters, see the ioo Command page of AIX® Commands Reference, Volume 3, i - m at:

  http://publib.boulder.ibm.com/infocenter/systems/scope/aix/topic/com.ibm.aix.cmds/doc/aixcmds3/ioo.htmOpens in a new tab

AIX® Tunable Parameters

None of the following listed parameters requires tuning because each is dynamically adjusted as needed by the kernel. See the appropriate AIX® operating system documentationOpens in a new tab for more information.

The following table lists the tunable parameters for the IBM pSeries AIX® 5.2 operating system.

maxuproc, which specifies the maximum number of processes than can be started by a single nonroot user, is a tunable parameter that can be adjusted as described in this subsection.

If this parameter is set too low then various components of the operating system can fail as more and more users attempt to start processes; these failures include loss of CSP pages, background tasks failing, etc. Therefore, you should set the maxuproc parameter to be higher than the maximum number of processes that might be started by a nonroot user (including interactive users, web server processes, and anything that might start a process).

InterSystems suggests that you set maxuproc to be double your expected maximum process count which gives a margin of error but still provides protection from runaway processes. For example, if your system has 1000 interactive users and often runs 500 background processes, then a value of at least 3000 would be a good choice.

The maxuproc value can be examined and changed either from the command line or from the smit/smitty administrator utilities, both as root user, as follows:

• From the command line, view the current setting:

  # lsattr -E -l sys0 -a maxuproc
  

  

  then modify the value:

  # chdev -l sys0 -a maxuproc=NNNNNN
  

  

  where NNNNNN is the new value.

• From the administrator utility smit (or smitty) choose System Environments > Change / Show Characteristics of Operating System > Maximum number of PROCESSES allowed per user.

If you increase the value of maxuproc, the change is effective immediately. If you decrease the value of maxuproc, the change does not take effect until the next system reboot. In both cases the change persists over system reboots.

Shared Memory Limit

The default shared memory limit (shmmax) on Linux platforms is 32 MB. This value is too small for Caché, but it can be changed in the proc file system without a restart.

For example, to allow 128 MB, type the following command:

$ echo 134217728 >/proc/sys/kernel/shmmax

You can put this command into a startup script.

Alternatively, you can use sysctl(8), if available, to control this parameter. Look for a file called /etc/sysctl.conf and add a line similar to the following:

kernel.shmmax = 134217728

This file is usually processed at startup, but sysctl can also be called explicitly later.

Other parameters are sufficiently sized for a Caché application. To view the values of other parameters, look in the files /usr/src/linux/include/asm-xxx/shmparam.h and /usr/src/linux/include/linux/sem.h.

For more information, reference “The proc File SystemOpens in a new tab” chapter of the Red Hat Enterprise Linux 4: Reference Guide.

Locked-in Memory

On Linux platforms, you can configure Caché to lock the shared memory segment in memory to prevent paging as described in the memlock entry of the Caché Parameter File Reference. If shared memory is allocated in Huge Pages, they are automatically locked in memory and no further action is required. Otherwise, you must increase the maximum size that may be locked into memory. The default value is 32 KB. View the current value using the ulimit command.

For example, to display all current limits:

bash$ ulimit -a 
core file size (blocks, -c) unlimited 
data seg size ( KBytes, -d) unlimited 
file size (blocks, -f) unlimited 
pending signals (-i) 1024 
max locked memory (KBytes, -l) 32 <---------- THIS ONE 
max memory size (KBytes, -m) unlimited 
open files (-n) 1024 
pipe size (512 bytes, -p) 8 
POSIX message queues (bytes, -q) 819200 
stack size ( KBytes, -s) 10240 
cpu time (seconds, -t) unlimited 
max user processes (-u) 49000 
virtual memory ( KBytes, -v) unlimited 
file locks (-x) unlimited 

To display only max-locked memory, use the -l option:

bash$ ulimit -l 
32 

If you have privileges, you can alter the value directly using the ulimit command; however, it is better to update the memlock parameter in the /etc/security/limits.conf file. If the memlock limit is too low, Linux reports a ENOMEM - "Not enough memory" error, which does not make the cause obvious. The actual memory is allocated; it is the lock that fails.

For more information, see memlock in the Caché Parameter File Reference.

Adjusting for Large Number of Concurrent Processes

Make the following adjustments if you are running a system that requires a large number of processes or telnet logins.

1. In the /etc/xinetd.d/telnet file, add the following line:

   instances = unlimited
   
   

   

2. In the /etc/xinetd.conf file, add or change the instances setting to:

   instances = unlimited
   
   

   

3. After you make these modifications, restart the xinetd services with:

   # service xinetd restart
   

   

4. The default pty (pseudo terminal connection) limit is 4096. If this is not sufficient, add or change the maximum pty line in the /etc/sysctl.conf file. For example:

   kernel.pty.max=10000
   

   

Dirty Page Cleanup

On large memory systems (for example, 8GB or larger), when doing numerous flat-file writes (for example, Caché backups or file copies), you can improve performance by adjusting the following parameters, which are located in proc/sys/vm/:

• dirty_background_ratio — Maximum percentage of active that can be filled with dirty pages before pdflush begins to write them. InterSystems recommends setting this parameter to 5.

• dirty_ratio — Maximum percentage of total memory that can be filled with dirty pages before processes are forced to write dirty buffers themselves during their time slice instead of being allowed to do more writes. InterSystems recommends setting this parameter to 10

You can set these variables by adding the following to your /etc/sysctl.conf file:

vm.dirty_background_ratio=5 
vm.dirty_ratio=10

These changes force the Linux pdflush daemon to write out dirty pages more often rather than queue large amounts of updates that can potentially flood the storage with a large burst of updates.”

Shared Memory Limits

The default shared memory limits (shhmax and shmall) on SUSE Linux 32-bit platforms are too small for Caché, and can be changed in the proc file system without a restart.

Caché uses shared memory for database buffers, global buffers, routine buffers, as well as license use. If the machine is being used only for Caché, InterSystems recommends setting the shared memory to approximately half the total memory. For more information, see the subsections of Determining Initial System Configuration in this chapter, and Determining License Capacity and Usage in the “Managing Caché Licensing” chapter of the Caché System Administration Guide.

For example, to allow 512 MB, type the following commands:

#sets shmall and shmmax shared memory
echo 536870912 >/proc/sys/kernel/shmall     #Sets shmall to 512 MB
echo 536870912 >/proc/sys/kernel/shmmax     #Sets shmmax to 512 MB

You can put these commands into a script that is run at startup. The SUSE Linux product documentation recommends you put the commands in the /etc/init.d/boot.local script file.

You can change the settings for the system memory user limits by modifying a file called /etc/profile.local. Add lines similar to the following:

#sets user limits (ulimit) for system memory resources
ulimit -v 512000     #set virtual (swap) memory to 512 MB 
ulimit -m 512000     #set physical memory to 512 MB

In this same file, you can permanently change the values for the PATH and CLASSPATH parameters by adding lines similar to the following:

#sets env values PATH and CLASSPATH
export PATH=$PATH:/usr/cache/bin:/path/to/j2sdk/bin:/.
export CLASSPATH=
      $CLASSPATH:/cache/dev/java/lib/JDK16/cachedb.jar:/path/to/otherjar/file:/.

Locked-in Memory

On Linux platforms, you can configure Caché to lock the shared memory segment in memory to prevent paging as described in the memlock entry of the Caché Parameter File Reference. If shared memory is allocated in Huge Pages, they are automatically locked in memory and no further action is required. Otherwise, see the Locked-in Memory section of the Red Hat Linux Platform Notes in this appendix.

Semaphore Deletion Setting

Under some circumstances, the OS may delete an instance’s semaphores when the instance owner connects to an Ubuntu host, for example using SSH. To prevent this, edit the /etc/systemd/logind.conf file and change RemoveIPC=yes (the default) to RemoveIPC=no.

Updating to a newer version of Ubuntu may revert RemoveIPC to the default value of yes. After updating Ubuntu, be sure to change RemoveIPC to avoid unwanted semaphore deletion.

System Requirements

For information about current system requirements, see the “Supported Technologies” chapter of the online InterSystems Supported PlatformsOpens in a new tab document for this release.

Required C/C++ Runtime Libraries

You must ensure that the required C/C++ runtime is installed on your IBM AIX® system before installing Caché.

Caché for AIX is compiled using the IBM XL C/C++ for AIX 13.1 compiler. If the system on which you are installing Caché does not have the corresponding version of the runtime already installed, you must install these three runtime file sets from runtime package IBM_XL_CPP_RUNTIME_V13.1.0.0_AIX.tar.Z:

• xlC.aix61.rte 13.1

• xlC.rte 13.1

• xlC.msg.en_US.rte 13.1

If these files are not present, Caché installation will not complete.

Full information about and download of this package is available at IBM XL C/C++ Runtime for AIX 13.1Opens in a new tab.

Shared Library Environment Variable for Caché Engine Link Libraries

The Caché Engine link libraries contain a batch file that references any installed C linker.

If you have either the standard UNIX® C libraries or any proprietary C libraries defined in the LIBPATH environment variable, then your environment is ready.

If not, append the paths for the standard UNIX® C libraries to LIBPATH; these paths are /usr/lib and /lib.

Use of Raw Ethernet

In order to use raw Ethernet, an IBM AIX® machine must have the DLPI (Data Link Provider Interface) packages installed. If the machine does not have the DLPI packages, obtain them from your IBM provider and create DLPI devices through the following procedure:

1. Log in as root.

2. In the PSE drivers section of the /etc/pse.conf file, uncomment the four lines that refer to the DLPI drivers.

3. Save the file.

4. Restart the computer.

If the DLPI devices are not installed, the EthernetAddress() method of the %SYSTEM.INetInfoOpens in a new tab class returns a null string rather than information about the Ethernet device.