Caché High Availability Guide
Using IBM PowerHA SystemMirror with Caché
[Home] [Back] [Next]
InterSystems: The power behind what matters   
Class Reference   
Search:    

Caché can be configured as a resource controlled by IBM PowerHA SystemMirror. This appendix highlights the key portions of the configuration of PowerHA including how to incorporate the custom Caché application controller and monitor script. Refer to your IBM documentation and consult with IBM on all cluster configurations.

When using Caché in a high availability environment controlled by IBM PowerHA SystemMirror:
  1. Install the hardware and operating system according to your vendor recommendations for high availability, scalability and performance; for more information, see Hardware Configuration.
  2. Configure IBM PowerHA SystemMirror with shared disk and virtual IP. Verify that common failures are detected and the cluster continues operating; for more information, see IBM PowerHA SystemMirror Configuration.
  3. Install Caché according to the guidelines in this appendix and verify connectivity to your application via the virtual IP; for more information, see Install Caché in the Cluster.
  4. Test disk failures, network failures, and system crashes. Test and understand your application’s response to such failures; for more information, see Test and Maintenance.
Hardware Configuration
Configure the hardware according to best practices for the application. In addition to adhering to the recommendations of IBM and your hardware vendor, consider the following:
IBM PowerHA SystemMirror Configuration
Prior to installing Caché and your Caché-based application, follow the recommendations described in this section when configuring IBM AIX® and PowerHA SystemMirror.
Note:
These recommendations assume a two-node cluster in which both nodes are identical. If your configuration is different, consult with IBM and the InterSystems Worldwide Response Center (WRC) for guidance.
IBM AIX®
When configuring AIX® on the nodes in the cluster, ensure that:
IBM PowerHA SystemMirror
This appendix assumes you are using IBM PowerHA SystemMirror version 7.1.0 or later.
Note:
Older versions also work, but they may have slightly different configuration options. If your version of IBM PowerHA SystemMirror is earlier than 7.1.0, consult with IBM and the InterSystems Worldwide Response Center (WRC) for guidance.
InterSystems provides a sample Caché application and monitor script as part of a development install of Caché. The sample script can be found in the dev/cache/HAcluster/IBM/ subdirectory under the Caché install directory after completing a development install. There is no requirement to do a development install in the cluster. Copy the dev/cache/HAcluster/IBM/cache.sh script to all cluster members from any development install. Make sure the ownerships and permissions on the copied cache.sh file are as required by PowerHA SystemMirror. This script should be sufficient for most cluster configurations but should be tested and may require modification to suit unique cluster topologies.
The procedures in this appendix assume the following configuration choices:
In general, do the following:
  1. Install and cable all hardware, including disks and network.
  2. Create a resource group (cacheprod_rg) that includes the network paths and volume groups of the shared disk. To configure the resource group properly:
Install Caché in the Cluster
After the resource group has been created and configured, install Caché in the cluster. The procedure is outlined below.
There are different procedures, depending on whether you are installing only one instance of Caché or multiple instances of Caché, as described in the following subsections:
Installing a single instance of Caché in the cluster is common in production clusters. In development and test clusters it is common to have multiple instances of Caché controlled by the cluster software. If it is possible that you will install multiple instances of Caché in the future, follow the procedure for multiple instances.
Note:
For information about upgrading Caché in an existing failover cluster, see Upgrading a Cluster in the “Upgrading Caché” chapter of the Caché Installation Guide.
Installing a Single Instance of Caché in the Cluster
Use the following procedure to install and configure a single instance of Caché and your application.
Note:
If any Caché instance that is part of a failover cluster is to be added to a Caché mirror, you must use the procedure described in Installing Multiple Instances of Caché in the Cluster, rather than the procedure in this section.
  1. Bring the Caché resource group (cacheprod_rg) online on one node. This mounts all required disks, allowing for the proper installation of Caché:
    1. Check the file and directory ownerships and permissions on all mount points and subdirectories.
    2. Prepare to install Caché by reviewing the Installing Caché on UNIX® and Linux chapter of the Caché Installation Guide.
  2. Create a link from /usr/local/etc/cachesys to the shared disk. This forces the Caché registry and all supporting files to be stored on the shared disk resource configured as part of cacheprod_rg.
    A good choice is to use a ./usr/local/etc/cachesys/ subdirectory under your install directory. For example, assuming Caché is installed in the /cacheprod/cachesys/ subdirectory, specify the following on all nodes in the cluster:
    mkdir –p /cacheprod/cachesys/usr/local/etc/cachesys
    mkdir –p /usr/local/etc/
    ln –s /cacheprod/cachesys/usr/local/etc/cachesys/ /usr/local/etc/cachesys
  3. Run Caché cinstall on the node with the mounted disks.
    Important:
    Be sure the users and groups (either default or custom) have already been created on all nodes in the cluster, and that they all have the same UID and GIDs.
  4. Stop Caché and move the resource group to the other nodes.
  5. On the other node in the cluster, create the link in /usr/local/etc and the links in /usr/bin for ccontrol and csession:
    mkdir -p /usr/local/etc/
    ln –s /cacheprod/cachesys/usr/local/etc/cachesys/ /usr/local/etc/cachesys 
    
    ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol 
    ln –s /usr/local/etc/cachesys/csession /usr/bin/csession 
  6. Test connectivity via the virtual IP address.
    Important:
    Be sure the application, all interfaces, any ECP clients, etc. use the virtual IP address as configured here to connect to the Caché resource.
  7. Place the cache.sh file in /etc/cluster/<app>/, where <app> is the application controller name (for more information, see Application Controllers and Monitors in this appendix). Ensure the permissions, owner and group allow this script to be executable.
  8. Test that the cache.sh script stops and starts the newly installed Caché instance. Assuming the application controller is named cacheprod:
  9. Offline the resource group to prepare to add the control scripts and monitors to your resource group; for more information, see Application Controllers and Monitors in this appendix.
Installing Multiple Instances of Caché in the Cluster
To install multiple instances of Caché and your application, use the following procedure.
Note:
If any Caché instance that is part of a failover cluster is to be added to a Caché mirror, the ISCAgent (which is installed with Caché) must be properly configured; see Configuring the ISCAgent in the “Mirroring” chapter of this guide for more information.
  1. Bring the Caché resource group online on one node. This mounts all required disks and allows for the proper installation of Caché:
    1. Check the file and directory ownerships and permissions on all mount points and subdirectories.
    2. Prepare to install Caché by reviewing the Installing Caché on UNIX® and Linux chapter of the Caché Installation Guide.
  2. Run Caché cinstall on the node with the mounted disks,
    Important:
    Be sure the users and groups (either default or custom) have already been created on all nodes in the cluster, and that they all have the same UID and GIDs.
  3. The /usr/local/etc/cachesys directory and all its files must be available to all nodes at all times. Therefore, after Caché is installed on the first node, copy /usr/local/etc/cachesys to each node in the cluster, as follows:
    cd /usr/local/etc/
    scp –p cachesys node2:/usr/local/etc/
  4. Verify that ownerships and permissions on this cachesys directory and its files are identical on all nodes.
    Note:
    Keep the Caché registries on all nodes in sync using ccontrol create and/or ccontrol update after Caché upgrades; for example:
    ccontrol create CSHAD directory=/myshadow/ versionid=2011.1.475
  5. Stop Caché and move the resource group to the other nodes.
  6. On the other nodes in the cluster, create links in /usr/bin for ccontrol and csession:
    ln –s /usr/local/etc/cachesys/ccontrol /usr/bin/ccontrol 
    ln –s /usr/local/etc/cachesys/csession /usr/bin/csession
  7. Test connectivity via the virtual IP address. Be sure the application, all interfaces, any ECP clients, etc. use the virtual IP address as configured here to connect to the Caché resource.
  8. Place the cache.sh file in /etc/cluster/<app>/, where <app> is the application controller name (for more information, see Application Controllers and Monitors in this appendix). Ensure the permissions, owner and group allow this script to be executable.
  9. Test that the cache.sh script stops and starts the newly installed Caché instance. Assuming the application controller is named cacheprod:
  10. Offline the resource group to prepare to add the control scripts and monitors to your resource group; for more information, see Application Controllers and Monitors in this appendix.
Application Controllers and Monitors
Once Caché is installed, configure the application controller resource. For example, assuming the resource group that contains all the disk and IP resources is called cacheprod_rg and the chosen Application Controller Name is cacheprod, you would configure the following in the application controller:
You can also configure the optional custom application monitors:
Note:
If you configure a long-running monitor, you must also configure startup monitor.
Long-running Monitor
Configure the long-running monitor with the following values:
A long-running monitor uses the monitor function of the cache.sh script. This function, in turn, performs a ccontrol list to determine the status of the application:
Startup Monitor
Configure the startup monitor with the following values:
The startup monitor uses the startmonitor function of the cache.sh script. This function, in turn, performs a ccontrol list to determine the status of the application:
Application Considerations
Consider the following for your applications:
Test and Maintenance
Upon first setting up the cluster, be sure to test that failover continues to work as planned. This also applies any time changes are made to the operating system, its installed packages, the disk, the network, Caché, or your application.
In addition to the topics described in this section, you should contact the InterSystems Worldwide Response Center (WRC) for assistance when planning and configuring your IBM PowerHA SystemMirror resource to control Caché. The WRC can check for any updates to the cache.sh, as well as discuss failover and HA strategies.
Typical Full Scale Testing Must Go Beyond a Controlled Service Relocation
While service relocation testing is necessary to validate that the package configuration and the service scripts are all functioning properly, be sure to also test response to simulated failures.
Be sure to test failures such as:
Testing should include a simulated or real application load, as follows:
Keep Patches and Firmware Up to Date
Avoid known problems by adhering to a patch and update schedule.
Use Caché Monitoring Tools
Keep an eye on the IBM PowerHA hacmp.out file. The Caché cache.sh script logs time-stamped information to this file during cluster events. To troubleshoot any problems, search for the phrases “CACHE ERROR” and/or “CACHE INFO” in the hacmp.out log to quickly find the period during which the cache.sh script was executing.
Use the Caché console log, the Caché Monitor and the Caché System Monitor to be alerted to problems with the database that may not be caught by the cluster software. (See the chapters Monitoring Caché Using the Management Portal, Using the Caché Monitor and Using the Caché System Monitor in the Caché Monitoring Guide for information about these tools.)