Skip to main content
************* PRERELEASE CONTENT *************
Previous sectionNext section

Configuring Mirroring

This chapter provides information and procedures for setting up, configuring and managing mirrors and mirror members.

Using InterSystems Cloud Manager to Deploy Mirrored Configurations

InterSystems recommends using InterSystems Cloud Manager (ICM) to deploy InterSystems IRIS, including mirrored configurations. By combining plain text declarative configuration files, a simple command line interface, the widely-used Terraform infrastructure as code tool, and InterSystems IRIS deployment in Docker containers, ICM provides you with a simple, intuitive way to provision cloud or virtual infrastructure and deploy the desired InterSystems IRIS architecture on that infrastructure, along with other services. ICM can significantly simplify the deployment process, especially for complex horizontal cluster configurations.

For more information on using ICM to deploy InterSystems IRIS in mirrored configurations, see Using ICM and ICM Cluster Topology and Mirroring in the InterSystems Cloud Manager Guide.

Mirror Configuration Guidelines

In order to provide a robust, economical HA solution, mirroring is designed to be adaptable to a wide range of system configurations and architectures. However, InterSystems recommends that you adhere to the following general configuration guidelines:

  • InterSystems IRIS instance and platform compatibility — Before identifying the systems to be added to a mirror, be sure to review the requirements described in InterSystems IRIS Instance Compatibility and Member Endianness Considerations.

  • Coequality of failover members — The two failover members in a mirror are assumed to be coequal. There is no way to configure a preference for one to be primary, and the primary and backup roles are reversed as circumstances require. For this reason, the best practice is for the failover system hosts to be as similar to each other as possible, in particular to be configured with similar computing resources; that is, the CPU and memory configuration as well as the disk configuration on the two systems should be comparable.

  • Primary instance configuration and security settings — The configurations of such elements as users, roles, namespaces, and mappings (including global mappings and package mappings) on the primary failover member are not replicated by the mirror on other mirror members. Therefore, any settings required to enable the backup failover members or DR async members to effectively take over from the primary must be manually replicated on those members and updated as needed.

  • Unmirrored data — Only data in mirrored databases on the primary failover member is replicated and synchronized on the backup failover member and async members. Therefore, any files required to enable the backup or a DR async to effectively take over from the primary (including, for example, those related to SQL gateway and webserver configuration) must be manually copied to those members and updated as needed.

    Note:

    A mirrored database’s file streams, located by default in the stream subdirectory of the database directory, are not mirrored. (For information about file streams, see the “Working with Streams” chapter of Defining and Using Classes.)

  • ICMP — Do not disable Internet Control Message Protocol (ICMP) on any system that is configured as a mirror member; mirroring relies on ICMP to detect whether or not members are reachable.

  • Network — InterSystems recommends that you use a high-bandwidth, low-latency, reliable network between the two failover members. If possible, it is desirable to create a private subnet for the two failover members such that the data and control-channel traffic can be routed exclusively on this private network. A slow network could impact the performance of both the primary and the backup failover members, and could directly impact the ability of the backup failover member to take over as primary in the event of a failover. See Network Configuration Considerations and Network Latency Considerations for further discussion of networking requirements and configuration. See also Configuring a Mirror Virtual IP (VIP) for important networking requirements and considerations when using a VIP.

  • Disk subsystem — In order for the backup failover member to keep up with the primary system, the disk subsystems on both failover members should be comparable; for example, if configuring a storage array on the first failover member, it is recommended that you configure a similar storage array on the second failover member. In addition, if network-attached storage (NAS) is used on one or both systems, it is highly recommended that separate network links be configured for the disk I/O and the network load from the mirror data to minimize the chances of overwhelming the network.

  • Journaling performance and journal storage — As journaling/dejournaling is the core of mirror synchronization, it is essential to monitor and optimize the performance of journaling on the failover members. In particular, InterSystems recommends that you increase the generic memory heap size on all mirror members. In the interests of both performance and recoverability, InterSystems also recommends placing the primary and alternate journal directories on storage devices that are separated from the devices used by databases and the write image journal (WIJ), as well as separated from each other. For details, see Journaling Best Practices and Restore Journal Files in the “Journaling” chapter of the Data Integrity Guide.

  • Virtualization — While using mirroring in a virtualized environment provides a hybrid high availability solution that combines the benefits of both, important recommendations apply; see Mirroring in a Virtualized Environment for more information.

  • Task scheduling — When you create a task on a mirror member using the task manager, you must specify whether the task can run on the primary only, any member other than the primary, or any mirror member. Tasks intended to be run on more than one mirror member must be either created separately on the members or exported from the Task Manager on one member and imported on the others. For more information on creating, importing, and exporting tasks, see Using the Task Manager in the “Managing InterSystems IRIS” chapter of the System Administration Guide.

  • Startup — On the primary failover member, you may want to move code from existing ^ZSTU or ^%ZSTART routines to the ^ZMIRROR routine so that it is not executed until the mirror is initialized. See Using the ^ZMIRROR Routine for more information.

Installing the Arbiter

To extend automatic failover to the widest possible range of outage scenarios, as described in Automatic Failover Mechanics, InterSystems recommends that you configure an arbiter for each mirror. As detailed in Locating the Arbiter to Optimize Mirror Availability, the recommended network location for the arbiter depends on the locations of the failover members. A single system can be configured as arbiter for multiple mirrors, provided its location is appropriate for each.

To act as arbiter, a system must have a running ISCAgent process. Because the ISCAgent is installed with InterSystems IRIS, any system that hosts one or more instances of InterSystems IRIS meets this requirement and can be configured as arbiter without further preparation; however, a system hosting one or more failover or DR async members of a mirror should not be configured as arbiter for that mirror.

Systems that do not host an InterSystems IRIS instance can be prepared to act as arbiter in either of these ways:

  • Install the ISCAgent using a kit.

    To prepare such a system, download the ISCAgent installation kit appropriate to your arbiter system’s platform from InterSystems and then, to install the ISCAgent:

    • On Windows systems, simply execute the installation file, for example ISCAgent-2018.1.0.540.0-win_x64.exe.

    • On UNIX®, Linux, and macOS systems, unpack the single file installation kit if necessary, then execute irisinstall at the top level of the installation kit, /ISCAgent. For example:

      [root@arbiterhost home]# gunzip ISCAgent-2018.1.0.540.0-lnxrhx64.tar.gz
      [root@arbiterhost home]# tar -xf ISCAgent-2018.1.0.540.0-lnxrhx64.tar
      [root@arbiterhost home]# ./ISCAgent/irisinstall
      
      Copy code to clipboard
  • Deploy the ISCAgent in a container.

    A containerized ISCAgent can be deployed on any Linux system to act as arbiter. For information about obtaining and using the arbiter image provided by InterSystems, see Mirroring with InterSystems IRIS Containers in Running InterSystems Products in Containers.

Important:

Ensure that the ISCAgent process on the arbiter system is configured to start at system startup; see Starting and Stopping the ISCAgent for more information.

Note:

There are ISCAgent installation kits for all platforms on which InterSystems IRIS is supported; see the online InterSystems Supported Platforms document for this release for a list of supported platforms.

The ISCAgent serving as arbiter need not be of the same InterSystem IRIS version as the members of a mirror for which it is configured. However, InterSystems recommends that you upgrade the arbiter when you upgrade the mirror so that you can be sure of having the latest version of the ISCAgent.

Starting the ISCAgent

An InterSystems IRIS instance cannot be added to a mirror as a failover or DR async member unless the ISCAgent process is running on its host system. The ISCAgent must be configured to start automatically at system startup; see Starting and Stopping the ISCAgent for more information.

Securing Mirror Communication with TLS Security

To provide security within a mirror, you can configure its members to use TLS when communicating with each other. When you require the use of TLS when creating the mirror, all members must use TLS for all communication between them.

See Creating a Mirror for information about creating a mirror with TLS security; see Editing or Removing a Failover Member for information about adding TLS security to an existing mirror.

For a single, comprehensive step-by-step guide to creating a mirror with TLS security, written by an InterSystems Senior Support Specialist, see Creating SSL-Enabled Mirror on InterSystems IRIS Using Public Key Infrastructure (PKI) on InterSystems Developer Community.

Important:

Use of TLS with mirroring is highly recommended. Disabling TLS for a mirror is strongly discouraged.

If an instance has journal or database encryption enabled and you make it the primary failover member of a mirror, you must configure the mirror to use TLS.

The use of TLS for mirror communication by a mirror member requires proper TLS setup on the system hosting the mirror member instance; see Configuring InterSystems IRIS to Use TLS with Mirroring in the “Using TLS with InterSystems IRIS” chapter of the Security Administration Guide for more information.

The use of encrypted journal files in mirroring also requires preparation; for detailed information about journal encryption, see Activating Journal Encryption in a Mirror in this chapter and the “Managed Key Encryption” chapter of the Security Administration Guide.

Using the ^MIRROR Routine

Most mirroring configuration, management and status operations are available in the management portal and also in the ^MIRROR routine, which is executed in the %SYS namespace. However, some operations are available only in the ^MIRROR routine, including forcing the backup failover member to become the primary failover member (see Unplanned Outage of Primary Without Automatic Failover). The procedures provided in this chapter describe the management portal operation if available, but the ^MIRROR option providing the equivalent operation is always noted.

Creating a Mirror

Creating a mirror involves configuring the primary failover member, typically a backup failover member (although this is not required), and optionally one or more async members. After the mirror is created, you can add databases to the mirror.

Important:

Before you can add an InterSystems IRIS instance to a mirror as failover member or async, you must ensure that the ISCAgent process has been started as described in the Starting and Stopping ISCAgent section in this chapter.

The procedure for adding backup and async members requires an additional step if, as recommended by InterSystems, you configure the mirror to use TLS (see Securing Mirror Communication with TLS Security). When this is the case, each new member must be approved on the primary before joining the mirror.

To create and configure a mirror, use the following procedures:

  1. Create a mirror and configure the first failover member

  2. Configure the second failover member

  3. Authorize the second failover member (TLS mirrors only)

  4. Review failover member status in the Mirror Monitor

  5. Configure async mirror members

  6. Authorize new async members (TLS mirrors only)

After you have created the mirror and configured the failover members and optionally one or more async members, add databases to the mirror using the procedures in the Adding databases to a mirror section of this chapter.

Important:

When you add a system task to an instance using the Task Manager (see Using the Task Manager in the “Managing InterSystems IRIS” chapter of the System Administration Guide), the How should task run for Mirror setting determines which mirror members the task runs on, as follows:

  • Runs on the primary failover member only

  • Runs on the backup failover member and async members only (all members except the primary)

  • Runs on all mirror members (primary, backup, and asyncs)

If the instance is not a mirror member, this setting has no effect. On a mirror member, however, if this setting is not specified for a user-defined task, the task will not run, and adding an instance to a mirror does not automatically update the setting.

Therefore you must do one of the following:

  • Always set How should task run for Mirror when you create a task, even if the instance is not (yet) in a mirror.

  • Make sure you review all user-defined tasks when an instance is added to a mirror and set How should task run for Mirror.

Create a Mirror and Configure the First Failover Member

The following procedure describes how to create a mirror and configure the first failover member.

  1. On the first failover member, navigate to the Create Mirror page of the Management Portal (System Administration > Configuration > Mirror Settings > Create a Mirror) and click Create a Mirror. If the option is not active, mirroring has not been enabled; first click Enable Mirror Service, then select the Service Enabled check box and click Save, then select the Create a Mirror option.

  2. On the Create Mirror page, enter the following information in the Mirror Information section:

    1. Mirror Name — Enter a name for the mirror.

      Note:

      Valid names must be from 1 to 15 alphanumeric characters; lowercase letters are automatically replaced with uppercase equivalents.

    2. Require SSL/TLS — Specify whether or not you want to require TLS security for all communication within the mirror (as recommended) by selecting or clearing the check box. If you select Require SSL/TLS and the instance does not already have a valid TLS configuration for mirroring, before completing the procedure you must click the Set up SSL/TLS link and create the needed TLS configuration on this member. (Instructions for creating the TLS configuration are contained in Creating and Editing a TLS Configuration for a Mirror in the “Using TLS with InterSystems IRIS” chapter of the Security Administration Guide. You can also cancel the Create Mirror procedure and navigate to the TLS Configurations page (System Administration > Security > SSL/TLS Configurations). If the instance does have a valid TLS configuration for mirroring, the link is Edit SSL/TLS instead, and you need not use it when selecting Require SSL/TLS unless you want to modify that configuration.

    3. Use Arbiter — Specify whether or not you want to configure an arbiter (as recommended) to enable automatic failover for the widest possible range of outage scenarios, as described in Automatic Failover Mechanics. If you select Use Arbiter, you must supply the hostname or IP address of the system you want to configure as arbiter and the port used by its ISCAgent process (2188 by default). See Locating the Arbiter to Optimize Mirror Availability and Installing the Arbiter for additional information about the arbiter.

    4. Use Virtual IP — Specify whether or not you want to use a Virtual IP address by selecting or clearing the check box. If you select Use Virtual IP, you are prompted for an IP address, Classless Inter-Domain Routing (CIDR) mask, and network interface.

      Important:

      See Configuring a Mirror Virtual IP (VIP) for requirements and important considerations before configuring a VIP.

    5. Compression Mode for Failover Members, Compression Mode for Async Members — Specify whether to compress journal data before transmission from the primary to the backup and to async members, respectively, and which compression type to use for each; see Journal Data Compression for more information. The default setting for both is System Selected, which optimizes for response time between the failover members and for network utilization between the primary and asyncs.

  3. Enter the following information in the Mirror Failover Information Section:

    • Mirror Member Name — A name for the failover member you are configuring on this system (defaults to a combination of the system host name and the InterSystems IRIS instance name). Mirror member names cannot contain spaces, tabs, or the punctuation characters that follow:

      : [ ] # ; / * = ^ ~ ,

      Alphabetic characters are converted to uppercase before saving. The maximum length of a mirror member name is 32 characters.

    • Superserver Address — The IP address or host name that external systems can use to communicate with this failover member; typically you can accept the default. For information on the Superserver address, see Mirror Member Network Addresses and Updating Mirror Member Network Addresses.

    • Agent Port — The port number of the ISCAgent on this failover member; accept the installed agent’s port as provided at the prompt. For information on the agent port, see the Configuring the ISCAgent.

  4. Click Advanced Settings to display and edit additional mirror settings, as follows:

  5. Click Save.

Note:

You can also use the ^MIRROR routine (see Using the ^MIRROR Routine) to create a mirror. When you execute ^MIRROR on an InterSystems IRIS instance without an existing mirror configuration, the Enable Mirror Service option is available if mirroring is not yet enabled. Once mirroring is enabled, the Create a Mirror option is available and provides an alternative means of creating a mirror and configuring the instance as the primary failover member. The SYS.Mirror.CreateNewMirrorSet() mirroring API method can also be used for this purpose.

Configure the Second Failover Member

Follow this procedure to configure the second failover member of the mirror.

  1. On the second failover member, navigate to Join Mirror as Failover page (System Administration > Configuration > Mirror Settings > Join as Failover). If the Join as Failover option is not available, mirroring has not been enabled; first click Enable Mirror Service, then select the Service Enabled check box and click Save, then select the Join as Failover option.

  2. On the Join Mirror as Failover page, in the Mirror Information section, enter the mirror name you specified when you configured the first failover member.

  3. Enter the following information in the Other Mirror Failover Member’s Info section:

    • Agent Address on Other System — Enter the Superserver Address you specified when you configured the first failover member.

    • Agent Port — Enter the port of the ISCAgent you specified when you configured the first failover member.

    • InterSystems IRIS Instance Name — Enter the name of the InterSystems IRIS instance configured as the first failover member.

  4. Click Next to retrieve and display information about the mirror and the first failover member. In the Mirror Failover Member Information section:

    • Mirror Member Name — Specify a name for the failover member you are configuring on this system (defaults to a combination of the system host name and the InterSystems IRIS instance name). Mirror member names cannot contain spaces, tabs, or the punctuation characters that follow:

      : [ ] # ; / * = ^ ~ ,

      Alphabetic characters are converted to uppercase before saving. The maximum length of a mirror member name is 32 characters.

    • Superserver Address — Enter the IP address or host name that external systems can use to communicate with this failover member; see Mirror Member Network Addresses and Updating Mirror Member Network Addresses for information.

    • Agent Port — Enter the port number of the ISCAgent on this failover member; accept the installed agent’s port as provided at the prompt. For information on the agent port, see the Configuring the ISCAgent.

    • Network Interface for Virtual IP — Displays the network interface you specified when you configured the first failover member; this setting cannot be changed on the second failover member.

    • SSL/TLS Requirement — Displays the setting you specified when you configured the first failover member. This setting cannot be changed on the second failover member.

      If the mirror requires TLS and the instance does not already have a valid TLS configuration for mirroring, before completing the procedure you must click the Set up SSL/TLS link and create the needed TLS configuration on this member. (Instructions for creating the TLS configuration are contained in Creating and Editing a TLS Configuration for a Mirror in the “Using TLS with InterSystems IRIS” chapter of the Security Administration Guide. You can also cancel the Join as Failover procedure and navigate to the TLS Configurations page of the Management Portal (System Administration > Security > SSL/TLS Configurations).

      If the instance does have a valid TLS configuration for mirroring, the link is Edit SSL/TLS instead, and you need not use it when TLS is required unless you want to modify that configuration.

    • Mirror Private Address — Enter the IP address or host name that the other failover member can use to communicate with this failover member; see Mirror Member Network Addresses and Updating Mirror Member Network Addresses.

    • Agent Address — Enter the address that other mirror members attempting to contact this member’s ISCagent will try first; see Mirror Member Network Addresses and Updating Mirror Member Network Addresses.

  5. Click Advanced Settings to display the Quality of Service Timeout setting you specified when you configured the first failover member; this setting cannot be changed on the second failover member.

  6. Click Save.

If you configured the mirror to require TLS, you are reminded that you must complete the process of adding the second failover member to the mirror by authorizing the second failover member on the first failover member, as described in the following section.

Note:

You can also use the ^MIRROR routine (see Using the ^MIRROR Routine) to configure the second failover member. When you execute ^MIRROR on an InterSystems IRIS instance without an existing mirroring configuration, the Enable Mirror Service option is available if mirroring is not yet enabled. Once mirroring is enabled, the Join Mirror as Failover Member option is available and provides an alternative means of both configuring the backup failover member and adding it to the mirror. The SYS.Mirror.JoinMirrorAsFailoverMember() mirroring API method can also be used for this purpose.

Authorize the Second Failover Member or Async (TLS Mirrors Only)

If you configured the mirror to require TLS, an additional step is needed after you configure the second failover member or configure an async member. On the system on which you created the mirror and configured the first failover member, you must authorize the new mirror member, as follows:

  1. Navigate to the Edit Mirror page (System Administration > Configuration > Mirror Settings > Edit Mirror).

  2. At the bottom of the page, a Pending New Members area lists members that have been added to the mirror. Select the members you want to authorize, click Authorize, and confirm. (The TLS certificate of the second failover member is automatically verified when the member is added.)

  3. The information in the Mirror Member Information section of the Edit Mirror page now includes the members you added. (See Mirror Member Network Addresses for information about the addresses displayed in this list.)

Note:

The Authorize/Reject Pending New Members option on the Mirror Configuration menu of the ^MIRROR routine on the first failover member can be also used to authorize new failover or async members in a mirror configured to require TLS.

The SYS.Mirror.AddFailoverMember() mirroring API method can be used to authorize the second failover member in a mirror configured to require TLS, and the Config.MapMirrors.Create() API method can be used to create an authorized member (failover or backup). The SYS.Mirror.VerifyMirrorSSLCertificates() can be used to verify mirror member TLS certificates.

For information about authorizing X.509 DN updates on members of a mirror requiring TLS (for example when a member’s certificate is replaced), see Authorizing X.509 DN Updates (TLS Only).

Review Failover Member Status in the Mirror Monitor

As described in Monitoring Mirrors, you can use the Mirror Monitor to see information about the failover members in a mirror, including their current status (role) in the mirror. Use the mirror monitor to confirm that your mirror and its failover members are now set up as intended, as follows:

  1. On the first failover member you configured, display the Mirror Monitor page (System Operation > Mirror Monitor).

  2. In the Mirror Failover Member Information area, the mirror member names and network address of the two failover members are listed.

  3. The Mirror Member Status area should show the first failover member you configured as Primary in the Status column, and the second as Backup. As discussed in Mirror Member Journal Transfer and Dejournaling Status, the Journal Transfer status of the backup should be Active, and its Dejournaling status should be Caught up.

  4. In the Arbiter Connection Status area, if you configured an arbiter, its network address and agent port number are displayed. Failover Mode should be Arbiter Controlled and Connection Status should be Both failover members are connected to the arbiter; if this is not the case, the arbiter may not have been correctly installed, its ISCAgent process may not be running, or the network address or port number you supplied may be incorrect. A network problem preventing contact with the arbiter by one or both failover members could also cause the Failover Mode to be Agent Controlled.

The same information is displayed in the Mirror Monitor on the backup failover member.

Configure Async Mirror Members

For each async member you want to configure, use the following procedure. A mirror with a failover pair can include up to 14 reporting or disaster recovery (DR) async members. A single InterSystems IRIS instance can be a reporting async member of up to 10 mirrors, but an instance can be a DR async in one mirror only. Once you have configured an instance as either a read-only or a read-write reporting async, it can be added to other mirrors only as a reporting async member of that type. (A reporting async member’s type can be changed for all mirrors to which it belongs, however, as described in Editing the Mirror Configuration on an Async Member.)

Note:

The procedure for adding an instance to a mirror as a reporting async member is the same whether you are using the Join as Async option as described here or the Join a Mirror button on the Edit Async Configurations page as described in Editing the Mirror Configuration on an Async Member, except that the Join a Mirror button on the Edit Async Configurations page can be used only on reporting async members, as a DR async can belong to one mirror only.

  1. Navigate to the Join Mirror as Async page (System Administration > Configuration > Mirror Settings > Join as Async); if the Join as Async option is not available, choose Enable Mirror Service and enable the service.

  2. On the Join Mirror as Async page, enter the mirror name you specified when you created the mirror at the Mirror Name prompt.

  3. Select either the primary or the backup failover member, and in the Mirror Failover Member’s Info section, enter the information for the member you selected at each of the following prompts:

    1. Agent Address on Failover System — Enter the Superserver Address you specified when you configured the selected failover member.

    2. Agent Port — Enter the ISCAgent port you specified for the selected failover member.

    3. InterSystems IRIS Instance Name — Enter the name of the InterSystems IRIS instance you configured as the selected failover member.

  4. Click Next to verify the failover member’s information and move to the Async Member Information section. In this section, enter the following information:

    1. Async Member Name — Specify a name for the async member you are configuring on this system (defaults to a combination of the system host name and the InterSystems IRIS instance name). Mirror member names can contain alphanumeric characters, underscores, and hyphens.

      Note:

      The mirror member name cannot be changed, and will therefore be used when a reporting async member joins additional mirrors in the future.

    2. Async Member Address — Enter the IP address or host name that external systems can use to communicate with this async member.

      Note:

      The Async Member Address you provide becomes the async member’s superserver address and mirror private address (see Mirror Member Network Addresses). If you want these to be different, for example when you want to place a DR async’s mirror private address on the mirror private network while leaving its superserver address on the external network, you can update the async’s addresses on the primary after adding it to the mirror; see Updating Mirror Member Network Addresses for more information.

    3. Agent Address — Enter the address that other mirror members attempting to contact this member’s ISCagent will try first; see Mirror Member Network Addresses and Updating Mirror Member Network Addresses.

    4. Async Member System Type — Select one of the following types from the drop-down list. A single InterSystems IRIS instance can be a reporting async member of multiple mirrors, but can be a DR async member of only one mirror.

      • Disaster Recovery (DR) — This option is for a system on which read-only copies of all of the mirrored databases in a single mirror are maintained, making it possible to promote the DR async member to failover member when one of the failover members fails. See Promoting a DR Async Member to Failover Member in this chapter for more information about DR async promotion.

        Important:

        When the mirror is configured to use VIP, a disaster recovery async member must have direct TCP/IP connectivity to the failover members; see Configuring a Mirror Virtual IP (VIP) for more information.

      • Read-Only Reporting — This option is used to maintain read-only copies of the mirrored databases (or a subset of the mirrored databases) from one or more mirrors for purposes of enterprise reporting and data mining in which there is no requirement for data to be modified or added.

      • Read-Write Reporting — This option is used to maintain read-write copies of the mirrored databases (or a subset of the mirrored databases) from one or more mirrors as data sources for reporting/business intelligence operations in which data modification or the addition of data during analysis must be enabled.

    5. Set up SSL/TLS — If the mirror requires TLS and the instance does not already have a valid TLS configuration for mirroring, an error message and this link are included. Before completing the procedure, you must click the link and create the needed TLS configuration on this member. (Instructions for creating the TLS configuration are contained in Creating and Editing a TLS Configuration for a Mirror in the “Using TLS with InterSystems IRIS” chapter of the Security Administration Guide. You can also cancel the Join as Async procedure and navigate to the TLS Configurations page of the Management Portal (System Administration > Security > SSL/TLS Configurations).

    6. Edit SSL/TLS — If the mirror requires TLS and the instance does have a valid TLS configuration for mirroring, this link is displayed instead of Set up SSL/TLS; you can use it to edit the existing TLS configuration if you wish. The instance’s X.509 Distinguished Name is also displayed.

  5. Click Save.

If you configured the mirror to require TLS, you are reminded that you must complete the process of adding the async member to the mirror by authorizing the async member on the first failover member, as described in Authorize the Second Failover Member or Async (TLS Only).

Note:

You can also use the ^MIRROR routine (see Using the ^MIRROR Routine) to configure async mirror members. When you execute ^MIRROR on an InterSystems IRIS instance for which mirroring is enabled, the Join Mirror as Async Member (or Join Another Mirror as Async Member) option on the Mirror Configuration menu is available and provides an alternative means of configuring an async member and adding it to the mirror. The SYS.Mirror.JoinMirrorAsAsyncMember() mirroring API method can also be used to configure an async member.

After an instance has been added to one mirror as an async member using the procedure described in this section, you can use the Join a Mirror button on the Edit Async page (see Editing the Mirror Configuration on an Async Member) to add it to additional mirrors, but as the same type of async only.

Adding Databases to a Mirror

Only a local database on the current primary failover member can be added to a mirror; it is added on the primary first, then on the backup, and then on any desired async members. All mirrored databases must be journaled.

You must add the same set of mirrored databases to both the primary and backup failover members, as well as to any DR async members; which mirrored databases you add to reporting async members depends on your reporting needs. The namespaces and global/routine/package mappings associated with a mirrored database must be the same on all mirror members, including all async members on which the database exists. The mirrored databases on the backup failover member must be mounted and caught up (see Activating and Catching up Mirrored Databases) to be able to take over as the primary in the event of a failover; the mirrored databases on a DR async member must be mounted and caught up to make it suitable for promotion to failover member.

The procedure for creating a mirrored database (that is, adding a new database containing no data) is different from that for adding an existing database to the mirror. Global operations on a database created as a mirrored database are recorded in mirror journal files from the beginning, and the mirror therefore has access to all the data it needs to synchronize the database across mirror members. But global operations on an existing database before it was added to a mirror are contained in nonmirror journal files, to which the mirror does not have access. For this reason, an existing database must be backed up on the primary failover member after it is added to the mirror and restored on the backup failover member and on any async members on which it is to be located. Once this is done, you must activate and catch up the database to bring it up to date with the primary.

Mirrored Database Considerations

Bear the following points in mind when creating and adding mirrored databases:

  • Only data in IRIS.DAT files can be mirrored. Data that is external (that is, stored on the file system) cannot be mirrored by InterSystems IRIS (for more information, see Mirror Configuration Guidelines).

  • System databases (IRISSYS, IRISLIB, IRISLOCALDATA, IRISTEMP, IRISAUDIT, and ENSLIB) cannot be mirrored. In addition, if you are using InterSystems IRIS for Health or HealthShare® Health Connect, do not mirror HSLIB.

  • Because more database directory information is stored for mirrored databases, they reduce the maximum number of databases that can be configured within an InterSystems IRIS instance. For more information, see Configuring Databases in the “Configuring InterSystems IRIS” chapter of the System Administration Guide.

  • The mirror automatically and continually synchronizes the following properties of a mirrored database on a backup or async with the properties of the database on the primary:

    • Maximum Size

    • Expansion Size

    • Resource Name

    • Collation

    For example, when the Maximum Size of a mirrored database is increased on the primary, it is automatically increased for that database on the other members to match the primary, if necessary; if Maximum Size is then reduced on an async, synchronization automatically increases it to the value on the primary. If database properties are changed on either the primary or another mirror member while that member is disconnected, they are automatically synchronized when the member reconnects to the mirror. There are two exceptions to automatic synchronization of these database properties, as follows:

    • The values of the Maximum Size and Expansion Size properties on an async can be increased by synchronization, but never reduced. For example, if the Maximum Size of a database on the primary is reduced, the value of this property is reduced on the backup, but not on any asyncs belonging to the mirror; if the Maximum Size of a database on an async is increased to be larger than on the primary, it is not reduced by synchronization to the value on the primary.

    • The Resource Name property of a database is synchronized with the primary on the backup, but not on async members.

    See Edit Mirrored Local Database Properties in the “Configuring InterSystems IRIS” chapter of the System Administration Guide for information about mirrored database properties.

Create a Mirrored Database

To create a mirrored database, follow this procedure.

Note:

You can also use the ^DATABASE routine to create mirrored databases; see Creating a Mirrored Database Using the ^DATABASE Routine in this chapter.

  1. On the current primary failover member, navigate to the Local Databases page of the Management Portal (System Administration > Configuration > System Configuration > Local Databases), and click the Create New Database button.

  2. Follow the procedure in the Create a Local Database section of the “Configuring InterSystems IRIS” chapter of the System Administration Guide. On the second panel, select Yes for Mirrored database? and enter a name for the database within the mirror; the default is the local database name you provided. The leading character of the mirror database name must be alphabetic or an underscore, and the rest must be alphanumeric characters, hyphens and underscores. Mirror database names are case-insensitive, thus two names cannot differ only in case; if you enter a mirror database name that is already included in the mirror, the new database cannot be added to the mirror and must be deleted. (Names of mirrored databases created under earlier versions of InterSystems IRIS may be stored in lowercase or mixed case, but the addition of databases with duplicate uppercase names is still precluded.)

    On an async member that belongs to more than one mirror, you must also select the mirror the database will belong to.

    Note:

    When you select Yes for Mirrored database, Journal globals is automatically locked to Yes.

  3. Confirm the procedure to create the database and add it to the mirror on the primary.

  4. On the backup failover member, and on each async member to which you want to add the mirrored database, follow the previous three steps, taking care to enter the correct mirror database name from the primary as the mirror database name on each of the other members. (The local database names do not have to match.)

    Note:

    If you attempt to add a new database to the mirror on a nonprimary member that was not created as a mirrored database on the primary, but rather added to the mirror after it was created, an error message notes this and you cannot complete the operation.

Important:

If the first mirror journal file for a mirrored database has been purged from the primary, the database can no longer be created as a mirrored database on another member; instead, you must make a backup on the primary and restore it on the backup or async, as described in Add an Existing Database to the Mirror. For this reason, it is best to create the database on the backup and async members as soon as possible after creating it on the primary. (For information about when mirror journal files are purged on the primary, see Purge Journal Files in the “Journaling” chapter of the Data Integrity Guide.)

Add an Existing Database to the Mirror

Use the procedure that follows to add one or more existing databases to a mirror.

Note:

The SYS.Mirror.AddDatabase() mirroring API method provides an alternative means of adding existing databases to a mirror.

  1. On the current primary failover member, navigate to the Local Databases page of the Management Portal (System Administration > Configuration > System Configuration > Local Databases) and click the Add to Mirror button.

  2. From the listed databases (nonsystem databases not already in the mirror) select those you want to add and click Add. You must enter a name for each database within the mirror; the default is the local database name you provided. The leading character of the mirror database name must be alphabetic or an underscore, and the rest must be alphanumeric characters, hyphens and underscores. Mirror database names are case-insensitive, thus two names cannot differ only in case; if you enter a mirror database name that is already included in the mirror, the operation fails. (Names of mirrored databases created under earlier versions of InterSystems IRIS may be stored in lowercase or mixed case, but the addition of databases with duplicate uppercase names is still precluded.)

    To run the task in the background, select Run add in the background; if you select five or more databases, the task is automatically run in the background. Confirm the procedure to add the selected databases to the mirror on the primary.

    You can also add an individual database to the mirror by clicking its name to edit its properties and clicking the Add to Mirror <mirrorname> link, then clicking Add and confirming the procedure. (If journaling is not enabled on the database, Databases must be journaled to be mirrored is displayed in place of this link; to enable it, select Yes from the Global Journal State drop-down list.) Alternatively, the Add Mirrored Database(s) option on the Mirror Management menu of the ^MIRROR routine also lets you add an individual database. In either case, you can accept the default of a mirror database name the same as the local name, or enter a different one.

    Note:

    If a backup failover member or async member has a different endianness than the primary failover member, you must use the procedure described in Member Endianness Considerations to add the database to the backup or async member after adding it to the primary, rather than adding it on that member as described in the following steps.

  3. Once the database has been added to the mirror, back it up on the primary failover member. Review Backup Strategies, Restoring from Backup, and Mirrored Database Considerations in the “Backup and Restore” chapter of the Data Integrity Guide for information about different backup techniques and the corresponding restore procedures.

    Important:

    If the database you are copying is encrypted on the primary, the key with which it is encrypted must also be activated on the backup (and asyncs, if any), or the database must be converted to use a key that is activated on the destination system, as described in Converting an Encrypted Database to Use a New Key in the “Performing Encryption Management Operations” chapter of the Security Administration Guide.

    Ensuring that a mirrored database is synchronized after it is restored from backup (see the following step) requires that the journal files from the time of the backup on the primary failover member are available and online; for example, if the relevant journal files have been purged, you must make and restore a more up to date backup. For general information about restoring mirror journal files see Restoring Mirror Journal Files in the “Journaling” chapter of the Data Integrity Guide; for information about purging mirror journal files see Purge Journal Files in the “Journaling” chapter of the Data Integrity Guide.

  4. On the backup failover member and each connected async member, do the following:

    1. If a local database with the same local name and database directory as the mirrored database you just added on the primary failover member does not already exist, create it.

    2. Restore the backup you made of the mirrored database on the primary, overwriting the existing database. The procedure for this depends on the restore method you are using, as follows:

      • online backup restore (^DBREST routine) — This routine automatically recognizes, activates and catches up a mirrored database on the backup and async members. For more information see Mirrored Database Considerations in the “Backup and Restore” chapter of the Data Integrity Guide.

        Note:

        When a mirrored database is restored on a nonprimary member, the data to begin the automatic synchronization process may not have been sent yet. If the required data does not arrive within 60 seconds, the process begins anyway; those databases may not catch up if the data does not arrive before it is required, however, in which case a message regarding the database(s) that had the problem is logged in the messages.log file. (During database creation this process would affect only one database, but it also applies to catching up in other situations in which multiple databases are involved.)

      • External backup restore or cold (offline) backup restore — Both of these methods require that you manually activate and catch up the mirrored databases after they are restored and mounted on the backup failover member or async member, as described in Activating and Catching up Mirrored Databases, immediately following.

As an alternative to the previous procedure, after adding an existing database to the mirror on the primary, you can copy the databases’s IRIS.DAT file from the primary to the backup and async members instead of backing up and restoring the database. To do so, use this procedure:

  1. Ensure that there is a placeholder target database on the backup and each async member.

  2. On both failover members and each aysnc member, make sure the source and target databases are not mounted (see Maintaining Local Databases in the “Managing InterSystems IRIS” chapter of the System Administration Guide).

  3. Copy the mirrored IRIS.DAT file from the primary failover member to the database directory of the placeholder target database on the backup and each async member, overwriting the existing IRIS.DAT file.

    Note:

    If the database you are copying is encrypted on the primary, the key with which it is encrypted must also be activated on the backup (and asyncs, if any), or the database must be converted to use a key that is activated on the destination system, as described in Converting an Encrypted Database to Use a New Key in the “Performing Encryption Management Operations” chapter of the Security Administration Guide.

  4. Mount the database on the all members.

  5. Activate and catch up the mirrored databases on the backup failover member and async member(s) as described in Activating and Catching up Mirrored Databases in this chapter.

Note:

When you are adding an existing mirrored database to an async member, you can back up the database on (or copy the IRIS.DAT file from) the backup failover member or another async member, assuming it is fully caught up, instead of the primary. This may be more convenient, for example if the primary is in a different data center than the async on which you will be restoring the backup. Do not use a member as the source, however, unless you have a high degree of confidence in the consistency of its databases.

Activating and Catching Up Mirrored Databases

You can activate and/or catch up mirrored databases on the backup failover member and async members using the Mirror Monitor.

As noted in Add an Existing Database to a Mirror, a newly added mirrored database containing data can be automatically synchronized with the primary through use of the ^DBREST routine to restore the backup from the primary failover member. If some other method is used, it must be activated and caught up on the backup failover member and async members.

To activate and catch up mirrored databases, do the following on the backup failover member and async members:

  1. Navigate to the Mirror Monitor page (System Operation > Mirror Monitor).

  2. On an async member, click the Details link for the mirror containing the database(s) you want to take action on, if necessary.

  3. The Mirrored databases list shows the status of each database, as described in Using the Mirror Monitor. Among other possible statuses, Needs Catchup indicates that the Catchup operation is needed, Needs Activation indicates that both the Activate and Catchup operations are needed, and Catchup Running shows that the Catchup operation is currently running on the database.

  4. Select the Activate or Catchup link to perform an operation on a single database, or select Activate or Catchup from the Select an action drop-down and click Go to open a dialog in which you can select multiple databases from a list of all those for which the action is appropriate to apply it to all of them at once. When you do this, the Activate and Catchup tasks are run in the background. When you select Catchup, databases of both Needs Activation and Needs Catchup status are displayed; both Activate and Catchup are applied to any Needs Activation databases you select.

You can also use the Mirrored databases list to mount or dismount one or more mirrored databases, or to remove one or more databases from the mirror as described in Removing Mirrored Databases from a Mirror.

Note:

If a mirrored database cannot be caught up due to an error in the database, the affected database will not be active if its host member becomes primary; as described in Automatic Failover Rules, if the database is marked Mount Required at Startup, this will prevent that member from becoming primary.

The Activate or Catchup mirrored database(s) option on the Mirror Management menu in the ^MIRROR routine and the SYS.Mirror.ActivateMirroredDatabase() and SYS.Mirror.CatchupDB() mirroring API methods provide alternative means of activating/catching up mirrored databases.

When you use the Mirrored databases list, the Databases page of the management portal (see the “Managing InterSystems IRIS” chapter of the System Administration Guide), or the ^DATABASE routine (see the “Using Character-based Security Management Routines” chapter of the Security Administration Guide) to mount a mirrored database, you can choose whether or not to catch up the database following the mount operation.

When parallel dejournaling (see Configuring Parallel Dejournaling) is enabled and supported by available resources, it is used when catching up mirrored databases.

Editing or Removing Mirror Members

The following procedures describe how to edit or remove the mirror configuration on a mirror member, including deleting a mirror altogether, and how to remove databases from a mirror when you are not removing a mirror configuration.

Note:

Several options on the Mirror Configuration menu of the ^MIRROR routine provide alternative means for editing mirror configurations. The specific options available depend on whether the routine is used on a failover member or async member.

Clearing the FailoverDB Flag on Reporting Async Mirror Members

As described in Async Mirror Members, an async member must be of one of three types:

  • Disaster Recovery (DR)—Maintains read-only copies of all mirrored databases on the primary; eligible to be promoted to failover member (see Promoting a DR Async to Failover Member for more information).

  • Read-Only Reporting—Maintains read-only copies of mirrored databases; not eligible to be promoted to failover member.

  • Read-Write Reporting—Maintains read-write copies of mirrored databases; not eligible to be promoted to failover member.

When a mirrored database is added to a DR or read-only reporting async, it is mounted as Read-Only, and the FailoverDB flag, which is set when the database is created on the primary, remains set on the async’s copy to

  • Ensure that the database remains read-only and therefore an exact mirror of the database on the primary (assuming dejournaling is caught up).

  • Indicate that the database can become the primary copy in the mirror in the event that a DR async member is promoted to failover member. A DR async member can be promoted only if includes all of the databases in the mirror and all of those databases have the FailoverDB flag set.

When a mirrored database is added to a read-write reporting async, on the other hand, the FailoverDB flag is cleared to allow Read-Write mounting of the database. A mirrored database with the FailoverDB flag cleared can never be used as the mirror’s primary copy.

On a DR async, the FailoverDB flag can never be cleared. The flag can be manually cleared on reporting asyncs, however.

On a read-only reporting async, clearing the FailoverDB flag changes the database to read-write, which is typically not desirable. In most cases, therefore, including when you change the async type from Disaster Recovery (DR) to Read-Only Reporting (see Editing or Removing an Async Member), you can leave the FailoverDB flag set on all databases on a read-only reporting async.

When you change an async member’s type from Disaster Recovery (DR) or Read-Only Reporting to Read-Write Reporting, you are offered the option of clearing all the FailoverDB flags. Because the FailoverDB flag on a mirrored database requires it to remain read-only, you will typically want to use this option. If you want to keep one or more mirrored databases read-only on the read-write reporting async, however, you can use the individual Clear Flag links in the Mirrored Databases list to make individual databases read-write and leave the rest as read-only.

Databases added to an async member after you change its type are mounted and flagged according to the member’s new type, as previously described. The Clear FailoverDB Flags button always allows you to clear the flag from all databases at any time on either type of reporting async.

You cannot manually set the FailoverDB flag; this flag is set only when a mirrored database is added to a DR or read-only reporting async.

Removing the Mirrored Database Attribute When Removing a Mirror Member

When you remove a member from a mirror, you are always given the option of removing the mirror attribute from the mirrored databases belonging to the mirror. The consequences are as follows:

  • If you retain the mirror attribute and later restore the InterSystems IRIS instance to the mirror, the databases are automatically added to the mirror but must be activated before they can be caught up and then synchronized (see Activating and Catching Up Mirrored Databases).

    However, if you retain the mirror attribute, you cannot delete that databases unless you first do one of the following:

    • Restore the member to the same mirror you removed it from. (If the member was the primary failover member, this is not an option, as the mirror no longer exists.) You can then remove one or more of the databases from the mirror (see Removing Mirrored Databases from a Mirror) and delete them if you wish.

    • Use the Remove one or more mirrored databases option of the ^MIRROR routine (see Using the ^MIRROR Routine) to remove the mirror attribute from one or more databases, then delete them if you wish.

  • If you remove the mirror attribute, the databases become permanently unmirrored and can be used like any local database; if you want to return them to the mirror after the instance rejoins as a mirror member, you must use the procedure for adding them to the mirror as existing databases for the first time.

When you remove an individual database from the mirror on a backup or async member, the mirrored database attribute is automatically removed.

Editing or Removing an Async Member

  1. Navigate to the Edit Async Configurations page (System Administration > Configuration > Mirror Settings > Edit Async).

  2. Use the Remove Mirror Configuration button to remove a DR async from its mirror or a reporting async from all mirrors to which it belongs and remove the instance’s mirror configuration entirely. (To remove a reporting async from a single mirror, use the Leave mirror link described later in this procedure.)

    You are given the option of removing the mirror attribute from the mirrored databases on the member; see Removing the Mirrored Database Attribute When Removing a Mirror Member for information about this decision.

    You are also given the option of removing the instance’s TLS configuration (see Securing Mirror Communication with TLS Security).

    After using the Remove Mirror Configuration button to remove the instance’s mirror configuration entirely, you must restart InterSystems IRIS.

    Note:

    The Remove Mirror Configuration option on the Mirror Configuration menu of the ^MIRROR routine (see Using the ^MIRROR Routine) and the SYS.Mirror.RemoveMirrorConfiguration() API call provide alternative options for removing an async member’s mirror configuration entirely.

  3. Use the Join a Mirror button to add a reporting async member to another mirror (it can belong to a maximum of 10); the procedure is the same as that described in Configure Async Mirror Members for adding an async member to its first mirror, except that the member name and async type (read-only or read-write) cannot be changed. This button is not available on a DR async member; to join another mirror, you must first change the Async Member System Type as described in a later step.

  4. As described in Clearing the FailoverDB Flag on Reporting Async Mirror Members, you can use the Clear FailoverDB Flags button to clear the FailoverDB flag on all mirrored databases on a read-only reporting async, or after you change the async system type from Disaster Recovery (DR) to Read-Write or Read-Only Reporting.

  5. The following settings in the Mirror Member Information section can be modified for the async member you are editing except the mirror member name. After you have changed one or more, click Save.

    • Mirror Member Name — The name provided when the async member joined its first mirror; cannot be changed.

    • Async Member System Type — You can change the type of an async member using this drop down. The following conditions apply:

      • If you change from Disaster Recovery (DR) to Read-Write Reporting, you are prompted to clear the FailoverDB flag for all mirrored databases, as described in Clearing the FailoverDB Flag on Reporting Async Mirror Members.

      • When you change from Read-Write Reporting to Read-Only Reporting or the reverse, the change is made for all mirrors to which the reporting async member belongs.

      • You cannot change from Read-Write or Read-Only Reporting to Disaster Recovery (DR) unless all of the following are true:

        • If journal encryption is in use, the async is using the same journal encryption key as the failover members (see Activating Journal Encryption in a Mirror).

        • The FailoverDB flag is set on all mirrored databases. (Once cleared, this flag cannot be reset. To address this, you can substitute a copy of the database taken from another member on which FailoverDB is set.)

        • The member does not belong to any other mirror.

        • The ISCAgent is running (see Starting and Stopping the ISCAgent).

        If a dejournaling filter is set on the async (see Using a Dejournal Filter on a Reporting Async), it is removed when you change the Async Member System Type to Disaster Recovery (DR).

        Important:

        Before converting a reporting async to DR async, ensure that the member is prepared to become a failover member should a disaster occur that calls for it to be promoted (see Promoting a DR Async Member to Failover Member). This includes confirming the following:

    • Mirror Journal File Retention (reporting asyncs only) — Whether mirror journal files are purged immediately after they are dejournaled or according to the instance’s local purge policies. This setting is available for reporting asyncs only. For information about how mirror journal files are purged, see Purging Mirror Journal Files in the “Journaling” chapter of the Data Integrity Guide.

    • SSL/TLS Configuration — If TLS is required (see Securing Mirror Communication with TLS Security) the X.509 Distinguished Name (DN) is displayed, as well as the Verify SSL button, which lets you verify the TLS certificates of all current mirror members that can be contacted by the async you are editing. If any certificate is not valid, an informational message is displayed. (Certificates can also be verified using the ^Mirror routine.)

      If the mirror does not use TLS, the SSL/TLS link is available, allowing you to configure TLS if you intend to add it to the mirror (see Editing or Removing a Failover Member).

      Note:

      The SYS.Mirror.UpdateMirrorSSL() mirroring API method and the ^SECURITY routine can also be used to update a mirror’s member’s TLS settings.

  6. The Mirrors this async member belongs to list shows you all the mirrors the instance belongs to as an async member. Each entry provides three links for changes.

    • Mirror Name — Click the name of the mirror shown in the Name column to open the Edit Mirror dialog, showing the instance directories and network addresses (see Mirror Member Network Addresses) of all members of the mirror.

      If the async is currently connected to the mirror, you cannot change any of the network information displayed except the async’s Superserver port; if the async member is disconnected and the network information for the primary has changed, you can update the primary’s information here so that the async can reconnect when desired. See Updating Mirror Member Network Addresses for important information about updating the network addresses of mirror members.

    • Leave Mirror — Removes the async member from the mirror for which you clicked the link, and from that mirror only. (In the case of a DR async, this would be the only mirror it belongs to.)

      You are given the option of removing the mirror attribute from the mirrored databases on the async member; see Retaining or Removing the Mirrored Database Attribute for information about this decision.

      Note:

      The Remove This Member From a Mirror option on the Mirror Configuration menu of the ^MIRROR routine (see Using the ^MIRROR Routine) and the SYS.Mirror.RemoveOneMirrorSet() API call provide alternative options for removing an async member from a mirror. You can also use the Remove Other Mirror Member button on the Edit Mirror page on a failover member to remove an async member from the mirror.

      On any async member, you can temporarily stop mirroring (for an individual mirror if a reporting async belongs to more than one); see Stopping Mirroring on Backup and Async Members for more information.

    • Edit Dejournal Filter (reporting asyncs only) — Lets you set or remove a dejournal filter on the async; see Using a Dejournal Filter on a Reporting Async for more information.

  7. The Mirrored Databases list shows you all mirrored databases on the async member. If the instance is a DR async member, these should include all mirrored databases on the mirror’s failover members, and the FailoverDB flag should be set on each.

  8. In a mirror that uses TLS, select Authorize Pending DN Updates (if it appears) to authorize pending DN updates from the primary so that the async can continue to communicate with the primary. See Authorizing X.509 DN Updates (TLS Only) for information about authorizing DN updates.

Editing or Removing a Failover Member

  1. Navigate to the Edit Mirror page (System Administration > Configuration > Mirror Settings > Edit Mirror).

  2. Use the Remove Mirror Configuration button on the backup failover member to remove it from the mirror and remove the InterSystems IRIS instance’s mirror configuration entirely.

    Important:

    To remove a mirror entirely, begin by removing all async members from the mirror, then remove the backup failover member, and finally remove the primary failover member.

    When removing a failover member from the mirror, you are given the option of removing the mirror attribute from the mirrored databases on the member; see Removing the Mirrored Database Attribute When Removing a Mirror Member for information about this decision. This is especially significant when you are removing the primary failover member, thereby permanently deleting the mirror.

    On the backup, you are also given the option of removing the instance’s TLS configuration (see Securing Mirror Communication with TLS Security).

    You can also use the Remove Other Mirror Member button on the primary to remove the backup or an async from the mirror. You can use the Remove Other Mirror Member button on the backup to remove an async from the mirror.

    After using the Remove Mirror Configuration button or the Remove Other Mirror Member button to remove an async or backup member’s mirror configuration entirely, you must restart InterSystems IRIS.

    Note:

    The Remove Mirror Configuration option on the Mirror Configuration menu of the ^MIRROR routine (see Using the ^MIRROR Routine) and the SYS.Mirror.RemoveMirrorConfiguration() API call provide alternative options for removing a failover member’s mirror configuration entirely.

    You can temporarily stop mirroring on the backup failover member; see Stopping Mirroring on Backup and Async Members.

  3. To remove the primary failover member from the mirror and remove the mirror entirely (which you can do only if the primary is the last member remaining in the mirror), use this procedure:

    1. Use the Remove Mirror Configuration button on the Edit Mirror page; a dialog displays that lets you clear the JoinMirror flag from the instance.

    2. After clearing the flag, restart the instance.

    3. Navigate to the Edit Mirror page and use the Remove Mirror Configuration button again.

  4. In the Mirror Information section, you cannot edit the Mirror Name; the remaining settings can be modified on the primary failover member only.

    • Use SSL/TLS — If you did not select TLS security when you created the mirror (see Securing Mirror Communication with TLS Security), you can add TLS security to the mirror by following this procedure:

      1. On each mirror member, including the primary, the backup, and all asynchs if any, edit the mirror, click the Set Up SSL/TLS link to the right of the Use SSL/TLS check box, and follow the instructions in the Creating and Editing a TLS Configuration for a Mirror section of the “Using TLS with InterSystems IRIS” chapter of the Security Administration Guide to create a mirror TLS configuration on the member. (If the link is Edit SSL/TLS instead of Set Up SSL/TLS, the configuration already exists and you do not need to create it on that member.)

      2. Edit the mirror on the primary and click the Verify SSL button, which lets you verify the certificates of all current mirror members that can be contacted by the failover member you are editing. (Certificates can also be verified using the ^MIRROR routine.) If any certificate is not valid, an informational message is displayed; check the configurations and replace certificates if necessary. Otherwise, proceed to the next step.

      3. Select the Use SSL/TLS check box and click the Save button.

      4. Authorize the backup and any async members as described in Authorize the Second Failover Member or Async Member (TLS only).

      Note:

      The mirror does not have to be running when you add TLS security using this procedure.

      The SYS.Mirror.UpdateMirrorSSL() mirroring API method and the ^SECURITY routine can also be used to update a mirror’s member’s TLS settings.

    • Use Arbiter — If you did not configure an arbiter when creating the mirror, you can do so by selecting this setting on the primary and entering the hostname or IP address of the system you want to configure as arbiter and the port used by its ISCAgent process (2188 by default). See Automatic Failover Mechanics, Locating the Arbiter to Optimize Mirror Availability, and Installing the Arbiter for additional information about the arbiter.

    • Use Virtual IP — Change whether or not you want to use a Virtual IP address by selecting or clearing this check box on the primary. If you select Use Virtual IP, you must provide (or can change if already provided) an IP address, Classless Inter-Domain Routing (CIDR) mask, and network interface.

      Important:

      See Configuring a Mirror Virtual IP (VIP) for requirements and important considerations before configuring a VIP.

    • Quality of Service Timeout (msec) — The maximum time, in milliseconds, that a failover member waits for a response from the other failover member before taking action; also applies to the arbiter’s wait for a failover member’s response. See Configuring the Quality of Service (QoS) Timeout Setting for more information. This setting can be changed on the primary failover member only.

    • Compression Mode for Failover Members, Compression Mode for Async Members — Change the settings that specify whether to compress journal data before transmission from the primary to the backup and to async members, respectively, and which compression type to use for each; see Journal Data Compression for more information. When you change one or both compression settings, the mirror connections of all affected members (backup and/or asyncs) are restarted so the new compression mode can be used immediately.

    • Allow Parallel Dejournaling — Change the setting that specifies whether to enable parallel dejournaling for the failover members and DR asyncs (the default), all members including reporting asyncs, or the failover members only. Parallel dejournaling increases mirror throughput, but may slightly increase the chances of inconsistent results from queries or reports involving multiple databases; see Configuring Parallel Dejournaling for more information.

  5. The Mirror Member Information section lists the member name and type, instance directory, and network addresses of each mirror member. On the primary, click a member name to update that member’s network information (except for the member’s Superserver port, which must be updated locally; see Updating Mirror Member Network Addresses).

    If the backup is currently connected to the mirror, you cannot change any network information except the backup’s Superserver port; if the backup is disconnected and network information for the primary has changed, you can update the primary’s information here so that the backup can reconnect when desired. See Updating Mirror Member Network Addresses for important information about updating the network addresses of mirror members.

  6. On the primary in a mirror that uses TLS, select the Authorize/Reject Pending New Members link (if it appears) to authorize new members so they can join the mirror, or the Authorize/Reject Pending DN Updates link (if it appears) to authorize DN updates on other members so that mirror communication can continue. On the backup, select Authorize Pending DN Updates (if it appears) to authorize pending DN updates from the primary so that the backup can continue to communicate with the primary. See Authorizing X.509 DN Updates (TLS Only) for information about authorizing DN updates.

  7. Click Save.

Note:

The Add New Async Member button on the Edit Mirror page on the primary is reserved for use with other InterSystems products. Do not use this button in this version of InterSystems IRIS.

Remove Mirrored Databases from a Mirror

You can convert a database from mirrored to unmirrored, local use by removing it from the mirror, which you do through the Mirror Monitor (see Monitoring Mirrors for more information about the Mirror Monitor).

Note:

Alternatively, you can remove mirrored databases from a mirror by selecting the Remove mirrored database(s) option from the Mirror Management main menu list of the ^MIRROR routine (see Using the ^MIRROR Routine), or by using the SYS.Mirror.RemoveMirroredDatabase() API call.

When you remove a database from a mirror on an async, the failover members are unaffected; the database remains a part of the functioning mirror. Once you have removed it from a failover member, however, it must be removed from the other failover member and any async members on which it is mirrored. To entirely remove a database from a mirror, start by removing it from the primary failover member, then the backup failover member, then any async members.

Important:

Removing a database from a mirror on the primary is a permanent action. Once a mirrored database is removed on the primary, returning it to the mirror later will require the procedures used for adding an existing database to the mirror for the first time.

To remove a database from a mirror, do the following on either failover system:

  1. Navigate to the Mirror Monitor page (System Operation > Mirror Monitor) on the primary failover member.

  2. In the Mirrored databases list, click Remove in the row of the database you wish to remove from the mirror.

    If you want to remove more than one database at a time, select Remove from the Select an action drop-down and click Go to open a dialog in which you can select multiple mirrored databases and remove all of them at once.

Using Managed Key Encryption in a Mirror

As described in the “Managed Key Encryption” chapter of the Security Administration Guide, you can secure individual InterSystems IRIS databases by encrypting them. You can also activate encryption of journal files on any InterSystems IRIS instance. The following sections explain how to use these features in a mirror:

Encrypting Mirrored Databases

While database encryption on a mirror member requires preparation as on any system, there are no specific mirroring-related requirements for database encryption. For the greatest possible security, however, InterSystems recommends that a mirrored database that is encrypted on the primary also be encrypted on all mirror members. For this reason, when you add a mirrored database that is encrypted on the primary to another member without encrypting it, a security warning is sent to the messages log.

Based on the best practice of coequality of failover members, as described in Mirror Configuration Guidelines, a given database is typically encrypted using the same encryption key on both failover members and on any DR async members that may be promoted to failover.

When at least one encryption key is activated, you have the option of encrypting any new databases you create. Therefore, when using the procedure in Create a Mirrored Database, select the encryption option when you create the database on each mirror members. If you add an existing database to a mirror on the primary, as described in Add an existing database to the mirror, and that database is encrypted, you must either activate the key with which it was encrypted on each member you add it to, or convert the database to a new key after adding it on the each mirror members. For the procedure for doing the latter, see Converting an Encrypted Database to Use a New Key in the “Performing Encryption Management Operations” chapter of the Security Administration Guide. (You can also use this procedure to switch one or more existing encrypted mirrored databases to a new encryption key, or to remove encryption from a database.)

To encrypt one or more unencrypted mirrored databases on the failover members of an existing mirror, use the following procedure:

  1. Load and activate the encryption key(s) to be used on both failover members, as described in Managing Keys in Key Files in the “Managed Key Encryption” chapter of the Security Administration Guide.

  2. On the backup, do the following

    1. Stop mirroring, as described in Stopping Mirroring on Backup and Async Members.

    2. Encrypt each database as described in Converting an Unencrypted Database to Be Encrypted in the “Performing Encryption Management Operations” chapter of the Security Administration Guide.

    3. Start mirroring.

    4. Go to the Mirror Monitor page (System Operation > Mirror Monitor) and wait until the status of all mirrored databases is Dejournaling, as described in Mirrored Database Status, before proceeding.

  3. Gracefully shut down the primary using the iris stop command (see Controlling InterSystems IRIS Instances in the “Using Multiple Instances of InterSystems IRIS” chapter of the System Administration Guide) so that the mirror fails over and the backup becomes the new primary.

  4. Restart the primary; when it becomes backup, follow the steps described for the original backup in an earlier step to encrypt the same databases using the same keys.

  5. Shut down the current backup so that the original primary once again becomes primary.

  6. Restart the original backup so that it once again becomes backup.

To encrypt mirrored databases on an async member, follow the steps described for the backup in the previous procedure — stop mirroring, encrypt the databases, and start mirroring. Remember that the best practice is to use the key(s) used on the failover members on any DR async that may be promoted to failover.

Activating Journal Encryption in a Mirror

When activating journal encryption on mirror members, bear in mind three important considerations:

  • You cannot activate journal file encryption on the failover members and DR asyncs unless the mirror requires TLS security.

  • If journal encryption is activated on the primary, it must be activated on any reporting asyncs belonging to the mirror. In addition, it is a best practice to activate journal encryption on the backup and on any DR asyncs as well, so that in the event of failover or DR promotion journal encryption will continue to be in force.

  • Journal encryption among failover members and DR asyncs requires that the encryption key used for journal encryption on one member be activated (although not necessarily used for journal encryption) on others, to be used to decrypt received journal files as needed. Specifically,

    • If journal encryption is activated on the primary, the key used for journal encryption on the primary must be loaded and activated on the backup and all DR asyncs. (If a reporting async that does not have the primary’s journal encryption key activated is changed to DR async, a warning is generated; the async can remain temporarily connected to the mirror, but will not be able to reconnect the next time this is required unless the key has been activated.)

    • If journal encryption is activated on the backup or a DR async, the key used for journal encryption on that member must be loaded and activated on the primary.

    Again, as a best practice in preparation for failover or promotion, if any member that is (primary, backup) or may become (DR async) a failover member has a journal encryption key designated, this key should be loaded on all other such members, including multiple DR asyncs.

    Note:

    If you activate journal encryption on reporting ayncs only, the mirror does not need to require TLS security, and the only encryption key requirement is that a key be selected for journal encryption on each reporting async for which it is activated.

The following procedure describes the steps for activating journal encryption on a mirror consisting of failover members A (current primary) and B (current backup), DR async D, and reporting async R:

  1. If the mirror does not currently require TLS security (see Securing Mirror Communication with TLS Security), configure it to do so using the procedure described in Editing or Removing a Failover Member.

  2. Select the encryption key or keys that will be used to encrypt journal files on A, B, and D. These can all be different if desired.

  3. On each of A, B, and D, and optionally on R if it may be converted to a DR async, perform the following steps:

    1. Load and activate all keys that will be used to encrypt journal files on A, B, and D (and optionally R), if they are not already activated.

    2. Select the desired key for journal encryption on the instance as described in Specifying the Default Database Encryption Key or Journal Encryption Key for an Instance in the “Managed Key Encryption” chapter of the Security Administration Guide.

  4. If you did not already do so in the previous step, load, activate, and select a journal encryption key on R.

  5. On A, B, D, and R, in that order, activate journal encryption as described in Configuring Encryption Startup Settings in the “Managed Key Encryption” chapter of the Security Administration Guide.

    When you activate journal encryption on an instance, encryption begins after the instance is restarted or the next journal switch, whichever comes first. To make the change immediate without restarting mirror members, you can manually switch journal files on each member, as described in Switch Journal Files in the “Journaling” chapter of the Data Integrity Guide.

    Note:

    The ^JOURNAL routine (see the “Journaling” chapter of the Data Integrity Guide) includes an option you can use to activate/deactivate journal encryption instead of using the Management Portal. When you activate encryption using this option, the instance immediately switches to an encrypted journal file, and sets the encryption startup setting to Interactive.

To switch journal encryption keys on an instance, load, activate, and select the new key, which will be used for encryption after the instance is restarted or the next journal switch, whichever comes first.

When adding a DR async to the mirror after journal encryption is activated, ensure that the journal encryption key or keys in use on A, B and D are activated on the new DR async before it is added.

Configuring Application Server Connections to a Mirror

When you deploy a distributed cache cluster with a mirrored data sever using InterSystems Cloud Manager (ICM), ICM takes care of all the needed configuration. When you deploy a cluster using the management portal, you must indicate that the data server is a mirror when adding it to each application server. When the data server is added in this way (by ICM or manually), each application server regularly collects updated information about the mirror from the primary, automatically detecting failover and redirecting connections to the new primary as needed.

For information about configuring a mirrored data server with ICM, see Deploy the Cluster with InterSystems Cloud Manager in the “Horizontally Scaling Systems for User Volume with InterSystems Distributed Caching” chapter of the Scalability Guide. To manually configure a mirror as the data server in a distributed cache cluster, use the following procedure:

  1. Prepare both of the failover members and any DR async members as data servers as described in Preparing the Data Server in the same chapter of the Scalability Guide. All of these instances must be configured with the same Maximum number of application servers setting.

  2. On each application server, do the following:

    • Add the data server as described in Configuring an Application Server in the same chapter of the Scalability Guide, being sure to select the Mirror Connection check box and entering the DNS name or IP address of the current primary failover member for Host DNS Name or IP Address, not the virtual IP address (VIP) of the mirror (if it has one).

    • Create one or more namespaces mapped to one or more remote databases on the data server, as described in Configuring an Application Server. You can select both mirrored databases (databases listed as :mirror:mirror_name:mirror_DB_name) and nonmirrored databases (databases listed as :ds:DB_name); only mirrored databases remain accessible to the application server in the event of mirror failover. When the data server is a failover member, mirrored databases are added as read-write, and nonmirrored databases are added as read-only, if journaled, or read-write, if not journaled; when the data server is a DR async member, all databases are added as read-only.

    Note:

    A mirrored database path in the format :mirror:mirror_name:mirror_DB_name: can also be used in an implied namespace extended global references (see Extended Global References in the “Global Structure” chapter of Using Globals).

Important:

A failover mirror member does not accept ECP connections that are not configured as mirror connections as described in the foregoing; a data server that is not a mirror member does not accept ECP connections that are configured as mirror connections. This means that if you add an existing data server to a mirror, or remove one from a mirror, the data server must be removed as a remote data server on all application servers and added again using the appropriate procedure, either as a mirror connection as described here, or with the Mirror Connection check box cleared if it is no longer a connection to a failover member.

After configuring application servers to connect to a mirror, perform redirection tests by gracefully shutting down the current primary to ensure that the application servers connect to the intended mirror member.

You can also identify a data server as a mirror connection while restricting the connection to the designated mirror member specified by the Address and Port properties of the application server’s ECPServer definition. This means that the application server does not redirect connections, even when the designated member is not primary. When you configure the connection in this way, the following rules apply:

  • If the designated member is primary, it accepts the connection from the application server as normal. If the member is a failover member but not primary (as when a former primary is restarted and becomes backup), it accepts the connection when it becomes primary.

  • If the designated member was formerly primary, and is restarted and becomes primary again, the application server’s connection to the member is recovered. If the member is a failover member but not primary, it does not accept the connection until it becomes primary.

  • If the designated member is a DR async, it accepts the connection and provides the application server with read-only access to mirrored databases (and to any other databases configured as remote databases on the application servers).

Restricting the connection to a designated mirror member is useful in certain special configurations when redirecting connections to other members is not desired, such as when this would entail high-latency ECP connections. Two examples of its use are as follows:

  • Suppose your mirror primary is in data center A (DCA) and a backup or DR async is in remote data center B (DCB). Each member has its own bank of application servers configured. Network load balancers direct connections to the correct data center. But if the primary becomes unavailable and the member in DCB becomes primary through failover or promotion, you do not want the application servers in DCA connecting to the member in DCB, which would lead to high-latency connections between DCA and DCB. In this situation, on the application servers in DCA, you would restrict the mirror connection to the primary, so that in the event of failover, so that they do not redirect to DCB and their connections can be recovered when the member in DCA becomes primary again.

  • Suppose your primary and backup are in DCA and a DR async, with its own application servers for use in the event of a disaster, in remote DCB. On the application servers in DCA, the primary would be configured with a standard mirror connection, since you want the connections to be redirected within DCA in the event of failover. On the application servers in DCB, however, the mirror connection would be restricted to the DR async. That way, you can test the mirror connection on a read-only basis either as part of disaster recovery preparation or before cutting over during an actual disaster. After the DR async is promoted, the application servers in DCA could redirect connections to the new primary in DCB (unless prevented at the network level), but if they are not already down they can be brought down to prevent this.

You cannot restrict the application server’s connection to a designated mirror member using the management portal. Instead, do the following:

  1. If you have not already done so, use the procedure described earlier in this section to prepare the failover members and any DR asyncs as data servers, and to configure a connection to the data server on each application server.

  2. Use the Config.ECPServer class to modify the application server’s MirrorConnection property, giving it a value of -1. You can also edit the application server instance’s iris.cpf file. In the [ECPServers] section of the file, change the third parameter from 0 to -1; see ECPServers in the Configuration Parameter File Reference for more information.

    Once you have modified the MirrorConnection property in one of these ways, you must not use the management portal to change the setting of the Mirror Connection check box.

Configuring a Mirror Virtual IP (VIP)

As described in Planning a Mirror Virtual IP (VIP), you can configure a mirror virtual address that allows external applications to interact with the mirror using a single address, ensuring continuous access on failover.

After configuring InterSystems IRIS for the mirror VIP and then configuring the mirror VIP, perform failover tests by gracefully shutting down the current primary (as described in Planned Outage Procedures) to ensure that applications can continue to connect to the mirror regardless of which failover member is primary.

Important:

Before configuring a mirror VIP on a Linux platform, ensure that the arping command is available by installing the appropriate package (for example, the Debian iputils-arping package).

If one or more of a mirror’s members is a nonroot InterSystems IRIS instance on a UNIX® or Linux system, as described in InterSystems IRIS Nonroot Installation in the chapter “Installing InterSystems IRIS on UNIX® and Linux” in the Installation Guide, a mirror VIP cannot be used.

Note:

See Promoting a DR Async Member to Failover Member for important information about promoting a DR async to primary when a VIP is in use.

Configuring InterSystems IRIS for a Mirror VIP

To ensure that the management portal and Studio can seamlessly access the mirror regardless of which failover member is currently the primary, it is recommended that the failover members be configured to use the same superserver and web server port numbers.

The application servers in a distributed cache cluster with mirrored data server do not use a mirror’s VIP. When adding a mirror as a data server (see Configuring Application Server Connections to a Mirror), do not enter the virtual IP address (VIP) of the mirror, but rather the DNS name or IP address of the current primary failover member. Because the application server regularly collects updated information about the mirror from the specified host, it automatically detects a failover and switches to the new primary failover member. For this reason, both failover members and any DR async members must be prepared a data servers with the same Maximum number of application servers setting; see Configuring Application Server Connections to a Mirror for further distributed caching considerations.

When configuring one or both failover members as license servers, as described in the “Managing InterSystems IRIS Licensing” chapter of the System Administration Guide, specify the actual hostname or IP address of the system you are configuring as the Hostname/IP Address; do not enter the VIP address.

Configuring a Mirror VIP

To configure a mirror VIP, you must enter the following information:

  • An available IP address to be used as the mirror VIP. It is important to reserve the VIP so that other systems cannot use it; for example, in a Dynamic Host Configuration Protocol (DHCP) network configuration, the VIP should be reserved and removed from the DNS tables so that it is not allocated dynamically to a host joining the network.

  • An appropriate network mask, which you must specify in Classless Inter-Domain Routing (CIDR) notation. The format for CIDR notation isip_address/CIDR_mask, where ip_address is the base IP address of system, and CIDR_mask is platform-dependent, as follows:

    • macOS — must be /32.

    • All other platforms — must match the mask of the IP address assigned to the base interface. For example:

      bash-2.05b# uname -a 
      AIX apis 3 5 00C0B33E4C00
      bash-2.05b# ifconfig en1
      en1:
      flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,
      GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>
              inet 10.0.2.11 netmask 0xffffff00 broadcast 10.0.2.255
              tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 
      
      Copy code to clipboard

      In this example, the en1 interface has a base address of 10.0.2.11 with a netmask of 0xffffff00, which translates to /24. Therefore, to assign 10.0.2.100 as the VIP to the en1 interface, you specify the network mask as follows (in CIDR notation): 10.0.2.100/24.

  • An available network interface on each of the failover members. The interfaces selected on the two systems must be on the same subnet as the VIP.

    When selecting a network interface, the following platform-specific rules must be followed to ensure correct behavior:

    • IBM AIX®, Linux (Red Hat, SuSE, Ubuntu), Apple macOS, and Windows — An existing physical network interface must be provided during VIP configuration. On these platforms, IP address aliasing is used to bind an IP address (that is, the VIP) to an existing physical network interface. This platform allows a single physical network interface to host multiple IP addresses.

Configuring the ISCAgent

The ISCAgent runs securely on a dedicated, configurable port (2188 by default) on each mirror member. When the agent receives an incoming network connection which directs it to a mirrored instance, it executes irisuxagent in that instance to escalate to the privileges necessary to administer the mirror member. If the mirror is configured to require TLS, the incoming connection is authenticated before any actions are performed.

When multiple InterSystems IRIS instances belonging to one or more mirrors are hosted on a single system, they share a single ISCAgent.

This section provides information on managing the ISCAgent in the following ways:

Starting and Stopping the ISCAgent

The ISCAgent, which is installed when you install or upgrade InterSystems IRIS, runs as user iscagent and as a member of the iscagent group by default. To acquire the group privilege, which is necessary to execute the irisuxagent utility that provides it with access to an InterSystems IRIS instance (as described in ISCAgent), the ISCAgent must be started automatically during system startup or by a user with root privileges. Once it has assigned itself the needed user and group privileges, the ISCAgent discards all root privileges.

The ISCAgent must be configured to start automatically when the system starts on each failover and DR async mirror member. InterSystems provides platform-specific control scripts that can be added to the initialization process by a system administrator, as described in the following sections. (Consult your operating system documentation for detailed system startup configuration procedures.)

Starting the ISCAgent on UNIX® and macOS Systems

On UNIX® and macOS platforms, run the ISCAgent start/stop script, which is installed in the following locations, depending on the operating system:

  • IBM AIX®: /etc/rc.d/init.d/ISCAgent

  • macOS: /Library/StartupItems/ISCAgent/ISCAgent

For example, to start the ISCAgent on the IBM AIX® platform, run the following command as root: /etc/rc.d/init.d/ISCAgent start; to stop it, run the command /etc/rc.d/init.d/ISCAgent stop.

Additional ISCAgent considerations on UNIX®/Linux platforms include the following:

  • As previously noted, the ISCAgent must be started automatically at system startup on each failover and DR async mirror member. There may also be times at which it is useful to have a user start, stop, or restart the agent. This can be done in the following ways:

    • Directly, by the root user, as described in the preceding.

    • Using the agentctrl executable in the InterSystems IRIS instance’s /bin directory, by any user who is able to start and stop the InterSystems IRIS instance. For example, to start the agent, execute the following command:

      /iris/bin$ ./agentctrl start
      

      The command also takes the arguments stop and restart.

  • InterSystems IRIS uses the iscagent.status file, located in the directory specified by the IRISSYS environment variable (see Installation Directory in the “Preparing to Install” chapter of the Installation Guide) or in /var/run if IRISSYS is not defined, to track the status of the ISCAgent. Because the agent must be able to gain an exclusive lock on this file, if the iscagent.status file is located in /var/run and /var/run is on an NFS-mounted filesystem, the NFS configuration must support fcntl file locking.

  • As described earlier, the ISCAgent obtains the privileges it needs to administer InterSystems IRIS instances using irisuxagent. By default, the agent has the privileges required (iscagent user/iscagent group) to execute irisuxagent, and under typical configurations, no change is necessary.

    Depending on your system’s security configuration, however, instances at your site may require additional privileges to navigate to the /bin directory of the mirrored instance in order to execute irisuxagent. If so, you must ensure that the ISCAgent’s privileges are sufficient to execute the command. To do so, you can modify the agent’s privileges using the following procedure:

    1. Create the file /etc/iscagent/iscagent.conf, or edit it if it already exists (for example, because you previously created it to customize the ISCAgent port number or interface).

    2. To add group privileges, add the following line, specifying one or more groups that are required to execute irisuxagent:

      privileges.group=iscagent,<groupname>[,<groupname>[,...]]
      

      Typically, adding group privileges is sufficient. Under some configurations, however, you may need to run the ISCAgent as a different user. This can also be done in /etc/iscagent/iscagent.conf, as follows:

      privileges.user= <username>
      
      Note:

      Because irisuxagent requires iscagent group privileges, iscagent must remain in the groups list.

  • The ISCAgent writes messages to the operating system’s system error log, for example /var/log/messages on Linux.

Starting the ISCAgent on Linux Systems

On Linux systems supporting systemd (such as SUSE Linux Enterprise Server 12, SP1 or later), the /etc/systemd/system/ISCAgent.service file is installed, providing support for management of the ISCAgent using systemd. On any such system, the following commands can be used to start, stop and display the status of the ISCAgent:

systemctl start ISCAgent.service
systemctl stop ISCAgent.service
systemctl status ISCAgent.service
Copy code to clipboard

To control whether the ISCAgent starts on system boot on a system that supports systemd, use the following commands:

sudo systemctl enable ISCAgent.service
sudo systemctl disable ISCAgent.service
Copy code to clipboard

By default, systemd services are disabled. You can use systemctl to start and stop the service on demand, even when it is disabled.

The ISCAgent.service file does not read the location of the InterSystems IRIS registry and shared support files from the IRISSYS environment variable (see Installation Directory in the “Preparing to Install” chapter of the Installation Guide), but instead is installed with /usr/local/etc/irissys as the location. You can edit ISCAgent.service to specify a different registry directory if required.

On all Linux systems, the ISCAgent start/stop script described in Starting the ISCAgent on UNIX® and macOS Systems is installed in /etc/init.d/ISCAgent. If systemd is not supported, use the commands described in that section to start and stop the ISCAgent.

The remainder of the information provided in Starting the ISCAgent on UNIX® and macOS Systems also applies to Linux systems supporting systemd.

Important:

Although it is possible to use either the systemctl commands or the /etc/init.d/ISCAgent script on a Linux system that supports systemd, you must choose one method and use it exclusively, without switching back and forth. The ISCAgent should always be stopped using the method with which it was started.

When you upgrade InterSystems IRIS on such a Linux system, a running ISCAgent is automatically restarted using systemd. If you are using the /etc/init.d/ISCAgent script to manage the ISCAgent, stop the agent before performing the upgrade so that it is not automatically restarted, then restart it using the script after the upgrade.

When changing from using the /etc/init.d/ISCAgent script to using systemctl commands, before starting the agent with systemctl for the first time, do the following as root:

  1. Run the command the following command:

    systemctl status ISCAgent
    
  2. If the output from the command contains this warning:

    Warning: Unit file changed on disk, 'systemctl daemon-reload' recommended.
    

    run the following command:

    systemctl daemon-reload
    
  3. When the previous command has completed, run systemctl status ISCAgent again to confirm that the warning does not appear.

Starting the ISCAgent for Nonroot Instances on UNIX®/Linux and macOS Systems

Although InterSystems IRIS is typically installed as root, on UNIX®/Linux and macOS Systems it is possible for an instance to be installed and run by another user. Nonroot installation is described in InterSystems IRIS Nonroot Installation in the “Installing InterSystems IRIS on UNIX® and Linux” chapter of the Installation Guide.

The ISCAgent for a nonroot instance is started by the installing user running the ISCAgentUser script, located in the directed defined by the IRISSYS environment variable, in the background, for example:

nohup <IRISSYS_directory>/ISCAgentUser &

While it may not be possible to configure the ISCAgent to start automatically when the system starts, this remains the first choice if it can be achieved. When the mirror includes two failover members, the best practice is to start the agent as soon as possible after the system boots, even if you do not intend to start InterSystems IRIS; this aids in recovery in certain situations, such as that described in Unplanned Outage of Both Failover Members.

Starting the ISCAgent on Microsoft WindowsSystems

On Microsoft Windows systems, start the ISCAgent process as follows:

  1. In the Microsoft Windows Control Panel, select Services from the Administrative Tools drop-down list, and double-click ISCAgent to display the ISCAgent Properties window.

  2. On the Extended tab, click Start to start, or Stop to stop ISCAgent.

  3. On the Extended tab, select Automatic from the Startup type drop-down list.

Customizing the ISCAgent

You can customize the following ISCAgent attributes:

  • Port Number

    As described earlier in this chapter, the default ISCAgent port is 2188. While this is typically all that is needed, you can change the port number if required.

  • Interface

    The ISCAgent binds to the default (or configured) port on all available interfaces. While this is typically all that is needed, you can change the ISCAgent to bind to the interface serving a specific address if required.

  • SYSLOG severity level

    By default, the ISCAgent sends all log messages to the InterSystems IRIS system error log, also known as SYSLOG (see InterSystems IRIS System Error Log in the “Monitoring InterSystems IRIS Using the Management Portal” chapter of the Monitoring Guide). If desired, you can configure a minimum severity setting, so that messages below this severity are not passed to the system error log.

To customize the ISCAgent, do the following:

  1. Create the iscagent.conf file, or edit it if it already exists:

    • UNIX/Linux/macOS: /etc/iscagent/iscagent.conf

    • Windows: windir\system32\iscagent.conf (where windir is the system root directory).

  2. To customize the port, add the following line, replacing <port> with the desired port number:

    application_server.port=<port>
    Copy code to clipboard
  3. To customize the interface, add the following line, replacing <ip_address> with the address served by the desired interface:

    application_server.interface_address=<ip_address>
    Copy code to clipboard

    To explicitly bind to all available interfaces (the default), specify * as the IP address.

  4. To customize the SYSLOG severity level, add the following line, replacing <severity> with the desired minimum severity level where 1=warning, 2=severe, and 3=fatal:

    logging.minimum_severity=<severity>
    Copy code to clipboard

Configuring the Quality of Service (QoS) Timeout Setting

The Quality of Service Timeout (QoS timeout) setting plays an important role in governing failover member and arbiter behavior by defining the range of time, in milliseconds, that a mirror member waits for a response from another mirror member before taking action. The QoS timeout itself represents the maximum waiting time, while the minimum is one half of that. A larger QoS timeout allows the mirror to tolerate a longer period of unresponsiveness from the network or a host without treating it as an outage; decreasing the QoS allows the mirror to respond to outages more quickly. Specifically, the QoS timeout affects the following situations:

  • If the backup failover member does not acknowledge receipt of data from the primary within the range defined by the QoS timeout, the primary disconnects the backup and acts in accord with a possible outage of the backup.

  • If the backup receives no message from the primary within the range defined by the QoS timeout, the backup disconnects and acts in accord with a possible outage of the primary.

  • If the arbiter does not receive a response from a failover member within the range defined by the QoS timeout, it considers its connection with that failover member to be lost.

  • If operations performed on a failover member’s host cause the host to be entirely unresponsive for a period within the range defined by the QoS timeout, unwanted failover or alerts may result. This is a particular concern where virtualization platform operations such as backup or migration are involved; see Mirroring in a Virtualized Environment for more information

See Automatic Failover Mechanics for complete and detailed information about the role the QoS timeout plays in the behavior of the failover members and the arbiter.

The default QoS is 8 seconds (8000 ms) to allow for several seconds of intermittent unresponsiveness that may occur on some hardware configurations. Typically, deployments on physical (nonvirtualized) hosts with a dedicated local network can reduce this setting if a faster response to outages is required.

The Quality of Service Timeout setting can be adjusted on the Create Mirror page or the primary failover member’s Edit Mirror page.

Note:

The QoS timeout can also be adjusted using the Adjust Quality of Service Timeout parameter option on the Mirror Configuration menu of the ^MIRROR routine (see Using the ^MIRROR Routine).

Configuring Parallel Dejournaling

As described in Mirror synchronization, the mirrored databases on the backup failover member and any async members of a mirror are kept synchronized with the primary through dejournaling, which is the application of database updates made on the primary and recorded in the primary’s journal files to the corresponding databases on the other members. If there are sufficient computing and memory resources available, up to 16 dejournaling jobs can perform the updates in parallel within a single dejournaling operation. Called parallel dejournaling, this feature increases the throughput of mirrors, especially those that have a typically high rate of database updates. For information about parallel dejournaling, which is also used in journal restores, see Restore Globals From Journal Files Using ^JRNRESTO in the “Journaling” chapter of the Data Integrity Guide.

Parallel dejournaling is always enabled for the failover members of a mirror, and thus is used when the needed resources are available. By default, it is also enabled for DR async members. You can either enable it for reporting asyncs as well (that is, for all members) or restrict it to the failover members only by changing the Allow Parallel Dejournaling setting when configuring the first failover member (see Create a Mirror and Configure the First Failover Member) or editing the mirror on the primary (see Editing or Removing a Failover Member). When enabled (and supported by available resources), parallel dejournaling is used when catching up multiple databases in one operation, as described in Activating and Catching Up Mirrored Databases.

While enabling parallel dejournaling for reporting asyncs is advantageous for performance, it may increase the occurrence of unexpected results in queries or reports. This is because databases or globals within a database being updated by separate dejournaling jobs are likely to be at slightly different places in the dejournaling sequence. For example, database A may contain updates made on the primary at 11:45:30 when database B is only up to the updates from 11:45:28; by the same token, one global may be updated to the former time while another in the same database may only be updated to the latter. However, the uncertainty introduced by parallel dejournaling is similar to the uncertainty that is always present when running reports or queries against changing data that is in the process of being dejournaled. InterSystems therefore expects most reporting applications to run against mirrored databases for which parallel dejournaling is enabled with negligible impact. Note that all updates to a single global within a database are always applied by a single dejournaling job, and those updates are applied in order.

Using the ^ZMIRROR Routine

The user-defined ^ZMIRROR routine allows you to implement your own custom, configuration-specific logic and mechanisms for specific mirroring events, such as a failover member becoming primary.

The ^ZMIRROR routine contains the following entry points. All provide appropriate defaults if they are omitted.

  • $$CanNodeStartToBecomePrimary^ZMIRROR() — This procedure is called when an instance has determined that

    • The other failover member is not currently acting as primary and cannot become primary without manual intervention.

    • The local member is eligible to become primary and is about to begin the process of taking over.

    CanNodeStartToBecomePrimary provides an entry point for logic to block failover members from automatically becoming the primary (either at startup or when connected as the backup) to provide manual control over failover, and is not part of most ^ZMIRROR routines.

    When CanNodeStartToBecomePrimary returns 1, the local instance is fully initialized as the primary failover member and can continue the process of becoming primary: all mirrored databases are read-write, ECP sessions have been recovered or rolled back, and local transactions (if any) from the former primary have been rolled back. No new work has been done because users are not allowed to log in, superserver connections are blocked, and ECP is still in a recovery state.

    If this entry point returns 0 (False), the instance enters a retry loop in which it continues to call CanNodeStartToBecomePrimary every 30 seconds until

    • CanNodeStartToBecomePrimary returns 1 and the local member continues the process of becoming primary.

    • The instance detects that the other failover member has become primary (which must be through manual intervention), at which point the local member becomes backup.

  • $$CheckBecomePrimaryOK^ZMIRROR() — This procedure is called after CanNodeStartToBecomePrimary returns 1 (True).

    If CheckBecomePrimaryOK exists and returns 1, the mirror resumes operation with the local member as primary; this is the point at which you can start any local processes or do any initialization required to prepare the application environment for users. Bear in mind, however, that only the process running CheckBecomePrimaryOK can actually write to mirrored databases until after it returns 1, at which point the mirrored databases are updated for general use.

    As with CanNodeStartToBecomePrimary, if CheckBecomePrimaryOK returns 0 (False), the instance aborts the process of becoming primary and retries CheckBecomePrimaryOK every 30 seconds until

    • The entry point returns 1 and the mirror resumes operation with the local member as primary.

    • The instance detects that the other failover member has become primary (which must be through manual intervention), at which point the local member becomes backup.

    In general CheckBecomePrimaryOK is successful; if there are “common cases” in which a node does not become the primary member, they should be handled by CanNodeStartToBecomePrimary rather than CheckBecomePrimaryOK.

    If you move code from existing ^ZSTU or ^%ZSTART routines on the primary to ^ZMIRROR so that it is not executed until the mirror is initialized, CheckBecomePrimaryOK is the best location for it. However, if you use the job command to start other jobs, those jobs should wait until $SYSTEM.Mirror.IsPrimary() returns true, which will happen after CheckBecomePrimaryOK returns 1; alternatively, you can start the jobs in $$NotifyBecomePrimary^ZMIRROR() instead.

    Note:

    If CheckBecomePrimaryOK returns False, ECP sessions are reset. When a node succeeds in becoming the primary, the ECP client reconnects and ECP transactions are rolled back (rather than preserved). Client jobs receive <NETWORK> errors until a TRollback command is explicitly executed (see the ECP Rollback Only Guarantee section in the “Horizontally Scaling Systems for User Volume with InterSystems Distributed Caching” chapter of the Scalability Guide).

  • $$NotifyBecomePrimary^ZMIRROR() — This procedure is called for informational purposes at the very end of the process of becoming the primary failover member (that is, after users have been allowed on and ECP sessions, if any, have become active). This entry point does not return a value. You can include code to generate any notifications or enable application logins if desired.

  • $$NotifyBecomePrimaryFailed^ZMIRROR() — This procedure is called for informational purposes when

    • A failover member starts up and fails to become the primary or backup member.

    • The backup detects that the primary has failed and attempts to become primary but fails.

    This entry point is called only once per incident; once it is called, it is not called again until the member either becomes primary or the primary is detected.