This chapter covers topics related to managing and maintaining operational InterSystems IRIS mirrors.
You can monitor the operation of an existing mirror using one of two methods:
Both methods display information about the operating status of a mirror and its members and the incoming journal transfer rate, as well as about mirrored database status. In addition, the Mirror Monitor lets you perform several operations on the mirrored databases.
Monitoring Mirroring Communication Processes describes the mirror communication processes that run on mirror members.
Basic mirror member information, including a link to the Mirror Monitor, also appears in the management portal home page message pane (see Management Portal Message Pane in the “Using the Management Portal” chapter of the System Administration Guide.
Many database and mirror-related actions, such as mounting or dismounting a database and adding a database to or removing it from a mirror, are logged in the messages log (see Monitoring Log Files in the “Monitoring InterSystems IRIS Using the Management Portal” chapter of the Log Monitoring Guide).
Using the Mirror Monitor
To display the Mirror Monitor, navigate to the System Operation > Mirror Monitor page on any mirror member.
On a failover member, the Mirror Monitor contains the following buttons and sections:
The View Mirror Journal Files button lets you view and search through the member’s mirror journal files or nonmirror journal files; see View Journal Files in the “Journaling” chapter of the Data Integrity Guide for more information.
The Stop Mirror On This Member button (backup only) temporarily stops mirroring on the backup, as described in Stopping Mirroring on Backup and Async Members.
One of two buttons to demote the backup to DR async, leaving the mirror with a single failover member; see Demoting the Backup to DR Async for more information.
Mirror Failover Member Information lists the member name, superserver address and mirror private address of each failover member (see Mirror member network addresses for information about these addresses).
Arbiter Connection Status shows the address (hostname and port) of the arbiter if one is configured, the current failover mode, and the status of the member’s arbiter connection, as follows:
Both failover members are connected to the arbiter
Only this member is connected to the arbiter
This member is not connected to the arbiter (if the connection has been lost or no arbiter is configured)
See Automatic Failover Mechanics for information about the arbiter and the meaning of this connection information.Note:
When a failover member loses contact with the arbiter a severity 2 message is sent to the messages log. If the member fails to rebuild the connection to the arbiter, another severity 2 message is logged.
Mirror Member Status provides the member type and status, journal transfer status, dejournaling status of each mirror member, as described in Mirror Member Journal Transfer and Dejournaling Status. This table also shows the X.509 DNs of members if configured.
Mirrored Databases lists information about each mirrored database on the current member, including its name and directory as well as its status and dejournaling status, as described in Mirrored Database Status. Mirrored Databases also lets you perform several operations on one or more databases.Note:
The Mirrored Databases list includes only databases that are included in the mirror on the current member. For example, the databases listed on different reporting async members may be different as they may have different sets of mirrored databases.
On an async member, the Mirror Monitor contains the following buttons and sections:
The View Journal Files button (DR asyncs only) lets you view and search through the member’s mirror journal files or nonmirror journal files; see View Journal Files in the “Journaling” chapter of the Data Integrity Guide for more information.
The Stop Mirror On This Member button (DR asyncs only) temporarily stops mirroring on the async, as described in Stopping Mirroring on Backup and Async Members.
The Promote to Failover Member button (DR asyncs only) promotes a DR async to failover member; see Promoting a DR Async Member to Failover Member for information about this operation and its uses.
The section below the buttons displays the member name, async type, and X.509 DN (if configured) of the member.
Mirrors this async member belongs to provides information about each mirror a reporting async member belongs to and the member’s status, journal transfer status, and dejournaling status in that mirror, as described in Mirror Member Journal Transfer and Dejournaling Status. Each line includes a Details link to display information about the members of that mirror and a Stop Mirroring On This Member link to cause the async member to stop mirroring for that mirror, as described in Stopping Mirroring on Backup and Async Members
Mirror Member Status provides the type, status, journal transfer latency, and dejournal latency of all members of the mirror selected in Mirrors this async member belongs to, including the current async member.
If the async member belongs to a single mirror (which is the case with all DR asyncs), that mirror is displayed in this section by default; if the member belongs to more than one mirror, this section and the Mirrored Databases for MIRROR section below it do not appear until you click the Details link for one of the mirrors listed in Mirrors this async member belongs to section.
Mirrored Databases for MIRROR section is as described for failover members, with the same operations available. For reporting asyncs, the databases displayed are those in the mirror selected in Mirrors this async member belongs to and displayed in Mirror Member Status.
Mirror Member Journal Transfer and Dejournaling Status
When an InterSystems IRIS instance belongs to a mirror, its member type and status, journal transfer status, and dejournaling status are displayed by the Mirror Monitor and the ^MIRROR routine Status Monitor option, as described in Monitoring Mirrors.
The following table describes the possible types and statuses displayed:
Connected to primary as backup.
As primary, in a trouble state due to lost connection with the backup; see Automatic Failover Mechanics for complete information about the varying circumstances under which the primary can enter a temporary or indefinite trouble state.
Connected to primary as async.
(any of the above)
In a transitional state that will soon change when initialization or another operation completes; this status prompts processes querying a member’s status to query again shortly.
When there is no operating primary, a failover member can report this status for an extended period while it retrieves and applies journal files in the process of becoming primary; if there is another failover member that is primary, the status is Synchronizing instead.
Starting up or reconnecting after being stopped or disconnected, retrieving and applying journal files in order to synchronize the database and journal state before becoming Backup or Connected.
Unable to complete an action, such as becoming primary or connecting to primary; will retry indefinitely, but user intervention may be required. See messages log for details.
Mirroring on member stopped indefinitely by user and will not start automatically; see messages log for details.
Mirror no longer running due to unexpected condition; see messages log for details.
|Error||An unexpected error occurred while fetching the member’s status.|
Displayed on other members for a member that is down or inaccessible.
The member is not initialized (the mirror configuration is not yet loaded).
Mirror member Type and Status can also be obtained using the $SYSTEM.Mirror.GetMemberType() and $SYSTEM.Mirror.GetMemberStatus() methods. Some combinations of Type and Status not listed above are reported by these calls, as follows:
Not Member and Not Initialized—Instance is configured as not a mirror member.
Read-Only or Read-Write Reporting and M/N Status—Instance is an async member of several mirrors; supply the mirrorname argument to get Status for a particular mirror.
For backup and async mirror members, Journal Transfer indicates whether a mirror member has the latest journal data from the primary and, if not, how far behind journal transfer is, while Dejournaling indicates whether all of the journal data received from the primary has been dejournaled (applied to the member’s mirrored databases) and, if not, how, how far behind dejournaling is. The following tables describe the possible statuses for these fields displayed by the Mirror Monitor and ^MIRROR. (These fields are always N/A for the primary.)
Active (backup only)
The backup has received the latest journal data from and is synchronized with the primary. (See in Backup Status and Automatic Failover for more information about Active backup status.) Note that the backup can be Active even if its Dejournaling status is not Caught up; as long as the backup has all the needed journal files, they can be dejournaled even after it has lost its connection to the primary.
On the backup, indicates that the backup has received the latest journal data from the primary, but is not fully synchronized in that the primary is not waiting for it to acknowledge receipt of journal data. This status is often transient, as when the backup reconnects to the mirror.
On an async, indicates that the async has received the latest journal data from and is synchronized with the primary.
The member is a specific amount of time behind the primary, with time representing the amount of time elapsed between the timestamp of the last journal block the member received and the current time.
Disconnected on time
The member was disconnected from the primary at the specified time.
As noted, Active in the Journal Transfer field indicates that the backup has received all journal data from and is synchronized with the primary, and is therefore capable of taking over from the primary during failover without contacting the primary’s ISCAgent to obtain additional journal data.
|All journal data received from the primary has been dejournaled (applied to the member’s mirrored databases).|
Some journal data received from the primary has not yet been dejournaled, with time representing the amount of time elapsed between the timestamp of the last dejournaled journal block and the last journal block received from the primary.
Disconnected on time
The member was disconnected from the primary at the specified time.
|Warning! Some databases need attention.||At least one mirrored database is not in a normal state; databases should be checked.|
|Warning! Dejournaling is stopped.||Dejournaling has been stopped by an operator or because of an error; see Managing Database Dejournaling.|
Caught Up in the Dejournaling field for an Active backup failover member or Caught Up in both the Dejournaling field and the Journal Transfer field for an async member indicates that the member has received the most recent journal data from the primary and applied the most recent global operations contained in that data. If the member is not caught up, the amount of time elapsed since generation of the most recent journal data or writing of the most recent operation on the primary is displayed instead.
Incoming Journal Transfer Rate
Below the mirror member status list on backup and async members, the rate at which journal data has arrived from the primary since the last time the Mirror Monitor was refreshed is displayed under Incoming Journal Transfer Rate for This Member.
When the Mirror Monitor page is first loaded, this area displays the text --- (will be displayed on refresh). When the page is next refreshed, the information displayed depends on whether the incoming journal data is compressed (see Journal Data Compression), as follows:
If the journal data is not compressed, the incoming journal data rate is provided in kilobytes (KB) per second, for example:
42345 KB/s (22s interval)
If the incoming journal data is compressed, the display includes the incoming compressed data rate, the incoming journal (uncompressed) data rate, and the ratio of the latter to the former, for example:
14288 KB/s network; 39687 KB/s journal; ratio 2.78:1 (143s interval)
Mirrored Database Status
On backup and DR async members, the Missing Mirrored Databases Report on the Mirror Monitor page alerts you to any mirrored databases that are present on the primary but not on the current member. This is very important, as the backup, or a DR async if promoted to backup, cannot successfully take over in the event of a primary outage if it does not have the full set of mirrored databases. The full mirror database name of each missing database is listed. The Missing Mirrored Databases Report is not displayed if there are no missing databases.
On all members, the Mirrored Databases list on the Mirror Monitor page displays one of the following statuses for each database listed:
|Normal (primary only)||The mirrored database is writable (if not a read-only database) and global updates are being journaled.|
Dejournaling (backup and async)
|The database has been activated and caught up and the mirror is applying journal data to the database.|
The database has been activated but not caught up yet; the user-initiated Catchup operation is needed.
|Needs Activation||The database has not been activated yet; the user-initiated Activate and Catchup operations are needed.|
|Catchup Running||The user-initiated Catchup operation is running on the database.|
|Dejournaling Stopped||Dejournaling has been stopped by an operator or an error; see Stopping Mirroring on Backup and Async Members and Managing Database Dejournaling.|
|Database Dismounted||The database is dismounted.|
|Orphaned (backup and async)||The mirrored database does not exist on the primary.|
|Obsolete||The mirrored database is obsolete and should be removed from the mirror.|
On the primary, the Next Record to Dejournal column contains N/A if the status of the database is Normal. Otherwise, the column includes the following:
Time is the timestamp at the beginning of the next journal record to be applied to the database, or Current if this matches the primary's current journal position.
FileName is the name of the mirror journal file containing the next journal record to be applied.
Offset is position within the journal file of the beginning of the next journal record to be applied.
The status of a database and the operations related to it (Activate and Catchup) are discussed in Activating and Catching Up Mirrored Databases; the operations are available in the drop-down below the list. You can also use the dropdown to mount dismounted databases (but not to dismount mounted databases). You can use the Remove link or select Remove from the drop-down to remove a listed database from the mirror; see Remove Mirrored Databases from a Mirror for more information.
Using the ^MIRROR Status Monitor
The ^MIRROR routine provides a character-based mirror status monitor. The ^MIRROR Status Monitor option displays the status of the mirror members including type, status, journal transfer latency and dejournal latency (see Mirror Member Journal Transfer and Dejournaling Status). The monitor can be run on any mirror member, but running it on a failover member provides information about the arbiter configuration and about all connected async members, which running it on an async member does not.
To start the status monitor, open a Terminal window, run the ^MIRROR routine (see Using the ^MIRROR Routine) in the %SYS namespace, and select Status Monitor from the Mirror Status menu. The following is a sample of output from the monitor when run on a failover member:
Status of Mirror MIR25FEB at 17:17:53 on 02/27/2018 Member Name+Type Status Journal Transfer Dejournaling -------------------------- --------- ---------------- -------------- MIR25FEB_A Failover Primary N/A N/A MIR25FEB_B Failover Backup Active Caught up MIR25FEB_C Disaster Recovery Connected Caught up Caught up MIR25FEB_D Read-Only Reporting Connected Caught up Caught up Arbiter Connection Status: Arbiter Address: 127.0.0.1|2188 Failover Mode: Arbiter Controlled Connection Status: Both failover members are connected to the arbiter Press RETURN to refresh, D to toggle database display, Q to quit, or specify new refresh interval <60>
When you run the status monitor on an async member, only the failover members and that async are listed, and the status of dejournaling on the async (running or stopped) is also shown, for example:
Status of Mirror MIR25FEB at 17:17:53 on 02/27/2018 Member Name+Type Status Journal Transfer Dejournaling -------------------------- --------- ---------------- -------------- MIR25FEB_A Failover Primary N/A N/A MIR25FEB_B Failover Backup Active Caught up MIR25FEB_C Disaster Recovery Connected Caught up Caught up Dejournal Status: running (process id: 12256) Press RETURN to refresh, D to toggle database display, Q to quit, or specify new refresh interval <60>
By default, information about mirrored databases is not displayed. Enter d at the prompt to list information about each database in the mirror, including name, directory, status, and next record to dejournal as described in Using the Mirror Monitor, for example:
Mirror Databases: Last Record Name Directory path Status Dejournaled ------------- ----------------------------------- ----------- ----------- MIR25FEB_DB1 C:\InterSystems\20182209FEB25A\Mgr\MIR25FEB_DB1\ Active Current,c:\intersystems\20182209feb25a\mgr\journal\MIRROR-MIR25FEB-20180227.001,40233316 MIR25FEB_DB2 C:\InterSystems\20182209FEB25A\Mgr\MIR25FEB_DB2\ Active Current,c:\intersystems\20182209feb25a\mgr\journal\MIRROR-MIR25FEB-20180227.001,40233316
Monitoring Mirroring Communication Processes
There are processes that run on each system (primary and backup failover members, and each connected async member) that are responsible for mirror communication and synchronization.
For more information, see the following topics:
Mirroring Processes on the Primary Failover Member
Running the system status routine (^%SS) on the primary failover member reveals the processes listed in the following table.
The CPU, Glob, and Pr columns have been intentionally omitted from the ^%SS output in this section.
The processes are defined as follows:
Mirror Master: This process, which is launched at system startup, is responsible for various mirror control and management tasks.
Mirror Primary: This is the outbound data channel; it is a one-way channel. There is one job per connected system (backup failover or async member).
Mirror Svr:Rd*: This is the inbound acknowledgment channel; it is a one-way channel. There is one job per connected system (backup failover or async member).
Each connected async member results in a new set of Mirror Master, Mirror Primary, and Mirror Svr:Rd* processes on the primary failover member.
Mirroring Processes on the Backup Failover/Async Member
Running the system status routine (^%SS) on the backup failover/async member reveals the processes listed in the following table.
The processes identified in this table also appear on each connected async member:
Mirror Master: This process, which is launched at system startup, is responsible for various mirror control and management tasks.
Mirror JrnRead (Mirror Journal Read): This process reads the journal data being generated on the backup into memory and queues up these changes to be dejournaled by the dejournal job.
Mirror Dejour (Mirror Dejournal): This is the dejournal job on the backup failover member; it issues the sets and kills from the received journal data to the mirrored databases.
Mirror Prefet* (Mirror Prefetch): These processes are responsible for pre-fetching the disk blocks needed by the dejournal job into memory before the dejournal job actually attempts to use them. This is done to speed up the dejournaling process. There are typically multiple Mirror Prefetch jobs configured on the system. Mirror Backup: This process is a two-way channel that writes the journal records received from the primary into the backup’s mirror journal files and returns acknowledgment to the primary.
Updating Mirror Member Network Addresses
When one or more of the network addresses of one or more mirror members (including the primary) must be updated, as described in Editing or Removing a Failover Member, this information is generally changed on the primary. When you save your changes, the primary propagates them to all connected mirror members (and to disconnected members when they reconnect). You cannot change any mirror member network addresses on a connected backup or async member, as mirror members must receive all such information from the primary. There are a few exceptions to the general case, however, as follows:
Because the Superserver port of an InterSystems IRIS instance is part of its general configuration, it must be changed locally. Thus the Superserver port of a mirror member is the only mirror network information that is always updated on the member itself. To change the primary’s Superserver port, go to the Edit Mirror page on the primary, to change the backup’s, go to the Edit Mirror page on the backup, and so on.Note:
When you click the Edit Port link for the local member’s Superserver port in the Edit Network Address dialog, a dialog containing the Memory and Startup page of the management portal appears so you can change the port number. Do not, however, go directly to this page to change the Superserver port of a mirror member; always use the Edit Mirror or Edit Async Configurations page and the Edit Network Address dialog to make this change.
When a failover member or async member is disconnected and the primary’s network addresses have changed, you must first ensure that all mirror network addresses are correct on the current primary, then update the primary’s network addresses on the disconnected member or members (see Editing or Removing a Failover Member or Editing or Removing an Async Member). It may be necessary to restart a disconnected member after updating the primary’s network information before the member can reconnect to the mirror.
In some cases in which neither failover member is operating as primary, you may need to update the network addresses on one of the failover members before it can become primary. Once it becomes primary, it propagates these addresses to other members as they connect. It may be necessary to restart the member after updating the network addresses before the member can become primary.
As described in Configure Async Mirror Members, the Async Member Address you provide when an async member joins a mirror becomes the async’s superserver address and mirror private address (see Mirror Member Network Addresses). If you want these to be different, for example when you want to place a DR async’s mirror private address on the mirror private network while leaving its superserver address on the external network, after adding the async to the mirror you can update its addresses as described here.
Resolving Network Address Validation Errors
When a mirror member is started, the mirror initialization procedure verifies that the instance can be reached at the mirror private address or Superserver address (see Mirror Member Network Addresses) configured for it on the primary. If this does not succeed, validation fails, and the member is prevented from joining the mirror. There are two situations that cause network address validation errors:
The instance contacted at the configured mirror private address of the local instance is not the local instance. This is very likely because the local instance was copied from a different host, for example through backup and restore or cloning of a VM.
No instance can be contacted at the configured mirror private address of the local instance. This can occur for either of the following reasons:
The network configuration of the mirror member host has changed, for example its IP address has been changed. In this case, you can update the configured addresses of the member on the primary as described in Updating Mirror Member Network Addresses, then restart the local instance.
The host at the configured address is down. When this is the case, the previous problem (address mismatch) is very likely to occur when it comes back online.
When one of these validation errors occurs, an appropriate message is displayed in the messages log, and the validation problem is described on the Mirror Monitor page and the Edit Mirror page. Both pages provide two links you can use to resolve the situation, as follows:
Use the Remove the local mirror configuration link to remove the local (invalid) mirror configuration, as described in Editing or Removing Mirror Configurations.
Use the Join the mirror as a new member link to add the instance to the mirror as a new member, as describe in Creating a Mirror; this replaces the invalid local mirror configuration with a valid one.
When a validation error of this nature occurs, the dialogs that display when you choose one of these options also include information describing the problem and instructions regarding its resolution.
An additional option is to do nothing for the time being. You might choose this while you investigate the problem, or update the configured network addresses of the mirror member.
You can also resolve network address validation errors using the ^Mirror routine (see Using the ^MIRROR Routine).
Authorizing X.509 DN Updates (SSL/TLS Only)
When you configure a mirror to use SSL/TLS, you must authorize the newly-added second failover member and each new async member on the first failover member before it can join the mirror, as described in Authorize the Second Failover Member or Async Member (SSL/TLS only). For similar reasons, when a member of a mirror using SSL/TLS updates its X.509 certificate and DN, this update must be propagated to and authorized on other members in one of the following ways:
An X.509 DN update on the primary is automatically propagated to and authorized on other mirror members that are connected to the primary at the time the update is made.
If a backup or async member is not connected to the primary when the primary updates its X.509 DN, the update is added to that member’s Authorize Pending DN Updates list the next time it connects to the primary. To enable the member to continue as part of the mirror, the update must be authorized by clicking the Authorize Pending DN Updates link on the Edit Mirror page (backup) or Edit Async Configurations page (async) of the management portal. A backup or async member cannot reject an X.509 DN update from the primary.
An X.509 DN update on a backup or async member appears in the primary’s Authorize/Reject Pending DN Updates list immediately, if the member is connected to the primary, or the next time the member connects to the primary. To enable the member to continue as part of the mirror, the update must be authorized by clicking the Authorize/Reject Pending DN Updates link on the Edit Mirror page on the primary and selecting Authorize.
The Authorize/Reject Pending DN Updates option (primary) or the Authorize Pending DN Updates option (backup or async) on the Mirror Configuration menu of the ^MIRROR routine can be also used to authorize X.509 DN updates.
Promoting a DR Async Member to Failover Member
A disaster recovery (DR) async mirror member can be promoted to failover member, replacing a current failover member if two are configured or joining the current member if there is only one. For example, when one of the failover members will be down for a significant period due to planned maintenance or following a failure, you can temporarily promote a DR async to take its place (see Temporary Replacement of a Failover Member with a Promoted DR Async). During true disaster recovery, when both failover members have failed, you can promote a DR to allow it to take over production as primary failover member, accepting the risk of some data loss; see Manual Failover to a Promoted DR Async During a Disaster for more information.
When a DR async is promoted to failover member, it is paired, if possible, with the most recent primary as failover partner; when this cannot be done automatically, you are given the option of choosing the failover partner. Following promotion, the promoted member communicates with its failover partner's ISCAgent as any failover member does at startup, first to obtain the most recent journal data, then to become primary if the failover partner is not primary, or to become backup if the failover partner is primary. The promoted member cannot automatically become primary unless it can communicate with its failover partner to obtain the most recent journal data.
When promoting a DR async to failover member, there are several important considerations to bear in mind:
Depending on the location of the DR async, network latency between it and the failover partner may be unacceptably high. See Network Latency Considerations for information about latency requirements between the failover members.
When the DR async becomes a failover member, the failover member compression setting is applied, rather than the async member compression setting as before the promotion (see Journal Data Compression for information about these settings). Depending on the network configuration, you may need to adjust the failover member compression setting, as described in Editing or Removing a Failover Member, for optimal mirror function.
When a mirror private network is used to connect the mirror private addresses of the failover members, as described in Sample Mirroring Architecture and Network Configurations, a DR async that is not connected to this network should be promoted only to function as primary, and this should be done only when no other failover member is in operation. If a DR async is promoted when a primary is in operation but does not have access to the primary’s mirror private address, it cannot become backup; it will, however, be able to obtain journal data from the primary’s agent and become primary with the most recent journal data when the primary is shut down.
If a mirror VIP is in use, and the promoted DR async is not on the VIP subnet, some alternative means must be used to redirect user connections to the promoted DR should it become primary; for example, manually updating the DNS name to point to the DR async’s IP instead of the VIP, or configuring one of the mechanisms discussed in Redirecting Application Connections Following Failover.
In some disaster recovery situations, however, the promoted DR async cannot contact any existing failover member’s agent. When this is the case, you have the option of promoting the DR with no failover partner, as described under Promotion With Partner Selected by User in this section. This means that the DR can become primary only, using only the journal data it already has and any more recent journal data that may be available on other connected mirror members, if any. When this happens, the new primary may not have all the journal data that has been generated by the mirror, and some application data may be lost. If you restart a former failover partner while a DR async promoted in this manner is functioning as primary, it may need to be rebuilt; see Rebuilding a Mirror Member for more information. Be sure to see the DR promotion procedure later in this section for details.
When the primary InterSystems IRIS instance is in an indefinite trouble state due to isolation from both the backup and the arbiter in arbiter controlled mode, as described in Automatic Failover Mechanics Detailed, you cannot promote a DR async to failover member.
When possible, the promoted DR async’s failover partner is selected automatically, as follows:
When there is a running primary failover member, the primary is automatically selected as failover partner; the promoted member obtains the most recent journal data from it and becomes the backup. If InterSystems IRIS is running on the current backup, that member is simultaneously demoted to DR async; if it is not, the member is demoted to DR async when InterSystems IRIS is restarted.
When InterSystems IRIS is not running on any failover member but the ISCAgents on both failover members (or one if there is only one) can be contacted, the most recent primary is automatically selected as failover partner and the promoted member obtains the most recent journal data from it and becomes the primary. When InterSystems IRIS is restarted on the former primary, it automatically becomes the backup; when InterSystems IRIS is restarted on the former backup, it automatically becomes a DR async.
When InterSystems IRIS is not running on any failover member and at least one ISCAgent cannot be contacted, the promotion procedure informs you of which agents cannot be contacted and gives you the option of choosing a failover partner. To avoid the possibility of data loss, you should select the failover member that was last primary, even if its agent cannot be contacted. The results differ depending on the selection you make and ISCAgent availability, as follows:
If the agent on the partner you select can be contacted, the promoted DR async obtains the most recent journal data from it and then becomes primary. When InterSystems IRIS is restarted on the partner, it automatically becomes backup.
If the agent on the partner you select cannot be contacted, the promoted DR async does not become primary until the partner’s agent can be contacted and the most recent journal data obtained. At any time before the partner’s agent becomes available, however, you can, force the promoted member to become primary (as described in Manual Failover to a Promoted DR Async During a Disaster) without obtaining the most recent journal data; some application data may be lost as a consequence.
If you choose no failover partner, the promoted DR async attempts to obtain the most recent available journal data from all other connected async mirror members before becoming primary. Because there may not be any connected members with more recent journal data than the promoted DR async, some application data may be lost.
When you make this choice, you have the option of setting the no failover state on the promoted DR async so that it will prepare to become primary, including obtaining journal data from other connected members, but not become primary until you clear no failover. This allows you to perform any additional verification you wish and to bring additional members online, if possible, to potentially make more journal data available before allowing the promoted DR async to become primary.Note:
Messages about successful and unsuccessful attempts to contact mirror members to review their journal data, as well as successful and unsuccessful attempts to retrieve recent data when it is identified, are posted in the messages log.Caution:
Do not restart InterSystems IRIS on a former failover member whose ISCAgent was down when the DR async was promoted until you have set ValidatedMember=0 in the [MirrorMember] section of the Configuration Parameter File for the InterSystems IRIS instance, as described in the DR promotion procedure that follows.
When the failover partner is not selected automatically, the following rules apply:
Any former failover member that is not selected as partner becomes a DR async member when InterSystems IRIS is restarted.
On any former failover member whose agent could not be contacted at the time the DR async was promoted, you must at earliest opportunity and before restarting InterSystems IRIS instance set ValidatedMember=0 in the [MirrorMember] section of the Configuration Parameter File for the InterSystems IRIS instance (see [MirrorMember] in the Configuration Parameter File Reference). This instructs the InterSystems IRIS instance to obtain its new role in the mirror from the promoted DR async, rather than reconnecting to the mirror in its previous role. The ^MIRROR routine lists the failover member(s) on which this change is required.
If the promoted DR async becomes primary or is forced to become primary without obtaining the most recent journal data, some global update operations may be lost and the other mirror members may need to be rebuilt (as described in Rebuilding a Mirror Member). Under some disaster recovery scenarios, however, you may have no alternative to promoting a DR async to primary without obtaining journal data. If you are uncertain about any aspect of the promotion procedure, InterSystems recommends that you contact the InterSystems Worldwide Response Center (WRC) for assistance.
To promote a DR async member to failover member, do the following:
On the DR async member that you are promoting to failover member, navigate to the System Operation > Mirror Monitor page to display the Mirror Monitor.
Click the Promote to Failover Member button at the top of the page.
Follow the instructions provided by the resulting dialog boxes. In the simplest case, this involves only confirming that you want to proceed with promotion, but it may include selecting a failover partner or no partner, as described earlier in this section.
If a VIP is configured for the mirror, the promoted DR async must have a network interface on the VIP’s subnet to be able to acquire the VIP in the event of becoming primary (due to manual failover or to a later outage of the primary while operating as backup).
If the DR async has exactly one interface on the VIP’s subnet, the procedure automatically selects this interface.
If the DR async has more than one interface on the VIP’s subnet, the procedure asks you to choose an interface.
If the DR async does not have an interface on the VIP’s subnet, the promotion procedure warns you that this is the case and asks you to confirm before proceeding. If you go ahead with the procedure and promote the DR async, you will have to make take manual steps to allow users and applications to connect to the new primary, for example updating the DNS name to point to the DR async’s IP instead of the VIP.
When a former failover member’s agent is available at the time a DR async is promoted, it automatically sets ValidatedMember=0 in the [MirrorMember] section of the Configuration Parameter File for the InterSystems IRIS instance (see [MirrorMember] in the Configuration Parameter File Reference). This instructs the InterSystems IRIS instance to obtain its new role in the mirror from the promoted DR async, rather than reconnecting to the mirror in its previous role.
If a former failover member’s agent cannot be contacted at the time of promotion, this change cannot be made automatically. Therefore, at the earliest opportunity and before InterSystems IRIS is restarted on any former failover member whose agent could not be contacted at the time of promotion, you must manually set ValidatedMember=0 by editing the Configuration Parameter File for the InterSystems IRIS instance. The instructions list the former failover member(s) on which this change must be made.Caution:
Restarting InterSystems IRIS on a mirror member whose agent was down at the time of DR async promotion without first setting ValidatedMember=0 may result in both failover members simultaneously acting as primary.
Demoting the Backup to DR Async
In addition to promoting a DR async to failover member, you can do the reverse — demote the failover member that is not the current primary to DR async, so the mirror is left with a single failover mirror. This is useful in planned outage situations when you do not want a failover member to respond to temporary changes in the mirror’s configuration. For example:
When you have shut down the backup failover member and its host system for maintenance and the InterSystems IRIS instance on the primary is restarted (for whatever reason), it cannot become primary after the restart because it cannot contact the backup instance or its ISCAgent and thus has no way of determining whether it was the most recent primary. However, if you demote the backup to DR async before bringing it down, as described in Maintenance of Backup Failover Member, you avoid this risk, as the primary knows there is no current backup and that it can therefore become primary after a restart. You can then promote the demoted DR async to backup (as described in Promoting a DR Async Member to Failover Member) after you restart it,
When you are testing your disaster recovery capability by deliberately failing over to a DR async, as described in Planned Failover to a Promoted DR Async, and shut down the primary instance to trigger failover, you may want to restart it to keep it synchronized without it automatically becoming backup (since in a real disaster it is not likely to be available). In this case, you can demote it to DR async (through its ISCAgent) before restarting it, and then later promote it to failover member when you are ready.
To demote a failover member, navigate to the Mirror monitor page (Home > System Operation > Mirror Monitor) on one of the failover members, as described in Using the Mirror Monitor. Then:
On the backup, use the Demote to DR Member button to demote the backup to DR async. (You would use this method in the first of the preceding examples.)
On the primary, use the Demote Other Member button to demote the backup to DR async. (You would use this method in the second of the preceding examples.) Demotion succeeds only if the current member is primary and either the backup instance or its ISCAgent is reachable.
You cannot demote the current primary when no failover is set, as described in Avoiding Unwanted Failover During Maintenance of Failover Members.
The Demote Backup member to Async DR member option on the Mirror Management menu in the ^MIRROR routine and the SYS.Mirror.Demote() and SYS.Mirror.DemotePartner() mirroring API methods provide alternative means of demoting the backup to DR async.
Rebuilding a Mirror Member
Under some circumstances following an outage or failure, particularly if manual procedures are used to return a mirror to operation, a member’s mirrored databases may no longer be synchronized with the mirror. For example, when a backup that did not automatically take over following a primary outage is forced to become primary without the most recent journal data (see Manual Failover When the Backup Is Not Active), one or more of the mirrored databases on the former primary may be inconsistent with the new primary’s databases.
In some cases the mirror is able to reconcile the inconsistency, but in others it cannot. When a mirror member whose data is irreparably inconsistent with the mirror is restarted and attempts to rejoin the mirror, the process is halted and the following severity 2 message is written to the messages log:
This member has detected that its data is inconsistent with the mirror MIRRORNAME. If the primary is running and has the correct mirrored data, this member, including its mirrored databases, must be rebuilt.
This message is preceded by a severity 1 message providing detail on the inconsistency.
When this message appears in the messages log, take the following steps:
Confirm that the functioning mirror has the desired version of the data, and that the member reporting the inconsistency should therefore be rebuilt. This will likely be true, for example, in any case in which this message appears when you are restarting the former primary after having chosen to manually cause another member to become primary without all of the most recent journal data. If this is the case, rebuild the inconsistent member using the steps that follow.
If you conclude instead that the member reporting the inconsistency has the desired version of the data, you can adapt this procedure to rebuild the other members.
If you are not certain which version of the data to use or whether it is desirable to rebuild the inconsistent member, contact the InterSystems Worldwide Response Center (WRC) for help in determining the best course of action.
Back up the mirrored databases on a member of the functioning mirror. You can also use an existing backup created on a member of the mirror, if you are certain that
the backup was created before the outage or failure that led to the data inconsistency.
the current primary has all of the journal files going back to when the backup was created.
Remove the inconsistent member from the mirror as described in Editing or Removing Mirror Configurations, retaining the mirrored DB attribute on the mirrored databases.
Restore the mirrored databases on the member from the backup you created or selected, as described in Add an existing database to the mirror.
Stopping Mirroring on Backup and Async Members
You can temporarily stop mirroring on the backup or an async member. For example, you may want to stop mirroring on the backup member for a short time for maintenance or reconfiguration, or during database maintenance on the primary, and you might temporarily stop mirroring on a reporting async member to reduce network usage. To do so, do the following:
Navigate to the System Operation > Mirror Monitor page for the member on which you want to stop mirroring
If the member is the backup failover member, click the Stop Mirroring On This Member button.
If the member is an async, click the Stop Mirroring On This Member link in the row for the mirror you want the async to stop mirroring. (Stopping mirroring of one mirror does not affect others a reporting async belongs to.)
The operation takes a few seconds. When you refresh the Mirror Monitor, the Stop Mirroring On This Member is replaced by Start Mirroring On This Member, which you can use to resume mirroring.
When you stop mirroring on a member, mirroring remains stopped until you explicitly started it again as described in the preceding. Neither reinitialization of the mirror or a restart of the member starts mirroring on the member.
Managing Database Dejournaling
As described in Mirror Synchronization, dejournaling is the process of synchronizing mirrored databases by applying journal data from the primary failover member to the mirrored databases on another mirror member. Although dejournaling is an automatic process during routine mirror operation, under some circumstances you may need or want to manage dejournaling using options provided by the ^MIRROR routine (see Using the ^MIRROR Routine). Because of the differences in purpose between the backup failover member, DR async members, and reporting async members, there are also some differences in dejournaling and dejournaling management, specifically in regard to interruptions in dejournaling, whether deliberate or caused by error. In addition, a user-defined filter can be applied to dejournaling for one or more of the mirrors a reporting async belongs to.
All types of mirror members continue to receive journal data even when dejournaling of one or all mirrored databases is paused.
The SYS.Mirror.AsyncDejournalStatus(), SYS.Mirror.AsyncDejournalStart(), SYS.Mirror.AsyncDejournalStop(), and SYS.Mirror.DejournalPauseDatabase() mirroring API methods can also be used to manage dejournaling.
Managing Dejournaling on the Backup or a DR Async
Because mirrored databases on the backup failover member and DR async members should always be as close as possible to caught up to support potential takeover as primary or use in disaster recovery, respectively, dejournaling is paused by error for only the affected mirrored database, while it continues for others.
For example, when there is a database write error such as <FILEFULL> on the backup or a DR async member, dejournaling of the database on which the write error occurred is automatically paused, but dejournaling of other mirrored databases continues. Dismount the database and correct the error, then remount the database and resume dejournaling by selecting the Activate or Catchup mirrored database(s) option from the Mirror Management menu of the ^MIRROR routine or catching up the database using the management portal (see Activating and Catching Up Mirrored Databases).
On a DR async, you also have the option of pausing dejournaling for all mirrored databases on the member using the Manage mirror dejournaling on async member option on the Mirror Management menu of the ^MIRROR routine. (This option is disabled on backup members.) You can use this following a dejournaling error or for maintenance purposes. For example, if you prefer to pause dejournaling for all databases in the mirror when a dejournaling error causes dejournaling to pause for one database only, you can do the following:
Select Manage mirror dejournaling on async member option from the Mirror Management menu of the ^MIRROR routine to pause dejournaling for all databases.
Dismount the problem database, correct the error, and remount the database.
Select Manage mirror dejournaling on async member option from the Mirror Management menu of the ^MIRROR routine to restart dejournaling for all databases. (This option automatically activates the database that had the error and catches it up to the same point as the most up to date database in the mirror.)
When you pause dejournaling on a DR async member using the Manage mirror dejournaling on async member option, dejournaling does not restart until you use the option again to restart it.
Managing Dejournaling on a Reporting Async
As described in Async Mirror Members, a reporting async member can belong to multiple mirrors. For each of these mirrors, you may want dejournaling of the databases to be continuous or you may want dejournaling to be conducted on a regular schedule, depending on the ways in which the databases are being used. For example, for a given mirror you may want to dejournal between midnight and 4:00am, allowing the databases to remain static for stable report generation over the rest of the day.
In addition, you may want different behavior for different mirrors when dismounting a database for maintenance or encountering an error during dejournaling. For one mirror, it may be most important that the database for which dejournaling is paused not fall behind the other databases in the mirror, in which case you will prefer to pause dejournaling for the entire mirror; for another, it may be most important that the databases in the mirror stay as up to date as possible, in which case you will want to pause only the database involved.
When you want to pause dejournaling for one or more mirrors on a reporting async as a one-time operation or on a regular basis, you can select the Manage mirror dejournaling on async member option from the Mirror Management menu of the ^MIRROR routine to pause dejournaling for all databases in any mirror you wish. When you want to restart dejournaling, use the Manage mirror dejournaling on async member option again. (This option is not available on backup members.)
Unlike backup and DR async members, when there is an error during dejournaling of a database on a reporting async member, dejournaling is automatically paused for all databases in that mirror. Depending on your needs and policies, you can either:
Dismount the database that encountered the error, select the Manage mirror dejournaling on async member option from the Mirror Management menu of the ^MIRROR routine to restart dejournaling for all other databases in the mirror, correct the error and mount the database, then resume dejournaling for that database by selecting the Activate or Catchup mirrored database(s) option from the Mirror Management menu of the ^MIRROR routine or catching up the database using the management portal (see Activating and Catching Up Mirrored Databases).
Allow dejournaling to remain paused for the entire mirror while you correct the error and remount the database, then use the Manage mirror dejournaling on async member option to restart dejournaling for the entire mirror (This option automatically activates the database that had the error and catches it up to the same point as the most up to date database in the mirror.)
When you want to perform maintenance on a mirrored database on a reporting async member, you can simply dismount the database, then mount the database again after maintenance and use the Activate or Catchup mirrored database(s) option or the management portal to catch up the database. (If the maintenance involves several such databases, use the Mirror Monitor to perform the operation on all of them at once, as described in Activating and Catching Up Mirrored Databases. This is more efficient and less time-consuming than catching up the databases individually.)
When dejournaling pauses for a mirror on a reporting async member due to an error, the member attempts to restart dejournaling for the mirror the next time its connection to the primary is rebuilt. When you pause dejournaling for a mirror on an async member using the Manage mirror dejournaling on async member option, dejournaling for the mirror does not restart until you use the option again to restart it.
Using a Dejournal Filter on a Reporting Async
On a reporting async only, you can set a user-defined dejournal filter on a given mirror, letting you execute your own code for each journal record to determine which records are applied to the Read-Write databases in that mirror. Once you have defined a filter, you can set it on as many mirrors as you want, and you can set, change and remove filters at any time.
This functionality is intended only for highly specialized cases. Alternatives should be carefully considered. For controlling which globals are replicated to mirror members, global mapping to nonmirrored databases provides a much simpler, lightweight solution. For monitoring updates to application databases, solutions built at the application level are typically more flexible.
A dejournal filter allows a reporting async to skip dejournaling of some of the records in a journal file received from the primary. However, this applies to Read-Write databases only—databases originally added to the mirror on a read-write reporting async, or from which the FailoverDB flag has been cleared since the database was added to the mirror as Read-Only. (See Clearing the FailoverDB Flag on Reporting Async Mirror Members for a detailed explanation of the FailoverDB flag and the mount status of mirrored databases on reporting asyncs.) If the FailoverDB flag is set on a database, which means that the database is mounted as Read-Only, the dejournal filter code still executes, but all records are always dejournaled on that database, regardless of what the filter code returns.
Setting a dejournal filter slows dejournaling for the mirror it is set on; this effect may be significant, depending on the contents of the filter.
To create a dejournal filter, extend the superclass SYS.MirrorDejournal to create a mirror dejournal filter class. The class name should begin with Z or z, so that it is preserved during an InterSystems IRIS upgrade.
To set a dejournal filter on a mirror on a reporting async, navigate to the Edit Async Configurations page (System Administration > Configuration > Mirror Settings > Edit Async), click the Edit Dejournal Filter link next to the desired mirror in the Mirrors this async member belongs to list, enter the name of a mirror dejournal filter class, and click Save. To remove a filter, do the same but clear the entry box before clicking Save. Whenever you add, change, or remove a journal filter on a mirror, dejournaling is automatically restarted for that mirror so the filter can be applied. However, if you modify and recompile a mirror dejournal filter class, you must manually stop and restart dejournaling on all mirrors it is set on using the Manage mirror dejournaling on async member option on the Mirror Management menu of the ^MIRROR routine.
General Mirroring Considerations
This section provides information to consider, recommendations, and best-practice guidelines for mirroring. It includes the following subsections:
The SYS.Mirror class provides methods for programmatically calling the mirror operations available through the management portal and the ^MIRROR routine (see Using the ^MIRROR Routine), as well as many queries. For example, the SYS.Mirror.CreateNewMirrorSet() method can be used to create a mirror and configure the first failover member, while the SYS.Mirror.MemberStatusList() query returns a list of mirror members and the journal latency status of each. See the SYS.Mirror class documentation for descriptions of these methods.
If you use an external script to perform backups, you can use the $SYSTEM.Mirror class methods to verify whether a system is part of a mirror and, if so, what its role is:
$System.Mirror.IsMember() $System.Mirror.IsPrimary() $System.Mirror.IsBackup() $System.Mirror.IsAsyncMember() $System.Mirror.MirrorName()
where $SYSTEM.Mirror.IsMember() returns 1 if this system is a failover member, 2 if this is an async mirror member, or 0 if this is not a mirror member; $SYSTEM.Mirror.IsPrimary() returns 1 if this system is the primary failover member, or 0 if it is not; $SYSTEM.Mirror.IsBackup() returns 1 if this system is the backup failover member, or 0 if it is not; $SYSTEM.Mirror.IsAsyncMember() returns 1 if this system is an async member, or 0 if it is not; $SYSTEM.Mirror.MirrorName() returns the name of the mirror if the instance is configured as a failover mirror member or NULL if it is not.
You can also use $SYSTEM.Mirror.GetMemberType() and $SYSTEM.Mirror.GetMemberStatus() to obtain information about the mirror membership (if any) of the current instance of InterSystems IRIS and its status in that role; see Mirror Member Journal Transfer and Dejournaling Status for more information.
External Backup of Primary Failover Member
When using the Backup.General.ExternalFreeze() method to freeze writes to a database on the primary failover member so an external backup can be performed, as described in the “Backup and Restore” chapter of the Data Integrity Guide, ensure that the external freeze does not suspend updates for longer than the specified ExternalFreezeTimeOut parameter of Backup.General.ExternalFreeze(). If this happens, the mirror may fail over to the backup failover member, thereby terminating the backup operation in progress.
Upgrading InterSystems IRIS on Mirror Members
To review options and considerations for upgrading InterSystems IRIS on a mirror member, see Minimum Downtime Upgrade with Mirroring in the “Upgrading InterSystems IRIS” chapter of the Installation Guide.
Database Considerations for Mirroring
This section provides information to consider when configuring and managing mirrored databases:
InterSystems IRIS Instance Compatibility
The InterSystems IRIS instances in a mirror must be compatible in several ways, as follows:
All InterSystems IRIS instances in a mirror must:
Use the same character width (8-bit or Unicode; see Character Width Setting in the “Preparing to Install” chapter of the Installation Guide).
Use the same locale (see Using the NLS Settings Page of the Management Portal in the “Configuring InterSystems IRIS” chapter of the System Administration Guide).
The one exception to these requirements is that an 8-bit instance using a locale based on the ISO 8859 Latin-1 character set is compatible with a Unicode instance using the corresponding wide character locale. For example, an 8–bit primary instance using the enu8 locale is compatible with a Unicode backup instance using the enuw locale. However, an 8–bit primary instance using the heb8 locale is not compatible with a Unicode backup instance using the hebw locale, as these locales are not based on ISO 8859 Latin- 1.
The failover members must have the same database block sizes enabled (see Large Block Size Considerations in the “Configuring InterSystems IRIS” chapter of the System Administration Guide). Additionally, the sizes enabled on the failover members must be enabled on async members. If the block size of a mirrored database that is added to the primary is not enabled on another member, the database cannot be added to the mirror on that member,
The failover members and any DR async member must be of the same InterSystems IRIS version; they can differ only for the duration of one of the upgrade procedures described in Minimum Downtime Upgrade with Mirroring in the “Upgrading InterSystems IRIS” chapter of the Installation Guide. Once an upgraded member becomes primary, you cannot make use of the other failover member and any DR async members (and in particular cannot allow them to become the primary) until the upgrade is completed.
Mirroring does not require reporting async members to be of the same InterSystems IRIS version as the failover members, although application functionality may require it.
Member Endianness Considerations
When creating a mirrored database or adding an existing database to a mirror, if a backup failover member or async member has a different endianness than the primary failover member, you cannot use the backup and restore procedure described in Add an existing database to the mirror; you must instead use the procedure in that section involving copying the database’s IRIS.DAT file. Additionally, when using that procedure, insert the following step after copying the IRIS.DAT file to all nonprimary members and before mounting the database on those members:
On the backup failover member and each async member, convert the copied IRIS.DAT files as described in the Using cvendian to Convert Between Big-endian and Little-endian Systems section of the “Migration and Conversion Utilities” chapter of Specialized System Tools and Utilities.
Creating a Mirrored Database Using the ^DATABASE Routine
You can create mirrored databases on mirror members using the ^DATABASE routine. (See ^DATABASE in the “Using Character-based Security Management Routines” chapter of the Security Administration Guide for information about the routine.) You must create the new mirrored database on the primary member before creating it on other mirror members. To create a mirrored database:
Run the ^DATABASE routine, and select the 1) Create a database option.
Enter the directory path at the Database directory? prompt.
Enter yes at the Change default database properties? prompt.
Enter 3 (Mirror DB Name:) at the Field number to change? prompt, and enter a mirror name for the mirrored database at the Mirror DB Name? prompt.Note:
If the member on which you are creating the mirrored database is a member of multiple mirrors and you are creating a mirrored database that is in a mirror that is different from the one that is listed by default, Enter (Mirror Set Name:) at the Field number to change? prompt, and choose the correct mirror name from the list. If the member on which you are running the routine is a member of only one mirror, this field cannot be changed.
Modify other fields as necessary for your database, then when you have finished making changes, press Enter at the Field number to change? prompt without specifying any option.
Enter the dataset name of the database at the Dataset name of this database in the configuration: prompt. This is the name that is displayed in the management portal.
Enter responses to the remaining prompts until the mirrored database is created.
When you create the mirrored databases on the backup and async members, they automatically catch up with the database you created on the primary member.
You cannot add an existing nonmirrored database to a mirror using the ^DATABASE routine; see Adding Databases to Mirror for the required procedure.
Recreating an Existing Mirrored Database Using the ^DATABASE Routine
The 10) Recreate a database option of ^DATABASE routine lets you clear the data in an existing database without changing the database’s name or size. (See ^DATABASE in the “Using Character-based Security Management Routines” chapter of the Security Administration Guide for information about the routine.) You can use this option with a mirrored database, but you must use it on every mirror member on which the database appears, and in the same order in which you use the Create a database option to create a new mirrored database—on the primary first, then the backup, then any asyncs on which the database is part of the mirror.
If you use the 10) Recreate a database option to recreate a database on the primary, you must repeat the operation on the backup and any DR asyncs in the mirror; if you do not, the database may become obsolete in the event of failover or disaster recovery. You are strongly encouraged to repeat the recreate operation on reporting asyncs as well.
Mounting/Dismounting Mirrored Databases
Mirrored databases can be mounted/dismounted on either failover member. If dismounted on the backup failover member, however, the database remains in a “stale” state until it is remounted, after which mirroring attempts to catch up the database automatically. If the required journal files are available on the primary failover member, the automatic update should succeed, but if any of the required journal files on the primary member have been purged, you must restore the database from a recent backup on the primary member.
Copying Mirrored Databases to Nonmirrored Systems
You can copy a mirrored database to a nonmirrored system and mount it read-write on that system by doing the following:
Back up the mirrored database on the primary or backup failover member and restore it on the nonmirrored system using the procedure described in Add an Existing Database to the Mirror (omit the step of manually activating and catching up the database following external backup restore or cold backup restore). Once restored, the database is still marked as mirrored and is therefore read-only.
On the nonmirrored system, use the ^MIRROR routine (see Using the ^MIRROR Routine) to remove the database from the mirror by selecting Remove one or more mirrored databases and following the instructions. Following this procedure the database is mounted read-write.
Production Considerations for Mirroring
This section discusses additional considerations that apply to InterSystems IRIS productions, including:
Creating an Interoperability-Enabled Namespace with Mirrored Data
Because creating an interoperability-enabled namespace requires database writes that enable the use of productions in the new namespace, an interoperability-enabled namespace with mappings from one or more mirrored databases must be created on the current primary mirror member and cannot be created on the backup, where mirrored databases are read-only.
How InterSystems IRIS Handles Interoperability-Enabled Namespaces with Mirrored Data
InterSystems IRIS examines the mappings in an interoperability-enabled namespace and determines whether that namespace contains any mappings from a mirrored database., with the following results:
When you start or upgrade a mirror member containing an interoperability-enabled namespace, productions are started on the primary only.
When you upgrade InterSystems IRIS, certain tasks require write access to the database; those tasks are performed only on the primary mirror member.
If a failover occurs and a member becomes the primary mirror member, any tasks that were skipped when it was upgraded (because it was not primary at the time) are performed before productions are started.
Recommended Mirroring Configuration for InterSystems IRIS Productions
Mirroring is intended to be a high availability solution and there should thus be minimal extraneous activity on either of the mirror instances. That is, you should mirror all databases on any mirrored instances.
Customers sometimes choose to have “less critical” productions running on either node without having that data mirrored. Such a configuration, however, creates operational complexity that may prove difficult to maintain. Consequently, InterSystems strongly recommends that you avoid such configurations and that you instead mirror all the databases.
How Production Autostart Works in a Mirrored Environment
When a mirror system starts up (at which point no member has yet become the primary failover member):
InterSystems IRIS does not start any production that accesses mirrored data even if the production is specified in ^Ens.AutoStart. If the member becomes the primary instance, these productions will be started at that time.
InterSystems IRIS determines if there are any namespaces on the instance that do not access mirrored data. As described previously, InterSystems recommends that only mirrored productions be installed on a mirror member. If you have, however, installed any production with nonmirrored databases, InterSystems IRIS starts the production specified in ^Ens.AutoStart. (This logic ensures that if you have installed a nonmirrored namespace on a mirror member, it is started on InterSystems IRIS startup.)
Later, when the member becomes the primary failover member, InterSystems IRIS finds the namespaces that do reference mirrored data so that it can start the productions in these namespaces. If you follow InterSystems recommendations, no production accessing mirrored data should be running before an instance becomes the primary mirror member. InterSystems IRIS first checks to see if a production is already running before starting it, specifically:
InterSystems IRIS determines whether the production is already running by counting the jobs that are running as the _Ensemble user in the namespace. If there are more than two such jobs, indicating that the production is already running, InterSystems IRIS logs a warning to the messages log and does not attempt to start the production.
If, as expected, the production is not running, InterSystems IRIS automatically starts the production specified in ^Ens.AutoStart.
For complete information about starting and stopping productions, see the “Starting and Stopping Productions” chapter of Managing Productions.