HSSYS Mirroring

Standard InterSystems IRIS® mirroring replicates mirrored database data, but InterSystems IRIS® for Health and Health Connect™ deployments have additional requirements to maintain consistency across mirror members. In particular, the HSSYS database contains healthcare-specific system state, including interoperability namespace information, the Mirror Monitor Agent, and installation/upgrade step tracking, that must be mirrored as part of a correctly configured mirrored environment.

This section describes HSSYS mirroring requirements for mirrored InterSystems IRIS for Health and Health Connect deployments, including configuration requirements, verification steps, and remediation procedures for environments where HSSYS was not mirrored or was introduced after the mirror was already in use. When HSSYS is not mirrored, or is mirrored only after a mirrored environment is already in use, mirror members can become misaligned and issues may surface during upgrades, configuration, or failover.

This guidance applies to mirrored deployments with interoperability-enabled productions. Mirroring is optional; when mirroring is used, the healthcare mirroring requirements are mandatory.

Frequently Asked Questions

Does mirroring IRIS for Health or Health Connect require any special steps?

Yes. Mirroring InterSystems IRIS for Health or Health Connect requires additional configuration beyond standard InterSystems IRIS mirroring. See Healthcare Mirroring for the complete list of required healthcare mirroring configurations.

Is mirroring required?

No. Mirroring is optional. However, when a mirrored deployment is used, the healthcare mirroring requirements, including mirroring HSSYS, are mandatory to maintain consistency across mirror members.

Does this apply in HL7-only environments?

Yes. If there are interoperability-enabled productions, HSSYS mirroring and the associated healthcare mirroring configuration are required even when the system only processes HL7 messages and does not use FHIR or other advanced healthcare features. In the rare case that IRIS for Health or Health Connect is deployed without interoperability-enabled productions, this advisory does not apply.

What is the HSSYS database used for?

The HSSYS database contains system-level data required by InterSystems IRIS for Health and Health Connect. It includes:

Interoperability and Foundation namespace details.
The Health Mirror Monitor Agent and related healthcare mirroring state.
Upgrade and installation step tracking, including logs of which configuration/upgrade steps have been run.
Information needed to keep mirrored environments consistent across mirror members.
Configuration and operational data, such as users, roles, and connectivity information, needed by the system.

Because this information is required for correct operation of mirrored healthcare deployments, HSSYS must be mirrored when the environment is mirrored.

What is the Health Mirror Monitor Agent?

The Health Mirror Monitor Agent synchronizes selected configuration changes that are stored in IRISSYS (which is not mirrored) from the primary mirror member to non-primary members. This synchronization runs on backup and DR async mirror members and includes configuration items such as roles, resources, and mappings.

Where can Mirror Monitor Agent information be viewed?

Beginning with InterSystems IRIS for Health and Health Connect version 2023.2, mirrored instances provide a user interface for the Mirror Monitor Agent in the top banner of the Management Portal, located next to the Installer Wizard from the Health menu. This interface shows information about the configuration steps that have been run.

What configuration is required for HSSYS mirroring?

HSSYS must be included in the mirror set, and the following supporting configuration is required for healthcare mirroring to function correctly:

Enable the HS_Services user and configure its interoperability credentials in the HSSYS namespace.
Schedule and verify the Mirror Monitor Agent task that synchronizes configuration data stored in IRISSYS.
Configure a stable network endpoint for downstream systems so clients consistently connect to the active primary member.
- This can be a virtual IP (VIP) or a proxy hostname/address that always routes to the current primary member.

In addition, healthcare mirroring has environmental constraints that may affect successful operation, including a web server requirement and a dependency on performing configuration through the Installer Wizard.

How can correct HSSYS mirroring be verified?

On each mirror member, open the Management Portal and navigate to System Operation > Mirror Monitor.

Confirm:

The HSSYS database appears in the mirrored database list.
The mirror status for HSSYS shows Caught Up.

How is Mirror Monitor Agent operation verified?

Beginning with InterSystems IRIS for Health and Health Connect 2023.2, mirrored instances provide a Mirror Monitor Agent user interface in the top banner of the Management Portal, located next to the Installer Wizard from the Health menu. Use this interface to view information about the configuration steps that have been run.

What validations are recommended before upgrades or failover?

Before performing an upgrade or planned failover, verify:

HSSYS is Caught Up on all mirror members.
The Mirror Monitor Agent task is scheduled and operating correctly.
Downstream connectivity uses a consistent endpoint (VIP or proxy hostname) that routes to the active primary.

What logs are used to diagnose installation or upgrade anomalies?

Review the following logs (and when multiple files exist, review the most recent):

messages.log
ensinstall.log (in the mgr directory)
HS.Util.Installer logs (in the mgr directory)

Scenario-Based Remediation

This section provides remediation procedures for mirrored IRIS for Health and Health Connect environments that are misaligned due to missing or late HSSYS mirroring. Before selecting a remediation path, review the following diagnostic inputs for evidence of a failed or partially completed installation or upgrade.

messages.log
ensinstall.log (in the mgr directory)
HS.Util.Installer logs (in the mgr directory)

Next, select the remediation scenario based on the state of HSSYS across mirror members:

Determine whether the ^HS.Upgrade global in HSSYS is consistent among all mirror members. Navigate to the Globals page in the Management Portal (System Explorer > Globals > View Global Data). Look in the HSSYS namespace/database and select View in the HS.Upgrade row. Repeat across mirror members and compare ^HS.Upgrade global content.

Before performing the cleanup, run the following queries in the HSSYS namespace to identify actions that failed or were never executed.

To identify actions that failed, run the following queries:

SELECT ID, ConfigStep, Error, ErrorCount, HasRun, InstanceName, IsError, IsIgnored, Timestamp
FROM HS_HC_SystemConfig_ConfigStep.Status
WHERE IsError = 1

SELECT ID, ConfigStep, Error, ErrorCount, HasRun, InstanceName, IsError, IsIgnored, Timestamp
FROM HS_HC_SystemConfig_ConfigStep.LocalStatus
WHERE IsError = 1

To identify actions that were never executed, run the following queries:

SELECT ID, AssociatedDatabase, Classname, MethodArguments, MethodName
FROM HS_HC_SystemConfig.ConfigStep
WHERE ID NOT IN ( SELECT ConfigStep AS ID FROM HS_HC_SystemConfig_ConfigStep.Status )

SELECT ID, AssociatedDatabase, Classname, MethodArguments, MethodName
FROM HS_HC_SystemConfig.ConfigStep
WHERE ID NOT IN ( SELECT ConfigStep AS ID FROM HS_HC_SystemConfig_ConfigStep.LocalStatus )

Remediation Scenario A: ^HS.Upgrade Consistent Across Mirror Members

This scenario applies when the ^HS.Upgrade global in HSSYS is consistent among all mirror members. Databases used by foundation namespaces must be mirrored on all mirror members.

System Configuration Consistency Requirements

Ensure the system configuration items are consistent among mirror members, including:

users
roles
resources
global mappings
tasks
web applications

Removal of Legacy SystemConfig Entries in HSSYS

As a recommended practice, delete all SystemConfig entries from the ^%SYS("HealthShare","SystemConfig*") globals in HSSYS. To do so safely, run the following commands:

Set status = ##class(HS.HC.SystemConfig.ConfigStep.Status).%KillExtent()
Set status = ##class(HS.HC.SystemConfig.ConfigStep.LocalStatus).%KillExtent()
Set status = ##class(HS.HC.SystemConfig.ConfigStep).%KillExtent()

This cleanup removes updates and actions stored from previous upgrades and prevents outdated or unexecuted steps from interfering with subsequent mirroring and upgrade operations.

HSSYS Mirroring Requirements and Special Configuration

Mirror HSSYS between the mirror members. This includes the special requirements for HSSYS:

Enable the HS_Services user and configure its interoperability credentials in the HSSYS namespace.
Schedule and verify the Mirror Monitor Agent task that synchronizes configuration data stored in IRISSYS.
Configure a stable network endpoint for downstream systems so clients consistently connect to the active primary member.
- This can be a virtual IP (VIP) or a proxy hostname/address that always routes to the current primary member.

Review of Prior Installation or Upgrade Failures

Check for previous interoperability installation failures in the following locations:

ensinstall.log files in the mgr directory (check the most recent file when multiple exist)
HS.Util.Installer logs in the mgr directory (check the most recent file when multiple exist)

If there have been partial or failed upgrades, once HSSYS is mirrored, run a Repair install on any mirror members that had installation issues.

Remediation Scenario B: ^HS.Upgrade Not Consistent Across Mirror Members

This scenario applies when the ^HS.Upgrade global in HSSYS is not consistent among mirror members. Databases used by foundation namespaces must be mirrored on all mirror members.

System Configuration Consistency Requirements

Ensure the system configuration items are consistent among mirror members, including:

users
roles
resources
global mappings
tasks
web applications

Removal of Legacy SystemConfig Entries in HSSYS

As a recommended practice, delete all SystemConfig entries from the ^%SYS("HealthShare","SystemConfig*") globals in HSSYS. To do so safely, run the following commands:

Set status = ##class(HS.HC.SystemConfig.ConfigStep.Status).%KillExtent()
Set status = ##class(HS.HC.SystemConfig.ConfigStep.LocalStatus).%KillExtent()
Set status = ##class(HS.HC.SystemConfig.ConfigStep).%KillExtent()

This cleanup removes updates and actions stored from previous upgrades and prevents outdated or unexecuted steps from interfering with subsequent mirroring and upgrade operations.

Reset of Upgrade Step Tracking State

Delete the ^HS.Upgrade global in the HSSYS database. This will allow the needed steps to be run again at install/upgrade time.

HSSYS Mirroring Requirements and Special Configuration

Mirror HSSYS between the mirror members. This includes the special requirements for HSSYS:

Enable the HS_Services user and configure its interoperability credentials in the HSSYS namespace.
Schedule and verify the Mirror Monitor Agent task that synchronizes configuration data stored in IRISSYS.
Configure a stable network endpoint for downstream systems so clients consistently connect to the active primary member.
- This can be a virtual IP (VIP) or a proxy hostname/address that always routes to the current primary member.

Controlled Repair Install and Failover Procedure

Now that HSSYS is mirrored, run a repair install to ensure the upgrade steps are run. Depending on the mirror configuration, this can be done on either a backup failover or DR member.

Set no failover on the primary member.
Shut down a non-primary node (example: backup failover mirror member).
Run a repair install on the non-primary member.
Check for a successful interoperability install using the ensinstall.log, HS.Util install logs, and messages.log.
Bring up the non-primary mirror member.
Clear no failover on the primary member.
Fail over from the current primary to the repaired member.

When the repaired member becomes primary, it will finish the upgrade steps it had deferred as a non-primary mirror member. This will recreate the ^HS.Upgrade global in HSSYS. Since HSSYS is mirrored, the needed data is replicated on all members of the mirror.

Review of Prior Installation or Upgrade Failures

Check for previous interoperability installation failures in the following locations:

ensinstall.log files in the mgr directory (check the most recent file when multiple exist)
HS.Util.Installer logs in the mgr directory (check the most recent file when multiple exist)

If there have been partial or failed upgrades, once HSSYS is mirrored, run a Repair install on any mirror members that had installation issues.

Post-Remediation Validation

After completing remediation, verify the following on each mirror member:

In the Management Portal, navigate to the Mirror Monitor Page (System Operation > Mirror Monitor).
Confirm that the HSSYS database appears in the mirrored database list and verify that its mirror status shows Caught Up.

To verify the Mirror Monitor Launch Task on each member:

In the Management Portal, navigate to the Task Schedule (System Operation > Task Manager > Task Schedule).
Verify that the Task Schedule lists the Mirror Monitor Launch Task configured to run every five minutes in the HSSYS namespace.
Check that the next scheduled run time is within the expected interval.

Finally, review the most recent versions of the following logs to confirm that interoperability installation and upgrade steps completed successfully and that no current errors are present:

messages.log
ensinstall.log (in the mgr directory)
HS.Util.Installer logs (in the mgr directory)