The HSSYS Mirroring Utility
The HSSYS database stores configuration data, metadata, and system-level information that is vital for the overall functionality of InterSystems IRIS for Health, specifically including a list of tasks to be executed on the backup member of a mirrored system.
You can automatically mirror HSSYS using the HSSYS mirroring terminal utility. The assumption is that mirroring is configured and currently in a good state, with both machines running. The utility validates that HSSYS can be mirrored, confirming that the data is consistent on each of the mirror members. There are options to validate only or to validate and then complete the mirroring of HSSYS.
Standard Validation and Mirroring
In most cases, you can use the HSSYS Mirroring Utility online. The utility can be used on a running system as long as no new namespaces or databases are being created and no namespace activations are taking place in the installer wizard during the execution.
The standard validation and mirroring procedure is only supported on systems version 2021.1 and later. If you are upgrading from an older version, you should instead follow the Offline Validation and Mirroring procedure.
For offline instructions see Offline Validation and Mirroring.
Follow these steps to validate and/or automatically mirror HSSYS using the HSSYS mirroring terminal utility online:
-
Enable the %Service_Bindings service. In non-Windows deployments, enable %Service_Terminal as well.
-
The standard usage of the HSSYS Mirroring Utility uses a JDBC connection between IRIS instances via the SQL Gateway. Ensure that a current Java Runtime Engine (JRE) is installed.
-
Download and install the most recent HSSYS Mirroring Utility code from the WRC.
-
On the backup member, switch to namespace HSLIB.
NAMESPACE> ZN "HSLIB"
-
Execute Run() or RunValidate():
-
RunValidate does not make any system changes; it only validates, reporting whether the HSSYS data is consistent across the primary and backup members.
HSLIB> Set return = ##class(HS.HC.SYS.Mirror).RunValidate("<ip_or_dn>",<server_port>,"HSLIB","<adm_user>","<adm_pwd>",<verbose>) -
Run performs validation and then, if validation succeeds, performs the actual mirror setup on HSSYS.
HSLIB> Set return = ##class(HS.HC.SYS.Mirror).Run("<ip_or_dn>",<server_port>,"HSLIB","<adm_user>","<adm_pwd>",<verbose>,<StopOnError>)The Backup / DR Member contacts the Primary Member to gather system information; once the information is validated and there are no inconsistencies, the Primary Member dismounts HSSYS, makes a copy as COPY_IRIS.DAT in the HSSYS database directory, and remounts HSSYS.
The COPY_IRIS.DAT is pulled down from the Primary to the Backup/DR Member. HSSYS is dismounted and renamed to OLD_IRIS.DAT and replaced with the contents of COPY_IRIS.DAT from the Primary Member.
-
The following argument descriptions apply to both methods:
ip_or_dnThe IP address or fully qualified domain name of the current primary member.
server_portThe Super Server port number of the primary member.
adm_userThe username of a user with the %ALL role.
adm_pwdThe privileged user’s password.
verboseTo view error messages in the terminal output, set this flag equal to 1. Otherwise, set it to 0. Regardless of the value of verbose, it is always possible to view results by interrogating the return variable.
StopOnErrorWhen set to 1 (the default), this flag prevents the automatic mirroring of HSSYS when blocking errors are present. When this flag is set to 0, the automatic mirroring of HSSYS will proceed as long as there are no blocking errors. In the terminal output, blocking error messages have a *** Blocking *** prefix. In the response JSON, blocking errors have the attribute "blocking":1. If there are any blocking errors, you must correct those before mirroring can take place.
When you are comfortable that all remaining errors fall into non-blocking categories, you can set the StopOnError flag to 0 to ignore them and proceed with mirroring. After mirroring, you can use the installer wizard to re-activate any namespaces that are in the report with non-blocking errors. This may correct some or all of the outstanding errors.
For information about blocking versus non-blocking errors and examples of each, see Mirroring Error Types.
-
-
With verbose set to 1, errors and messages will be returned to the terminal output. Both the Run() and RunValidate() methods return a JSON object that can be evaluated:
HSLIB> ZW return
See Reading the Output of the HSSYS Mirroring Utility for information about how to interpret the return variable.
Note:If validation fails, mirroring setup will also fail. The return value will include helpful messages indicating how to resolve the issues causing setup to fail. Correct the issues and repeat these steps. In verbose mode, error messages will be displayed in the terminal output as well.
Offline Validation and Mirroring
If you are upgrading from a version prior to 2021.1, or in situations where it is impossible or undesirable to allow the members to communicate directly with one another, you can perform the validation and mirroring process off line. Follow these instructions if, for example, you do not wish to send credentials over the network, or if the backup or DR member is unable to communicate with the primary member.
-
If your deployment is non-Windows, enable the %Service_Terminal service.
-
Download and install the most recent HSSYS Mirroring Utility code from the WRC.
-
On the primary member, in HSLIB, gather data from the primary system into a stream, then output the stream to a file:
HSLIB> Set config = ##class(HS.HC.SYS.Mirror).Configuration() HSLIB> Write config.%Save() HSLIB> Write config.Filename
The first Write command should output 1 to the terminal display. The second Write command should output the full path and filename.
-
Move the file to the DR or Backup Member.
-
On the DR or Backup Member, get a stream instance of the file (the Primary Member configuration):
HSLIB> Set primaryFile = ##class(%Stream.FileCharacter).%New() HSLIB> Do primaryFile.LinkToFile(<fullPath>)
fullPathThe full path and filename to the location on the current system where you saved the file containing data from the primary member.
-
Gather the data for the Backup or DR Member. On the DR or Backup Member:
HSLIB> Set config = ##class(HS.HC.SYS.Mirror).Configuration()
-
Execute the comparison between the Primary member and the Backup or DR member:
HSLIB> Set return = {"verbose":true,"thisSystem":({}.%FromJSON(config)),"remoteSystem":({}.%FromJSON(primaryFile))} HSLIB> do ##class(HS.HC.SYS.Mirror).Comparison(return) -
Display the error message to the terminal output if desired:
HSLIB> do ##class(HS.HC.SYS.Mirror).FinalOutput(return)
You can also interrogate the return variable directly if you wish. See Reading the Output of the HSSYS Mirroring Utility for information about how data is stored in that variable.
-
Resolve blocking errors as needed and repeat the comparison and display steps until all blocking errors have been resolved.
-
When all blocking errors have been resolved, add HSSYS to the mirror set on the primary member.
-
Copy HSSYS to the current member.
-
Restore HSSYS from the Primary Member to the Backup/DR Member. See Restore the Mirrored Database on the Backup and Async Members for instructions.
-
After mirroring, you can use the installer wizard to re-activate any namespaces that are in the report with non-blocking errors. This may correct some or all of the outstanding errors.
For information about blocking versus non-blocking errors and examples of each, see Mirroring Error Types.
Reading the Output of the HSSYS Mirroring Utility
Both the Run() and RunValidate() methods return output as a JSON object with the following main sections:
-
thisSystem contains system information and errors found in the primary mirror member.
-
remoteSystem contains system information and errors found in the backup mirror member.
-
errors[] lists all errors found in either system, as well as errors that are visible only when you compare both systems. Blocking errors have the attribute "blocking":1.
-
messages[] contains messages such as status updates and success notifications.
System information contained in thisSystem and remoteSystem includes blocks for namespaces, roles, resources, applications, tasks, and users.
In verbose mode (that is, when the verbose flag is set to 1), you can also view errors in the terminal output.
Mirroring Error Types
This section is a list of some of the error types the HSSYS Mirroring Utility is likely to find.
You can set the ignore flag to ignore non-fatal errors, which will allow the utility to proceed with mirroring unless a fatal error is present.
Non-fatal error types include the following:
-
A role is missing from the specified mirror member.
ERROR <HSSYSToolsLoc>Missing: roles 'TEST' does not exist on '<server>'.
-
A resource is missing from the specified mirror member.
ERROR <HSSYSToolsLoc>Missing: resources 'TEST' does not exist on '<server>'.
-
A user is missing from the specified mirror member.
ERROR <HSSYSToolsLoc>Missing: users 'Tester' does not exist on '<server>'.
-
A mapping is missing from the specified mirror member.
ERROR <HSSYSToolsLoc>RoutineMapMissing: Routine mapping 'HS.*' of type '' to 'HSLIB' does not exist in TESTING on '<server>'.
-
There is a discrepancy between the details of the two mirror members. The <details of Server> values will show the particular discrepancy. For example, if serverA has "authenticationMethods":"Password,Unauthenticated" and serverB has "authenticationMethods":"Password,Delegated", you can see that discrepancy by comparing <details of serverA> to <details of serverB>.
ERROR <HSSYSToolsLoc>Difference: applications '<test>' is different between '<serverA>' ({<details of serverA>}) and '<serverB>' with ({<details of serverB>}).
Blocking errors cannot be ignored, even when you set the ignore flag to 1. In the terminal output, these errors have the prefix *** Blocking *** . In the response JSON, they have the attribute "blocking":1.
The following list contains some of the blocking errors you might run into.
-
A username and password are required for the JDBC connection.
*** Blocking *** ERROR <HSSYSToolsLoc>UsernamePasswordReqd: Username and Password are required for JDBC Connection.
-
Code must be executed on the backup mirror member.
*** Blocking *** ERROR <HSSYSToolsLoc>ExecuteOnBackup: Please execute code on the backup member, in HSLIB.
-
FHIR SQL data is permitted only on the primary mirror member.
*** Blocking *** ERROR <HSSYSToolsLoc>FHIRSQLDataExists: FHIRSQL Data exists on '<backupServer>'.