Skip to main content

%SYSTEM.Cluster

class %SYSTEM.Cluster extends %SYSTEM.Help

This class provides an API for node-level configuration of sharded clusters.

You can use it via the special $SYSTEM object, for example:

set status = $SYSTEM.Cluster.Initialize()

API Usage

Call Initialize() to set up an InterSystems IRIS instance as the first node of a new nonmirrored sharded cluster (that is, a sharded cluster in which all data nodes are nonmirrored).

Call InitializeMirrored() to set up an InterSystems IRIS instance as the first node of a new mirrored sharded cluster (that is, a sharded cluster in which all data nodes are mirrored).

Call AttachAsDataNode() to add an instance to a cluster as a nonmirrored data node.

Call AttachAsMirroredNode() to add an instance to a cluster as a mirrored data node.

Call AttachAsComputeNode() to add an instance to a cluster as a compute node.

Call ListNodes() to print a list of a cluster's nodes.

Call GetMetadata() to retrieve an overview of a cluster's metadata.

Call ClusterNamespace() to get the name of the cluster namespace for the current instance.

Terminology

Sharded Cluster
An interconnected set of nodes which provides horizonal scalability of users and data.
Node
An InterSystems IRIS instance that participates in a sharded cluster. Nodes are of two types, "data" and "compute", but all nodes can be used interchangeably as application servers, providing transparent access to all data in the cluster. Data nodes can be mirrored or nonmirrored. Compute nodes are always non-mirrored.
Data Node
A node that stores a partition (shard) of sharded data locally.
Compute Node
A node that does not store sharded data locally, but provides compute access to the sharded data on a corresponding data node, via global mapping.
Cluster URL
The address of a node, specified in the format "IRIS://host:port/cluster-namespace".
Cluster Namespace
A namespace that exists, with the same name, on every node of a sharded cluster, providing transparent access to all sharded and nonsharded data, and all code, on the cluster.
Master Namespace
A namespace that exists only on node 1 of a sharded cluster. Metadata and nonsharded data stored in its globals databases, and code stored in its routines database, are accessible in the cluster namespace on every node of the cluster, via database mapping.
Shard Database
A database on each data node that stores a partition (shard) of sharded data. A shard database is not the default database of any namespace, but globals stored in it are visible within the cluster namespace on the same data node, or any associated compute node, via global mapping.

Architecture of a Sharded Cluster

Sharded clusters configured using methods of this class have the following architecture:

Prequisites for Using This API

Method Inventory

Methods

classmethod AttachAsComputeNode(pClusterURL As %String, pHostIPAddress As %String = "") as %Status
Attaches the current InterSystems IRIS instance to a specified cluster as a compute node.

This method automatically and transparently performs all steps needed to enable this instance as a compute node of the specified cluster (subject to these prerequisites), including:

  • Enables the ECP and sharding services, and sets these services' lists of allowed connections to match the lists on the node specified by pClusterURL.
  • Creates the cluster namespace if it doesn't already exist, configuring it to match the settings on the node specified by pClusterURL.
  • Associates this compute node with a data node that previously had the minumum number of associated compute nodes, so as to automatically balance compute nodes across data nodes.
  • Creates all necessary mappings from the cluster namespace to the globals and routines databases of the master namespace on node 1, and to the shard database on the the associated data node.
  • Propagates any user-defined mappings from the master namespace on node 1 to the cluster namespace on the current instance.
  • Sets all SQL configuration options to match their settings on the node specified by pClusterURL.

This method returns an error if the current InterSystems IRIS instance is already a node of a sharded cluster.

Parameters:

pClusterURL
Identifies the cluster to which to attach, by specifying the address of any existing node of the cluster, in the format "IRIS://host:port/cluster-namespace". Namespace may optionally be omitted, since there can be at most one cluster namespace in an InterSystems IRIS instance.
pHostIPAddress
IP address other nodes of this cluster should use to connect to this node. Specify a value for for this parameter if the default hostname known to InterSystems IRIS does not resolve to an appropriate address, or no hostname is available. Default: Default hostname will be used for inter-node connections.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • A cluster namespace, with same name as on all other nodes of this cluster, is created if a namespace of that name doesn't already exist. Note that if it does already exist, its globals database is "orphaned", as the global database of the cluster namespace is mapped to the globals database of the master namespace on node 1, and a compute node does not need to reuse this database as a shard database.
  • If the cluster namespace doesn't exist, it is created with interoperability enabled or not according to whether it is enabled for the cluster namespace on the node specified by pClusterURL.
  • All SQL configuration settings are set to match those on the node specified by pClusterURL.
  • Compute nodes are assigned a numeric id, in sequence starting from 1001, in the order they are attached to the cluster. Nodes are listed by id by ListNodes() and GetMetadata().
classmethod AttachAsDataNode(pClusterURL As %String, pHostIPAddress As %String = "") as %Status
Attaches the current InterSystems IRIS instance to a specified cluster as a nonmirrored data node.

This method automatically and transparently performs all steps needed to enable this instance as a data node of the specified cluster (subject to these prerequisites), including:

  • Enables the ECP and sharding services, and sets these services' lists of allowed connections to match the lists on the node specified by pClusterURL.
  • Creates the cluster namespace and shard database if they don't already exist, configuring them to match the settings on the node specified by pClusterURL.
  • Creates all necessary mappings from the cluster namespace to the globals and routines databases of the master namespace on node 1, and to the shard database on the current instance.
  • Propagates any user-defined mappings from the master namespace on node 1 to the cluster namespace on the current instance.
  • Sets all SQL configuration options to match their settings on the node specified by pClusterURL.

This method returns an error in any of the following cases:

  • The current InterSystems IRIS instance is already a node of a sharded cluster.
  • The current instance is a mirror member.
  • The specified cluster namespace already exists, and its globals database is mirrored.
  • The node specified by pClusterURL is mirrored.

Parameters:

pClusterURL
Identifies the cluster to which to attach, by specifying the address of any existing node of the cluster, in the format "IRIS://host:port/cluster-namespace". Namespace may optionally be omitted, since there can be at most one cluster namespace in an InterSystems IRIS instance.
pHostIPAddress
IP address other nodes of this cluster should use to connect to this node. Specify a value for for this parameter if the default hostname known to InterSystems IRIS does not resolve to an appropriate address, or no hostname is available. Default: Default hostname will be used for inter-node connections.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • A cluster namespace, with same name as on all other nodes of this cluster, is created if a namespace of that name doesn't already exist. If it does already exist, its globals database becomes the shard database for this node.
  • If the node specified by pClusterURL is a data node, it is used as the template for configuration settings on this node. If the specified node is a compute node, the corresponding data node is used as the template.
  • If the cluster namespace doesn't exist, it is created with interoperability enabled or not according to whether it is enabled for the cluster namespace on the template node, and with its globals database settings, including directory path, matching those of the shard database on the template node.
  • All SQL configuration settings are set to match those on the template node.
  • Data nodes are assigned a numeric id, in sequence starting from 1, in the order they are attached to the cluster. A data node's id is the same as its shard number. Nodes are listed by id by ListNodes() and GetMetadata().
classmethod AttachAsMirroredNode(pClusterURL As %String, pMemberRole As %String, pHostIPAddress As %String = "", pArbiterIP As %String = "", pArbiterPort As %Integer = 0, pSSLDirectory As %String = "", pMirrorName As %String = "", pMemberName As %String = "") as %Status
Attaches the current InterSystems IRIS instance to a specified cluster as a mirrored data node.

This method automatically and transparently performs all steps needed to enable this instance as a data node of the specified cluster (subject to these prerequisites), including:

  • Enables the ECP and sharding services, and sets the sharding services' lists of allowed connections to match the lists on the node specified by pClusterURL.
  • If the current instance is not already a member of a mirror, performs mirror configuration according to role specified by pMemberRole:
    • "primary" Creates a new mirror set, and configures the current instance as its first failover member.
    • "backup" Configures the current instance as the backup failover member of the mirror set whose primary failover member is specified by pClusterURL.
    • "drasync" Configures the current instance as a DR async member of the mirror set whose primary failover member is specified by pClusterURL.
  • Creates the cluster namespace and shard database if they don't already exist, configuring them to match the settings on the node specified by pClusterURL. If pMemberRole is "backup" and pClusterURL specifies node 1 of this cluster, also creates the master namespace and master database if they don't already exist.
  • Creates all necessary mappings from the cluster namespace to the globals and routines databases of the master namespace on node 1, and to the shard database on the current instance. (Mappings are not created when pMemberRole is "drasync".)
  • Propagates any user-defined mappings from the master namespace on node 1 to the cluster namespace on the current instance. (Mappings are not propagated when pMemberRole is "drasync".)
  • Sets all SQL configuration options to match their settings on the node specified by pClusterURL.

This method returns an error in any of the following cases:

  • The current InterSystems IRIS instance is already a node of a sharded cluster.
  • The specified cluster namespace already exists, and its globals database is not mirrored.
  • pMemberRole is "backup", pClusterURL specifies node 1 of this cluster, and a namespace with the same name as the master namespace already exists, and is not mirrored.
  • The node specified by pClusterURL is not mirrored.
  • pMemberRole is "backup", and the node specified by pClusterURL belongs to a mirror set which already has two failover members.
  • The current instance is a member of an existing mirror set, in a role different from the one specified by pMemberRole.
  • The current instance is a backup or DR async member of a different mirror set than the node specified by pClusterURL.
  • pMemberRole is "backup", and one of the backup databases created by this method could be caught up within two minutes. This is rare, and is most likely due to a delay in this instance becoming an active mirror backup member (check messages.log for possible causes). In this event, the error message "Cannot catch up database dbname, manual catchup required" is returned. The method CatchupDB() can be used to perform manual catchup.

Parameters:

pClusterURL
Identifies the cluster to which to attach, by specifying the address of am existing node of the cluster, in the format "IRIS://host:port/cluster-namespace". Namespace may optionally be omitted, since there can be at most one cluster namespace in an InterSystems IRIS instance. If pMemberRole is "primary", pClusterURL may specify any existing node. If pMemberRole is "backup", pClusterURL must specify the node which is the primary failover member of the mirror set for which the current instance will be attached as the backup failover member.
pMemberRole This node's role as a mirror member:
  • "primary" Attach as primary failover member.
  • "backup" Attach as backup failover member of mirror whose primary failover member is specified by pClusterURL.
  • "drasync" Attach as DR async member of mirror whose primary failover member is specified by pClusterURL.
pHostIPAddress
IP address other nodes of this cluster should use to connect to this node. Specify a value for for this parameter if the default hostname known to InterSystems IRIS does not resolve to an appropriate address, or no hostname is available. Default: Default hostname will be used for inter-node connections.
pArbiterIP
Arbiter's hostname or IP address. Specify a non-null value to configure this mirror to use an arbiter, overriding the value inherited from the node specified by pClusterURL. Ignored when attaching a backup failover member.
pArbiterPort
Arbiter's port. Specify a non-zero value to configure this mirror to use an arbiter, overriding the value inherited from the node specified by pClusterURL. Ignored when attaching a backup failover member.
pSSLDirectory
SSL/TLS keys directory. Specify a non-null value to configure this mirror to use SSL/TLS, overriding the value inherited from the node specified by pClusterURL. Directory is expected to contain three files named "CAFile.pem", "CertificateFile.pem, and "PrivateKeyFile.pem".
pMirrorName
Name of mirror set to create. Default: IRISMIRRORN, where N is a mirror naming sequence number unique within this cluster (usually the same as the node id).
pMemberName
Name by which this instance is known to the mirror. Default: The mirror name, with the letter 'A' appended for the primary failover member, or 'B' for the backup failover member.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • Mirrored data nodes are always paired, and both members of a pair are assigned the same node id. Each instance of the pair is attached by a separate call to AttachAsMirroredNode, specifying pMemberRole as "primary" for the first node attached, and as "backup" for the second node attached. These specify the initial roles, which are swapped when the mirror fails over. Use ListNodes() to determine the current member roles of all mirrored nodes.
  • If the current instance is already a mirror member, the mirror configuration parameters (pArbiterIP, pArbiterPort, pSSLDirectory, pMirrorName, and pMemberName) are ignored, and the mirror configuration is not modified.
  • If any of the arguments pArbiterIP, pArbiterPort, or pSSLDirectory is not specified, the values configured for the node specified by pClusterURL are used.
  • Recommended best practice is to attach both failover members of a mirror set, before storing any data in the cluster database. In the case of creating a new mirror set, data stored in the primary member is not backed up until the backup member has been attached. In the case of attaching the failover members of an existing mirror set, if only the primary failover member has been attached, and the mirror fails over, sharded operations on the cluster cannot access data on other failover member that has been promoted from backup to primary, until it has been attached to the cluster.
  • When attaching the failover members of an existing mirrored set, attach the current primary member first, then the current backup member. The node being attached as primary must actually be primary at the time it is attached, and the node being attached as backup must actually be backup at the time it is attached.
  • A cluster namespace, with same name as on all other nodes of this cluster, is created if a namespace of that name doesn't already exist. If it does already exist, its globals database becomes the shard database for this node.
  • If the node specified by pClusterURL is a data node, it is used as the template for configuration settings on this node. If the specified node is a compute node, the corresponding data node is used as the template.
  • If the cluster namespace doesn't exist, it is created with interoperability enabled or not according to whether it is enabled for the cluster namespace on the template node, and with its globals database settings, including directory path, matching those of the shard database on the template node.
  • All SQL configuration settings are set to match those on the template node.
  • Data nodes are assigned a numeric id, in sequence starting from 1, in the order they are attached to the cluster. A data node's id is the same as its shard number. Nodes are listed by id by ListNodes() and GetMetadata().
classmethod CatchupDB(pDbName As %String) as %Status
Catches up the database specified by pDbName. Call this method if AttachAsMirroredNode() returns the error message "Cannot catch up database dbname, manual catchup required".

Parameters:

pDbName
The name of the database to catch up.

Returns:

Status code reporting success or failure of this API call.
classmethod ClusterNamespace() as %String
Gets the name of the cluster namespace on the current InterSystems IRIS instance.

Returns:

Name of the cluster namespace, or empty string if this instance is not a node in a cluster.
classmethod CreateMirror(pMirrorSetName As %String = "", pPrimaryName As %String = "", pPrimaryIP As %String = "", pBackupName As %String = "", pBackupIP As %String = "", pAgentPort As %Integer = "", pNodeType As %String = "", pInstance As %String = "", pArbiterIP As %String = "", pArbiterPort As %Integer = 0, pSSLdir As %String = "") as %String
Create a mirror member on the designated instance
pMirrorSetName Mirror Set name
pPrimaryName Primary member name
pPrimaryIP Primary member IP address
pBackupName Backup member name
pBackupIP Backup member IP address
pAgentPort ISC Agent Port
pNodeType "primary" | "backup" | "async"
pInstance Instance name
pArbiterIP Arbiter Node IP address
pArbiterPort Arbiter Node port number
pSSLdir SSL/TLS keys directory
classmethod GetMetadata(ByRef pMetadata) as %Status
Retrieves an overview of the metadata of the cluster to which the current InterSystems IRIS instance belongs.

The metadata is returned in an array, passed by reference, under the following subscripts:

  • ClusterNamespace - the name of the cluster namespace, visible on all nodes.
  • MasterNamespace - the name of the master namespace on node 1.
  • MasterGlobalsDatabase - the name of the globals database of the master namespace.
  • MasterRoutinesDatabase - the name of the routines database of the master namespace.
  • Node - information specific to individual nodes, stored under two additional subscripts: the node's numeric id (with ":Primary" or ":Backup" appended for mirrored nodes), and one of the following:
    • NodeType - "Data" or "Compute".
    • DataNodeId - for compute nodes only, the id of the corresponding data node.
    • ShardDatabase - for data nodes only, the name of the database containing this node's portion of sharded data.
    • Host - the hostname or IP address of the machine hosting the node's InterSystems IRIS instance.
    • SuperServerPort - the default port (superserver port) of the InterSystems IRIS instance.
    • WebServerPort - the web server port of the InterSystems IRIS instance.
    • IRISVersion - the InterSystems IRIS version number.
    • ShardingVersion - the sharding manager version number.
Parameters:
pMetadata
Array in which metadata is returned, passed by reference.

Returns:

Status code reporting success or failure of this API call.
classmethod Initialize(pClusterNamespace As %String = "IRISCLUSTER", pMasterNamespace As %String = "IRISDM", pAllowedConnections As %String = "", pHostIPAddress As %String = "") as %Status
Sets up the current InterSystems IRIS instance as the first data node of a new cluster. This method automatically and transparently performs all steps needed to enable this instance as the first node of a cluster (subject to these prerequisites), including:
  • Enables the ECP and sharding services, and sets these services' lists of allowed connections, if specified via the pAllowedConnections argument.
  • Creates the cluster and master namespaces and their databases if they don't already exist.
  • Creates all necessary mappings from the cluster namespace to the globals and routines databases of the master namespace, and to the shard database.

This method returns an error in any of the following cases:

  • The current InterSystems IRIS instance is already a node of a sharded cluster.
  • The current instance is a mirror member.
  • The specified cluster namespace already exists, and its globals database is mirrored.

Parameters:

pClusterNamespace
Cluster namespace, visible on all nodes of this cluster, providing access to both sharded and nonsharded data. If specified namespace does not exist, it is created. Default name: IRISCLUSTER.
pMasterNamespace
Master namespace for this cluster. If specified namespace does not exist, it is created. Default name: IRISDM.
pAllowedConnections
List of hosts allowed to participate in this cluster, specified as a semi-colon-separated list of IP addresses or hostnames. Default: No restriction of which hosts can participate.
pHostIPAddress
IP address other nodes of this cluster should use to connect to this node. Specify a value for for this parameter if the default hostname known to InterSystems IRIS does not resolve to an appropriate address, or no hostname is available. Default: Default hostname will be used for inter-node connections.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • If either the cluster namespace or the master namespace already exists, the existing namespace is used. If one of them exists and the other doesn't, the globals database for the one that doesn't exist is created using all of the same configuration settings as the globals database of the namespace that already exists (other than directory path). If neither namespace exists, databases for both are created using default settings.
  • If neither namespace exists, the namespaces are created without enabling interoperability. If one namespace exists, the other namespace is created with interoperability enabled or not, according to whether it is enabled in the existing namespace.
  • If the cluster namespace exists, its globals database becomes the shard database for node 1. The globals database of the master namespace is used to store all nonsharded data for the cluster, and its routines database is use to store all code for the cluster.
classmethod InitializeMirrored(pClusterNamespace As %String = "IRISCLUSTER", pMasterNamespace As %String = "IRISDM", pAllowedConnections As %String = "", pHostIPAddress As %String = "", pArbiterIP As %String = "", pArbiterPort As %Integer = 0, pSSLDirectory As %String = "", pMirrorName As %String = "IRISMIRROR1", pMemberName As %String = "IRISMIRROR1A") as %Status
Sets up the current InterSystems IRIS instance as the first data node of a new mirrored cluster. This method automatically and transparently performs all steps needed to enable this instance as the first node of a mirrored cluster (subject to these prerequisites), including:
  • Enables the ECP and sharding services, and sets these services' lists of allowed connections, if specified via the pAllowedConnections argument.
  • If the current instance is not already the primary failover member of a mirror, creates a new mirror set, and configures the current instance as its first failover member.
  • Creates the cluster and master namespaces and their databases if they don't already exist.
  • Creates all necessary mappings from the cluster namespace to the globals and routines databases of the master namespace, and to the shard database.

This method returns an error in any of the following cases:

  • The current InterSystems IRIS instance is already a node of a sharded cluster.
  • The current instance is already a mirror member, but is not currently the primary failover member.
  • The specified cluster namespace or master namespace already exists, but its globals database is not mirrored.

Parameters:

pClusterNamespace
Cluster namespace, visible on all nodes of this cluster, providing access to both sharded and nonsharded data. If specified namespace does not exist, it is created. Default name: IRISCLUSTER.
pMasterNamespace
Master namespace for this cluster. If specified namespace does not exist, it is created. Default name: IRISDM.
pAllowedConnections
List of hosts allowed to participate in this cluster, specified as a semi-colon-separated list of IP addresses or hostnames. Default: No restriction of which hosts can participate.
pHostIPAddress
IP address other nodes of this cluster should use to connect to this node. Specify a value for for this parameter if the default hostname known to InterSystems IRIS does not resolve to an appropriate address, or no hostname is available. Default: Default hostname will be used for inter-node connections.
pArbiterIP
Arbiter's hostname or IP address. Specify a non-null value to configure this mirror set to use an arbiter.
pArbiterPort
Arbiter's port. Specify a non-zero value to configure this mirror to use an arbiter.
pSSLDirectory
SSL/TLS keys directory. Specify a non-null value to configure this mirror to use SSL/TLS. Directory is expected to contain three files named "CAFile.pem", "CertificateFile.pem, and "PrivateKeyFile.pem".
pMirrorName
Name of mirror set to create. Default: IRISMIRROR1.
pMemberName
Name by which this instance is known to the mirror. Default: IRISMIRROR1A.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • If the current instance is already a mirror member, the mirror configuration parameters (pArbiterIP, pArbiterPort, pSSLDirectory, pMirrorName, and pMemberName) are ignored, and the mirror configuration is not modified.
  • If either the cluster namespace or the master namespace already exists, the existing namespace is used. If one of them exists and the other doesn't, the globals database for the one that doesn't exist is created using all of the same configuration settings as the globals database of the namespace that already exists (other than directory path). If neither namespace exists, databases for both are created using default settings.
  • If neither namespace exists, the namespaces are created without enabling interoperability. If one namespace exists, the other namespace is created with interoperability enabled or not, according to whether it is enabled in the existing namespace.
  • If the cluster namespace exists, its globals database becomes the shard database for node 1. The globals database of the master namespace is used to store all nonsharded data for the cluster, and its routines database is use to store all code for the cluster.
classmethod IsNodeReady(pClusterURL As %String = "") as %Status
classmethod ListNodes(pOutputFile As %String = "") as %Status
Lists the nodes of the cluster to which the current InterSystems IRIS instance belongs, to the console or to a specified output file.

The list contains a row for each node, with information in columns under the following headings:

  • NodeId - a numeric id. For data nodes, this is identical to the shard number.
  • NodeType - "Data" or "Compute".
  • DataNodeId - for compute nodes only, the id of the corresponding data node.
  • Host - the hostname or IP address of the machine hosting the node's InterSystems IRIS instance.
  • Port - the default port (superserver port) of the InterSystems IRIS instance.
  • Mirror - the mirror name, if the node is mirrored.
  • Failover - for mirrored nodes only, the type of failover member, either "Primary" or "Backup".
  • VIP - for mirrored nodes only, the VIP, if one is configured.
Parameters:
pOutputFile
The pathname of a file to which output is written. Default: Prints output to the console.

Returns:

Status code reporting success or failure of this API call.

Inherited Members

Inherited Methods

FeedbackOpens in a new tab