%SYSTEM.Sharding

class %SYSTEM.Sharding extends %SYSTEM.Help

This class provides an API for "manually" configuring sharding, at the level of individual InterSystems IRIS Data Platform instances.

It can be used via the special $SYSTEM object, for example:

set status = $SYSTEM.Sharding.EnableSharding()

The $SYSTEM.Sharding API provides an alternative to using ICM (InterSystems Cloud Manager) to provision and configure sharded clusters, for scenarios ICM does not support, or situations where step-by-step manual control, or avoiding the use of containers, is preferred. Unlike ICM, which manages the entire process of installing and configuring InterSystems IRIS instances to function together as sharded clusters, the $SYSTEM.Sharding API adds only sharding-specific functionality, depending on other InterSystems IRIS tools and APIs for tasks such as installing InterSystems IRIS instances, configuring mirroring, or creating namespaces and databases.

Most of the $SYSTEM.Sharding API calls operate at the level of a specified master namespace. The two exceptions are EnableSharding() and SetNodeIPAddress(), which operate on an entire InterSystems IRIS instance (which may contain multiple master namespaces).

Terminology

Sharding
Transparent horizontal partitioning of tables across a set of shards. Each shard can be hosted on a separate InterSystems IRIS instance, on a separate machine, providing horizontal performance scaling for queries and data ingestion. Horizontal partitioning means partitioning by rows, as opposed to vertical partitioning by columns. Each partition of a horizontally partitioned table contains a subset of the table's rows, and each row is contained in exactly one partition.
Sharded Cluster
A set of InterSystems IRIS instances configured to work together to support sharding. A sharded cluster is comprised of one shard master, one or more shard servers, each of which hosts one or more shards that are assigned to the master namespace hosted on the shard master, and zero or more master app servers, across which applications can be load balanced.
Shard Master
A InterSystems IRIS instance, or a mirrored pair of InterSystems IRIS instances, which hosts a master namespace. The set of InterSystems IRIS instances comprising a sharded cluster consists of the shard master itself, the set of shard server instances hosting shards assigned to the master namespace, and any shard app servers that have been configured with the shard master as their data server, and with a namespace whose globals and routine databases are mapped to those of the master namespace on the shard master. A single InterSystems IRIS instance can host more than one master namespace, in which case each of these master namespaces defines a separate sharded cluster, and the shard master instance participates in each of these sharded clusters. If a shard master is mirrored, then the globals and routine databases of each master namespace must be mirrored.
Shard Server
A InterSystems IRIS instance, or a mirrored pair of InterSystems IRIS instances, which hosts one or more shards (either data shards, query shards, or both).
Master App Server
A InterSystems IRIS instance that is configured as an ECP application server with the shard master as its data server, and hosting a namespace whose globals and routine databases are mapped to those of the master namespace on the shard master. Any sharding operation (including configuration, data definition, and data manipulation) that can be performed in the master namespace on the shard master can be performed identically in the namespace on a master app server whose database are mapped to those of the master namespace. The sole exception: the first-ever call to AssignShard() for a given master namespace must be made locally on the shard master (and will return an error if attempted on a namespace whose globals database is not local).
Master Namespace
A InterSystems IRIS namespace which has been assigned one or more shards, and can therefore contain sharded (that is, horizontally partitioned) tables. Other than the added ability to contain sharded tables, a master namespace has all of the characteristics and capabilities of an "ordinary" namespace (for example, it can contain tables that are not sharded; it can have mappings to globals residing in databases other than its default globals database; globals in its default globals database can be transparently accessed from a namespace on an ECP app server, whose globals database is a remote database mapped to the globals database of the master namespace; etc.). Any namespace that is not a shard namespace can become a master namespace by being assigned one or more shards.
Shard Namespace
A InterSystems IRIS namespace which has been designated to play the role of shard, by being specified (by host and port of the InterSystems IRIS instance on which it resides, and the name of the namespace) in a call to AssignShard(). Shard namespaces are transparent to end users of sharding, who access sharded tables via a master namespace, without needing to be aware of the names or locations of the shard namespaces assigned to that master namespace. Shard namespaces are visible to users who administer and configure sharding via this $SYSTEM.Sharding API. As namespaces, they are created and managed in the same ways as any other namespaces; their roles as shards are managed using this $SYSTEM.Sharding API. A namespace cannot be both a master namespace and a shard namespace. As recommended best practice, shard namespaces should not be accessed directly by end users, and should not be used for any purpose other than as shards of the master namespaces to which they are assigned.
Shard
A partition of a master namespace, that contains one horizontal partition of each sharded table in the master namespace to which the shard is assigned. A shard is implemented as as shard namespace. A shard is either a data shard or a query shard. A data shard may or may not be a mirrored shard, but a query shard is never a mirrored shard.
Note: The term shard can also be used to refer to a horizontal partition of an individual sharded table, but the context of the $SYSTEM.Sharding API, it refers to a partition of the master namespace.
Data Shard
A shard in which data is stored for one horizontal partition of each sharded table in the master namespace to which the data shard is assigned. A data shard may or may not be a mirrored shard. If it is a mirrored shard, its default globals database is a mirrored database. Otherwise, its default globals database is a non-mirrrored local database, or a remote database mapped to a non-mirrored database. Data shards are assigned a shard number 1 through number of shards, in the order that they are assigned calling AssignShard().
Query Shard
A shard which does not store data, but provides remote access via ECP to the data stored in a data shard to which it is assigned. Zero or more query shards may be assigned to each data shard, by specifying the shard number of corresponding data shard in a call to AssignShard(). A shard namespace used as a query shard must have as its default globals database a remote database that is mapped to the default globals database of the data shard to which the query shard is assigned. When SQL operations are executed on sharded tables, read-only queries are automatically executed on query shards for any shards that have one or more query shards assigned to them, but are executed on data shards if those data shards have no query shards assigned to them. If more than one query shard has been assigned to a shard, queries are automatically load balanced among them. Write operations (insert, update, delete, and DDL operations) are automatically executed on data shards. Query shards can be used to minimize interference betwe en query and data ingestion work loads, and to increase the band width of a sharded configuration for high volume multi-user query work loads.
Mirrored Shard
A data shard whose default globals database is mirrored. Use of mirrored shards provides high availability for sharded configurations, with transparent failover between mirror failover members and transparent completion of query operations, in the event of failover of one or more shards occurring during execution of a query. When configuring shard namespaces for mirrored shards, the shard namespace for a given shard must have the same name on both mirror failover members, and its default globals database must be the same mirrored database.

API Usage

A InterSystems IRIS instance is enabled to act as a shard master or shard server by calling EnableSharding().

The set of shards belonging to a master namespace is defined by making repeated calls to AssignShard(), one call for each shard.

Once shards have been assigned, VerifyShards() can be called to verify that they are reachable and correctly configured.

If additional shards are assigned to a namespace that already contains sharded tables, and the new shards can't be reached for automatic verification during the calls to AssignShard(), ActivateNewShards() can be called to activate them, once they are reachable.

A shard can be removed from the set belonging to a master namespace by calling DeassignShard().

An existing data shard can be assigned a different shard namespace address by calling ReassignShard().

All the shards assigned to a master namespace can be listed by calling ListShards().

A non-default IP address for connecting an InterSystems IRIS instance to other instances of sharded cluster can be specified by calling SetNodeIPAddress().

Sharding configuration options can be set by calling SetOption(), and their values can be retrieved by calling GetOption().

Method Inventory (Including Private)

Methods

classmethod ActivateNewShards(MasterNamespace As %String = $namespace) as %Status [ Language = objectscript ]
Activates shards that could not be activated by prior calls to AssignShard().

If new data shards are assigned when sharded tables already exist, AssignShard() attempts to connect to the newly assigned shards to verify them before they are activated. If this fails (either because the new shards are not reachable, or because they fail verification), AssignShard() returns an error indicating that ActivateNewShards() must be called to activate the new shards.

If this API call returns an error, it can be called again, after correcting the problem reported in the error, until it succeeds. If there are multiple shards requiring activation, none of them is activated until all of them can be successfully activated.

Newly assigned shards are always automatically activated, except in the case where sharded tables already exist and the new shards cannot be reached or cannot be verified. Therefore, ActivateNewShards() never needs to be called unless new shards are assigned while sharded tables already exist.

Parameters:

MasterNamespace
The master namespace whose new shards are to be activated. Defaults to the current namespace.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • When new shards have not yet been successfully activated, the master namespace's shard count has not been incremented to include them. This ensures that all sharded operations can continue executing on activated shards with no problems, without being affected by the existence of shards that have not yet been activated. When this API call succeeds in activating new shards, it increments the shard count by the number of newly-activated shards, making them available for use in all sharded operations. Existing data stored in other shards is not moved to newly activated shards, but data inserted after they are activated is evenly balanced across all shards including newly activated shards.
  • Shards which have been assigned but not yet activated are listed by ListShards() with their shard number in parentheses and followed by an asterisk, for example "(25*)".

Examples:

  • Activate shards newly assigned to the current namespace:
    set status = $SYSTEM.Sharding.ActivateNewShards()
classmethod AddDatabasesToMirrors(MasterNamespace As %String = $namespace) as %Status [ Language = objectscript ]
classmethod AssignShard(MasterNamespace As %String = $namespace, ShardHost As %String, ShardPort As %Integer, ShardNamespace As %String, ShardNumber As %Integer = "", ShardMirrorName As %String = "", ShardBackupHost As %String = "", ShardBackupPort As %Integer = "", ShardVIP As %String = "") as %Status [ Language = objectscript ]
Assigns a shard to a master namespace.

This API call can be used to assign a data shard or a query shard.

A data shard can be a namespace on a single InterSystems IRIS instance, or it can be a namespace whose globals database is mirrored.

A query shard must be a namespace on a single InterSystems IRIS instance, whose globals database is a remote database mapped to the globals database of the corresponding data shard. When assigning a query shard, the ShardNumber of the corresponding data shard must be specified, and that data shard must already have been assigned.

New data shards cannot be assigned if any sharded tables with user-defined shard keys already exist in the specified master namespace. If new data shards are assigned when sharded tables without user-defined shard keys already exist, this API call attempts to connect to the newly assigned shards to verify them before they are activated; if this fails (either because the new shards are not reachable, or because they fail verification), this API call returns an error indicating that ActivateNewShards() must be called to activate the new shards.

New query shards can be assigned at any time, regardless of whether sharded tables already exist.

Parameters:

MasterNamespace
The master namespace to which the shard is assigned. Defaults to the current namespace.
ShardHost
The machine hosting the shard namespace, specified by hostname or IP address.
ShardPort
The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
ShardNamespace
The namespace being assigned as a shard.
ShardNumber
Specifying a ShardNumber indicates that this shard is being assigned as a query shard, corresponding to the data shard with the specified ShardNumber, which must already have been assigned. Do not specify a value for ShardNumber when assigning a data shard.
ShardMirrorName
For a mirrored shard, ShardMirrorName specifies the mirror name (also known as mirror set name) of the mirror hosting the shard. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
ShardBackupHost
For a mirrored shard, the machine hosting the backup failover member of the mirror, specified by hostname or IP address. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
ShardBackupPort
For a mirrored shard, the default port (superserver port) of the backup failover member of the mirror. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
ShardVIP
For a mirrored shard, the Virtual IP address of the mirror, if one has been configured (optional).

Returns:

Status code reporting success or failure of this API call.

Notes:

  • This API call does not create or configure InterSystems IRIS instances, namespaces, mirrors, or remote database mappings, it just assigns specified namespaces on specified single or mirrored instances to play roles as shards of a specified master namespace. The specified namespaces, instances, mirrors, and remote database mappings must be configured separately using appropriate APIs in Management Portal or in classes in the Config package, either before or after calling AssignShard(). The requisites are:
  • Create InterSystems IRIS instance and namespace specified by ShardHost, ShardPort, and ShardNamespace.
  • For mirrored shard, create InterSystems IRIS instances specified by ShardHost and ShardPort, and by ShardBackupHost and ShardBackupPort, and configure them as the failover members of a mirrored set. (Note: It doesn't matter which one is actually the primary failover member at the time AssignShard() is called.) Create namespace specified by ShardNamespace on both failover members, with its globals database configured as a mirrored database.
  • For query shard, create namespace specified by ShardNamespace with remote database mapped to the globals database of the namespace of the corresponding data shard.
  • Enable sharding on the shard instance, by calling EnableSharding(). For mirrored shard, enable sharding on both failover members.
  • This API call returns an error if the namespace specified by ShardHost, ShardPort, and ShardNamespace, or by ShardBackupHost, ShardBackupPort, and ShardNamespace, has already been assigned as a data or query shard, or as the backup of a mirrored data shard, or if it is the same as the master namespace.
  • By default, this API call does not attempt to connect to the newly assigned shard to confirm that it is reachable and is correctly configured, except in the case where a data shard is assigned and sharded tables already exist. This call can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace), or verification can be performed in a separate call to VerifyShards().
  • Shard numbers are assigned to data shards sequentially, starting from 1, in the order they are assigned. The shard number of a given shard can be determined by calling ListShards(). If shards are de-assigned by calling DeassignShard(), the shard numbers of any high-numbered shards are decremented, so that there are never any gaps in the sequence.
  • As a convenience, "localhost" can be specified as the value of ShardHost, for a shard that resides on the current machine. This is always translated internally to the actual hostname or IP address. Sharding never uses the "loop-back" IP address 127.0.0.1.

Examples:

  • Assign a single-instance data shard:
    set status = $SYSTEM.Sharding.AssignShard("MASTER","machine1",1972,"SHARD1")
  • Assign a mirrored shard to the current namespace:
    set status = $SYSTEM.Sharding.AssignShard(,"machine2",1972,"SHARD2",,"MIRROR1","machine3",1972,"123.45.67.89")
  • Assign a query shard:
    set status = $SYSTEM.Sharding.AssignShard("MASTER","machine4",1972,"SHARD1",1)
classmethod ClusterIsNodeLevel(ClusterNamespace As %String = $namespace) as %Boolean [ Language = objectscript ]
classmethod DeassignShard(MasterNamespace As %String = $namespace, ShardHost As %String, ShardPort As %Integer, ShardNamespace As %String) as %Status [ Language = objectscript ]
De-assigns a shard from a master namespace to which it had previously been assigned. This removes the shard from the set of shards belonging to the master namespace.

This API call can be used to de-assign a data shard or a query shard.

A data shard may only be de-assigned if no sharded tables exist in the master namespace. A query shard may be de-assigned at any time.

When de-assigning a data shard to which any query shards are currently assigned, those query shards are automatically de-assigned as well.

When de-assigning a mirrored data shard, the host and port of either failover member may be specified.

Parameters:

MasterNamespace
The master namespace to which the shard is currently assigned. Defaults to the current namespace.
ShardHost
The machine hosting the shard namespace, specified by hostname or IP address.
ShardPort
The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
ShardNamespace
The namespace being de-assigned as a shard.

Returns:

Status code reporting success or failure of this API call.

Notes:

  • De-assigning a shard does not delete the shard namespace, or any data that may have been created in that namespace while it was serving as a shard. It simply causes that namespace to no longer serve as a shard of the specified master namespace, and makes the namespace available to be assigned as a shard of different master namespace. When a data shard is de-assigned, the shard count is decremented, and the shard numbers of any higher-numbered shards are decremented, so that there are no gaps in the sequence of shard numbers. When a query shard is de-assigned, this simply reduces by one the number of query shards assigned to the data shard to which that query shard had been assigned.
  • De-assigning data shards in production systems is not recommended. The ability to de-assign data shards is provided primarily as a convenience during initial configuration and testing of sharded clusters. De-assigning data shards while any sharded tables exist in the master namespace is not permitted, because this would make unreachable the data from each sharded table that had been stored in that data shard. If there is a need to reconfigure a master namespace to have fewer or different shards, after sharded tables already exist, all of the sharded tables can be dropped, and shards can then be de-assigned.
  • Query shards can be de-assigned at any time, just as they can be assigned at any time. This provides a dynamic means of adjusting the query throughput capacity of a sharded cluster as multi-user workloads grow or shrink. Note that this does not change the degree of parallelism for the execution of an individual query (which always equals the number of data shards), but it changes the multi-user throughput by enabling different concurrent queries to execute on different sets of query shards. When a query shard is de-assigned, it continues to be used by any active user connections already in progress, but is not used by any new user connections. Therefore, query shards can safely be de-assigned while query applications are running.
  • This API call returns an error if the the shard specified by ShardHost, ShardPort, and ShardNamespace is not currently assigned to the specified master namespace as either a query shard, a data shard, or either failover member of a mirrored data shard.

Examples:

  • De-assign a shard from a specified master namespace:
    set status = $SYSTEM.Sharding.DeassignShard("MASTER","machine1",1972,"SHARD1")
  • De-assign a shard from the current namespace:
    set status = $SYSTEM.Sharding.DeassignShard(,"machine1",1972,"SHARD1")
classmethod EnableSharding(MaxConn As %Integer = 64, EnableAsShardServer As %Boolean = 1, AllowedConnections As %String = "") as %Status [ Language = objectscript ]
Enables the current InterSystems IRIS instance to act as a shard master or shard server.

This API call provides a convenient way to perform several configuration steps which would otherwise need to be performed separately:

  • Enables the ECP service.
  • Sets the config options MaxServers and MaxServerConn.
  • Optionally enables the sharding service, so this InterSystems IRIS instance can act as a shard server.
  • Optionally configures a list of allowed connections for the sharding and ECP services.

    Parameters:
    MaxConn
    Maximum number of ECP connections needed for this InterSystems IRIS instance to communicate with other instances in this sharded cluster (default 64). If non-zero, this must be greater than or equal to the total number of InterSystems IRIS instances participating in the sharded cluster, but must be at least 2 (even if there is only one instance).
    The value specified for MaxConn is used to set the config options MaxServers and MaxServerConn. Specifying 0 means "do not change these config options", in which case they are assumed to be already set to appropriate values.
    EnableAsShardServer
    TRUE(1)/FALSE(0). If EnableAsShardServer is TRUE(1) (the default), the sharding service (%Service_Sharding) is enabled for this InterSystems IRIS instance, enabling it to act as a shard server or a shard master. If EnableAsShardServer is FALSE(0), the sharding service is disabled for this instance, enabling this instance to act as a shard master, but not as a shard server.
    AllowedConnections
    List of hosts allowed to connect to this InterSystems IRIS instance in its role as shard server, specified as a semi-colon-separated list of IP addresses or hostnames. If this list is specified, the listed hosts are configured as the allowed incoming connections for the sharding service and the ECP service (replacing any lists of allowed connections previously configured for those services); else the sharding and ECP services are configured to have no lists of allowed incoming connections (this causes there to be no restriction on which hosts may connect to this InterSystems IRIS instance via these services). If AllowedConnections is specified, the list should include all hosts participating in the sharded cluster as masters, shard servers, master app servers, or query shard servers, and should include both failover members of any mirrored pairs.
    Note: AllowedConnections only needs to be specified on InterSystems IRIS instances playing the role of shard server (either data or query shard server). If EnableAsShardServer is FALSE(0), the ECP service's list of allowed connections is not modified.

    Returns:

    Status code reporting success or failure of this API call.

    Notes:

    • A user must have administrative privileges in order to execute this API call.
    • After this API call, the InterSystems IRIS instance must be restarted for all of the changes to take effect.
    • This API call affects an entire InterSystems IRIS instance. If this instance participates in more than one sharded cluster (e.g. contains more than one master namespace, or shards belonging to more than one master namespace), the MaxConn and AllowedConnections arguments must be sufficient for all of the clusters in which this instance participates.

    Examples:

    • Enable as shard server or shard master, MaxServers=MaxServerConn=64, no restriction of allowed connections:
      set status = $SYSTEM.Sharding.EnableSharding()
    • Enable as shard server or shard master, MaxServers=MaxServerConn=3, two allowed connections specified:
      set status = $SYSTEM.Sharding.EnableSharding(3,1,"172.16.120.119;172.16.120.120")
    • Enable as shard master only, MaxServers=MaxServerConn=64:
      set status = $SYSTEM.Sharding.EnableSharding(,0)
    • classmethod GetOption(MasterNamespace As %String = $namespace, OptionName As %String, ByRef OptionValue As %Integer) as %Status [ Language = objectscript ]
    Gets the value of a sharding configuration option specified by name, within the scope of a specified master namespace.

    Parameters:
    MasterNamespace
    The master namespace within whose scope the option's value is determined.
    OptionName
    The name of the option (case insensitive), validated against a list of supported options.
    OptionValue (Output)
    This parameter is set to the option's value.

    Returns:

    Status code reporting success or failure of this API call.

    Notes:

    • See SetOption() for a table of supported options, the meanings of their values, and the default behavior if they are not set.

    Examples:

    • Set the variable connectTimeout to the value of of the option "ConnectTimeout" in the namespace "MASTER":
      set status = $SYSTEM.Sharding.GetOption("MASTER","ConnectTimeout",connectTimeout)
    classmethod ListShards(MasterNamespace As %String = $namespace) as %Status [ Language = objectscript ]
    Lists the shards assigned to a specified master namespace, to the console or current device.

    The list contains a row for each shard, with information in columns under the following headings:

  • Shard - the shard number (1 through number of shards).
  • Host - the hostname or IP address of the machine hosting the shard.
  • Port - the default port (superserver port) of the InterSystems IRIS instance hosting the shard.
  • Namespace- the shard namespace.
  • Mirror - the mirror name, if the shard is mirrored.
  • Role - the shard's role:
  • (blank) - ordinary data shard.
  • "Query" - query shard.
  • "Primary" - primary failover member hosting mirrored shard.
  • "Backup - backup failover member hosting mirrored shard.
  • VIP - VIP for mirrored shard, if one is configured.

    Parameters:
    MasterNamespace
    The master namespace whose shards are listed. Defaults to the current namespace.

    Returns:

    Status code reporting success or failure of this API call.

    Notes:

    • This API call can be used to determine the shard number corresponding to a shard's host, port, and namespace, for use as the ShardNumber parameter to AssignShard(), when assigning a query shard to a shard specified by shard number.
    • For mirrored shards, indicated primary and backup members are those at the time of initial shard assignment, or of the most recent operation that required connecting to shards. It is possible that the mirror has failed over since then, in which case the indicated primary member is now the backup member and vice versa.
    • Shards which have been assigned but not yet activated are listed with their shard number in parentheses and followed by an asterisk, for example "(25*)". (See ActivateNewShards() for an explanation of activating shards.)

    Examples:

    • List shards assigned to the current namespace to the console:
      set status = $SYSTEM.Sharding.ListShards()
    • List shards asssigned to namespace "MASTER" to the file "shards.list" in the current directory:
      open "shards.list":"NW"
      use "shards.list" s status=$SYSTEM.Sharding.ListShards()
      close "shards.list"
    • classmethod PurgeShard(MasterNamespace As %String = $namespace, ShardHost As %String, ShardPort As %Integer, ShardNamespace As %String, ShardNumber As %Integer) as %Status [ Language = objectscript ]
    classmethod ReassignShard(MasterNamespace As %String = $namespace, ShardHost As %String, ShardPort As %Integer, ShardNamespace As %String, ShardNumber As %Integer, ShardMirrorName As %String = "", ShardBackupHost As %String = "", ShardBackupPort As %Integer = "", ShardVIP As %String = "") as %Status [ Language = objectscript ]
    Re-assigns an existing data shard. This API call assigns a different shard namespace (specified by host, port, and namespace) to a shard number to which a data shard has previously been assigned.

    The newly specified shard namespace can be a namespace on a single InterSystems IRIS instance, or it can be a namespace whose globals database is mirrored. It is expected to contain identical data to the namespace previously assigned to the specified ShardNumber. The caller is responsible for ensuring this.

    This API call has the following use cases:

    • The globals database of the shard namespace is being relocated on a different InterSystems IRIS instance and/or host machine. The move itself is done outside of this API, and can be done in two general ways:

    • Dismount the source database, copy its IRIS.DAT file to its target location, and configure the new shard namespace with a globals database configured to use the copied IRIS.DAT file. This requires a maintenance window during which no applications access sharded tables in this master namespace.

    • Temporarily configure the source and target InterSystems IRIS instances as a mirrored set, with the target instance as backup failover member, backup the source database to the target database, and then promote the target instance to primary failover member. Applications can continue to access sharded tables throughout this process.

      • The source and target versions of the relocated shard can each either be mirrored or not. If the target is mirrored, the relocation can be done in multiple steps: first call ReassignShard to specify the target shard as a namespace on a single InterSystems IRIS instance, then after configuring the target as a mirror and backing up the target database to the backup failover member, call ReassignShard to re-specify the target shard as a mirrored namespace. If the relocation is done during a maintenance window, then ReassignShard only needs to be called once, after the target mirror is configured and the target database is up to date on both failover members.

      The ability to relocate shards can be used to allow for future horizontal scaling, by initially configuring more shards than the number of of host machines, which multiple shards on each machine (each hosted on a different InterSystems IRIS instance), and later adding machines to the sharded cluster, and relocating shards to separate host machines.

    • A shard that was previously not mirrored is being reconfigured to be mirrored. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current shard namespace, specify the ShardMirrorName of the newly configured mirror, and specify the ShardBackupHost, ShardBackupPort, and (optionally) ShardVIP of the newly configured backup failover member.

    • A shard that was previously mirrored is being reconfigured as not mirrored, leaving what was previously the primary failover member as the sole instance hosting the shard. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current primary failover member, and do not specify values for ShardMirrorName, ShardBackupHost, ShardBackupPort, or ShardVIP.

    • A new backup failover member is being specified for a mirrored shard, following failure and replacement of the previous backup failover member. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current primary failover member, specify current ShardMirrorName, and specify the ShardBackupHost, ShardBackupPort, and (optionally) ShardVIP of the newly configured backup failover member.
    Parameters:
    MasterNamespace
    The master namespace to which the shard is assigned. Defaults to the current namespace.
    ShardHost
    The machine hosting the shard namespace, specified by hostname or IP address.
    ShardPort
    The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
    ShardNamespace
    The namespace being assigned as a shard.
    ShardNumber
    The shard number to which a new shard namespace is assigned. There must already be a data shard assigned to this shard number.
    ShardMirrorName
    For a mirrored shard, ShardMirrorName specifies the mirror name (also known as mirror set name) of the mirror hosting the shard. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
    ShardBackupHost
    For a mirrored shard, the machine hosting the backup failover member of the mirror, specified by hostname or IP address. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
    ShardBackupPort
    For a mirrored shard, the default port (superserver port) of the backup failover member of the mirror. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
    ShardVIP
    For a mirrored shard, the Virtual IP address of the mirror, if one has been configured (optional).

    Returns:

    Status code reporting success or failure of this API call.

    Notes:

    • This API call does not create or configure InterSystems IRIS instances, namespaces, or mirrors, or copy, backup, mount, or dismount databases. These operations must be performed separately using appropriate APIs in Management Portal or in classes in the Config package, either before or after calling ReassignShard().
    • If any query shards have previously been configured and assigned to the ShardNumber that is being re-assigned, they must be re-configured so that their globals databases are mapped to the re-configured database of the data shard. This does not require de-assigning or re-assigning the query shards through this API, since this API associates query shards with data shards by shard number, not by database mappings.
    • This API call returns an error if the namespace specified by ShardHost, ShardPort, and ShardNamespace, or by ShardBackupHost, ShardBackupPort, and ShardNamespace, has already been assigned as a data shard, or as the backup of a mirrored data shard, with a shard number other than the specified ShardNumber, or if it has been assigned as a query shard with any shard number.
    • By default, this API call does not attempt to connect to the re-assigned shard to confirm that it is reachable and is correctly configured . This call can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace), or verification can be performed in a separate call to VerifyShards().
    • The ShardNumber to pass to this API call, if not already known, can be determined by calling ListShards().
    • As a convenience, "localhost" can be specified as the value of ShardHost, for a shard that resides on the current machine. This is always translated internally to the actual hostname or IP address. Sharding never uses the "loop-back" IP address 127.0.0.1.

    Examples:

    • Re-assign shard number 1 as a single-instance data shard:
      set status = $SYSTEM.Sharding.ReassignShard("MASTER","machine1",1972,"SHARD1",1)
    • Re-assign shard number 1 as a mirrored data shard:
      set status = $SYSTEM.Sharding.ReassignShard("MASTER","machine2",1972,"SHARD2",1,"MIRROR1","machine3",1972,"123.45.67.89")
    classmethod Rebalance(MasterNamespace As %String = $namespace, TimeLimit As %Integer = 0, ByRef Report, MinBuckets As %Integer = 1) as %Status [ Language = objectscript ]
    classmethod SetNodeIPAddress(NodeIPAddress As %String) as %Status [ Language = objectscript ]
    Specifies the IP address to be used by this InterSystems IRIS instances, when interconnecting with other instances in a sharded cluster, rather than using the IP address resulting from resolving this machine's hostname.

    This API call does not need to be used in most sharded cluster deployment scenarios. It is only needed when neither DNS resolution of the hostname, nor calling $SYSTEM.TCPDevice.LocalAddr() on an open TCP device, return an IP address that is usable for interconnecting nodes of a cluster.

    When this API call is needed, it must be called in each instance of a sharded cluster, before any shards are assigned (in the case of a shard master instance), before any namespace in the current instance is assigned as a shard (in the case of a data shard or query shard instance), or before any sharded operations or other sharding API calls are made in this instance (in the case of a master app server instance). The same IP address specified for a shard instance in this API call should also be passed as the ShardHost argument to AssignShard() when assigning a namespace in the instance as a shard, rather than passing hostname.

    Parameters:

    NodeIPAddress
    The IP address to be used by this InterSystems IRIS instances, when interconnecting with other instances in a sharded cluster.

    Returns:

    Status code reporting success or failure of this API call.

    Notes:

    • A user must have administrative privileges in order to execute this API call.
    • This API call affects an entire InterSystems IRIS instance. If this instance participates in more than one sharded cluster (e.g. contains more than one master namespace, or shards belonging to more than one master namespace), this call specifies the IP address used for this instance in all of the clusters in which this instance participates.
    • This API call provides similar functionality to that provided by the setting the deprecated sharding option "MasterIPAddress", but this API call works in a wider range of scenarios. Sharded clusters that are already working using "MasterIPAddress" do not need to be modified, but it is recommended that SetNodeIPAddress be used when configuring new sharded clusters.

    Example:

    • Set this instance's IP address to "10.0.0.76":
      set status = $SYSTEM.Sharding.SetNodeIPAddress("10.0.0.76")
    classmethod SetOption(MasterNamespace As %String = $namespace, OptionName As %String, OptionValue As %Integer = "") as %Status [ Language = objectscript ]
    Sets a specified sharding configuration option to a specified value, within the scope of a specified master namespace.

    All option values are integers, except in the case of MasterIPAddress. The supported options, with their allowed values and defaults, are as follows:

    Option NameDescriptionAllowed ValuesDefault
    AutoVerify

    Should AssignShard() automatically call VerifyShards()? 1: Yes. 0: No.

    		0/1No
    ConnectTimeout

    Timeout when connecting to a shard, in seconds.

    		>=160
    Debug

    Enables debug trace. 1: Enable all debug trace messages. Higher numbers: enable increasingly selective debug trace. Only recommended when working directly with InterSystems support.

    		1-10No debug trace.
    DropIgnoreError

    Should errors occurring during DROP TABLE be ignored? 1: Yes. 0: No, return the error.

    		0/1No
    MasterIPAddress

    IP address to use for master data server, rather than using DNS resolution on hostname. Deprecated, use SetNodeIPAddress() instead.

    		Valid IP addressUse hostname.
    MirrorConnectAttempts

    Number of times to retry connecting to a mirrored shard.

    		>=11
    QuiesceAllowReads

    Should Backup.ShardedCluster.Quiesce allow reads? 1: Yes. 0: No, block reads as well as writes.

    		0/1No
    RunQueriesAsync

    (DEPRICATED) Should queries be run asynchronously? 1: Yes. 0: No, run queries synchronously. This permits transparent completion in event of shard failover, when executing queries on mirrored data shards, but it prevents cancelling queries and increases the risk of filling the IRISTEMP database. This option is depricated. The recommended approach for transparent query failover handling is to assign a query shard for each data shard.

    		0/1Run queries asynchronously.

    Parameters:
    MasterNamespace
    The master namespace within whose scope the option is set.
    OptionName
    The name of the option (case insensitive), validated against a list of supported options.
    OptionValue
    The value to which to set the option, validated against allowed values for the specified option. Omitting this parameter, or specifying "", causes the specified option to be undefined, resulting in the default behavior described in the table above.

    Returns:

    Status code reporting success or failure of this API call.

    Examples:

    • Set the option "ConnectTimeout" to 15 in the namespace "MASTER":
      set status = $SYSTEM.Sharding.SetOption("MASTER", "ConnectTimeout", 15)
    classmethod SetShardBrokeredHost(Namespace As %String = "", ShardNumber As %Integer = 0, BrokeredHost As %String = "", Backup As %Boolean = 0) as %Status [ Language = objectscript ]
    classmethod VerifyShards(MasterNamespace As %String = $namespace, ShardNumber As %Integer = "", QueryShardNumber As %Integer = "", ReturnFirstError As %Boolean = 0) as %Status [ Language = objectscript ]
    Verifies that assigned shards are reachable and are correctly configured.

    Verifies either all shards that have been assigned to a specified master namespace (the simplest and recommended usage), or a specified data shard, or a specified query shard of a specified data shard.

    For each shard verified, this API call verifies the following, and returns a specific error for each failure:

  • The shard is reachable (the InterSystems IRIS instance hosting the shard is started, and can be reached via TCP/IP).
  • The ECP and sharding services are enabled on the instance hosting the shard.
  • If the sharding service on that instance has a list of allowed incoming connections, the host on which this API call is made is included in the list.
  • The instance hosting the shard has the config parameters MaxServers and MaxServerConn set to sufficiently high values for the currently assigned set of shards. This checks for a value that is at least as great as the total number of InterSystems IRIS instances participating in the sharded cluster, including the shard master and each additional instance that hosts one or more shards, but not including any master app servers. For this purpose, a mirror counts as one instance. (If there are any master app servers participating in the sharded cluster, the number of master app servers must be added to the MaxServer and MaxServerConn settings on each instance in the sharded cluster, beyond the minimum value that is checked by this API call.)
  • The instance hosting the shard does not require restart due to changes having been made to the CPF file and not yet activated. (Note that changes to MaxServers and MaxServerConn require restart before they are effective.)
  • The shard namespace exists.
  • The shard namespace is not the master namespace of some other sharded cluster, and has not already been assigned as a shard of some other sharded cluster. (AssignShard() ensures that the same namespace is not assigned twice as a shard of the present sharded cluster.)
  • For query shards, the shard namespace has as its default globals database a remote database that is mapped to the default globals database of the data shard to which the query shard is assigned.
  • For mirrored shards, the instance assigned as primary failover member really is a mirror failover member, and really does have, as its backup failover member, the instance assigned as backup failover member (for this purpose, it does not matter which failover member is currently the primary; if the mirror is correctly configured, this API call transparently connects to whichever failover member is currently the primary, and verifies that the other failover member's host and port were correctly specified to AssignShard()).

    This API call also verifies that the master instance on which it is invoked has the ECP service enabled, has sufficient values for the config parameters MaxServers and MaxServerConn, and does not require restart due to changes not yet activated in the CPF file.

    Parameters:
    MasterNamespace
    The master namespace whose shards are to be verified. Defaults to the current namespace.
    ShardNumber
    The shard number of the shard to be verified (1 through number of data shards). By default, verifies all shards.
    QueryShardNumber
    Query shard number, among the query shards of the specified shard, of the shard to be verified. ShardNumber must be specified if QueryShardNumber is specified. By default, verifies all query shards of the specified shard, or all query shards of all shards if ShardNumber is not specified.
    ReturnFirstError
    TRUE(1)/FALSE(0). If ReturnFirstError is TRUE(1), this API call will return an error after the first shard that fails verification. If ReturnFirstError is FALSE(0) (the default), this API call will attempt to verify all data shards, and if any of them fail verification, it will return a nested status including the error for each shard which failed verification. If all data shards are successfully verified, this API call with then attempt to verify all query shards, and if any of them fail verfication, it will return a nested status including the error for each query shard which failed verification.

    Returns:

    Status code reporting success or failure of this API call. If multiple shards fail verification, this is a nested status code indicating how many shards failed, which shards failed, and the specific error for each failed shard.

    Notes:

    • The API call AssignShard() can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace). This has the advantage of avoiding the need for a separate call to VerifyShards(), but means that shard instances and namespaces must already be reachable and correctly configured when calling AssignShard(), or it will return verification errors, requiring a separate call to be made to VerifyShards() once all shard instances and namespaces have been correctly configured and made reachable. Separating these two API calls provides the option of configuring shard instances and namespaces either before or after assigning them as shards.

    Examples:

    • Verify all shards assigned to the current namespace:
      set status = $SYSTEM.Sharding.VerifyShards()
    • Verify shard 1 assigned to namespace "MASTER":
      set status = $SYSTEM.Sharding.VerifyShards("MASTER",1)
    • Verify the second query shard assigned to shard 3 of the current namespace:
      set status = $SYSTEM.Sharding.VerifyShards(,3,2)
    • Verify all shards assigned to namespace "MASTER", but stop and return error for the first failure:
      set status = $SYSTEM.Sharding.VerifyShards("MASTER",,,1)

    • Inherited Members

    Inherited Methods (Including Private)

    Feedback