%SYSTEM.Sharding
class %SYSTEM.Sharding extends %SYSTEM.Help
This class provides an API for "manually" configuring sharding, at the level of individual InterSystems IRIS Data Platform instances.It can be used via the special $SYSTEM object, for example:
The $SYSTEM.Sharding API provides an alternative to using ICM (InterSystems Cloud Manager) to provision and configure sharded clusters, for scenarios ICM does not support, or situations where step-by-step manual control, or avoiding the use of containers, is preferred. Unlike ICM, which manages the entire process of installing and configuring InterSystems IRIS instances to function together as sharded clusters, the $SYSTEM.Sharding API adds only sharding-specific functionality, depending on other InterSystems IRIS tools and APIs for tasks such as installing InterSystems IRIS instances, configuring mirroring, or creating namespaces and databases.
Most of the $SYSTEM.Sharding API calls operate at the level of a specified master namespace. The two exceptions are EnableSharding() and SetNodeIPAddress(), which operate on an entire InterSystems IRIS instance (which may contain multiple master namespaces).
Terminology
- Sharding
- Transparent horizontal partitioning of tables across a set of shards. Each shard
can be hosted on a separate InterSystems IRIS instance, on a separate machine, providing horizontal
performance scaling for queries and data ingestion. Horizontal partitioning means partitioning
by rows, as opposed to vertical partitioning by columns. Each partition of a horizontally
partitioned table contains a subset of the table's rows, and each row is contained in exactly one
partition.
- Sharded Cluster
- A set of InterSystems IRIS instances configured to work together to support sharding. A sharded cluster is comprised of one shard master, one or more shard servers, each of which hosts one or more shards that are assigned to the master namespace hosted on the shard master, and zero or more master app servers, across which applications can be load balanced.
- Shard Master
- A InterSystems IRIS instance, or a mirrored pair of InterSystems IRIS instances,
which hosts a master namespace. The set of InterSystems IRIS instances
comprising a sharded cluster consists of the shard master itself, the set of
shard server instances hosting shards assigned to the master namespace,
and any shard app servers that have been configured with the shard master as
their data server, and with a namespace whose globals and routine databases are mapped to
those of the master namespace on
the shard master. A single InterSystems IRIS instance can host more than one master
namespace, in which case each of these master namespaces defines a separate
sharded cluster, and the shard master instance participates in each of these
sharded clusters. If a shard master is mirrored, then the globals and
routine databases of each master namespace must be mirrored.
- Shard Server
- A InterSystems IRIS instance, or a mirrored pair of InterSystems IRIS instances, which hosts one or
more shards (either data shards, query shards, or both).
- Master App Server
- A InterSystems IRIS instance that is configured as an ECP application server with
the shard master as its data server, and hosting a namespace whose globals and
routine databases are mapped to those of the master namespace on the shard master.
Any sharding operation (including configuration, data definition, and data manipulation) that
can be performed in the master namespace on the shard master can be performed
identically in the namespace on a master app server whose database are mapped to
those of the master namespace. The sole exception: the first-ever call to
AssignShard() for a given master namespace must be made locally on the
shard master (and will return an error if attempted on a namespace whose globals
database is not local).
- Master Namespace
- A InterSystems IRIS namespace which has been assigned one or more shards, and can therefore contain
sharded (that is, horizontally partitioned) tables. Other than the added ability to contain
sharded tables, a master namespace has all of the characteristics
and capabilities of an "ordinary" namespace (for example, it can contain tables that are not sharded;
it can have mappings to globals residing in databases other than its default globals database;
globals in its default globals database can be transparently accessed from a namespace on an ECP
app server, whose globals database is a remote database mapped to the globals database of the
master namespace;
etc.). Any namespace that is not a shard namespace can become a master namespace
by being assigned one or more shards.
- Shard Namespace
- A InterSystems IRIS namespace which has been designated to play the role of shard, by
being specified (by host and port of the InterSystems IRIS instance on which it resides, and the name of
the namespace) in a call to AssignShard(). Shard namespaces are transparent to
end users of sharding, who access sharded tables via a master namespace, without needing to
be aware of the names or locations of the shard namespaces assigned to that master
namespace. Shard namespaces are visible to users who administer and configure
sharding via this $SYSTEM.Sharding API. As namespaces, they are created and managed in the same
ways as any other namespaces; their roles as shards are managed using this $SYSTEM.Sharding API. A
namespace cannot be both a master namespace and a shard namespace. As recommended best practice,
shard namespaces should not be accessed directly by end users, and should not be used for
any purpose other than as shards of the master namespaces to which they are assigned.
- Shard
- A partition of a master namespace, that contains one horizontal partition
of each sharded table in the master namespace to which the shard is
assigned. A shard is implemented as as shard namespace.
A shard is either a data shard or a query shard. A data shard may or
may not be a mirrored shard, but a query shard is never a mirrored shard.
Note: The term shard can also be used to refer to a horizontal partition of an individual sharded table, but the context of the $SYSTEM.Sharding API, it refers to a partition of the master namespace.
- Data Shard
- A shard in which data is stored for one horizontal partition of each sharded table in the
master namespace to which the data shard is assigned. A data shard may or may not
be a mirrored shard. If it is a mirrored shard, its default globals database is a mirrored database.
Otherwise, its default globals database is a non-mirrrored
local database, or a remote database mapped to a non-mirrored database. Data shards are
assigned a shard number 1 through number of shards, in the order that they are assigned calling
AssignShard().
- Query Shard
- A shard which does not store data, but provides remote access via ECP to the data stored
in a data shard to which it is assigned. Zero or more query shards may be assigned to
each data shard, by specifying the shard number of corresponding data shard in a call to
AssignShard(). A shard namespace used as a query shard must have
as its default globals database a remote database that is mapped to the default globals database
of the data shard to which the query shard is assigned. When SQL operations are executed
on sharded tables, read-only queries are automatically executed on query shards for any shards that
have one or more query shards assigned to them, but are executed on data shards if
those data shards have no query shards assigned to them. If more than one query shard has been
assigned to a shard, queries are automatically load balanced among them. Write operations
(insert, update, delete, and DDL operations) are automatically executed on data shards.
Query shards can be used to minimize interference betwe en query and data ingestion work
loads, and to increase the band width of a sharded configuration for high volume multi-user
query work loads.
- Mirrored Shard
- A data shard whose default globals database is mirrored. Use of mirrored shards
provides high availability for sharded configurations, with transparent failover between mirror
failover members and transparent completion of query operations, in the event of failover of one
or more shards occurring during execution of a query. When configuring shard namespaces
for mirrored shards, the shard namespace for a given shard must have the same
name on both mirror failover members, and its default globals database must be the same mirrored
database.
API Usage
A InterSystems IRIS instance is enabled to act as a shard master or shard server by calling EnableSharding().
The set of shards belonging to a master namespace is defined by making repeated calls to AssignShard(), one call for each shard.
Once shards have been assigned, VerifyShards() can be called to verify that they are reachable and correctly configured.
If additional shards are assigned to a namespace that already contains sharded tables, and the new shards can't be reached for automatic verification during the calls to AssignShard(), ActivateNewShards() can be called to activate them, once they are reachable.
A shard can be removed from the set belonging to a master namespace by calling DeassignShard().
An existing data shard can be assigned a different shard namespace address by calling ReassignShard().
All the shards assigned to a master namespace can be listed by calling ListShards().
A non-default IP address for connecting an InterSystems IRIS instance to other instances of sharded cluster can be specified by calling SetNodeIPAddress().
Sharding configuration options can be set by calling SetOption(), and their values can be retrieved by calling GetOption().
Method Inventory (Including Private)
- ActivateNewShards()
- AssignShard()
- DeassignShard()
- EnableSharding()
- GetOption()
- ListShards()
- ReassignShard()
- Rebalance()
- SetNodeIPAddress()
- SetOption()
- SetShardBrokeredHost()
- VerifyShards()
Methods (Including Private)
If new data shards are assigned when sharded tables already exist, AssignShard() attempts to connect to the newly assigned shards to verify them before they are activated. If this fails (either because the new shards are not reachable, or because they fail verification), AssignShard() returns an error indicating that ActivateNewShards() must be called to activate the new shards.
If this API call returns an error, it can be called again, after correcting the problem reported in the error, until it succeeds. If there are multiple shards requiring activation, none of them is activated until all of them can be successfully activated.
Newly assigned shards are always automatically activated, except in the case where sharded
tables already exist and the new shards cannot be reached or cannot be verified. Therefore,
ActivateNewShards() never needs to be called unless new shards are assigned while
sharded tables already exist.
Parameters:
- MasterNamespace
- The master namespace whose new shards are to be activated. Defaults to the current namespace.
Returns:
Notes:
- When new shards have not yet been successfully activated, the master namespace's shard count has not been incremented to include them. This ensures that all sharded operations can continue executing on activated shards with no problems, without being affected by the existence of shards that have not yet been activated. When this API call succeeds in activating new shards, it increments the shard count by the number of newly-activated shards, making them available for use in all sharded operations. Existing data stored in other shards is not moved to newly activated shards, but data inserted after they are activated is evenly balanced across all shards including newly activated shards.
- Shards which have been assigned but not yet activated are listed by ListShards() with their shard number in parentheses and followed by an asterisk, for example "(25*)".
Examples:
- Activate shards newly assigned to the current namespace:
set status = $SYSTEM.Sharding.ActivateNewShards()
This API call can be used to assign a data shard or a query shard.
A data shard can be a namespace on a single InterSystems IRIS instance, or it can be a namespace whose globals database is mirrored.
A query shard must be a namespace on a single InterSystems IRIS instance, whose globals database is a remote database mapped to the globals database of the corresponding data shard. When assigning a query shard, the ShardNumber of the corresponding data shard must be specified, and that data shard must already have been assigned.
New data shards cannot be assigned if any sharded tables with user-defined shard keys already exist in the specified master namespace. If new data shards are assigned when sharded tables without user-defined shard keys already exist, this API call attempts to connect to the newly assigned shards to verify them before they are activated; if this fails (either because the new shards are not reachable, or because they fail verification), this API call returns an error indicating that ActivateNewShards() must be called to activate the new shards.
New query shards can be assigned at any time, regardless of whether sharded tables
already exist.
Parameters:
- MasterNamespace
- The master namespace to which the shard is assigned. Defaults to the current namespace.
- ShardHost
- The machine hosting the shard namespace, specified by hostname or IP address.
- ShardPort
- The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
- ShardNamespace
- The namespace being assigned as a shard.
- ShardNumber
- Specifying a ShardNumber indicates that this shard is being assigned as a query
shard, corresponding to the data shard with the specified ShardNumber, which must already
have been assigned. Do not specify a value for ShardNumber when assigning a data shard.
- ShardMirrorName
- For a mirrored shard, ShardMirrorName specifies the
mirror name (also known as mirror set name) of the mirror hosting the shard. This parameter must
be specified when assigning a mirrored shard, and must not be specified otherwise.
- ShardBackupHost
- For a mirrored shard, the machine hosting the backup failover member of the mirror, specified
by hostname or IP address. This parameter must be specified when assigning a mirrored shard,
and must not be specified otherwise.
- ShardBackupPort
- For a mirrored shard, the default port (superserver port) of the backup failover member of the mirror. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
- ShardVIP
- For a mirrored shard, the Virtual IP address of the mirror, if one has been configured (optional).
Returns:
Notes:
- This API call does not create or configure InterSystems IRIS instances, namespaces, mirrors, or remote database
mappings, it just assigns specified namespaces on specified single or mirrored instances to play
roles as shards of a specified master namespace. The specified namespaces, instances, mirrors,
and remote database mappings must be configured separately using appropriate APIs in Management Portal
or in classes in the Config package, either before or after calling AssignShard(). The
requisites are:
- Create InterSystems IRIS instance and namespace specified by ShardHost, ShardPort, and ShardNamespace.
- For mirrored shard, create InterSystems IRIS instances specified by ShardHost and ShardPort, and by ShardBackupHost and ShardBackupPort, and configure them as the failover members of a mirrored set. (Note: It doesn't matter which one is actually the primary failover member at the time AssignShard() is called.) Create namespace specified by ShardNamespace on both failover members, with its globals database configured as a mirrored database.
- For query shard, create namespace specified by ShardNamespace with remote database mapped to the globals database of the namespace of the corresponding data shard.
- Enable sharding on the shard instance, by calling EnableSharding(). For mirrored shard, enable sharding on both failover members.
- This API call returns an error if the namespace specified by ShardHost, ShardPort, and ShardNamespace, or by ShardBackupHost, ShardBackupPort, and ShardNamespace, has already been assigned as a data or query shard, or as the backup of a mirrored data shard, or if it is the same as the master namespace.
- By default, this API call does not attempt to connect to the newly assigned shard to confirm that it is reachable and is correctly configured, except in the case where a data shard is assigned and sharded tables already exist. This call can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace), or verification can be performed in a separate call to VerifyShards().
- Shard numbers are assigned to data shards sequentially, starting from 1, in the order they are assigned. The shard number of a given shard can be determined by calling ListShards(). If shards are de-assigned by calling DeassignShard(), the shard numbers of any high-numbered shards are decremented, so that there are never any gaps in the sequence.
- As a convenience, "localhost" can be specified as the value of ShardHost, for a shard that resides on the current machine. This is always translated internally to the actual hostname or IP address. Sharding never uses the "loop-back" IP address 127.0.0.1.
Examples:
- Assign a single-instance data shard:
set status = $SYSTEM.Sharding.AssignShard("MASTER","machine1",1972,"SHARD1") - Assign a mirrored shard to the current namespace:
set status = $SYSTEM.Sharding.AssignShard(,"machine2",1972,"SHARD2",,"MIRROR1","machine3",1972,"123.45.67.89") - Assign a query shard:
set status = $SYSTEM.Sharding.AssignShard("MASTER","machine4",1972,"SHARD1",1)
This API call can be used to de-assign a data shard or a query shard.
A data shard may only be de-assigned if no sharded tables exist in the master namespace. A query shard may be de-assigned at any time.
When de-assigning a data shard to which any query shards are currently assigned, those query shards are automatically de-assigned as well.
When de-assigning a mirrored data shard, the host and port of either failover member may be
specified.
Parameters:
- MasterNamespace
- The master namespace to which the shard is currently assigned. Defaults to the current namespace.
- ShardHost
- The machine hosting the shard namespace, specified by hostname or IP address.
- ShardPort
- The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
- ShardNamespace
- The namespace being de-assigned as a shard.
Returns:
Notes:
- De-assigning a shard does not delete the shard namespace, or any data that may have been created in that namespace while it was serving as a shard. It simply causes that namespace to no longer serve as a shard of the specified master namespace, and makes the namespace available to be assigned as a shard of different master namespace. When a data shard is de-assigned, the shard count is decremented, and the shard numbers of any higher-numbered shards are decremented, so that there are no gaps in the sequence of shard numbers. When a query shard is de-assigned, this simply reduces by one the number of query shards assigned to the data shard to which that query shard had been assigned.
- De-assigning data shards in production systems is not recommended. The ability to de-assign data shards is provided primarily as a convenience during initial configuration and testing of sharded clusters. De-assigning data shards while any sharded tables exist in the master namespace is not permitted, because this would make unreachable the data from each sharded table that had been stored in that data shard. If there is a need to reconfigure a master namespace to have fewer or different shards, after sharded tables already exist, all of the sharded tables can be dropped, and shards can then be de-assigned.
- Query shards can be de-assigned at any time, just as they can be assigned at any time. This provides a dynamic means of adjusting the query throughput capacity of a sharded cluster as multi-user workloads grow or shrink. Note that this does not change the degree of parallelism for the execution of an individual query (which always equals the number of data shards), but it changes the multi-user throughput by enabling different concurrent queries to execute on different sets of query shards. When a query shard is de-assigned, it continues to be used by any active user connections already in progress, but is not used by any new user connections. Therefore, query shards can safely be de-assigned while query applications are running.
- This API call returns an error if the the shard specified by ShardHost, ShardPort, and ShardNamespace is not currently assigned to the specified master namespace as either a query shard, a data shard, or either failover member of a mirrored data shard.
Examples:
- De-assign a shard from a specified master namespace:
set status = $SYSTEM.Sharding.DeassignShard("MASTER","machine1",1972,"SHARD1") - De-assign a shard from the current namespace:
set status = $SYSTEM.Sharding.DeassignShard(,"machine1",1972,"SHARD1")
This API call provides a convenient way to perform several configuration steps which would otherwise need to be performed separately:
Parameters:
- MaxConn
- Maximum number of ECP connections needed for this InterSystems IRIS instance to communicate with other
instances in this sharded cluster (default 64). If non-zero, this must be greater than or equal to
the total number of InterSystems IRIS instances participating in the sharded cluster, but must be at least 2 (even if there is only one instance).
The value specified for MaxConn is used to set the config options MaxServers and MaxServerConn. Specifying 0 means "do not change these config options", in which case they are assumed to be already set to appropriate values.
- EnableAsShardServer
- TRUE(1)/FALSE(0). If EnableAsShardServer is TRUE(1) (the default), the sharding service
(%Service_Sharding) is enabled
for this InterSystems IRIS instance, enabling it to act as a shard server or a shard master. If EnableAsShardServer is
FALSE(0), the sharding service is disabled for this instance, enabling this instance to act as a shard master, but not as
a shard server.
- AllowedConnections
- List of hosts allowed to connect to this InterSystems IRIS instance in its role as shard server, specified
as a semi-colon-separated list of IP addresses or hostnames. If this list is specified, the listed hosts are
configured as the allowed incoming connections for the sharding service and the ECP service (replacing any
lists of allowed connections previously configured for those services); else the sharding
and ECP services are configured
to have no lists of allowed incoming connections (this causes there to be
no restriction on which hosts may connect to this InterSystems IRIS instance via these services).
If AllowedConnections is specified, the list should include all hosts participating in the
sharded cluster as masters, shard servers, master app servers, or query
shard servers, and should include both failover members of any mirrored pairs.
Note: AllowedConnections only needs to be specified on InterSystems IRIS instances playing the role of shard server (either data or query shard server). If EnableAsShardServer is FALSE(0), the ECP service's list of allowed connections is not modified.
Returns:
Notes:
- A user must have administrative privileges in order to execute this API call.
- After this API call, the InterSystems IRIS instance must be restarted for all of the changes to take effect.
- This API call affects an entire InterSystems IRIS instance. If this instance participates in more than one sharded cluster (e.g. contains more than one master namespace, or shards belonging to more than one master namespace), the MaxConn and AllowedConnections arguments must be sufficient for all of the clusters in which this instance participates.
Examples:
- Enable as shard server or shard master, MaxServers=MaxServerConn=64, no
restriction of allowed connections:
set status = $SYSTEM.Sharding.EnableSharding() - Enable as shard server or shard master, MaxServers=MaxServerConn=3, two allowed connections specified:
set status = $SYSTEM.Sharding.EnableSharding(3,1,"172.16.120.119;172.16.120.120") - Enable as shard master only, MaxServers=MaxServerConn=64:
set status = $SYSTEM.Sharding.EnableSharding(,0)
Parameters:
- MasterNamespace
- The master namespace within whose scope the option's value is determined.
- OptionName
- The name of the option (case insensitive), validated against a list of supported options.
- OptionValue (Output)
- This parameter is set to the option's value.
Returns:
Notes:
- See SetOption() for a table of supported options, the meanings of their values, and the default behavior if they are not set.
Examples:
- Set the variable connectTimeout to the value of of the option "ConnectTimeout"
in the namespace "MASTER":
set status = $SYSTEM.Sharding.GetOption("MASTER","ConnectTimeout",connectTimeout)
The list contains a row for each shard, with information in columns under the following headings:
Parameters:
- MasterNamespace
- The master namespace whose shards are listed. Defaults to the current namespace.
Returns:
Notes:
- This API call can be used to determine the shard number corresponding to a shard's host, port, and namespace, for use as the ShardNumber parameter to AssignShard(), when assigning a query shard to a shard specified by shard number.
- For mirrored shards, indicated primary and backup members are those at the time of initial shard assignment, or of the most recent operation that required connecting to shards. It is possible that the mirror has failed over since then, in which case the indicated primary member is now the backup member and vice versa.
- Shards which have been assigned but not yet activated are listed with their shard number in parentheses and followed by an asterisk, for example "(25*)". (See ActivateNewShards() for an explanation of activating shards.)
Examples:
- List shards assigned to the current namespace to the console:
set status = $SYSTEM.Sharding.ListShards() - List shards asssigned to namespace "MASTER" to the file "shards.list" in the current directory:
open "shards.list":"NW"
use "shards.list" s status=$SYSTEM.Sharding.ListShards()
close "shards.list"
The newly specified shard namespace can be a namespace on a single InterSystems IRIS instance, or it can be a namespace whose globals database is mirrored. It is expected to contain identical data to the namespace previously assigned to the specified ShardNumber. The caller is responsible for ensuring this.
This API call has the following use cases:
- The globals database of the shard namespace is being relocated on a different InterSystems IRIS instance
and/or host machine. The move itself is done outside of this API, and can be done in two general ways:
- Dismount the source database, copy its IRIS.DAT file to its target location, and configure the new shard namespace with a globals database configured to use the copied IRIS.DAT file. This requires a maintenance window during which no applications access sharded tables in this master namespace.
- Temporarily configure the source and target InterSystems IRIS instances as a mirrored set, with the target instance as backup failover member, backup the source database to the target database, and then promote the target instance to primary failover member. Applications can continue to access sharded tables throughout this process.
The source and target versions of the relocated shard can each either be mirrored or not. If the target is mirrored, the relocation can be done in multiple steps: first call ReassignShard to specify the target shard as a namespace on a single InterSystems IRIS instance, then after configuring the target as a mirror and backing up the target database to the backup failover member, call ReassignShard to re-specify the target shard as a mirrored namespace. If the relocation is done during a maintenance window, then ReassignShard only needs to be called once, after the target mirror is configured and the target database is up to date on both failover members.
The ability to relocate shards can be used to allow for future horizontal scaling, by initially configuring more shards than the number of of host machines, which multiple shards on each machine (each hosted on a different InterSystems IRIS instance), and later adding machines to the sharded cluster, and relocating shards to separate host machines.
- A shard that was previously not mirrored is being reconfigured to be mirrored. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current shard namespace, specify the ShardMirrorName of the newly configured mirror, and specify the ShardBackupHost, ShardBackupPort, and (optionally) ShardVIP of the newly configured backup failover member.
- A shard that was previously mirrored is being reconfigured as not mirrored, leaving what was previously the primary failover member as the sole instance hosting the shard. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current primary failover member, and do not specify values for ShardMirrorName, ShardBackupHost, ShardBackupPort, or ShardVIP.
- A new backup failover member is being specified for a mirrored shard, following failure and replacement of the previous backup failover member. In this case, specify the ShardHost, ShardPort, and ShardNamespace of the current primary failover member, specify current ShardMirrorName, and specify the ShardBackupHost, ShardBackupPort, and (optionally) ShardVIP of the newly configured backup failover member.
- MasterNamespace
- The master namespace to which the shard is assigned. Defaults to the current namespace.
- ShardHost
- The machine hosting the shard namespace, specified by hostname or IP address.
- ShardPort
- The default port (superserver port) of the InterSystems IRIS instance hosting the shard namespace.
- ShardNamespace
- The namespace being assigned as a shard.
- ShardNumber
- The shard number to which a new shard namespace is assigned. There must already be a data
shard assigned to this shard number.
- ShardMirrorName
- For a mirrored shard, ShardMirrorName specifies the
mirror name (also known as mirror set name) of the mirror hosting the shard. This parameter must
be specified when assigning a mirrored shard, and must not be specified otherwise.
- ShardBackupHost
- For a mirrored shard, the machine hosting the backup failover member of the mirror, specified
by hostname or IP address. This parameter must be specified when assigning a mirrored shard,
and must not be specified otherwise.
- ShardBackupPort
- For a mirrored shard, the default port (superserver port) of the backup failover member of the mirror. This parameter must be specified when assigning a mirrored shard, and must not be specified otherwise.
- ShardVIP
- For a mirrored shard, the Virtual IP address of the mirror, if one has been configured (optional).
Returns:
Notes:
- This API call does not create or configure InterSystems IRIS instances, namespaces, or mirrors, or copy, backup, mount, or dismount databases. These operations must be performed separately using appropriate APIs in Management Portal or in classes in the Config package, either before or after calling ReassignShard().
- If any query shards have previously been configured and assigned to the ShardNumber that is being re-assigned, they must be re-configured so that their globals databases are mapped to the re-configured database of the data shard. This does not require de-assigning or re-assigning the query shards through this API, since this API associates query shards with data shards by shard number, not by database mappings.
- This API call returns an error if the namespace specified by ShardHost, ShardPort, and ShardNamespace, or by ShardBackupHost, ShardBackupPort, and ShardNamespace, has already been assigned as a data shard, or as the backup of a mirrored data shard, with a shard number other than the specified ShardNumber, or if it has been assigned as a query shard with any shard number.
- By default, this API call does not attempt to connect to the re-assigned shard to confirm that it is reachable and is correctly configured . This call can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace), or verification can be performed in a separate call to VerifyShards().
- The ShardNumber to pass to this API call, if not already known, can be determined by calling ListShards().
- As a convenience, "localhost" can be specified as the value of ShardHost, for a shard that resides on the current machine. This is always translated internally to the actual hostname or IP address. Sharding never uses the "loop-back" IP address 127.0.0.1.
Examples:
- Re-assign shard number 1 as a single-instance data shard:
set status = $SYSTEM.Sharding.ReassignShard("MASTER","machine1",1972,"SHARD1",1) - Re-assign shard number 1 as a mirrored data shard:
set status = $SYSTEM.Sharding.ReassignShard("MASTER","machine2",1972,"SHARD2",1,"MIRROR1","machine3",1972,"123.45.67.89")
This API call does not need to be used in most sharded cluster deployment scenarios. It is only needed when neither DNS resolution of the hostname, nor calling $SYSTEM.TCPDevice.LocalAddr() on an open TCP device, return an IP address that is usable for interconnecting nodes of a cluster.
When this API call is needed, it must be called in each instance of a sharded cluster, before any shards are assigned (in the case of a shard master instance), before any namespace in the current instance is assigned as a shard (in the case of a data shard or query shard instance), or before any sharded operations or other sharding API calls are made in this instance (in the case of a master app server instance). The same IP address specified for a shard instance in this API call should also be passed as the ShardHost argument to AssignShard() when assigning a namespace in the instance as a shard, rather than passing hostname.
Parameters:
- NodeIPAddress
- The IP address to be used by this InterSystems IRIS instances, when interconnecting with other instances in a sharded cluster.
Returns:
Notes:
- A user must have administrative privileges in order to execute this API call.
- This API call affects an entire InterSystems IRIS instance. If this instance participates in more than one sharded cluster (e.g. contains more than one master namespace, or shards belonging to more than one master namespace), this call specifies the IP address used for this instance in all of the clusters in which this instance participates.
- This API call provides similar functionality to that provided by the setting the deprecated sharding option "MasterIPAddress", but this API call works in a wider range of scenarios. Sharded clusters that are already working using "MasterIPAddress" do not need to be modified, but it is recommended that SetNodeIPAddress be used when configuring new sharded clusters.
Example:
- Set this instance's IP address to "10.0.0.76":
set status = $SYSTEM.Sharding.SetNodeIPAddress("10.0.0.76")
All option values are integers, except in the case of MasterIPAddress. The supported options, with their allowed values and defaults, are as follows:
Option Name | Description | Allowed Values | Default |
---|---|---|---|
AutoVerify | Should AssignShard() automatically call VerifyShards()? 1: Yes. 0: No. | 0/1 | No |
ConnectTimeout | Timeout when connecting to a shard, in seconds. | >=1 | 60 |
Debug | Enables debug trace. 1: Enable all debug trace messages. Higher numbers: enable increasingly selective debug trace. Only recommended when working directly with InterSystems support. | 1-10 | No debug trace. |
DropIgnoreError | Should errors occurring during DROP TABLE be ignored? 1: Yes. 0: No, return the error. | 0/1 | No |
MasterIPAddress | IP address to use for master data server, rather than using DNS resolution on hostname. Deprecated, use SetNodeIPAddress() instead. | Valid IP address | Use hostname. |
MirrorConnectAttempts | Number of times to retry connecting to a mirrored shard. | >=1 | 1 |
QuiesceAllowReads | Should Backup.ShardedCluster.Quiesce allow reads? 1: Yes. 0: No, block reads as well as writes. | 0/1 | No |
RunQueriesAsync | Should queries be run asynchronously? 1 forces queries to be run asynchronously. 0 forces queries to be run synchronously. If not set, queries are run synchronously when any shards are mirrored (permitting transparent completion in event of failover), otherwise they are run asynchronously (faster in some cases). | 0/1 | Sync if shards mirrored else async |
Parameters:
- MasterNamespace
- The master namespace within whose scope the option is set.
- OptionName
- The name of the option (case insensitive), validated against a list of supported options.
- OptionValue
- The value to which to set the option, validated against allowed values for
the specified option. Omitting this parameter, or specifying "",
causes the specified option to be undefined, resulting in the default behavior
described in the table above.
Returns:
Examples:
- Set the option "ConnectTimeout" to 15 in the namespace "MASTER":
set status = $SYSTEM.Sharding.SetOption("MASTER", "ConnectTimeout", 15)
Verifies either all shards that have been assigned to a specified master namespace (the simplest and recommended usage), or a specified data shard, or a specified query shard of a specified data shard.
For each shard verified, this API call verifies the following, and returns a specific error for each failure:
This API call also verifies that the master instance on which it is invoked has the ECP service enabled, has sufficient values for the config parameters MaxServers and MaxServerConn, and does not require restart due to changes not yet activated in the CPF file.
Parameters:
- MasterNamespace
- The master namespace whose shards are to be verified. Defaults to the current namespace.
- ShardNumber
- The shard number of the shard to be verified (1 through number of data shards). By default,
verifies all shards.
- QueryShardNumber
- Query shard number, among the query shards of the specified shard, of the shard to be verified.
ShardNumber must be specified if QueryShardNumber is specified. By default, verifies
all query shards of the specified shard, or all query shards of all shards if ShardNumber
is not specified.
- ReturnFirstError
- TRUE(1)/FALSE(0). If ReturnFirstError is TRUE(1), this API call will return an error after the
first shard that fails verification. If ReturnFirstError is FALSE(0) (the default), this
API call will attempt to verify all data shards, and if any of them fail verification, it will
return a nested status including the error for each shard which failed verification. If all
data shards are successfully verified, this API call with then attempt to verify all query shards,
and if any of them fail verfication, it will return a nested status including the error for each
query shard which failed verification.
Returns:
Notes:
- The API call AssignShard() can be made to automatically verify all assigned shards by calling SetOption()(masterNamespace,"AutoVerify",1) (where masterNamespace is set to the appropriate master namespace). This has the advantage of avoiding the need for a separate call to VerifyShards(), but means that shard instances and namespaces must already be reachable and correctly configured when calling AssignShard(), or it will return verification errors, requiring a separate call to be made to VerifyShards() once all shard instances and namespaces have been correctly configured and made reachable. Separating these two API calls provides the option of configuring shard instances and namespaces either before or after assigning them as shards.
Examples:
- Verify all shards assigned to the current namespace:
set status = $SYSTEM.Sharding.VerifyShards() - Verify shard 1 assigned to namespace "MASTER":
set status = $SYSTEM.Sharding.VerifyShards("MASTER",1) - Verify the second query shard assigned to shard 3 of the current namespace:
set status = $SYSTEM.Sharding.VerifyShards(,3,2) - Verify all shards assigned to namespace "MASTER", but stop and return error for the first failure:
set status = $SYSTEM.Sharding.VerifyShards("MASTER",,,1)