ICM Reference

General ParametersCopy link to this section

The fields in the following table are all used with all cloud providers, and some are used with vSphere and Preexisting as well.

The two rightmost columns indicate whether each parameter is required in every deployment or optional, and whether it must be included (when used) in either defaults.json or definitions.json, is recommended for one file or the other, or can be used in either. For example,

• A single deployment is always on a single selected provisioning platform (even if subsequently merged with another to create a multiprovider deployment), therefore the Provider parameter is required and must be in the defaults file.

• Each node type must be specified but a deployment can include multiple node types, thus the Role parameter is required in each definition in the definitions file.

• Because each node that runs InterSystems IRIS must have a license, but other nodes don’t need one, the LicenseKey setting is required and generally appears in the appropriate definitions in the definitions file.

• At least one container must be deployed on each node in the deployment, but a single container may be deployed on all the nodes (for instance iris/iris-arm64 across a sharded cluster consisting of DATA nodes only) or different containers on different node types (iris/iris-arm64 on DM and AM, webgateway on WS, arbiter on AR in a distributed cache cluster). For this reason the DockerImage parameter is required and can appear in the defaults file, the definitions file, or both (to specify a default image but override it for one or more node types).

• Like the image to be deployed, the size of the OS volume can be specified for all nodes in the defaults file, for one or more node types in the definitions file, or in both, but because it has a default it is optional.

Provider	Definition	Use is ...	Config file
Provider	Platform to provision infrastructure on; see Provisioning Platforms.	required	defaults
Label Tag	Fields in naming scheme for provisioned cloud nodes: Label-Role-Tag-NNNN, for example ANDY-DATA-TEST-0001; should indicate ownership and purpose, to avoid conflicting with others. Multiple deployments should not share the same Label and Tag. Cannot contain dashes.	required	defaults
LicenseDir	Location of InterSystems IRIS license keys staged in the ICM container and individually specified by the LicenseKey field (below); see InterSystems IRIS Licensing for ICM.	required	defaults
LicenseKey	License key for the InterSystems IRIS instance on one or more provisioned DATA, COMPUTE, DM, AM, DS, or QS nodes, staged within the ICM container in the location specified by the LicenseDir field (above). In a configuration containing only DM and AM nodes, a standard license can be used; for all others (that is, sharded clusters), a sharding-enabled license is required.	required	definitions recommended
Region (Azure equivalent: Location)	Geographical region of provider’s compute resources in which infrastructure is provisioned. For information on deploying a single configuration in more than one region, see Deploying Across Multiple Regions or Providers. Provider-specific information, including provider documentation: AWS — Example: us-west-1; see About AWS Regions and Availability ZonesOpens in a new window. GCP — Example: us-east1; see Regions and ZonesOpens in a new window. Can be a comma-separated list for mutliregion deployment. Azure — Example: Central US; see Azure geographiesOpens in a new window. (Use Location instead of Region.) Tencent — Example: na-siliconvalley (West US/Silicon Valley); see Regions and Availability ZonesOpens in a new window.	required	defaults
Zone	Availability zone within the specified region (see above) in which to locate a node or nodes to be provisioned. For information on deploying a single configuration in more than one zone, see Deploying Across Multiple Zones. Provider-specific information: AWS — Example: us-west-1c in region us-west-1; see Regions and ZonesOpens in a new window. GCP — Example: us-east1-b in region us-east1; see Regions and ZonesOpens in a new window. Azure — Example: 1 in region Central US; see What are Availability Zones in Azure?Opens in a new window. Note: Some Azure regions do not support availability zones. To deploy to such a regionset Zone to the empty string or omit it altogether: Tencent — Example: na-siliconvalley-1 in region na-siliconvalley; see Regions and Availability ZonesOpens in a new window.	required	defaults
ZoneMap	When deploying across multiple zones (see Deploying Across Multiple Zones), specifies which nodes are deployed in which zones. Default: 0,1,2,...,255.	optional	definitions
Mirror	If true, InterSystems IRIS instances on DATA, DM, and DS nodes are deployed as mirrors; see Mirrored Configuration Requirements. Default: false.	optional	defaults
MirrorMap	Determines mirror member types of mirrored DATA, DS, and DM nodes, enabling deployment of DR async mirror members; see Rules for Mirroring. Default: primary,backup; the term async can be added one or more times to this, for example primary,backupo,async,async.	optional	definitions
ISCPassword	Password that will be set for the predefined user accounts on the InterSystems IRIS instances on one or more provisioned nodes. Corresponding command-line option: -iscPassword. If both parameter and option are omitted, ICM prompts for the password. For more information see The icm run Command.	optional	defaults
Namespace	Namespace to be created on deployed InterSystems IRIS instances. This namespace is the default namespace for the icm session and icm sql commands, and can also be specified or overridden by the command-line option -namespace. Default: IRISCLUSTER.	optional	defaults
DockerImage	Docker image to be used for in deployment by icm run command. Must include the repository name (see RepositoriesOpens in a new window in the Docker documentation). Can be specified for all nodes in defaults.json and optionally overridden for specific node definitions in definitions.json. Can also be specified or overridden using the command-line option -image.	required
DockerRegistry	DNS name of the server hosting the Docker repository storing the image specified by DockerImage (see About RegistryOpens in a new window in the Docker documentation). If not included, ICM uses Docker’s public registry at docker.comOpens in a new window. For information about the InterSystems Container Registry (ICR), see Downloading the ICM Image in the “Using ICM” chapter.	required	defaults
DockerUsername	Username to use along with DockerPassword (below) for logging into the Docker repository specified in DockerImage (above) on the registry specified by DockerRegistry (above). Not required for public repositories. If not included and the repository specified by DockerImage is private, login fails.	required	defaults
DockerPassword	Password to use along with DockerUsername (above) for logging into the Docker registry. Not required for public repositories. If this field is not included and the repository specified by DockerImage is private. ICM prompts you (with masked input) for a password. (If the value of this field contains special characters such as $, |, (, and ), they must be escaped with two \ characters; for example, the password abc$def must be specified as abc\\$def.)	required	defaults
DockerVersion	Version of Docker installed on provisioned nodes. The version in each /Samples/.../defaults.json is generally correct for the platform; however. if your organization uses a different version of Docker, you may want that version installed on the nodes instead. Important: Container images from InterSystems comply with the Open Container Initiative (OCIOpens in a new window) specification, and are built using the Docker Enterprise Edition engine, which fully supports the OCI standard and allows for the images to be certifiedOpens in a new window and featured in the Docker Hub registry. InterSystems images are built and tested using the widely popular container Ubuntu operating system, and are therefore supported on any OCI-compliant runtime engine on Linux-based operating systems, both on premises and in public clouds.	optional	defaults
DockerURL	URL of the Docker Enterprise Edition repository associated with your subscription or trial; when provided, triggers installation of Docker Enterprise Edition on provisioned nodes, instead of Docker Community Edition. For more information about Docker EE see Docker EnterpriseOpens in a new window in the Docker documentation.	optional	defaults
Overlay	Determines the Docker overlay network type; normally "weave", but may be set to "host" for development, performance, or debug purposes, or when deploying on a preexisting cluster. Default: weave (host when deploying on a preexisting cluster). For more information see Use overlay networksOpens in a new window in the Docker documentation and How the Weave Net Docker Network Plugins WorkOpens in a new window in the Weave documentation.	optional	defaults
DockerStorageDriver	Determines the storage driver used by Docker (see Docker storage driversOpens in a new window in the Docker documentation). Values include overlay2 (the default) and btrfs. If set to overlay2, FileSystem (see below) must be set to xfs; if set to btrfs, FileSystem must be set to btrfs..	optional	defaults
FileSystem	Type of file system to use for persistent volumes on provisioned nodes. Valid values are xfs and btrfs. Default: xfs. If DockerStorageDriver (above) is set to overlay2, FileSystem must be set to xfs; if DockerStorageDriver is btrfs, FileSystem must be btrfs.	optional	defaults recommended
OSVolumeSize	Size (in GB) of the OS volume for a node or nodes in the deployment. Default: 32. May be limited by or ignored in favor of settings specific to the applicable parameters specifying machine image or template, instance type, or OS volume type parameters (see Provider-Specific Parameters).	optional
DataVolumeSize WIJVolumeSize Journal1VolumeSize Journal2VolumeSize	Size (in GB) of the corresponding persistent storage volume to create for iris containers. For example, DataVolumeSize determines the size of the data volume. Default: 10, although DataVolumeSize must be at least 60 for Tencent deployments. May be limited by the applicable volume type parameter (see Provider-Specific Parameters). Each volume also has a corresponding device name parameter (for example, DataDeviceName; see Device Name Parameters) and mount point parameter (for example, DataMountPoint; see immediately below and Storage Volumes Mounted by ICM).	optional
DataMountPoint WIJMountPoint Journal1MountPoint Journal2MountPoint	The location within iris containers at which the corresponding persistent volume is mounted. For example, DataMountPoint determines the location for the data volume. For more information, see Storage Volumes Mounted by ICM. Defaults: /irissys/{ data | wij | journal1j | journal2j }. Each volume also has a corresponding device name parameter (for example, DataDeviceName; see Device Name Parameters) and size parameter (for example, DataVolumeSize; see above).	optional
Containerless	If true, enables containerless mode, in which InterSystems IRIS is deployed from an installation kit rather than a container; see the appendix Containerless Deployment. Default: false.	optional	defaults
Role	Role of the node or nodes to be provisioned by a given entry in the definitions file, for example DM or DATA; see ICM Node Types.	required	definitions
Count	Number of nodes to provision from a given entry in the definitions file. Default: 1.	required	definitions
StartCount	Numbering start for a particular node definition in the definitions file. For example, if the DS node definition includes "StartCount": "3", the first DS node provisioned is named Label-DS-Tag-0003.	optional	definitions
LoadBalancer	If true in definitions of node type DATA, COMPUTE, AM, or WS, a predefined load balancer is automatically provisioned on providers AWS, GCP, Azure, and Tencent (see Predefined Load Balancer). If true in definitions of node type CN or VM, a generic load balancer is added if other parameters are included in the definition (see Generic Load Balancer). Default: false.	optional	definitions
AlternativeServers	Remote server selection algorithm for definitions of type WS (see Node Type: Web Server). Valid values are LoadBalancing and FailOver. Default: LoadBalancing.	optional	definitions
ApplicationPath	Application path to create for definitions of type WS. Do not include a trailing slash.	optional	definitions
IAMImage	InterSystems API Manager (IAM) image; no default.	optional	definitions
PostgresImage	Postgres image (optional IAM component); default: postgres:11.6.	optional	definitions
PrometheusImage	Prometheus image (System Alerting and Monitoring [SAM] component); default: prom/prometheus:v2.17.1.	optional	definitions
AlertmanagerImage	Alertmanager image (SAM component); default: prom/alertmanager:v0.20.0.	optional	definitions
GrafanaImage	Grafana image (SAM component); default: grafana/grafana:6.7.1.	optional	definitions
NginxImage	Nginx image (SAM component); default: nginx:1.17.9-alpine.	optional	definitions
UserCPF	Configuration merge file to be used to customize the CPFs InterSystems IRIS instances during deployment (see Deploying with Customized InterSystems IRIS Configurations).	optional
SystemMode	String to be shown in the masthead of the Management PortalOpens in a new window of the InterSystems IRIS instances on one or more provisioned nodes. Certain values (LIVE, TEST, FAILOVER, DEVELOPMENT) trigger additional changes in appearance. Default: blank. This setting can also be specified by adding [Startup]/SystemMode to the configuration merge file (see previous entry).	optional

Security-related ParametersCopy link to this section

The parameters in the following table are used to provide access and identify required files and information so that ICM can communicate securely with the provisioned nodes and deployed containers. They are all required, in the defaults file only.

• For information about using scripts provided with ICM to generate these files, see Obtain Security-Related Files in the “Using ICM” chapter.

• For information about how ICM uses the security files you provide to communicate securely with provisioned nodes and services on them, see ICM Security in this chapter

• For general information about using the SSH protocol, see SSH PROTOCOLOpens in a new window from SSH Communications Security.

• For information about Docker security. including the use of TLS certificates with Docker, see Docker securityOpens in a new window in the Docker documentation.

• For general information about using TLS with InterSystems IRIS, see InterSystems TLS GuideOpens in a new window and The InterSystems Public Key InfrastructureOpens in a new window. For information about the contents of the file identified by the SSLConfig parameter, see Create a Client ConfigurationOpens in a new window.

• For information about the use of TLS to secure connections between mirror members, see Securing Mirror Communication with TLS SecurityOpens in a new window in the High Availability Guide.

Port and Protocol ParametersCopy link to this section

Typically, the defaults for these parameters are sufficient. For information about two use cases in which you may need to specify some of these parameters, see Ports in the appendix “Using ICM with Custom and Third-Party Containers” and Ports in the appendix “Deploying on a Preexisting Cluster”.

* If ICM is in container mode (Containerless is false or absent) and Overlay is set to weave (see General Parameters), this port is closed in the node’s firewall.

CPF ParametersCopy link to this section

When using a configuration merge file specified by the UserCPF property to customize the CPF of one or more InterSystems IRIS instances during deployment, as described in Deploying with Customized InterSystems IRIS Configuration Parameters, you cannot include certain CPF settings, because ICM needs to read their values before it adds them to the CPF at a later stage. You should therefore customize these settings by specifying the following parameters (described in General Parameters and Port and Protocol Parameters) in your configuration files:

Provider-Specific ParametersCopy link to this section

This tables in this section list parameters used by ICM that are specific to each provider, as follows:

• Selecting Machine Images

• Amazon Web Serverices (AWS) Parameters

• Google Cloud Platform (GCP) Parameters

• Microsoft Azure (Azure) Parameters

• Tencent Cloud (Tencent) Parameters

• VMware vSphere (vSphere) Parameters

Some of the parameters listed are used with more than one provider; for example, the InstanceType, ElasticIP, and VPCId parameters can be used in both AWS and Tencent deployments. Some provider-specific parameters have different names but the same purpose, for example AMI and InstanceType for AWS, Image and MachineType for GCP, and ImageId and InstanceType for Tencent, whereas there are four Azure parameters corresponding to each of these.

Like the General Parameters table, the tables in this section indicate whether each parameter is required in every deployment or optional, and whether it must be included (when used) in either defaults.json or definitions.json, is recommended for one file or the other, or can be used in either. For examples of each type, see General Parameters.

Amazon Web Services (AWS) ParametersCopy link to this section

Parameter	Definition	Use is ...	Config file
Credentials	Path to a file containing the public/private keypair for an AWS account. To download, after logging into the AWS management console, open Managing Access Keys for IAM UsersOpens in a new window in the AWS documentation and follow the procedure for managing access keys in the AWS console.	required	defaults
AMI	AMI (machine image) to use as platform and OS template for nodes to be provisioned; see Amazon Machine Images (AMI)Opens in a new window in the AWS documentation. Example: ami-a540a5e1. To list public AMIs available, in the EC2 Console, select AMIs in the navigation pane and filter for Public AMIs.	required
InstanceType	Instance type to use as compute resources template for nodes to be provisioned on AWS and Tencent; see Amazon EC2 Instance TypesOpens in a new window in the AWS documentation. Example: m4.large. (Some instance types may not be compatible with some AMIs.)	required
ElasticIP	Enables the Elastic IP feature on AWS and Tencent to preserve IP address and domain name across host node restart (see Host Node Restart and Recovery). Default: false.	optional	defaults
VPCId	Existing Virtual Private Cloud (VPC) to be used in the deployment on AWS and Tencent, instead of allocating a new one; the specified VPC is not deallocated during unprovision. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new VPC is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network. Note: Internal parameter net_subnet_cidr must be provided if the VPC is not created in the default address space 10.0.%d.0/24; for example, for a VPC in the range 172.17.0.0/24, you would need to specify net_subnet_cidr as 172.17.%d.0/24.	optional	defaults
SubnetIds	When deploying on an existing private subnet on AWS or Tencent, comma-separated list of subnet IDs, one for each element specified by the Zone parameter (see General Parameters).	optional	defaults
RouteTableId	When deploying on an existing private subnet, theroute table to use for access to the ICM host; if provided, ICM uses this instead of allocating its own (and does not deallocate during unprovision). No default.	optional	defaults
InternetGatewayID	When deploying on an existing private subnet, the Internet gateway to use for access to the ICM host; if provided, ICM uses this instead of allocating its own (and does not deallocate during unprovision). No default.	optional	defaults
OSVolumeType	Determines disk type of the OS volume for a node or nodes in the deployment, which in turn determines the maximum value for the OSVolumeSize parameter (see General Parameters), which sets the size of the OS volume. See Amazon EBS Volume TypesOpens in a new window in the AWS documentation. Tencent uses the same parameter name. Default: standard.	optional
DataVolumeType WIJVolumeType Journal1VolumeType Journal2VolumeType	Determines disk type of the corresponding persistent storage volume for iris containers (see Storage Volumes Mounted by ICM), which in turn determines the maximum size of the volume. For example, DataVolumeType determines the maximum value for the DataVolumeSize parameter (see General Parameters), which detemines the size of the data volume. See Amazon EBS Volume TypesOpens in a new window in the AWS documentation. Tencent uses the same parameter name. Default: standard.	optional
OSVolumeIOPS	Determines IOPS count for the OS volume for a node or nodes in the deployment; see I/O Characteristics and MonitoringOpens in a new window in the AWS documentation. Default: 0.	optional
PlacementGroups	A comma-separated list of placement groups to create (see  Placement groupsOpens in a new window. If blank or omitted, no placement groups are created. Default: none.
PlacementStrategy	Strategy for placing instances in the groups specified by PlacementGroups (above). Valid values are cluster, partition, and spread. Default: cluster. Note: For placement strategy partition, Terraform does not currently allow users to specify the partition count, which is fixed at two per group, or to map instances to partitions.
PlacementMap	Specifies the mapping between the values of PlacementGroups and the nodes within a given definition. Instances will be assigned in the order in which they occur in PlacementGroups (with wraparound). Default: 0,1,2,3,...,256.
DataVolumeIOPS WIJVolumeIOPS Journal1VolumeIOPS Journal2VolumeIOPS	Determines IOPS count for the corresponding persistent storage volume for iris containers (see Storage Volumes Mounted by ICM). For example, DataVolumeIOPS determines the IOPS count for the data volume. See I/O Characteristics and MonitoringOpens in a new window in the AWS documentation. Must be nonzero when the corresponding volume type (see the immediately preceding) is io1. Default: 0.	optional

Google Cloud Platform (GCP) ParametersCopy link to this section

Parameter	Definition	Use is ...	Config file
Credentials	Path to a JSON file containing the service account key for a GCP account. To download, after logging into the GCP console and selecting a project, open Creating and managing service account keysOpens in a new window in the GCP documentation and follow the procedure for creating service account keys in the GCP console.	required	defaults
Project	GCP project ID; see Creating and Managing ProjectsOpens in a new window in the GCP documentation.	required	defaults
Image	Source machine image to use as platform and OS template for provisioned nodes; see ImagesOpens in a new window in the GCP documentation. Example: ubuntu-os-cloud/ubuntu-1804-bionic-v20190911.	required
MachineType	Machine type to use as compute resources template for nodes to be provisioned; see Machine typesOpens in a new window in the GCP documentation. Example: n1-standard-1.	required
RegionMap	When deploying across multiple regions (see Deploying Across Multiple Regions on GCP), specifies which nodes are deployed in which regions. Default: 0,1,2,...,255.	optional	definitions
Network	Existing Virtual Private Cloud (VPC) to be used in the deployment, instead of allocating a new one; the specified VPC is not deallocated during unprovision. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new VPC is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network.	optional	defaults
Subnet	Existing private subnet to be used in the deployment, instead of allocating a new one; not deallocated during unprovision. For multiregion deployments (see Deploying Across Multiple Regions on GCP), value must be a comma-separated list, one for each region specified. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new VPC is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network.	optional	defaults
OSVolumeType	Determines disk type for the OS volume for a node or nodes in the deployment; see Storage OptionsOpens in a new window in the GCP documentation. Default: pd-standard.	optional
DockerVolumeType	Determines disk type for the block storage device used for the Docker thin pool on a node or nodes in the deployment; see Storage OptionsOpens in a new window in the GCP documentation. Default: pd-standard.	optional
DataVolumeType	Determines disk type for the corresponding persistent storage volume for iris containers (see Storage Volumes Mounted by ICM). For example, DataVolumeType determines the disk type for the data volume. See Storage OptionsOpens in a new window in the GCP documentation. Default: pd-standard.	optional
WIJVolumeType
Journal1VolumeType
Journal2VolumeType

Microsoft Azure (Azure) ParametersCopy link to this section

Parameter	Definition	Use is ...	Config file
SubscriptionId	A unique alphanumeric string that identifies a Microsoft Azure subscription; to display, on the Azure portal select Subscriptions or type “subscriptions” into the search box, and use the Subscription ID displayed for SubscriptionId.	required	defaults
TenantId	A unique alphanumeric string that identifies the Azure Active Directory directory in which an application was created; to display, on the Azure portal select Azure Active Directory in the nav pane and then Properties on the nav pane for that page, and use the Directory ID displayed for TenantId.	required	defaults
ClientId	Credentials identifying and providing access to an Azure application; to creat them: Follow the procedure in Quickstart: Register an application with the Microsoft identity platformOpens in a new window to create a new application registration. Use the Application ID displayed on the App Registration tab for ClientId. Select Settings > Keys to generate a key and use the key value displayed for ClientSecret.	required	defaults
ClientSecret
Location	Region in which to provision a node or nodes; see the Region parameter in General Parameters.	required	defaults
LocationMap	When deploying across multiple regions (see Deploying Across Multiple Regions on Azure), specifies which nodes are deployed in which regions. Default: 0,1,2,...,255.	optional	definitions
PublisherName	Entity providing a given Azure machine image to use as platform and OS template for provisioned nodes. Example: OpenLogic.	required
Offer	Operating system of a given Azure machine image. Example: UbuntuServer.	required
Sku	Major version of the operating system of a given Azure machine image. Example: 7.2.	required
Version	Build version of a given Azure machine image. Example: 7.2.20170105.	required
CustomImage	Image to be used to create the OS disk, in place of the Azure machine image described by the PublisherName, Offer, Sku, and Version fields. Value is an Azure URI of the form: /subscriptions/subscription/resourceGroups/resource_group/providers /Microsoft.Compute/images/image_name	optional
Size	Machine size to use as compute resources template for nodes to be provisioned; see Sizes for virtual machines in AzureOpens in a new window in the Azure documentation. Example: Standard_DS1.	required
ResourceGroupName	Existing resource group to be used in the deployment, instead of allocating a new one; the specified group is not deallocated during unprovision. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new resource group is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network.	optional	defaults
VirtualNetworkName	Existing private subnet to be used in the deployment, instead of allocating a new one; not deallocated during unprovision. For multiregion deployments (see Deploying Across Multiple Regions on Azure), value must be a comma-separated list, one for each region specified. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new VPC is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network. Note: The net_subnet_cidr parameter (see Security-related Parameters) must be provided if the network is not created in the default address space 10.0.%d.0/24.	optional	defaults
SubnetName	Name of an existing subnet to be used in the deployment, instead of allocating a new one; not deallocated during unprovision. For multiregion deployments (see Deploying Across Multiple Regions on Azure), value must be a comma-separated list, one for each region specified. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new subnet is allocated for the deployment and deallocated during unprovision. Note: When provisioning on a private network, unique SubnetName and net_subnet_cidr parameters must be provided for each entry in the definitions file (but ResourceGroupName and VirtualNetworkName remain in the defaults file). This includes the bastion host definition when deploying a bastion host (see Deploy on a Private Network Through a Bastion Host).	optional	definitions
AccountTier	Storage account performance tier (see Azure storage account overviewOpens in a new window in the Azure documentation); either HDD (Standard) or SSD (Premium).	optional
AccountReplicationType	Storage account replication type: locally-redundant storage (LRS), geo-redundant storage (GRS), zone-redundant storage (ZRS), or read access geo-redundant storage (RAGRS).	optional

Tencent Cloud (Tencent) ParametersCopy link to this section

Parameter	Definition	Use is ...	Config file
SecretID SecretKey	Unique alphanumeric strings that identify and provide access to a Tencent Cloud account. To download, open SignatureOpens in a new window in the Tencent Cloud documentation and follow the procedure in “Applying for Security Credentials”.	required	defaults
ImageId	Machine image to use as platform and OS template for provisioned nodes; see Image OverviewOpens in a new window in the Tencent documentation. Example: img-pi0ii46r.	required (see below)
OSName	If ImageId (above) is not provided, ICM searches for an image matching this field. Note that this field supports regexp. Default: ubuntu.	required (see above)
InstanceFamily	Instance family from which to select instance type; if InstanceType (below) is not provided, ICM searches for an instance type matching InstanceFamily, CPUCoreCount, and MemorySize (below). Default: S3.	required (see below)
InstanceType	Instance type to use as compute resources template for nodes to be provisioned on AWS and Tencent; see Instance TypesOpens in a new window in the Tencent documentation. Example: S2.MEDIUM4.	required (see above)
ElasticIP	Enables the Elastic IP feature on AWS and Tencent to preserve IP address and domain name across host node restart (see Host Node Restart and Recovery). Default: false.	optional	defaults
VPCId	Existing Virtual Private Cloud (VPC) to be used in the deployment on AWS and Tencent, instead of allocating a new one; the specified VPC is not deallocated during unprovision. If not specified when PrivateSubnet (see Security-related Parameters) is true, a new VPC is allocated for the deployment and deallocated during unprovision. For more information, see Deploying Within an Existing Private Network. Note: Internal parameter net_subnet_cidr must be provided if the VPC is not created in the default address space 10.0.%d.0/24; for example, for a VPC in the range 172.17.0.0/24, you would need to specify net_subnet_cidr as 172.17.%d.0/24.	optional	defaults
SubnetIds	When deploying on an existing private subnet on AWS or Tencent, comma-separated list of subnet IDs, one for each element specified by the Zone parameter (see General Parameters).	optional	defaults
CPUCoreCount	CPU core to match when selecting instance type; if InstanceType (above) is not provided, ICM searches for an instance type matching InstanceFamily, CPUCoreCount, and MemorySize (above). Default: 2.	optional
MemorySize	Memory size to match when selecting instance type; if InstanceType (above) is not provided, ICM searches for an instance type matching InstanceFamily, CPUCoreCount, and MemorySize (above). Default: 4 GB.	optional
OSVolumeType	Determines disk type for the OS volume for a node or nodes in the deployment; see Data Types: DataDiskOpens in a new window in the Tencent documentation. AWS uses the same parameter name. Default: CLOUD_BASIC.	optional
DockerVolumeType	Determines disk type for the block storage device used for the Docker thin pool on a node or nodes in the deployment; see Data Types: DataDiskOpens in a new window in the Tencent documentation. AWS uses the same parameter name. Default: CLOUD_BASIC.	optional
DataVolumeType	Determines disk type for the corresponding persistent storage volume for iris containers (see Storage Volumes Mounted by ICM). For example, DataVolumeType determines the disk type for the data volume. AWS uses the same parameter names. See Data Types: DataDiskOpens in a new window in the Tencent documentation. Default: CLOUD BASIC.	optional
WIJVolumeType
Journal1VolumeType
Journal2VolumeType

VMware vSphere (vSphere) ParametersCopy link to this section

Parameter	Definition	Use is ...	Config file
Server	Name of the vCenter server. Example: tbdvcenter.iscinternal.com.	required	defaults
Datacenter	Name of the datacenter.	required	defaults
DatastoreCluster	Collection of datastores where virtual machine files will be stored; see Creating a Datastore ClusterOpens in a new window in the VMware documentation. Example: DatastoreCluster1.	required	defaults
VSphereUser	Credentials for vSphere operations; see About vSphere AuthenticationOpens in a new window in the VMware documentation.	required	defaults
VSpherePassword
DNSServers	List of DNS servers for the virtual network. Example: 172.16.96.1,172.17.15.53	required	defaults
DNSSuffixes	List of name resolution suffixes for the virtual network adapter. Example: iscinternal.com	required	defaults
Domain	FQDN for a node or nodes to be provisioned. Example: iscinternal.com	required	defaults
NetworkInterface	Label to assign to a network interface. Example: VM Network	optional	defaults
ResourcePool	Name of a vSphere resource pool; see Managing Resource PoolsOpens in a new window in the VMware documentation. Example: ResourcePool1.	optional	defaults
Template	Virtual machine master copy (machine image) to use as platform and OS template for nodes to be provisioned. Example: ubuntu1804lts	required
VCPU	Number of CPUs in a node or nodes to be provisioned. Example: 2.	optional
Memory	Amount of memory (in MB) in a node or nodes to be provisioned. Example: 4096.	optional
GuestID	Guest ID for the operating system type. See Enum - VirtualMachineGuestOsIdentifierOpens in a new window on the VMware support website. Default: other3xLinux64Guest.	optional
WaitForGuestNetTimeout	Time (in minutes) to wait for an available IP address on a virtual machine. Default: 5.	optional
ShutdownWaitTimeout	Time (in minutes) to wait for graceful guest shutdown when making necessary updates to a virtual machine. Default: 3.	optional
MigrateWaitTimeout	Time (in minutes) to wait for virtual machine migration to complete. Default: 10.	optional
CloneTimeout	Time (in minutes) to wait for virtual machine cloning to complete. Default: 30.	optional
CustomizeTimeout	Time (in minutes) that Terraform waits for customization to complete. Default: 10.	optional
DiskPolicy	Disk provisioning policy for the deployment (see About Virtual Disk Provisioning PoliciesOpens in a new window in the VMware documentation). Values are: thin — Thin Provision lazy — Thick Provision Lazy Zeroed eagerZeroedThick — Thick Provision Eager Zeroed Default: lazy.	optional
SDRSEnabled	If specified, determines whether Storage DRS (see Enable and Disable Storage DRSOpens in a new window in the VMware documentation) is enabled for a virtual machine; otherwise, use current datastore cluster settings. Default: Current datastore cluster settings.	optional
SDRSAutomationLevel	If specified, determines Storage DRS automation level for a virtual machine; otherwise, use current datastore cluster settings. Values are automated or manual. Default: Current datastore cluster settings.	optional
SDRSIntraVMAffinity	If provided, determines Intra-VM affinity setting for a virtual machine (see Override VMDK Affinity RulesOpens in a new window in the VMware documentation); otherwise, use current datastore cluster settings. Values include: true — All disks for this virtual machine will be kept on the same datastore. false — Storage DRS may locate individual disks on different datastores if it helps satisfy cluster requirements. Default: Current datastore cluster settings.	optional
SCSIControllerCount	Number of SCSI controllers for a given host node; must be between 1 and 4. The OS volume is always be placed on the first SCSI controller. vSphere may not be able to create more SCSI controllers than were present in the template specified by the Template field. Default: 1	optional
DockerVolumeSCSIController	SCSI controller on which to place the Docker volume. Must be between 1 and 4 and may not exceed SCSIControllerCount. Default: 1	optional
DataVolumeSCSIController	SCSI controller on which to place the corresponding volume in iris containers; for example, DataVolumeSCSIController determines the controller for data volume. Must be between 1 and 4 and may not exceed SCSIControllerCount. Default: 1	optional
WIJVolumeSCSIController
Journal1VolumeSCSIController
Journal2VolumeSCSIController

Note:

The requirements for the VMware vSphere template specified by the Template property are similar to those described in Host Node Requirements in the appendix “Deploying on a Preexisting Cluster” (for example, passwordless sudo access).

To address the needs of the many users who rely on VMware vSphere, it is supported by this release of ICM. Depending on your particular vSphere configuration and underlying hardware platform, the use of ICM to provision virtual machines may entail additional extensions and adjustments not covered in this guide, especially for larger and more complex deployments, and may not be suitable for production use. Full support is expected in a later release.

Device Name ParametersCopy link to this section

The parameters listed in the following specify the device files under /dev that represent the persistent volumes created by ICM for use by InterSystems IRIS. For information about these persistent volumes and a table of provider and OS-specific default values for these parameters, see Storage Volumes Mounted by ICM. For PreExisting deployments, see Storage Volumes in the “Deploying on a Preexisting Cluster” appendix.

Alphabetical List of User ParametersCopy link to this section

The following table lists all of the parameters discussed in the preceding tables in this section in alphabetical order, with links to the table(s) containing their definition.

Role DATA: Sharded Cluster Data NodeCopy link to this section

As described in Overview of InterSystems IRIS ShardingOpens in a new window and other sections of the “Horizontally Scaling for Data Volume with Sharding” chapter of the Scalability Guide, a typical sharded cluster consists only of data nodes, across which the sharded data is partitioned, and therefore requires only a DATA node definition in the definitions.json file. If DATA nodes are defined, the deployment must be a sharded cluster, and the only other node type that can be defined with them is COMPUTE.

DATA nodes can be mirrored if provisioned in a number matching the MirrorMap setting in their definition, as described in Rules for Mirroring. The DATA nodes in a cluster must be either all mirrored or all nonmirrored.

The only distinction between data nodes in a sharded cluster is that the first node configured (known as node 1) stores all of the nonsharded data, metadata, and code for the cluster in addition to its share of the sharded data. The difference in storage requirements, however, is typically very small. Because all data, metadata, and code is visible on any node in the cluster, application connections can be load balanced across all of the nodes to take greatest advantage of parallel query processing and partitioned caching. A load balancer may be assigned to DATA nodes; see Role LB: Load Balancer.

Role COMPUTE: Sharded Cluster Compute NodeCopy link to this section

For advanced use cases in which extremely low query latencies are required, potentially at odds with a constant influx of data, compute nodes can be added to a sharded cluster to provide a transparent caching layer for servicing queries, separating the query and data ingestion workloads and improving the performance of both. (For more information see Deploy Compute Nodes for Workload Separation and Increased Query ThroughputOpens in a new window in the Scalability Guide.)

Adding compute nodes yields significant performance improvement only when there is at least one compute node per data node, so you should define at least as many COMPUTE nodes as DATA nodes; if the number of DATA nodes in the definitions file is greater than the number of COMPUTE nodes, ICM issues a warning. Configuring multiple compute nodes per data node can further improve the cluster’s query throughput, and the recommended best practice when doing so is to configure the same number of compute nodes for each data node, so ICM distributes the defined COMPUTE nodes as evenly as possible across the DATA nodes.

Because COMPUTE nodes support query execution only and do not store any data, their instance type and other settings can be tailored to suit those needs, for example by emphasizing memory and CPU and keeping storage to the bare minimum. Because they do not store data, COMPUTE nodes cannot be mirrored.

A load balancer may be assigned to COMPUTE nodes; see Role LB: Load Balancer.

Role DM: Distributed Cache Cluster Data Server, Standalone Instance, Shard Master Data ServerCopy link to this section

If multiple nodes of role AM and a DM node (nonmirrored or mirrored) are specified, they are deployed as an InterSystems IRIS distributed cache cluster, with the former serving as application servers and the latter as an data server.

A node of role DM (nonmirrored or mirrored) deployed by itself becomes a standalone InterSystems IRIS instance.

If a DM node (mirrored or nonmirrored), DS nodes (mirrored or nonmirrored), and (optionally) QS nodes are specified, they are deployed as a namespace-levelOpens in a new window sharded cluster.

Role DS: Shard Data ServerCopy link to this section

Under the namespace-levelOpens in a new window architecture, a data shard stores one horizontal partition of each sharded table loaded into a sharded cluster. A node hosting a data shard is called a shard data server. A cluster can have two or more shard data servers up to over 200. Shard data servers can be mirrored by deploying an even number and specifying mirroring.

Role QS: Shard Query ServerCopy link to this section

Under the namespace-level architecture, shard query servers provides query access to the data shards to which they are assigned, minimizing interference between query and data ingestion workloads and increasing the bandwidth of a sharded cluster for high volume multiuser query workloads. If shard query servers are deployed they are assigned round-robin to the deployed shard data servers. Shard query servers automatically redirect application connections when a mirrored shard data server fails over.

Role AM: Distributed Cache Cluster Application ServerCopy link to this section

If multiple nodes of role AM and a DM node are specified, they are deployed as an InterSystems IRIS distributed cache cluster, with the former serving as application servers and the latter as a data server. When the data server is mirrored, application connection redirection following failover is automatic.

A load balancer may be assigned to AM nodes; see Role LB: Load Balancer.

Role AR: Mirror ArbiterCopy link to this section

When DATA nodes (sharded cluster DATA nodes), a DM node (distributed cache cluster data server, stand-alone InterSystems IRIS instance, or namespace-level shard master data server), or DS nodes (namespace-level shard data servers) are mirrored, deployment of an arbiter node to facilitate automatic failover is highly recommended. One arbiter node is sufficient for all of the mirrors in a cluster; multiple arbiters are not supported and are ignored by ICM, as are arbiter nodes in a nonmirrored cluster.

The AR node does not contain an InterSystems IRIS instance, using a different image to run an ISCAgent container. This arbiter image must be specified using the DockerImage field in the definitions file entry for the AR node; for more information, see The icm run Command.

For more information about the arbiter, see the “MirroringOpens in a new window” chapter of the High Availability Guide.

Role WS: Web ServerCopy link to this section

A deployment may contain any number of web servers. Each web server node contains an InterSystems Web Gateway installation along with an Apache web server. ICM populates the remote server list in the InterSystems Web Gateway as follows:

• If DATA and COMPUTE nodes are deployed (node-level sharded cluster), all of the DATA and COMPUTE nodes, or all of the DATA nodes if no COMPUTE nodes are deployed.

• If AM nodes are deployed (distributed cache cluster), all of the AM nodes.

• Otherwise, the DM node (standalone instance or namespace-level sharded cluster).

  Note:

  If deploying a namespace-level sharded cluster with a web server tier, you can manually deploy a custom or third-party load balancer to distribute connections across the DS and (if they exist) QS nodes of the cluster (as recommended in Deploying the Namespace-level ArchitectureOpens in a new window in the Scalability Guide) and manually edit the Web Gateway configurations to populate the remote server lists with the load balancer address (for more information, see Web Gateway Operation and ConfigurationOpens in a new window in the Web Gateway Configuration Guide).

For mirrored DATA and DM nodes, a mirror-aware connection is created, and application connection redirection following failover is automatic. Communication between the web server and the remote servers is configured to run in TLS mode.

A load balancer may be assigned to WS nodes; see Role LB: Load Balancer.

The WS node does not contain an InterSystems IRIS instance, using a different image to run a Web Gateway container. As described in The icm run Command, the webgateway image can be specified by including the DockerImage field in the WS node definition in the definitions.json file, for example:

{
    "Role": "WS",
    "Count": "3",
    "DockerImage": "intersystems/webgateway:2021.1.0.205.0",
    "ApplicationPath": "/acme",
    "AlternativeServers": "LoadBalancing"
}

Copy code to clipboard

If the ApplicationPath field is provided, its value is used to create an application path for each instance of the Web Gateway. The default server for this application path is assigned round-robin across Web Gateway instances, with the remaining remote servers making up the alternative server pool. For example, if the preceding sample WS node definition were part of a distributed cache cluster with three AM nodes, the assignments would be like the following:

The AlternativeServers field determines how the Web Gateway distributes requests to its target server pool. Valid values are LoadBalancing (the default) and FailOver. This field has no effect if the ApplicationPath field is not specified.

For information about using the InterSystems Web Gateway, see the Web Gateway Configuration GuideOpens in a new window.

Role SAM: System Alerting and Monitoring NodeCopy link to this section

Defining a SAM node adds the System Alerting and Monitoring (SAM) cluster monitoring solution to a deployment. For information about adding SAM, see Monitoring in ICM; for complete information about SAM, see the System Alerting and Monitoring GuideOpens in a new window.

Role LB: Load BalancerCopy link to this section

ICM automatically provisions a predefined load balancer node when the provisioning platform is AWS, GCP, Azure, or Tencent, and the definition of nodes of type DATA, COMPUTE, AM, or WS in the definitions file sets LoadBalancer to true. For a generic load balancer for VM or CN nodes, additional parameters must be provided.

Predefined Load BalancerCopy link to this section

For nodes of role LB, ICM configures the ports and protocols to be forwarded as well as the corresponding health checks. Queries can be executed against the deployed load balancer the same way one would against a data node in a sharded cluster or a distributed cache cluster application server.

To add a load balancer to the definition of DATA, COMPUTE, AM, or WS nodes, add the LoadBalancer field, for example:

{
    "Role": "AM",
    "Count": "2",
    "LoadBalancer": "true"
}

Copy code to clipboard

The following example illustrates the nodes that would be created and deployed given this definition:

$ icm inventory
Machine           IP Address    DNS Name                              Region   Zone
-------           ----------    --------                              ------   ----
ANDY-AM-TEST-0001 54.214.230.24 ec2-54-214-230-24.amazonaws.com       us-west1 c
ANDY-AM-TEST-0002 54.214.230.25 ec2-54-214-230-25.amazonaws.com       us-west1 c
ANDY-LB-TEST-0000 (virtual AM)  ANDY-AM-TEST-1546467861.amazonaws.com us-west1 c

Copy code to clipboard

Queries against this cluster can be executed against the load balancer the same way they would be against the AM nodes.

The LoadBalancer field can be added to more than one definition in a deployment; for example a distributed cache cluster can contain AM nodes receiving load-balanced connections from a WS tier that receives load-balanced application connections. Currently, a single automatically provisioned load balancer cannot serve multiple node types (for example, both DATA and COMPUTE nodes), so each requires its own load balancer. This does not preclude the user from manually deploying a custom or third-party load balancer to serve the desired roles.

Generic Load BalancerCopy link to this section

A load balancer can be added to VM (virtual machine) and CN (container) nodes by providing the following additional keys:

• ForwardProtocol

• ForwardPort

• HealthCheckProtocol

• HealthCheckPath

• HealthCheckPort

The following is an example:

{
    "Role": "VM",
    "Count": "2",
    "LoadBalancer": "true",
    "ForwardProtocol": "tcp",
    "ForwardPort": "443",
    "HealthCheckProtocol": "http",
    "HealthCheckPath": "/csp/status.cxw",
    "HealthCheckPort": "8080"
}

Copy code to clipboard

More information about these keys can be found in the Ports and Protocol Parameters table in “ICM Configuration Parameters”.

Note:

A load balancer does not require (or allow) an explicit entry in the definitions file.

Some cloud providers create a DNS name for the load balancer that resolves to multiple IP addresses; for this reason, the value displayed by the provider interface as DNS Name should be used. If a numeric IP address appears in the DNS Name column, it simply means that the given cloud provider assigns a unique IP address to their load balancer, but doesn't give it a DNS name.

Because the DNS name may not indicate to which resources a given load balancer applies, the values displayed under IP Address are used for this purpose.

Load balancers on different cloud providers may behave differently; be sure to acquaint yourself with load balancer details on platforms you provision on.

Avoid provisioning a load balancer for mirrored DATA nodes on Tencent; load balancers provisioned on Tencent are not currently able to determine which side of a mirrored DATA node is primary, which could result in errors performing read/write operations through the load balancer. 

For providers VMware and PreExisting, you may wish to deploy a custom or third-party load balancer.

Role VM: Virtual Machine NodeCopy link to this section

A cluster may contain any number of virtual machine nodes. A virtual machine node provides a means of allocating host nodes which do not have a predefined role within an InterSystems IRIS cluster. Docker is not installed on these nodes, though users are free to deploy whatever custom or third-party software (including Docker) they wish.

The following commands are supported on the virtual machine node:

• icm provision

• icm unprovision

• icm inventory

• icm ssh

• icm scp

A load balancer may be assigned to VM nodes; see Role LB: Load Balancer.

Role CN: Container NodeCopy link to this section

A cluster may contain any number of container nodes. A container node is a general purpose node with Docker installed.

You can add InterSystems API Manager (IAM) to any deployment by defining a CN node and including the IAM and IAMImage fields; for more information, see Deploying InterSystems API Manager (IAM) in the “ICM Reference” chapter. You can also deploy any custom and third-party containers you wish on a CN node; iris (InterSystems IRIS) containers will not be deployed if specified. All ICM commands are supported for container nodes, but most will be filtered out unless they use the -container option to specify a container other than iris, or either the -role or -machine option to limit the command to CN nodes (see ICM Commands and Options).

A load balancer may be assigned to CN nodes; see Role LB: Load Balancer. CN nodes cannot be deployed in containerless mode.

Role BH: Bastion HostCopy link to this section

You may want to deploy a configuration that offers no public network access. If you have an existing private network, you can launch ICM on a node on that network and deploy within it. If you do not have such a network, you can have ICM configure a private subnet and deploy your configuration on it. Since ICM is not running within that private subnet, however, it needs a means of access to provision, deploy, and manage the configuration. The BH node serves this purpose.

A bastion host is a host node that belongs to both the private subnet configured by ICM and the public network, and can broker communication between them. To use one, you define a single BH node in your definitions file and set PrivateSubnet to true in your defaults file. For more information, see Deploying on a Private Network.

Rules for MirroringCopy link to this section

All data nodes in a sharded cluster must be mirrored, or all unmirrored. This requirement is reflected in the following ICM topology validation rules.

When the Mirror field is set to false in the defaults file (the default), mirroring is never configured, and provisioning fails if more than one DM node is specified in the definitions file.

When the Mirror field is set to true, mirroring is configured where possible, and the mirror roles of the DATA, DS, or DM nodes (primary, backup, or DR async) are determined by the value of the MirrorMap field (see General Parameters) in the node definition, as follows:

• If MirrorMap is not included in the relevant node definition, the nodes are configured as mirror failover pairs using the default MirrorMap value, primary,backup:

  • If an even number of DATA or DS nodes is defined, they are all configured as failover pairs; for example, specifying six DATA nodes deploys three data node mirrors containing failover pairs and no DR asyncs. If an odd number of DATA or DS nodes is defined, provisioning fails.

  • If two DM nodes are defined, they are configured as a failover pair; if any other number is defined, provisioning fails.

• If MirrorMap is included in the node definition, the nodes are configured according to its value, as follows:

  • The number of DATA or DS nodes must be a multiple of the number of roles specified in the MirrorMap value or fewer. For example, suppose the MirrorMap value, is primary,backup,async, as shown:

    "Role": "DATA",
    "Count": "",
    "MirrorMap": "primary,backup,async"
    

    Copy code to clipboard

    In this case, DATA or DS nodes would be configured as follows:

    Value of Count	Result
    3 or multiples of 3	One or more mirrors containing a failover pair and a DR async
    2	A single mirror containing a failover pair
    1, 4 or more but not multiples of 3	Provisioning fails

  • The number of DM nodes must be the same as the number of roles specified in the MirrorMap value or fewer; if a single DM node is specified, provisioning fails.

• If more than one AR (arbiter) node is specified, provisioning fails. (While a best practice, use of an arbiter is optional, so an AR node need not be included in a mirrored configuration.)

All asyncs deployed by ICM are DR asyncs; reporting asyncs are not supported. Up to 14 asyncs can be included in a mirror. For information on mirror members and possible configurations, see Mirror ComponentsOpens in a new window in the High Availability Guide.

There is no relationship between the order in which DATA, DS, or DM nodes are provisioned or configured and their roles in a mirror. Following provisioning, you can determine which member of each pair is the intended primary failover member and which the backup using the icm inventory command. To see the mirror member status of each node in a deployed configuration when mirroring is enabled, use the icm ps command.

Nonmirrored Configuration RequirementsCopy link to this section

A nonmirrored cluster consists of the following:

• One or more DATA (data nodes in a sharded cluster).

• If DATA nodes are included, zero or more COMPUTE (compute nodes in a sharded cluster); best practices are at least as many COMPUTE nodes as DATA nodes and the same number of COMPUTE nodes for each DATA node.

• If no DATA nodes are included:

  • Exactly one DM (distributed cache cluster data server, standalone InterSystems IRIS instance, shard master data server in namespace-levelOpens in a new window sharded cluster).

  • Zero or more AM (distributed cache cluster application server).

  • Zero or more DS (shard data servers in namespace-level sharded cluster).

  • Zero or more QS (shard query servers in namespace-level sharded cluster).

• Zero or more WS (web servers).

• Zero or more LB (load balancers).

• Zero or more VM (virtual machine nodes).

• Zero or more CN (container nodes).

• Zero or one BH (bastion host).

• Zero AR (arbiter node is for mirrored configurations only).

The relationships between some of these nodes types are pictured in the following examples.

Mirrored Configuration RequirementsCopy link to this section

A mirrored cluster consists of:

• If DATA nodes (data nodes in a node-level sharded cluster) are included:

  • A number of DATA matching the MirrorMap value, default or explicit, as described in Rules for Mirroring.

  • Zero or more COMPUTE (compute nodes in a node-level sharded cluster); best practices are at least one COMPUTE node per DATA node mirror, and the same number of COMPUTE nodes for each DATA node mirror.

• If no DATA nodes are included:

  • Two DM as a mirrored shard master data server in a namespace-level sharded cluster, data server in a distributed cache cluster, or standalone InterSystems IRIS instance, or more than two if DR asyncs are specified by the MirrorMap field, as described in Rules for Mirroring.

  • If a namespace-level sharded cluster:

    • A number of DS (shard data servers) matching the MirrorMap value, default or explicit, as described in Rules for Mirroring.

    • Zero or more QS (shard query servers), as described in the foregoing for COMPUTE nodes.

  • Zero or more AM as application servers in a distributed cache cluster.

• Zero or one AR (arbiter node is optional but recommended for mirrored configurations).

• Zero or more WS (web servers).

• Zero or more LB (load balancers).

• Zero or more VM (virtual machine nodes).

• Zero or more CN (container nodes).

• Zero or one BH (bastion host).

The following fields are required for mirroring:

• Mirroring is enabled by setting key Mirror in your defaults.json file to true.

  "Mirror": "true"

  Copy code to clipboard

• To include DR asyncs in DATA, DS, or DM mirrors, you must include the MirrorMap field in your definitions file to specify that those beyond the first two are DR async members. The value of MirrorMap must always begin with primary,backup, for example:

  "Role": "DM",
  "Count": "5”,
  "MirrorMap": "primary,backup,async,async,async",
  ...

  Copy code to clipboard

  For information on the relationship between the MirrorMap value and the number of DATA, DS, or DM nodes defined, see Rules for Mirroring. MirrorMap can be used in conjunction with the Zone and ZoneMap fields to deploy async instances across zones; see Deploying Across Multiple Zones.

Automatic LB deployment (see Role LB: Load Balancer) is supported for providers AWS, GCP, Azure, and Tencent; when creating your own load balancer, the pool of IP addresses to include are those of DATA, COMPUTE, AM, or WS nodes, as called for by your configuration and application.

The relationships between some of these nodes types are pictured in the following examples.

Host Node CommunicationCopy link to this section

A host node is the host machine on which containers are deployed. It may be virtual or physical, running in the cloud or on-premises.

ICM uses SSH to log into host nodes and remotely execute commands on them, and SCP to copy files between the ICM container and a host node. To enable this secure communication, you must provide an SSH public/private key pair and specify these keys in the defaults.json file as SSHPublicKey and SSHPrivateKey. During the configuration phase, ICM disables password login on each host node, copies the private key to the node, and opens port 22, enabling clients with the corresponding public key to use SSH and SCP to connect to the node.

Other ports opened on the host machine are covered in the sections that follow.

DockerCopy link to this section

During provisioning, ICM downloads and installs a specific version of Docker from the official Docker web site using a GPG fingerprint. ICM then copies the TLS certificates you provide (located in the directory specified by the TLSKeyDir field in the defaults file) to the host machine, starts the Docker daemon with TLS enabled, and opens port 2376. At this point clients with the corresponding certificates can issue Docker commands to the host machine.

Weave NetCopy link to this section

During provisioning, ICM launches Weave Net with options to encrypt traffic and require a password (provided by the user) from each machine joining the Weave network. To enable these options, set WeavePassword to the any value other than null in the defaults.json file.

InterSystems IRISCopy link to this section

For a comprehensive overview of InterSystems IRIS security, see About InterSystems SecurityOpens in a new window.

Private NetworksCopy link to this section

ICM can deploy on an existing private network (not accessible from the Internet) if you configure the access it requires. ICM can also create a private network on which to deploy and configure its own access through a bastion host. For more information on using private networks, see Deploying on a Private Network.

Deploying Across Multiple Regions on GCPCopy link to this section

To deploy across multiple regions on GCP, specify the desired regions as a comma-separated list in the Region field in the defaults file, as shown:

{
    "Provider": "GCP",
    "Label": "Acme",
    "Tag": "multi",
    "Region": "us-east1,us-west1",
    "Zone": "us-east1-b,us-west1-a",
    ...
}

Copy code to clipboard

By default, nodes within each definition are assigned a region in round-robin fashion. For example, suppose you are deploying with the fields shown above in defaults.json and the following definitions.json:

[
  {
    "Role": "DATA",
    "Count": "2"
  },
  {
    "Role": "AR",
    "Count": "1",
    "DockerImage": "intersystems/arbiter:2021.1.0.205.0"
  }
]

Copy code to clipboard

In this case, the output of the icm inventory command might look like this:

$ icm inventory
Machine               IP Address    DNS Name              Region    Zone
-------               ----------    --------              ------    ----
Acme-AR-multi-0001    35.179.173.90 acmear1.google.com    us-east1     b
Acme-DATA-multi-0001- 35.237.131.39 acmedata1.google.com  us-east1     b
Acme-DATA-multi-0002+ 35.233.223.64 acmedata2.google.com  us-west1     a

Copy code to clipboard

For control over the regions that nodes are deployed in, you can use the RegionMap field to map the defined nodes to the specified regions. When RegionMap is included in a node definition, ZoneMap (described in the preceding section, Deploying Across Multiple Zones) must also be included to map the node or nodes to the desired zone or zones. For example, suppose you are deploying a mirror containing a failover pair and a DR async with an arbiter, and you want the failover pair in one region but in different zones, and the async and arbiter in a different region and also in different zones. The files you might use and the output you might see from the icm inventory and icm ps commands are shown in the following:

defaults.json

{
    "Provider": "GCP",
    "Label": "Acme",
    "Tag": "multi",
    "Region": "us-east1,us-west1",
    "Zone": "us-east1-a,us-east1-b,us-west1-a,us-west1-b",
    ...
}

Copy code to clipboard

definitions.json

[
  {
    "Role": "DATA",
    "Count": "3",
    "MirrorMap": "primary,backup,async",
    "RegionMap": "1,1,2",
    "ZoneMap": "1,2,3",
  },
  {
    "Role": "AR",
    "Count": "1",
    "DockerImage": "intersystems/arbiter:2021.1.0.205.0",
    "RegionMap": "2",
    "ZoneMap": "4"
  }
]

Copy code to clipboard

icm inventory

Machine               IP Address    DNS Name              Region    Zone
-------               ----------    --------              ------    ----
Acme-AR-multi-0001    35.179.173.90 acmear1.google.com    us-west1     b
Acme-DATA-multi-0001+ 35.237.131.39 acmedata1.google.com  us-east1     b
Acme-DATA-multi-0002- 35.233.223.64 acmedata2.google.com  us-east1     a
Acme-DATA-multi-0003  35.166.127.82 acmedata3.google.com  us-west1     a

Copy code to clipboard

icm ps

Machine              IP Address    Container Status Health  Mirror    Image
-------              ----------    --------- ------ ------  ------    -----
Acme-AR-multi-0001   35.179.173.90 arbiter   Up     healthy           intersystems/arbiter:2021.1.0.205.0
Acme-DATA-multi-0001 35.237.131.39 iris      Up     healthy PRIMARY   intersystems/iris:2021.1.0.205.0
Acme-DATA-multi-0002 35.233.223.64 iris      Up     healthy BACKUP    intersystems/iris:2021.1.0.205.0
Acme-DATA-multi-0003 35.166.127.82 iris      Up     healthy CONNECTED intersystems/iris:2021.1.0.205.0

Copy code to clipboard

To use the Network field (see Google Cloud Platform (GCP) Parameters) to specify an existing network to use in a multiregion deployment, you must also use the GCP Subnet field to specify a unique subnet for each region specified by the Region field. For example, for a deployment on regions us-west1 and us-east1, as illustrated here, you might include the following in your defaults file:

"Network": "acme-network",
"Subnet": "acme-subnet-data-east,acme-subnet-data-west"

Copy code to clipboard

Deploying Across Multiple Regions on AzureCopy link to this section

To deploy across multiple regions on Azure, specify the desired regions as a comma-separated list in the Location field in the defaults file, as shown:

{
    "Provider": "Azure",
    "Label": "Acme",
    "Tag": "multi",
    "Location": "East US,Central US",
    ...
}

Copy code to clipboard

By default, nodes within each definition are assigned a location in round-robin fashion. For example, suppose you are deploying with the fields shown above in defaults.json and the following definitions.json:

[
  {
    "Role": "DATA",
    "Count": "2"
  },
  {
    "Role": "AR",
    "Count": "1",
    "DockerImage": "intersystems/arbiter:2021.1.0.205.0"
  }
]

Copy code to clipboard

In this case, the output of the icm inventory command might look like this:

$ icm inventory
Machine               IP Address    DNS Name             Region    Zone
-------               ----------    --------             ------    ----
Acme-AR-multi-0001    35.179.173.90 acmear1.azure.com    East US    1
Acme-DATA-multi-0001- 35.237.131.39 acmedata1.azure.com  East US    1
Acme-DATA-multi-0001+ 35.233.223.64 acmedata2.azure.com  Central US 1

Copy code to clipboard

For control over the regions that nodes are deployed in, you can use the LocationMap field to map the defined nodes to the specified regions. When LocationMap is included in a node definition, ZoneMap (described in the preceding section, Deploying Across Multiple Zones) must also be included to map the node or nodes to the desired zone or zones. For example, suppose you are deploying a mirror containing a failover pair and a DR async with an arbiter, and you want the failover pair in one region but in different zones, and the async and arbiter in a different region and also in different zones. The files you might use and the output you might see from the icm inventory and icm ps commands are shown in the following. (Note that Azure zones are identified by the same integers in every region, so ZoneMap identifies only the desired zone within whatever region is specified by LocationMap.)

defaults.json

{
    "Provider": "Azure",
    "Label": "Acme",
    "Tag": "multi",
    "Location": "East US,Central US",
    "Zone": "1,2",
    ...
}

Copy code to clipboard

definitions.json

[
  {
    "Role": "DATA",
    "Count": "3",
    "MirrorMap": "primary,backup,async",
    "LocationMap": "1,1,2",
    "ZoneMap": "1,2,1"
  },
  {
    "Role": "AR",
    "Count": "1",
    "DockerImage": "intersystems/arbiter:2021.1.0.205.0",
    "RegionMap": "2",
    "ZoneMap": "2"
  }
]

Copy code to clipboard

icm inventory

Machine               IP Address    DNS Name             Region     Zone
-------               ----------    --------             ------     ----
Acme-AR-multi-0001    35.179.173.90 acmear1.azure.com    Central US 2
Acme-DATA-multi-0001+ 35.237.131.39 acmedata1.azure.com  East US    1
Acme-DATA-multi-0002- 35.233.223.64 acmedata2.azure.com  East US    2
Acme-DATA-multi-0003  35.166.127.82 acmedata3.google.com Central US 1

Copy code to clipboard

icm ps

Machine              IP Address    Container Status Health  Mirror    Image
-------              ----------    --------- ------ ------  ------    -----
Acme-AR-multi-0001   35.179.173.90 arbiter   Up     healthy           intersystems/arbiter:2021.1.0.205.0
Acme-DATA-multi-0001 35.237.131.39 iris      Up     healthy PRIMARY   intersystems/iris:2021.1.0.205.0
Acme-DATA-multi-0002 35.233.223.64 iris      Up     healthy BACKUP    intersystems/iris:2021.1.0.205.0
Acme-DATA-multi-0003 35.166.127.82 iris      Up     healthy CONNECTED intersystems/iris:2021.1.0.205.0

Copy code to clipboard

If you want to use an existing virtual network in a multiregion Azure deployment, you must include the ResourceGroupName and VirtualNetwork fields (see Microsoft Azure (Azure) Parameters) in the defaults file to specify a network for each region specified in the Location field, for example:

{
    "Provider": "Azure",
    "Label": "Acme",
    "Tag": "multi",
    "Location": "East US,Central US",
    "Zone": "1,2",
    "ResourceGroupName": "acme-resource-group",
    "VirtualNetworkName": "acme-vnet-east,acme-vnet-central"
    ...
}

Copy code to clipboard

The specified networks must have nonoverlapping address spaces. In the accompanying definitions file, each definition must include the Subnetname field specifying a unique subnet for each region specified by the Location field. For example, for the mirrored deployment illustrated in this section, if the defaults file included the ResourceGroupName and VirtualNetwork fields, the definitions file might look like the following. Because the AR definition deploys in the Central US region only, just one subnet is required in that definition.

[
  {
    "Role": "DATA",
    "Count": "3",
    "MirrorMap": "primary,backup,async",
    "RegionMap": "1,1,2",
    "ZoneMap": "1,2,1",
    "SubnetName": "acme-subnet-data-east,acme-subnet-data-central"
  },
  {
    "Role": "AR",
    "Count": "1",
    "DockerImage": "intersystems/arbiter:2021.1.0.205.0",
    "RegionMap": "2",
    "ZoneMap": "2",
    "SubnetName": "acme-subnet-arbiter-central"
  }
]

Copy code to clipboard

Deploying Across Multiple Regions on AWS and TencentCopy link to this section

To deploy across multiple regions on AWS and Tencent, ICM first provisions the needed infrastructure in the separate regions, then merges that infrastructure and deploys services on it as if it were preexisting infrastructure.

This procedure can also be used to deploy across multiple providers. In this discussion, “region” is used to indicate “region or provider”, with differences between multiprovider and AWS/Tencent multiregion noted as needed.

The procedure for creating merged multiregion deployments involves the following steps:

1. Provision the infrastructure in each region in separate ICM sessions.

2. Merge the multiregion infrastructure using the icm merge command.

3. Review the merged definitions.json file to reorder and update as needed.

4. Reprovision the merged infrastructure using the icm provision command.

5. Deploy services on the merged infrastructure as a Preexisting deployment using the icm run command.

6. When unprovisioning the infrastructure, issue the icm unprovision command separately in the original session directories.

Provision the InfrastructureCopy link to this section

The separate sessions for provisioning infrastructure in each region (specified by the Region field) should be conducted in separate working directories within the same ICM container. For example, you could begin by copying the provided /Samples/AWS directory (see Define the Deployment in the “Using ICM” chapter) to /Samples/AWS/us-east-1 and /Samples/AWS/us-west–1. Specify the desired region, node definitions, and features to match the eventual multiregion deployment in the defaulta and definitions file for each. For example, if you want to deploy a mirror failover pair in one region and a DR async member of the mirror in another, include the appropriate region and zones and "Mirror": "true" in the defaults files, and define two DMs (for the failover pair) in one region in its definitions file, a third DM (for the async) in the other, and a single AR (arbiter) node in one or the other. Each defaults file in a multiregion deployment should have a unique Label and/or Tag to prevent resource conflicts; this is not necessary for multiprovider deployments. This example is shown in the following.

Note:

If a given definition doesn't satisfy topology requirements for a single-region deployment, for example a single DM node defined when Mirror is set to true, disable topology validation by including "SkipTopologyValidation": "true" in the defaults file, as shown in the /Samples/AWS/us-west-1/defaults.json.

/Samples/AWS/us-east-1

defaults.json

{
    "Provider": "AWS",
    "Label": "Acme",
    "Tag": "east1",
    "Region": "us-east-1",
    "Zone": "us-east1-a,us-east1-b",
    "Mirror": "true",
    ...
}

Copy code to clipboard

definitions.json

[
  {
    "Role": "DM",
    "Count": "2",
    "ZoneMap": "1,2"
  }
]

Copy code to clipboard

/Samples/AWS/us-west-1/

defaults.json

{
    "Provider": "AWS",
    "Label": "Acme",
    "Tag": "west1",
    "Region": "us-west-1",
    "Zone": "us-west1-a,us-west1-b",
    "Mirror": "true",
    "SkipTopologyValidation": "true"},
    ...
}

Copy code to clipboard

definitions.json

[
  {
    "Role": "DM",
    "Count": "1",
    "ZoneMap": "1"
  }
  {
    "Role": "AR",
    "Count": "1",
    "ZoneMap": "2"
  }
]

Copy code to clipboard

Use the icm provision command in each working directory to provision the infrastructure in each region. The output of the icm inventory command, executed in each directory, shows you the infrastructure you are working with, for example:

/Samples/AWS/us-east-1

$ icm inventory
Machine             IP Address    DNS Name                        Region    Zone
-------             ----------    --------                        ------    ----
Acme-DM-east1-0001+ 54.214.230.24 ec2-54-214-230-24.amazonaws.com us-east-1 a
Acme-DM-east1-0002- 54.129.103.67 ec2-54-129-103-67.amazonaws.com us-east-1 b

Copy code to clipboard

/Samples/AWS/us-west-1

$ icm inventory
Machine             IP Address    DNS Name                        Region    Zone
-------             ----------    --------                        ------    ----
Acme-AR-west1-0001  54.181.212.79 ec2-54-181-212-79.amazonaws.com us-west-1 b
Acme-DM-west1-0002  54.253.103.21 ec2-54-253-103-21.amazonaws.com us-west-1 a

Copy code to clipboard

Merge the Provisioned InfractructureCopy link to this section

The icm merge command scans the configuration files in the current working directory and those in the additional directory or directories specified to create merged configuration files that can be used for a Preexisting deployment in a specified new directory. For example, to merge the definitions and defaults files in /Samples/AWS/us-east–1 and /Samples/AWS/us-west–1 into a new set in /Samples/AWS/merge, you would issue the following commands:

$ cd /Samples/AWS/us-east-1
$ mkdir ../merge
$ icm merge -options ../us-west1 -localPath /Samples/AWS/merge

Copy code to clipboard

In the icm merge command, --options specifies a comma-separated list of the provisioning directories to be merged with the local one, and --localPath specifies the destination directory for the merged definitions.

Review the Merged Definitions FileCopy link to this section

When you examine the new configuration files, you will see that Provider has been changed to PreExisting in the merged defaults file. (The previous Provider field and others have been moved into the definitions file; they are displayed by the icm inventory command, but otherwise have no effect.) The Label and/or Tag can be modified if desired.

The definitions in the merged definitions file have been converted for use with provider PreExisting. As described in Definitions File for PreExisting in the appendix “Deploying on a Preexisting Cluster”, the definitions.json file for a Preexisting deployment contains exactly one entry per node (rather than one entry per role with a Count field to specify the number of nodes of that role). Each node is identified by its IP address or fully-qualified domain name. Either the IPAddress or DNSName field must be included in each definition, as well as the SSHUser field. (The latter specifies a nonroot user with passwordless sudo access, as described in SSH in “Deploying on a Preexisting Cluster”.) In the merged file, the definitions have been grouped by region, or by provider in multiprovider deployments; they should be reordered to reflect desired placement of mirror members, if necessary, and a suitable mirror map defined (see Mirrored Configuration Requirements and Deploying Across Multiple Zones). After review, the definitions file for our example would look like this:

[
    {
        "Role":"DM",
        "IPAddress":"54.214.230.24",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"DM",
        "IPAddress":"54.129.103.67",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"DM",
        "IPAddress":"54.253.103.21",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"AR",
        "IPAddress":"54.181.212.79",
        "SSHUser": "icmuser",
        "StartCount": "4"
    }
]

Copy code to clipboard

Reprovision the Merged InfrastructureCopy link to this section

Reprovision the merged infrastructure by issuing the icm provision command in the new directory (/Samples/AWS/merge in the example). The output of the icm inventory command shows the merged infrastructure in one list:

$ icm inventory
Machine             IP Address     DNS Name                       Region    Zone
-------             ----------     --------                       ------    ----
Acme-DM-east1-0001+ 54.214.230.24 ec2-54-214-230-24.amazonaws.com us-east-1 a
Acme-DM-east1-0002- 54.129.103.67 ec2-54-129-103-67.amazonaws.com us-east-1 b
Acme-AR-west1-0001  54.181.212.79 ec2-54-181-212-79.amazonaws.com us-west-1 b
Acme-DM-west1-0002  54.253.103.21 ec2-54-253-103-21.amazonaws.com us-west-1 a

Copy code to clipboard

Deploy Services on the Merged InfrastructureCopy link to this section

Use the icm run command to deploy services on your merged infrastructure, as you would for any deployment, for example

$ icm run
...
-> Management Portal available at: http://112.97.196.104.google.com:52773/csp/sys/UtilHome.csp
$ icm ps
Machine            IP Address    Container Status Health  Mirror    Image
-------            ----------    --------- ------ ------  ------    -----
Acme-AR-multi-0001 35.179.173.90 arbiter   Up     healthy           intersystems/arbiter:2021.1.0.205.0
Acme-DM-multi-0001 35.237.131.39 iris      Up     healthy PRIMARY   intersystems/iris:2021.1.0.205.0
Acme-DM-multi-0002 35.233.223.64 iris      Up     healthy BACKUP    intersystems/iris:2021.1.0.205.0
Acme-DM-multi-0003 35.166.127.82 iris      Up     healthy CONNECTED intersystems/iris:2021.1.0.205.0

Copy code to clipboard

Unprovision the Merged InfrastructureCopy link to this section

When the time comes to unprovision the multiregion deployment, return to the original working directories to issue the icm unprovision command, and then delete the merged working directory. In our example, you would do the following:

$ cd /Samples/AWS/us-east-1
$ icm unprovision -force -cleanUp
...
...completed destroy of Acme-east1
$ cd /Samples/AWS/us-west-1
$ icm unprovision -force -cleanUp
...
...completed destroy of Acme-west1
$ rm -rf /Samples/AWS/merge

Copy code to clipboard

Deploy Within an Existing Private NetworkCopy link to this section

If you deploy ICM on an existing private network and want to provision and deploy on that network, as shown in the following illustration, you need to add fields to the defaults and definitions files for the configuration you want to deploy.

To deploy on an existing private network, follow these steps:

1. Obtain access to a node that resides within the private network. This may require use of a VPN or intermediate host.

2. Install Docker and ICM on the node as described in Launch ICM in the “Using ICM” chapter.

3. Add the following fields to the defaults.json file:

   "PrivateSubnet": "true",
   "net_vpc_cidr": "10.0.0.0/16",
   "net_subnet_cidr": "10.0.2.0/24"

   Copy code to clipboard

   The net_vpc_cidr and net_subnet_cidr fields (shown with sample values) specify the CIDRs of the private network and the node’s subnet within that network, respectively.

4. Add the appropriate common and provider-specific fields to the defaults.json file, as follows:

   Provider	Key	Description
   all	PrivateSubnet	Must be set to true
   	net_vpc_cidr	CIDR of the private network
   	net_subnet_cidr	CIDR of the ICM node’s subnet within the private network (see Note)
   GCP	Network	Google VPC
   	Subnet	Google subnetwork
   Azure	ResourceGroupName	AzureRM resource group
   	VirtualNetworkName	AzureRM virtual network
   	SubnetName	AzureRM subnet (see Note)
   AWS (see Note)	VPCId	AWS VPC ID
   	SubnetIds	Comma-separated list of AWS subnet IDs, one for each element specified by the Zone field.
   Tencent	VPCId	Tencent VPC ID
   	SubnetIds	Comma-separated list of Tencent subnet IDs, one for each element specified by the Zone field.

   Note:

   On Azure, ICM assigns a security group to the subnet specified by SubnetName, which could affect the behavior of unrelated machines on the subnet. For this reason, a dedicated subnet (as specified by a unique SubnetName and corresponding net_subnet_cidr) must be provided for every entry in the definitions file (but ResourceGroupName and VirtualNetworkName remain in the defaults file). This includes the BH definition when deploying a bastion host, as described in the following section.

   To deploy IRIS within an existing private VPC on AWS, you must create a node within that VPC on which you can deploy and use ICM. If you want to reach this ICM host from outside the VPC, you can specify a route table and Internet gateway for ICM to use instead of creating its own. To do this, add the RouteTableId and InternetGatewayId fields to your defaults.json file, for example:

   "RouteTableID": "rtb-00bef388a03747469",
   "InternetGatewayId": "igw-027ad2d2b769344a3"

   Copy code to clipboard

   When provisioning on GCP, the net_subnet_cidr field is descriptive, not proscriptive; it should be an address space which includes the node’s subnet, as well as any others within the network should have access to the deployed configuration.

5. Use icm provision and icm run to provision and deploy your configuration.

Bear the following in mind when deploying on a private network.

• Viewing web pages on any node within the private network, for example the Management Portal, requires a browser that also resides within the private network, or for which a proxy or VPN has been configured.

• Any DNS name shown in the output of ICM commands is just a copy of the local IP address.

• Private network deployment across regions or providers is currently not supported.

Deploy on a Private Network Through a Bastion HostCopy link to this section

If you set the PrivateSubnet field to true in the defaults file but don't include the fields required to use an existing network, ICM creates a private network for you. You cannot complete the provisioning phase in this situation, however, because ICM is unable to configure or otherwise interact with the machines it just allocated. To enable its interaction with nodes on the private network it creates, ICM can optionally create a bastion host, a host node that belongs to both the private subnet and the public network and can broker communication between them.

To create a private network and a bastion host providing ICM with access to that network, add a definition for a single node of type BH to the definitions.json file, for example:

   {
       "Role": "DATA",
       "Count": "3"
   },
   {
       "Role": "BH",
       "Count": "1",
       "StartCount: 4"
   }

Copy code to clipboard

To deploy and use a bastion host with an existing private network, add a BH definition to the definitions file, as above, and include the fields necessary to specify the network in the defaults file (as describe in the previous section). ICM automatically sets the "PrivateSubnet" option to "true" when a BH node definition is included in definitions.json

The bastion host can be accessed using SSH, allowing users to tunnel SSH commands to the private network. Using this technique, ICM is able to allocate and configure compute instances within the private network from outside, allowing provisioning to succeed, for example:

$ icm inventory
Machine             IP Address     DNS Name                     Region   Zone
-------             ----------     --------                     ------   ----
Acme-BH-TEST-0004   35.237.125.218 218.125.237.35.bc.google.com us-east1 b
Acme-DATA-TEST-0001 10.0.0.2       10.0.0.2                     us-east1 b
Acme-DATA-TEST-0002 10.0.0.3       10.0.0.3                     us-east1 b
Acme-DATA-TEST-0003 10.0.0.4       10.0.0.4                     us-east1 b

Copy code to clipboard

Once the configuration is deployed, it is possible to run the ssh command against any node, for example:

# icm ssh -role DATA -interactive
ubuntu@ip-10.0.0.2:~$

Copy code to clipboard

If you examine the command being run, however, you can see that it is routed through the bastion host:

$ icm ssh -role DATA -interactive -verbose
ssh -A -t -i /Samples/ssh/insecure -p 3022 ubuntu@35.237.125.218
ubuntu@ip-10.0.0.2:~$

Copy code to clipboard

On the other hand, for other commands to succeed, ICM needs access to ports and protocols besides SSH. To do this, ICM configures tunnels between the bastion host and nodes within the cluster for Docker, JDBC, and HTTP. This allows commands such as icm run, icm exec, and icm sql to succeed.

Bear the following in mind when deploying a bastion host:

• The address of the configuration’s Management Portal is that of the bastion host.

• For security reasons, no private keys are stored on the bastion host.

• Any DNS name shown in the output of ICM commands is just a copy of the local IP address.

• Provisioning of load balancers in a deployment that includes a bastion host is not supported.

• Use of a bastion host with multiregion deployments (see Deploying Across Multiple Regions or Providers) and in distributed management mode (see the appendix “Sharing ICM Deployments”) are currently not supported.

  Note:

  When you create a custom VPC in the Google portal, you are required to create a default subnet. If you are provisioning with a bastion host and will use the subnet created by ICM, you should delete this default subnet before provisioning (or give it an address space that won't collide with the default address space 10.0.0.0/16).

System Alerting and MonitoringCopy link to this section

System Alerting and Monitoring, or SAM, is a cluster monitoring solution for InterSystems IRIS® data platform. Whatever configuration and platform your InterSystems IRIS-based application runs on, you can monitor with SAM. For complete information about SAM, see the System Alerting and Monitoring GuideOpens in a new window.

SAM is included in your ICM deployment when you include a SAM node in your definitions file; the DockerImage field is required and must specify the InterSystems sam image, as shown in the following:

[
    {
        "Role": "DM",
        "Count": "4",
        "LicenseKey": "ubuntu-sharding-iris.key"
    },
    {
        "Role": "SAM",
        "Count": "1",
        "DockerImage": "intersystems/sam:2.0"
    }
]

Copy code to clipboard

The SAM application comprises five containers. The SAM Manager container is deployed during the deployment phase (see The icm run Command).

The other four containers — Prometheus, Alertmanager, Grafana, and Nginx — are deployed during the provisioning phase (see The icm provision Command). The images from which these containers are deployed can be specified using the PrometheusImage, AlertmanagerImage, GrafanaImage, and NginxImage fields; their default values are shown in General Parameters.

Following successful deployment, a message like the example below is displayed:

$ icm run
...
-> SAM Portal available at: http://112.97.196.104.google.com:8080/api/sam/app/index.csp#

Copy code to clipboard

Deploying Third-party Monitoring with ICMCopy link to this section

You can deploy the third-party monitoring package of your choice (or any other third-party package) as part of your ICM configuration. The following example shows how to use the icm ssh command to add Weave Scope monitoring to all of the hosts in a deployment:

icm ssh -command "sudo curl -L git.io/scope -o /usr/local/bin/scope 2>&1"
icm ssh -command "sudo chmod +x /usr/local/bin/scope"
icm ssh -command "sudo /usr/local/bin/scope launch 2>&1"

Copy code to clipboard

Following these commands, you can access Weave Scope through port 4040 on any of the hosts displayed by the icm inventory command, that is, at http://hostname:4040.

Host Node Restart and RecoveryCopy link to this section

When a cloud host node is shut down and restarted due to an unplanned outage or to planned action by the cloud provider (for example, for preventive maintenance) or user (for example, to reduce costs), its IP address and domain name may change, causing problems for both ICM and deployed applications (including InterSystems IRIS).

This behavior differs by cloud provider. GCP and Azure preserve IP address and domain name across host node restart by default, whereas this feature is optional on AWS and Tencent (see Elastic IP Feature).

Reasons a host node might be shut down include the following:

• Unplanned outage

  • Power outage

  • Kernel panic

• Preventive maintenance initiated by provider

• Cost reduction strategy initiated by user

Methods for intentionally shutting down host nodes include:

• Using the cloud provider user interface

• Using ICM:

  icm ssh -command 'sudo shutdown'

  Copy code to clipboard

Correcting Time SkewCopy link to this section

If the system time within the ICM containers differs from Standard Time by more than a few minutes, the various cloud providers may reject requests from ICM. This can happen if the container is unable to reach an NTP server on startup (initial or after being stopped or paused). The error appears in the terraform.err file as some variation on the following:

Error refreshing state: 1 error(s) occurred:

    # icm provision
    Error: Thread exited with value 1
    Signature expired: 20170504T170025Z is now earlier than 20170504T171441Z (20170504T172941Z   15 min.)
    status code: 403, request id: 41f1c4c3-30ef-11e7-afcb-3d4015da6526 doesn’t run for a period of time

Copy code to clipboard

The solution is to manually run NTP, for example:

ntpd -nqp pool.ntp.org

Copy code to clipboard

and verify that the time is now correct. (See also the discussion of the --cap-add option in Launch ICM.)

Timeouts Under ICMCopy link to this section

When the target system is under extreme load, various operations in ICM may time out. Many of these timeouts are not under direct ICM control (for example, from cloud providers); other operations are retried several times, for example SSH and JDBC connections.

SSH timeouts are sometimes not identified as such. For instance, in the following example, an SSH timeout manifests as a generic exception from the underlying library:

# icm cp -localPath foo.txt -remotePath /tmp/
2017-03-28 18:40:19 ERROR Docker:324 - Error: 
java.io.IOException: com.jcraft.jsch.JSchException: channel is not opened. 
2017-03-28 18:40:19 ERROR Docker:24 - java.lang.Exception: Errors occurred during execution; aborting operation 
        at com.intersystems.tbd.provision.SSH.sshCommand(SSH.java:419) 
        at com.intersystems.tbd.provision.Provision.execute(Provision.java:173) 
        at com.intersystems.tbd.provision.Main.main(Main.java:22)

Copy code to clipboard

In this case the recommended course of action is to retry the operation (after identifying and resolving its proximate cause).

Note that for security reasons ICM sets the default SSH timeout for idle sessions at ten minutes (60 seconds x 10 retries). These values can be changed by modifying the following fields in the/etc/ssh/sshd_config file:

ClientAliveInterval 60
ClientAliveCountMax 10

Copy code to clipboard

Docker Bridge Network IP Address Range ConflictCopy link to this section

For container networking, Docker uses a bridge network (see Use bridge networksOpens in a new window in the Docker documentation) on subnet 172.17.0.0/16 by default. If this subnet is already in use on your network, collisions may occur that prevent Docker from starting up or prevent you from being able to reach your deployed host nodes. This problem can arise on the machine hosting your ICM container, your InterSystems IRIS cluster nodes, or both.

To resolve this, you can edit the bridge network’s IP configuration in the Docker configuration file to reassign the subnet to a range that is not in conflict with your own IP addresses (your IT department can help you determine this value). To make this change, add a line like the following to the Docker daemon configuration file:

"bip": "192.168.0.1/24"

Copy code to clipboard

If the problem arises with the ICM container, edit the file /etc/docker/daemon.json on the container’s host. If the problem arises with the host nodes in a deployed configuration, edit the file /ICM/etc/toHost/daemon.json in the ICM container; by default this file contains the value in the preceding example, which is likely to avoid problems with any deployment type except PreExisting.

Detailed information about the contents of the daemon.json file can be found in Daemon configuration fileOpens in a new window in the Docker documentation; see also Configure and troubleshoot the Docker daemonOpens in a new window.

Weave Network IP Address Range ConflictCopy link to this section

By default, the Weave network uses IP address range 10.32.0.0/12. If this conflicts with an existing network, you may see an error such as the following in log file installWeave.log:

Network 10.32.0.0/12 overlaps with existing route 10.0.0.0/8 on host
ERROR: Default --ipalloc-range 10.32.0.0/12 overlaps with existing route on host.
You must pick another range and set it on all hosts.

Copy code to clipboard

This is most likely to occur with provider PreExisting if the machines provided have undergone custom network configuration to support other software or local policies. If disabling or moving the other network is not an option, you can change the Weave configuration instead, using the following procedure:

1. Edit the following file local to the ICM container:

   /ICM/etc/toHost/installWeave.sh

   Copy code to clipboard

2. Find the line containing the string weave launch. If you're confident there is no danger of overlap between Weave and the existing network, you can force Weave to continue use the default range by adding the underscored text in the following:

   sudo /usr/local/bin/weave launch --ipalloc-range 10.32.0.0/12 --password $2 

   Copy code to clipboard

   You can also simply move Weave to another private network, as follows:

   sudo /usr/local/bin/weave launch --ipalloc-range 172.30.0.0/16 --password $2

   Copy code to clipboard

3. Save the file.

4. Reprovision the cluster.

Huge PagesCopy link to this section

On certain architectures you may see an error similar to the following in the InterSystems IRIS messages log:

0 Automatically configuring buffers 
1 Insufficient privileges to allocate Huge Pages; non-root instance requires CAP_IPC_LOCK capability for Huge Pages. 
2 Failed to allocate 1316MB shared memory using Huge Pages. Startup will retry with standard pages. If huge pages 
  are needed for performance, check the OS settings and consider marking them as required with the InterSystems IRIS 
  'memlock' configuration parameter.

Copy code to clipboard

This can be remedied by providing the following option to the icm run command:

-options "--cap-add IPC_LOCK"

Copy code to clipboard