ICM Reference

Role DATA: Sharded Cluster Data Node

Role COMPUTE: Sharded Cluster Compute Node

Role DM: Distributed Cache Cluster Data Server, Standalone Instance, Shard Master Data Server

Role DS: Shard Data Server

Role QS: Shard Query Server

Role AM: Distributed Cache Cluster Application Server, Shard Master Application Server

Role AR: Mirror Arbiter

Role WS: Web Server

{
    "Role": "WS",
    "Count": "3",
    "DockerImage": "intersystems/webgateway:stable",
    "ApplicationPath": "/acme",
    "AlternativeServers": "LoadBalancing"
}

Role LB: Load Balancer

{
    "Role": "AM",
    "Count": "2",
    "LoadBalancer": "true"
}

$ icm inventory
Machine           IP Address    DNS Name                              Region   Zone
-------           ----------    --------                              ------   ----
ACME-AM-TEST-0001 54.214.230.24 ec2-54-214-230-24.amazonaws.com       us-west1 c
ACME-AM-TEST-0002 54.214.230.25 ec2-54-214-230-25.amazonaws.com       us-west1 c
ACME-LB-TEST-0000 (virtual AM)  ACME-AM-TEST-1546467861.amazonaws.com us-west1 c

{
    "Role": "VM",
    "Count": "2",
    "LoadBalancer": "true",
    "ForwardProtocol": "tcp",
    "ForwardPort": "443",
    "HealthCheckProtocol": "http",
    "HealthCheckPath": "/csp/status.cxw",
    "HealthCheckPort": "8080"
}

Role VM: Virtual Machine Node

• icm provision
• icm unprovision
• icm inventory
• icm ssh
• icm scp

Role CN: Container Node

Role BH: Bastion Host

Rules for Mirroring

• All DATA nodes and DS nodes are configured as mirror failover pairs, for example specifying six DATA nodes deploys three mirrored data nodes; if an odd number of DATA or DS nodes is defined, provisioning fails. Mirrors in sharded configurations cannot contain async members.
• If two DM nodes are specified in the definitions file, they are configured as a mirror failover pair using the default MirrorMap value, primary,backup. If one DM is specified, provisioning fails.
• If more than two DMs are specified, and the MirrorMap field in the node definition matches the number of nodes specified and indicates that those beyond the failover pair are disaster recovery (DR) asyncs, they are configured as a failover pair and the specified number of DR asyncs. For example, the following definition creates a mirror consisting of a failover pair and two DR asyncs:

  "Role": "DM",
  "Count": "4",
  "MirrorMap": "primary,backup,async,async"
  
  

  The number of DM nodes can also be less than the number of elements in MirrorMap; in the example above, changing Count to 2 would deploy a primary and backup, while making it 3 would deploy the failover pair and one async.
  All asyncs deployed by ICM are DR asyncs; reporting asyncs are not supported. Up to 14 asyncs can be included in a mirror. For information on mirror members and possible configurations, see Mirror Components in the “Mirroring” chapter of the High Availability Guide.
  DM node mirrors in sharded configurations cannot contain async members.
• If more than one AR (arbiter) node is specified, provisioning fails.

Nonmirrored Configuration Requirements

• One or more DATA (data nodes in a sharded cluster).
• If DATA nodes are included, zero or more COMPUTE (compute nodes in a sharded cluster); best practices are at least as many COMPUTE nodes as DATA nodes and the same number of COMPUTE nodes for each DATA node.
• If no DATA nodes are included:
  • Exactly one DM (distributed cache cluster data server, standalone InterSystems IRIS instance, shard master data server in namespace-level sharded cluster).
  • Zero or more AM (distributed cache cluster application server, shard master application server in namespace-level sharded cluster).
  • Zero or more DS (shard data servers in namespace-level sharded cluster).
  • Zero or more QS (shard query servers in namespace-level sharded cluster).
• Zero or more WS (web servers).
• Zero or more LB (load balancers).
• Zero or more VM (virtual machine nodes).
• Zero or more CN (container nodes).
• Zero or one BH (bastion host).
• Zero AR (arbiter node is for mirrored configurations only).

Mirrored Configuration Requirements

• An even number of DATA (data nodes in a sharded cluster).
• If DATA nodes are included, zero or more COMPUTE (compute nodes in a sharded cluster); best practices are at least as many COMPUTE nodes as DATA nodes and the same number of COMPUTE nodes for each DATA node.
• If no DATA nodes are included:
  • Two DM (distributed cache cluster data server, standalone InterSystems IRIS instance) or more, if DR asyncs are specified by the MirrorMap field; exactly two DM (shard master data server in namespace-level sharded cluster).
  • Zero or more AM (distributed cache cluster application server, shard master application server in namespace-level sharded cluster).
  • Even number of DS (shard data servers in namespace-level sharded cluster).
  • Zero or more QS (shard query servers in namespace-level sharded cluster).
• Zero or one AR (arbiter node is optional but recommended for mirrored configurations)
• Zero or more WS (web servers).
• Zero or more LB (load balancers).
• Zero or more VM (virtual machine nodes).
• Zero or more CN (container nodes).
• Zero or one BH (bastion host).

• Mirroring is enabled by setting key Mirror in your defaults.json file to true.

  "Mirror": "true"
  

• To deploy more than two DM nodes, you must include the MirrorMap field in your definitions file to specify that those beyond the first two are DR async members, as follows:

  "MirrorMap": "primary,backup,async,..."
  

  The value of MirrorMap must always begin with primary,backup, and the number of elements in the field value must match the number of DM nodes, for example:

  "Role": "DM",
  "Count": "5”,
  "MirrorMap": "primary,backup,async,async,async",
  ...
  

  MirrorMap can be used in conjunction with the Zone and ZoneMap fields to deploy async instances across zones; see Deploying Across Multiple Zones.
  Note:

  Async mirror members are not currently supported in node-level or namespace-level sharded clusters.

Host Node Communication

Docker

Weave Net

Weave Scope, Rancher

InterSystems IRIS

Private Networks

Provision the Infrastructure

# defaults.json us-east1:

"Provider": "GCP",
"Label": "Acme",
"Tag": "EAST",
"Region": "us-east1",
"Zone": "us-east1b",
"Mirror": "true",

# defaults.json us-west1:

"Provider": "GCP",
"Label": "Acme",
"Tag": "WEST",
"Region": "us-west1",
"Zone": "us-west1a",
"Mirror": "true",

# definitions.json us-east1:

"Role": "DM",
"Count": "2",
...
"Role": "AR"
"StartCount": "3",

# definitions.json us-west1:

"Role": "DM",
"Count": "1",

$ icm provision
...
Machine            IP Address     DNS Name                   Region    Zone
-------            ----------     --------                   ------    ----
Acme-DM-EAST-0001+ 104.196.97.112 112.97.196.104.google.com  us-east1  b
Acme-DM-EAST-0002- 104.196.97.113 113.97.196.104.google.com  us-east1  b
Acme-AR-EAST-0003  104.196.97.114 114.97.196.104.google.com  us-east1  b

$ icm provision
...
Machine            IP Address     DNS Name                   Region    Zone
-------            ----------     --------                   ------    ----
Acme-DM-WEST-0001+ 104.196.91.134 134.91.196.104.google.com  us-west1  a

Merge the Provisioned Infractructure

$ cd /Samples/GCP/us-east1
$ mkdir ../merge
$ icm merge -options ../us-west1/instances.json -localPath /Samples/GCP/merge

Review the Merged Definitions File

[
    {
        "Role":"DM",
        "IPAddress":"104.196.97.112",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"DM",
        "IPAddress":"104.196.97.113",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"DM",
        "IPAddress":"104.196.91.134",
        "LicenseKey": "ubuntu-sharding-iris.key",
        "SSHUser": "icmuser",
        "MirrorMap": "primary,backup,async"
    },
    {
        "Role":"AR",
        "IPAddress":"104.196.97.114",
        "SSHUser": "icmuser",
        "StartCount": "4"
    }

]

Reprovision the Merged Infrastructure

$ icm provision
...
Machine             IP Address     DNS Name                   Region    Zone
-------             ----------     --------                   ------    ----
Acme-DM-MERGE-0001+ 104.196.97.112 112.97.196.104.google.com  us-east1  b
Acme-DM-MERGE-0002- 104.196.97.113 113.97.196.104.google.com  us-east1  b
Acme-DM-MERGE-0003  104.196.91.134 134.91.196.104.google.com  us-west1  a
Acme-AR-MERGE-0004  104.196.97.114 114.97.196.104.google.com  us-east1  b

Deploy Services on the Merged Infrastructure

$ icm run
. . .
-> Management Portal available at: http://112.97.196.104.google.com:52773/csp/sys/UtilHome.csp
$ icm ps
Machine            IP Address     Container  Status Health  Mirror    Image
-------            ----------     ---------  ------ ------  ------    -----
Acme-DM-MERGE-0001 104.196.97.112 iris       Up     healthy PRIMARY   isc/iris:stable
Acme-DM-MERGE-0002 104.196.97.113 iris       Up     healthy BACKUP    isc/iris:stable
Acme-DM-MERGE-0003 104.196.91.134 iris       Up     healthy CONNECTED isc/iris:stable
Acme-AR-MERGE-0004 104.196.97.114 arbiter    Up     healthy           isc/arbiter:stable

Unprovision the Merged Infrastructure

$ cd /Samples/GCP/us-east1
$ icm unprovision -force -cleanUp
...
...completed destroy of Acme-EAST
$ cd /Samples/GCP/us-west1
$ icm unprovision -force -cleanUp
...
...completed destroy of Acme-WEST
$ rm -rf /Samples/GCP/merge

Deploy Within an Existing Private Network

1. Obtain access to a node that resides within the private network. This may require use of a VPN or intermediate host.
2. Install Docker and ICM on the node as described in Launch ICM in the “Using ICM” chapter.
3. Add the following fields to the defaults.json file:

   "PrivateSubnet": "true",
   "net_vpc_cidr": "10.0.0.0/16",
   "net_subnet_cidr": "10.0.2.0/24"
   

   The net_vpc_cidr and net_subnet_cidr fields (shown with sample values) specify the CIDRs of the private network and the node’s subnet within that network, respectively.
4. Add the appropriate provider-specific fields to the defaults.json file, as follows:

   Provider	Key	Description
   all	PrivateSubnet	Must be set to true
   	net_vpc_cidr	CIDR of the private network (see Note)
   	net_subnet_cidr	CIDR of the ICM node’s subnet within the private network (see Note)
   GCP	Network	Google VPC
   	Subnet	Google subnetwork
   Azure	ResourceGroupName	AzureRM resource group
   	VirtualNetworkName	AzureRM virtual network
   	SubnetId	AzureRM subnet ID (see Note)
   AWS (see Note)	VPCId	AWS VPC ID
   	SubnetIds	Comma-separated list of AWS subnet IDs, one for each element specified by the Zone field.

   Note:

   When provisioning on Azure, a unique SubnetId and corresponding net_subnet_cidr must be provided for every entry in the definitions file (but ResourceGroupName and VirtualNetworkName remain in the defaults file). This includes the BH definition when deploying a bastion host, as described in the following section.

   To deploy IRIS within an existing private VPC on AWS, you must create a node within that VPC on which you can deploy and use ICM. If you want to reach this ICM host from outside the VPC, you can specify a route table and Internet gateway for ICM to use instead of creating its own. To do this, add the RouteTableId and InternetGatewayId fields to your defaults.json file, for example:

   "RouteTableID": "rtb-00bef388a03747469",
   "InternetGatewayId": "igw-027ad2d2b769344a3"
   

   When provisioning on GCP, the net_subnet_cidr field is descriptive, not proscriptive; it should be an address space which includes the node’s subnet, as well as any others within the network should have access to the deployed configuration.
5. Use icm provision and icm run to provision and deploy your configuration.

• Viewing web pages on any node within the private network, for example the Management Portal, requires a browser that also resides within the private network, or for which a proxy or VPN has been configured.
• Any DNS name shown in the output of ICM commands is just a copy of the local IP address.
• Private network deployment across regions is currently not supported.

Deploy on a Private Network Through a Bastion Host

   {
       "Role": "DATA",
       "Count": "3"
   },
   {
       "Role": "BH",
       "Count": "1",
       "StartCount: 4"
   }

$ icm inventory
Machine             IP Address     DNS Name                     Region   Zone
-------             ----------     --------                     ------   ----
Acme-BH-TEST-0004   35.237.125.218 218.125.237.35.bc.google.com us-east1 b
Acme-DATA-TEST-0001 10.0.0.2       10.0.0.2                     us-east1 b
Acme-DATA-TEST-0002 10.0.0.3       10.0.0.3                     us-east1 b
Acme-DATA-TEST-0003 10.0.0.4       10.0.0.4                     us-east1 b

# icm ssh -role DATA -interactive
ubuntu@ip-10.0.0.2:~$

$ icm ssh -role DATA -interactive -verbose
ssh -A -t -i /Samples/ssh/insecure -p 3022 ubuntu@35.237.125.218
ubuntu@ip-10.0.0.2:~$

• The address of the configuration’s Management Portal is that of the bastion host.
• For security reasons, no private keys are stored on the bastion host.
• Any DNS name shown in the output of ICM commands is just a copy of the local IP address.
• Provisioning of load balancers in a deployment that includes a bastion host is not supported.
• Use of a bastion host with multiregion deployments (see Deploying Across Multiple Regions) and in distributed management mode (see the appendix “Sharing ICM Deployments”) are currently not supported.
  Note:

  When you create a custom VPC in the Google portal, you are required to create a default subnet. If you are provisioning with a bastion host and will use the subnet created by ICM, you should delete this default subnet before provisioning (or give it an address space that won't collide with the default address space 10.0.0.0/16).

Weave Scope

"Monitor": "scope"

"ProxyImage": "intersystems/https-proxy-auth:stable"

Weave Scope available at https://54.191.23.24:4041

Rancher

"Monitor": "rancher"

"MonitorUsername": "<username>",
"MonitorPassword": "<password>"

Rancher Server available at: http://00.186.24.14:8080/env/1a5/infra/hosts

Host Node Restart and Recovery

• Unplanned outage
  • Power outage
  • Kernel panic
• Preventive maintenance initiated by provider
• Cost reduction strategy initiated by user

• Using the cloud provider user interface
• Using ICM:

  icm ssh -command 'sudo shutdown'
  

Correcting Time Skew

Error refreshing state: 1 error(s) occurred:

    # icm provision
    Error: Thread exited with value 1
    Signature expired: 20170504T170025Z is now earlier than 20170504T171441Z (20170504T172941Z   15 min.)
    status code: 403, request id: 41f1c4c3-30ef-11e7-afcb-3d4015da6526 doesn’t run for a period of time

ntpd -nqp pool.ntp.org

Timeouts Under ICM

# icm cp -localPath foo.txt -remotePath /tmp/
2017-03-28 18:40:19 ERROR Docker:324 - Error: 
java.io.IOException: com.jcraft.jsch.JSchException: channel is not opened. 
2017-03-28 18:40:19 ERROR Docker:24 - java.lang.Exception: Errors occurred during execution; aborting operation 
        at com.intersystems.tbd.provision.SSH.sshCommand(SSH.java:419) 
        at com.intersystems.tbd.provision.Provision.execute(Provision.java:173) 
        at com.intersystems.tbd.provision.Main.main(Main.java:22)

ClientAliveInterval 60
ClientAliveCountMax 10

Docker Bridge Network IP Address Range Conflict

"bip:" "192.168.0.1/24"

Weave Network IP Address Range Conflict

Network 10.32.0.0/12 overlaps with existing route 10.0.0.0/8 on host
ERROR: Default --ipalloc-range 10.32.0.0/12 overlaps with existing route on host.
You must pick another range and set it on all hosts.

1. Edit the following file local to the ICM container:

   /ICM/etc/toHost/installWeave.sh
   

2. Find the line containing the string weave launch. If you're confident there is no danger of overlap between Weave and the existing network, you can force Weave to continue use the default range by adding the underscored text in the following:

   sudo /usr/local/bin/weave launch --ipalloc-range 10.32.0.0/12 --password $2 
   

   You can also simply move Weave to another private network, as follows:

   sudo /usr/local/bin/weave launch --ipalloc-range 172.30.0.0/16 --password $2
   

3. Save the file.
4. Reprovision the cluster.

Huge Pages

0 Automatically configuring buffers 
1 Insufficient privileges to allocate Huge Pages; non-root instance requires CAP_IPC_LOCK capability for Huge Pages. 
2 Failed to allocate 1316MB shared memory using Huge Pages. Startup will retry with standard pages. If huge pages 
  are needed for performance, check the OS settings and consider marking them as required with the InterSystems IRIS 
  'memlock' configuration parameter.

-options "--cap-add IPC_LOCK"

General Parameters

Security-Related Parameters

• For information about using scripts provided with ICM to generate these files, see Obtain Security-Related Files in this document.
• For information about how ICM uses the security files you provide to communicate securely with provisioned nodes and services on them, see ICM Security.
• For general information about using the SSH protocol, see SSH PROTOCOL from SSH Communications Security.
• For information about using SSL/TLS with Docker, see Protect the Docker daemon socket in the Docker documentation.
• For general information about using SSL/TLS with InterSystems IRIS, see Using SSL/TLS with InterSystems IRIS and The InterSystems Public Key Infrastructure in the Security Administration Guide. For information about the contents of the file identified by the SSLConfig parameter, see Creating a Client Configuration in the same document.
• For information about the use of SSL/TLS to secure connections between mirror members, see Securing Mirror Communication with SSL/TLS Security in the High Availability Guide.

Provider-Specific Parameters

• Selecting Machine Images
• Amazon Web Services (AWS) Parameters
• Google Cloud Platform (GCP) Parameters
• Microsoft Azure (Azure) Parameters
• VMware vSphere (vSphere) Parameters
• PreExisting Cluster (PreExisting) Parameters

Device Name Parameters

Generated Parameters