docs.intersystems.com
Home  /  Architecture  /  InterSystems Cloud Manager Guide  /  Using ICM


InterSystems Cloud Manager Guide
Using ICM
[Back]  [Next] 
InterSystems: The power behind what matters   
Search:  


This chapter explains how to use ICM to deploy an InterSystems IRIS™ configuration in a public cloud, as follows. The sections that follow explain the steps involved in using ICM to deploy a sample InterSystems IRIS configuration on AWS, as follows:
For comprehensive lists of the ICM commands and command-line options covered in detail in the following sections, see ICM Commands and Options.
ICM Use Cases
This chapter is focused on two typical ICM use cases, deploying the following two InterSystems IRIS configurations:
Most of the steps in the deployment process are the same for both configurations. The primary difference lies in the definitions files; see Define the Deployment for detailed contents. Output shown for the provisioning phase (see The icm provision Command) is from the ECP configuration; output shown for the deployment phase (see Deploy and Manage Services) is for the sharded cluster.
Launch ICM
ICM is provided as a Docker image. Everything required by ICM to carry out its provisioning, deployment, and management tasks — for example Terraform, the Docker client, and templates for the configuration files — is included in the ICM container. Therefore the only requirement for the Linux, macOS or Microsoft Windows system on which you launch ICM is that Docker is installed.
Note:
ICM requires Docker CE 17.06.
To download and run the ICM image, and to deploy the InterSystems IRIS container and others in the cloud using ICM, you need to identify the repository in which these images are located and the credentials you need to log into that repository. The repository must be accessible from the internet (that is, not behind a firewall) in order for the cloud provider to download images.
InterSystems images are distributed as Docker tar archive files, available in the InterSystems Worldwide Response Center (WRC) download area. Your enterprise may have already have added these images to its Docker repository; in this case, you should get the location of the repository and the needed credentials from the appropriate IT administrator. If your enterprise has a Docker repository but has not yet added the InterSystems images, get the location of the repository and the needed credentials, obtain the tar archive files containing the ICM and InterSystems IRIS images from the WRC, and add each of them to the repository using the following steps on the command line:
  1. For example:
    docker tag docker.intersystems.com/intersystems/icm:2018.1.0.583 acme/icm:2018.1.0.583
  2. For example:
    docker login docker.acme.com
    Username: gsanchez@acme.com
    Pasword: **********
  3. For example:
    docker push acme/icm:2018.1.0.583 
If your organization does not have an internet accessible Docker repository, you can use the free (or extremely low cost) Docker Hub for your testing.
To launch ICM on the command line, run the ICM image from your repository on a system on which Docker is installed, for example:
docker run --name icm -it --cap-add SYS_TIME acme/icm:2018.1.0.583
The -i option makes the command interactive and the -t option opens a pseudo-TTY, giving you command line access to the container. From this point on, you can interact with ICM by invoking ICM commands on the pseudo-TTY command line. The --cap-add SYS_TIME option allows the container to interact with the clock on the host system, avoiding clock skew that may cause the cloud service provider to reject API commands.
The ICM container includes a /Samples directory that provides you with samples of the elements required by ICM for provisioning, configuration, and deployment. The /Samples directory makes it easy for you to provision and deploy using ICM out of the box; eventually, however, you will probably want to use locations outside the container to store these elements and InterSystems IRIS licenses.
Of course, the ICM image can also be run by custom tools and scripts, and this can help you accomplish goals such as making these external locations available within the container, and saving your configuration files and your state directory (which is required to remove the infrastructure and services you provision) to persistent storage the container as well. A script, for example, could do the latter by capturing the current working directory in a variable and using it to mount that directory as a storage volume when running the ICM container, as follows:
#!/bin/bash
clear

# extract the basename of the full pwd path
MOUNT=$(basename $(pwd))
docker run --name icm -it --volume $PWD:$MOUNT --cap-add SYS_TIME iscrepo/icm:stable 
printf "\nExited icm container\n"
printf "\nRemoving icm container...\nContainer removed:  "
docker rm icm
You can mount multiple external storage volumes when running the ICM container (or any other). For detailed information about the InterSystems IRIS feature that lets you store instance-specific data outside the container, see Durable %SYS for Persistent Instance Data.
Note:
On a Windows host, you must enable the local drive on which the directory you want to mount as a volume is located using the Shared Drives option on the Docker Settings ... menu.
Important:
When an error occurs during an ICM operation, ICM displays a message directing you to the log file in which information about the error can be found. Before beginning an ICM deployment, familiarize yourself with the log files and their locations as described in Log Files and Other ICM Files.
Obtain Security-Related Files
ICM communicates securely with the cloud provider on which it provisions the infrastructure, with Docker on the provisioned nodes, and with several InterSystems IRIS services following container deployment. Before defining your deployment, you must obtain the credentials and other files needed to enable secure communication.
Cloud Provider Credentials
To use ICM with one of the public cloud platforms, you must create an account and download administrative credentials. To do this, follow the instructions provided by the cloud provider; you can also find information about how to download your credentials once your account exists in the Provider-Specific Parameters section. In the ICM configuration files, you identify the location of these credentials using the Credentials parameter.
When using ICM with a vSphere private cloud, you can use an existing account with the needed privileges, or create a new one. You specify these using the Username and Password fields.
SSH and SSL/TLS Keys
ICM uses SSH to provide access to the operating system of a provisioned node, and SSL/TLS to establish secure connections to Docker, InterSystems Web Gateway, JDBC, and mirrored InterSystems IRIS databases. The locations of the files needed to enable this secure communication are specified using several ICM parameters.
You can create these files, either for use with ICM, or to review them in order to understand which are needed, using two scripts provided with ICM, located in the directory /ICM/bin in the ICM container.The keygenSSH.sh script creates the needed SSH files and places them in the directory /Samples/ssh in the ICM container. The keygenTLS.sh script creates the needed SSL/TLS files and places them in /Samples/tls. You can then specify these locations when defining your deployment, or obtain your own files based on the contents of these directories. For more information about the files generated by these scripts, see Security-Related Parameters in the “ICM Reference” chapter.
Important:
The keys generated by these scripts, as well as your cloud provider credentials, must be fully secured, as they provide full access to any ICM deployments in which they are used.
These generated keys are intended as a convenience for your use in your initial test deployments. (Some have strings specific to InterSystems Corporation.) In production, the needed keys should be generated or obtained in keeping with your company's security policies.
Define the Deployment
To provide the needed parameters to ICM, you must select values for a number of fields, based on your goal and circumstances, and then incorporate these into the defaults and definitions files to be used for your deployment. (See Configuration, State and Log Files for information about these files.) You can begin with the template defaults.json and definitions.json files provided in the ICM container in the /Samples directory tree, for example /Samples/AWS.
The difference in the deployment processes for the two target configurations described at the start of this chapter lies primarily in their separate definitions files. Field values they can share are included in a joint defaults file, while the nodes that must be provisioned and configured for each deployment are specified in their definitions files.
The following sections provide the content of both the shared defaults file and the separate definitions files. Each field/value pair is shown as it would appear in the configuration file. Note that the fields included here do not represent an exhaustive list of all applicable fields; see ICM Configuration Parameters for information about these and all ICM fields.
In general, which fields are included in defaults.json and which in definitions.json depends on your needs, but as noted in Configuration, State and Log Files, defaults.json is often used to provide values for multiple deployments in a particular category, for example those that are provisioned on the same platform, while definitions.json provides values for a particular deployment.
In this case, because both deployments are on AWS, if we assume that they also use the same credentials, deploy the same InterSystems IRIS image, and include Weave Scope monitoring, those fields can be included in defaults.json, while the fields that differ go in definitions.json. The tables are ordered to reflect this approach. As previously noted, some fields, such as Provider, must appear in the defaults files. The separate definitions files can be specified in an option on the provisioning command line, or the appropriate file can be swapped in as definitions.json in the current working directory before the icm provision command is executed (see Provision the Infrastructure).
For more information about the parameters listed in these tables, see ICM Configuration Parameters.
Docker Repository and Credentials
To download and run the ICM image, and to deploy the InterSystems IRIS image and other images provided by InterSystems, you need to identify the repository in which these images are located and the credentials needed to log into that repository.
InterSystems images are distributed as Docker tar archive files, available in the InterSystems Worldwide Response Center (WRC) download area. Your enterprise may have already have added these images to its Docker repository; in this case, you should get the location of the repository and the needed credentials from the appropriate IT administrator. If your enterprise has a Docker repository but has not yet added the InterSystems images, get the location of the repository and the needed credentials, obtain the tar archive files containing the ICM and InterSystems IRIS images from the WRC, and add each of them to the repository using the following steps on the command line:
  1. For example:
    docker tag docker.intersystems.com/intersystems/icm:2018.1.0.583 acme/icm:2018.1.0.583
  2. For example:
    docker login docker.acme.com
    Username: gsanchez@acme.com
    Pasword: **********
  3. For example:
    docker push acme/icm:2018.1.0.583 
Shared Defaults File
The field/value pairs shown in the following table represent the contents of a defaults.json file that can be used for both the ECP deployment and the sharded cluster deployment.
Note:
The pathnames provided in the fields specifying security files in this sample defaults file assume you have placed your AWS credentials in the /Samples/AWS directory, and used the keygen*.sh scripts to generate the keys, as decribed in Obtain Security-Related Files. If you have generated or obtained your own keys, these may be internal paths to external storage volumes mounted when the ICM container is run, as described in the Launch ICM. See Security-Related Parameters for information about these files.
Requirement Definition Field: Value
Provisioning platform
Platform details
Amazon Web Services
Required details
"Provider": "AWS"
"Zone": "us-west-1c"
"Region": "us-west-1"
Default machine image and sizing
Default AWS AMI and instance type to provision
Default size of OS storage volume (overriding ICM default)
"AMI": "ami-d1315fb1"
"InstanceType": "m4.large"
"OSVolumeSize": "20"
SSHUser
Nonroot account with sudo access used by ICM for access to provisioned nodes.
"SSHUser": "ec2-user"
(For AWS, the required value depends on the AMI; see Provider-Specific Parameters for more information.)
Locations of security files
Locations of needed credentials and key files .
(For information about the SSH and SSL/TLS-related fields, see)
"Credentials": "/Samples/AWS/credentials"
"SSHKey": "/Samples/ssh/secure-ssh2.pub"
"KeyFile": "/Samples/ssh/secure-ssh2"
"TLSKeyDir": "/Samples/tls"
"SSLConfig": "/Samples/tls/SSLConfig.Properties"
Monitoring
Install Weave Scope
"Monitor": "scope"
Image to deploy and repository credentials
Latest InterSystems IRIS image and credentials to log into Docker repository (see Launch ICM
"DockerImage": "iscrepo/iris:stable"
"DockerUsername": "..."
"DockerPassword": "..."
InterSystems IRIS settings
Password for deployed instances
Password specified on the deployment command line (see Deploy and Manage Services) to avoid displaying the password in a configuration file
ECP Definitions File
The definitions.json file for the ECP configuration must define the following nodes:
This configuration is illustrated in the following:
ECP Configuration to be Deployed by ICM
In addition, the Mirror field must be set to True in the definitions file, as it is not used in the sharded cluster deployment and thus cannot appear in the defaults file.
The table that follows lists the field/value pairs that are required for this configuration.
Requirement Definition Field: Value
Mirroring
If Mirror is True, when two DM nodes are defined, they are mirrored.
"Mirror": "true"
Nodes for target InterSystems IRIS configuration, including provider-specific characteristics
Two data servers using a master InterSystems IRIS license. Instance type and OS volume size override defaults file (see Shared Defaults File); data volume size overrides the ICM default. Automatically configured as a mirror due to Mirror setting (next row).
"Role": "DM"
"Count": "2"
"ISCLicense": "/License/master/license.key"
"InstanceType": "m4.xlarge"
"OSVolumeSize": "32"
"DataVolumeSize": "15"
Three load-balanced ECP application servers, using separate standard InterSystems IRIS licenses. Load balancer is provisioned automatically when LoadBalancer is True.
"Role": "AM"
"Count": "3"
"StartCount": "3"
"ISCLicense": "/License/isc1.key,/License/isc2.key,/License/isc3.key"
"LoadBalancer": "true"
Arbiter node for data server mirror. Use of arbiter Docker image overrides iris image specified in defaults file. No license needed. Instance type and OS volume size override defaults file. "Role": "AR"
"Count": "1"
"StartCount": "6"
"DockerImage": "iscrepo/arbiter:stable"
"InstanceType": "t2.small"
"OSVolumeSize": "5"
Naming scheme for provisioned nodes
ECP-role-TEST-NNNN
"Label": "ECP"
"Tag": "TEST"
Sharded Cluster Definitions File
The definitions.json file for the sharded cluster configuration must define the following nodes:
These are illustrated in the following:
Sharded Cluster to be Deployed by ICM
The table that follows lists the field/value pairs that are required for this configuration.
Requirement Definition Field: Value
Nodes for target InterSystems IRIS configuration, including provider-specific characteristics
Shard master data server using an InterSystems IRIS sharding license.
Instance type overrides defaults file due to database cache requirements.
Master namespace specified.
Database cache size specified (8KB block size).
Data volume size overrides defaults file.
"Role": "DM"
"ISCLicense": "/License/master/license.key"
"InstanceType": "m4.4xlarge"
"Namespace": "master"
"ISCglobals": "0,0,40,960,0,0,0"
DataVolumeSize": "60"
Three shard data servers, no license is required and none should be specified.
Instance type overrides defaults file due to database cache requirements.
Shard namespace specified.
Database cache size specified for 8KB blocks.
Data volume size overrides defaults file.
"Role": "DS"
"Count": "4"
"InstanceType": "m4.10xlarge"
"Namespace": "shard"
"ISCglobals": "0,0,143360,0,0,0"
"DataVolumeSize": "115"
Naming scheme for provisioned nodes
SHARDING-role-TEST-NNNN
"Label": "SHARDING"
"Tag": "TEST"
Note:
For more detailed information about the specifics of deploying a sharded cluster, such as database cache size and data volume size requirements, see Deploying the Sharded Cluster in the “Horizontally Scaling InterSystems IRIS for Data Volume with Sharding” chapter of the Scalability Guide.
Provision the Infrastructure
ICM provisions cloud infrastructure using the HashiCorp Terraform tool.
Note:
ICM gives you the option of provisioning your own existing cloud or virtual compute nodes or physical servers to deploy containers on; see Deploying on a Preexisting Cluster for more information.
The icm provision Command
The icm provision command allocates and configures compute nodes, using the field values provided in the definitions.json and defaults.json files, as well as default values for unspecified parameters where applicable. By default, the input files in the current working directory are used; you can specify another location using the -definitions and -defaults options. In the case of the separate definitions files for the two target configurations (see Define the Deployment), the appropriate file can be swapped in as definitions.json in the current working directory before the icm provision command is executed.
Note:
If you use the -definitions or -defaults options to specify a nondefault location for one or both of these configuration files, you must also do so for all subsequent ICM commands you run for this deployment. For example, if you execute icm provision -defaults ./config_files, you must add -defaults ./config_files to all subsequent commands you issue for that deployment.
While the provisioning operation is ongoing, ICM provides status messages regarding the plan phase (the Terraform phase that validates the desired infrastructure and generates state files) and the apply phase (the Terraform phase that accesses the cloud provider, carries out allocation of the machines, and updates state files). Because ICM runs Terraform in multiple threads, the order in which machines are provisioned and in which additional actions applied to them is not deterministic. This is illustrated in the sample output that follows.
At completion, ICM also provides a summary of the compute nodes and associated components that have been provisioned, and outputs a command line which can be used to delete the infrastructure at a later date.
Important:
Unprovisioning public cloud compute nodes in a timely manner avoids unnecessary expense. Because the -stateDir option to the icm unprovision command is mandatory, you may find it convenient to copy the icm unprovision command provided in the output, so you can easily replicate it when unprovisioning. This output also appears in the icm.log file.
The following example if excerpted from the output of provisioning of the ECP configuration described in Define the Deployment.
$ icm provision -definitions definitions_ECP.json
Starting plan of ECP-TEST...
...completed plan of ECP-TEST
Starting apply of ECP-TEST...
...completed apply of ECP-TEST
Starting plan of ECP-DM-TEST...
...
Starting apply of ECP-DM-TEST...
...
Copying files to ECP-DM-TEST-0002...
...
Configuring ECP-AM-TEST-0003...
...
Mounting volumes on ECP-AM-TEST-0004...
...
Installing Docker on ECP-AM-TEST-0003...
...
Installing Weave Net on ECP-DM-TEST-0001...
...
Collecting Weave info for ECP-AR-TEST-0006...
...
...collected Weave info for ECP-AM-TEST-0005
...installed Weave Net on ECP-AM-TEST-0004

Machine            IP Address       DNS Name                      
-------             ---------        -------                      
ECP-DM-TEST-0001   00.53.183.209    ec2-00-53-183-209.us-west-1.compute.amazonaws.com
ECP-DM-TEST-0002   00.53.183.185    ec2-00-53-183-185.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0003   00.56.59.42      ec2-00-56-59-42.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0005   00.67.1.11       ec2-00-67-1-11.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0003   00.193.117.217   ec2-00-193-117-217.us-west-1.compute.amazonaws.com
ECP-LB-TEST-0002   (virtual AM)     ECP-00-TEST-1687579230.us-west-1.elb.amazonaws.com
ECP-AR-TEST-0006   00.53.201.194    ec2-00-53-201-194.us-west-1.compute.amazonaws.com
To destroy: icm unprovision -stateDir /Samples/AWS/ICM-8620265620732464265 [-cleanUp] [-force]

During the provisioning operation, ICM creates or updates state and log files in the state directory (created by ICM, with a name beginning with ICM-) and when finished creates the instances.json file, which serves as input to subsequent deployment and management commands. (See The Instances File for more information about this file.) By default, the instances file is created in the current working directory; you can change this using the -instances option, but note that if you do you must supply the alternate location by using the -instances option with all subsequent commands.
Because interactions with cloud providers sometimes involve high latency leading to timeouts and internal errors on the provider side, the icm provision command is fully reentrant to make the provisioning process as resilient as possible. If errors are encountered during provisioning, the command can be issued multiple times until ICM completes all the required tasks for all the specified nodes without error. When this happens, however, you must use the -stateDir option to specify the state directory (see The State Directory and State Files) in your repeated execution of the command, to indicate that provisioning is already in process and provide the needed information about what has been done and what hasn’t. For example, suppose you encounter the problem in the following example:
$ icm provision
Starting plan of ECP-DM-TEST...
...completed plan of ECP-DM-TEST
Starting apply of ECP-AM-TEST...
Error: Thread exited with value 1
See /Samples/AWS/ICM-1105110161490759817/Sample-DS-TEST/terraform.err
To reprovision, specify --stateDir=/Samples/AWS/ICM-3078941885014382438
Review the indicated errors, fix as needed, then run icm provision again with the -stateDir option, as in the following
$ icm provision --stateDir=/Samples/AWS/ICM-3078941885014382438
Starting plan of ECP-DM-TEST...
...completed plan of ECP-DM-TEST
Starting apply of ECP-DM-TEST...
...completed apply of ECP -DM-TEST
[...]
To destroy: icm unprovision -stateDir /tmp/ICM-3078941885014382438 [-cleanUp] [-force]

Even when provisioning is successful, you can run the icm provision command again (with the -stateDir option) after making changes to your configuration files to alter the provisioned infrastructure. For example, you can change storage volume sizes for some of the nodes in the definitions file and execute icm provision again to reprovision with the new sizes.
By default, when issuing the icm provision command to modify existing infrastructure, ICM prompts you to confirm; you can avoid this by using the -force option, for example when using a script.
Infrastructure Management Commands
The commands in this section are used to manage the infrastructure you have provisioned using ICM.
icm inventory
The icm inventory command lists the provisioned nodes, as at the end of the provisioning output, based on the information in the instances.json file. For example:
$ icm inventory
Machine            IP Address       DNS Name                      
-------            ----------       --------                      
ECP-DM-TEST-0001   00.53.183.209    ec2-52-53-183-209.us-west-1.compute.amazonaws.com
ECP-DM-TEST-0002   00.53.183.185    ec2-52-53-183-185.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0003   00.56.59.42      ec2-13-56-59-42.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0005   00.67.1.11       ec2-54-67-1-11.us-west-1.compute.amazonaws.com
ECP-AM-TEST-0003   00.193.117.217   ec2-54-193-117-217.us-west-1.compute.amazonaws.com
ECP-LB-TEST-0002   (virtual AM)     ECP-AM-TEST-1687579230.us-west-1.elb.amazonaws.com
ECP-AR-TEST-0006   00.53.201.194    ec2-52-53-201-194.us-west-1.compute.amazonaws.com
icm ssh
The icm ssh command runs an arbitrary command on the specified compute nodes. Because mixing output from multiple commands would be hard to interpret, the output is written to files and a list of output files provided, for example:
$ icm ssh -command "ping -c 5 intersystems.com" -role DM
Executing command 'ping -c 5 intersystems.com' on ECP-DM-TEST-0001...
Executing command 'ping -c 5 intersystems.com' on ECP-DM-TEST-0002...
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-DM-TEST-0001/ssh.out
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-DM-TEST-0002/ssh.out
However, when the -machine or -role options are used to specify exactly one node, as in the following, the output is also written to the console:
$ icm ssh -command "df -k" -machine ECP-DM-TEST-0001
Executing command 'df -k' on ECP-DM-TEST-0001...
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-DM-TEST-0001/ssh.out 

Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs          10474496 2205468   8269028  22% /
tmpfs            3874116       0   3874116   0% /dev
tmpfs            3874116       0   3874116   0% /sys/fs/cgroup
/dev/xvda2      33542124 3766604  29775520  12% /host
/dev/xvdb       10190100   36888   9612540   1% /intersys/data
/dev/xvdc       10190100   36888   9612540   1% /intersys/wij
/dev/xvdd       10190100   36888   9612540   1% /intersys/journal1
/dev/xvde       10190100   36888   9612540   1% /intersys/journal2
shm                65536     492     65044   1% /dev/shm
The icm ssh command can also be used in interactive mode to execute long-running, blocking, or interactive commands on a compute node. Unless the command is run on a single-node deployment, the -interactive flag must be accompanied by a -role or -machine option restricting the command to a single node. If the -command option is not used, the destination user's default shell (for example bash) is launched.
See icm exec for an example of running a command interactively.
Note:
Two commands described in Service Management Commands, icm exec (which runs an arbitrary command on the specified containers) and icm terminal (which opens a Terminal for the InterSystems IRIS instance on the specified node) can be grouped with icm ssh as a set of powerful tools for interacting with your ICM deployment.
icm scp
The icm scp command securely copies a file or directory from the local ICM container to the host OS of the specified node or nodes. The command syntax is as follows:
icm scp localPath -local-path [-remotePath remote-path]
Both localPath and remotePath can be either files or directories. If remotePath is a directory, it must contain a trailing forward slash (/), or it will be assumed to be a file. If both are directories, the contents of the local directory are recursively copied; if you want the directory itself to be copied, remove the trailing slash (/) from localPath.
The default for the optional remote-path argument is /home/ssh-user. The root directory of this path, /home, is the default home directory; to change it, specify a different root directory using the Home field. The user specified by the SSHUser field must have the needed permissions for remotePath.
Note:
See also the icm cp command, which copies a local file or directory on the specified node into the specified container.
Deploy and Manage Services
ICM carries out deployment of software services using Docker images, which it runs as containers by making calls to Docker. Containerized deployment using images supports ease of use and DevOps adaptation while avoiding the risks of manual upgrade. In addition to Docker, ICM also carries out some InterSystems IRIS-specific configuration over JDBC.
There are many container management tools available, and these can be used to extend ICM’s deployment and management capabilities.
The icm run Command
The icm run command pulls, creates, and starts a container from the specified image on each of the provisioned nodes. By default, the image specified by the DockerImage field in the configuration files is used; by including this field in node definitions in the definitions.json file, you can run different images on different node types. For example, you could run an arbiter (ISCAgent) image on the AR node and a csp (Web Gateway) image on WS nodes while running the iris image on the others.
You can also use the -image command-line option to specify a different image. This allows you to deploy multiple containers created from multiple images on each provisioned node by using the icm run command multiple times. For example, you might use icm run once to run the images specified by the DockerImage fields in the node definitions, as described in the previous paragraph, and a second time with the -image option to run a custom image on all of the nodes. Each container running on a given node must have a unique name, so when repeating the icm run command to deploy multiple containers on one or more nodes, you must also use the -container option to specify different names for containers created from different images.
By default, the name of the deployed container is iris. This name is reserved for and should be used only for containers derived from the following InterSystems images (or images based on these images):
Typically, then, multiple container deployment proceeds as follows:
The -machine and -role options can also be used to restrict container deployment to a particular node, or to nodes of a particular type, for example, when deploying your own custom container on a specific provisioned node.
Another frequently used option, -iscPassword, specifies the InterSystems IRIS password to set for all deployed InterSystems IRIS containers; this value could be included in the configuration files, but the command line option avoids committing a password to a plain-text record. If the InterSystems IRIS password is not provided by either method, ICM prompts for it (with typing masked).
Given all of the preceding, consider the following three examples of container deployment using the icm run command. (These do not present complete procedures, but are limited to the procedural elements relating to the deployment of particular containers on particular nodes.)
Bear in mind the following further considerations:
Additional Docker options, such as --volume, can be specified on the icm run command line using the -options option, for example:
icm run -options "--volume /shared:/host" image iscrepo/iris:stable
For more information on the -options option, see Using ICM with Custom and Third-Party Containers.
The -command option can be used with icm run to provide arguments to (or in place of) the Docker entry point; for more information, see Overriding Default Commands.
Because ICM issues Docker commands in multiple threads, the order in which containers are deployed on nodes is not deterministic. This is illustrated in the sample output that follows.
Important:
The icm run command is not reentrant. If the command fails on one or more nodes due to factors such as network latency and disconnects or interruptions in cloud provider service, you must manually roll back deployment on all nodes before executing icm run again. The procedure involves commands described in detail later in this section, but is summarized as follows:
  1. Run the icm stop command on all nodes to stop any containers that were successfully deployed.
  2. Run the icm rm command on all nodes to remove the deployed containers.
  3. Execute the following icm ssh command to remove InterSystems IRIS data written to the external volumes, checking the output files when the command completes to verify successful completion on each node:
    icm ssh -command "sudo rm -rf /intersys/*/*"
  4. Issue icm run again.
In the following example, representing output from deployment of the sharded cluster configuration described in Define the Deployment, repetitive lines are omitted for brevity.
$ icm run -definitions definitions_cluster.json
Executing command 'docker login' on SHARDING-DM-TEST-0001...
...output in /Samples/AWS/ICM-8620265620732464265/Sample-DM-TEST/SHARDING-DM-TEST-0001/docker.out
...
Running container iris on SHARDING-DS-TEST-0004...
...
Waiting for InterSystems IRIS to start on SHARDING-DS-TEST-0002...
...
Configuring SSL on SHARDING-DM-TEST-0001...
...
Changing passwords on SHARDING-DS-TEST-0004...
...
Enabling ECP on SHARDING-DS-TEST-0003...
...
Licensing shards on SHARDING-DS-TEST-0002...
...
Loading %ZSTART routine on SHARDING-DS-TEST-0002...
...
Creating namespace 'NS2' on SHARDING-DM-TEST-0001...
...
Enabling shard service on SHARDING-DM-TEST-0001...
...
Assigning shards on SHARDING-DM-TEST-0001...
...
Configuring ODBC on SHARDING-DM-TEST-0003...
...
Management Portal available at: http://ec2-00-153-49-109.us-west-1.compute.amazonaws.com:57772/csp/sys/UtilHome.csp 
At completion, ICM outputs a link to the Management Portal of the appropriate InterSystems IRIS instance. In this case, the provided Management Portal link is for the shard master data server running in the InterSystems IRIS container on SHARDING-DM-TEST-001.
Container Management Commands
The commands in this section are used to manage the containers you have deployed on your provisioned infrastructure.
icm ps
When deployment is complete, the icm ps command shows you the run state of all containers running on the nodes, for example:
$ icm ps -container iris
Machine                  IP Address      Container    Status   Image
-------                  ----------      ---------    ------   -----
SHARDING-DS-TEST-0004    00.56.140.23    iris        Up       iscrepo/iris:stable
SHARDING-DS-TEST-0003    00.53.190.37    iris        Up       iscrepo/iris:stable
SHARDING-DS-TEST-0002    00.67.116.202   iris        Up       iscrepo/iris:stable
SHARDING-DM-TEST-0001    00.153.49.109   iris        Up       iscrepo/iris:stable

If the -container restriction is omitted, all containers running on nodes are (including any run outside of ICM). In this case, the Weave network containers would also be shown (along with any other you might have run since the end of ICM deployment).
Additional deployment and management phase commands are listed in the following. For complete information about these commands, see ICM Reference.
icm stop
The icm stop command stops the specified containers (or iris by default) on the specified nodes, or on all nodes if no machine or role constraints provided). For example, to stop the InterSystems IRIS containers on the application servers in the ECP configuration:
$ icm stop -container iris -role DS

Stopping container iris on SHARDING-DS-TEST-0002...
Stopping container iris on SHARDING-DS-TEST-0004...
Stopping container iris on SHARDING-DS-TEST-0003...
...completed stop of container iris on SHARDING-DS-TEST-0004
...completed stop of container iris on SHARDING-DS-TEST-0002
...completed stop of container iris on SHARDING-DS-TEST-0003
icm start
The icm start command starts the specified containers (or iris by default) on the specified nodes, or on all nodes if no machine or role constraints provided). For example, to restart one of the stopped application server InterSystems IRIS containers:
$ icm start -container iris -machine SHARDING-DS-TEST-0002...
Starting container iris on SHARDING-DS-TEST-0002...
...completed start of container iris on SHARDING-DS-TEST-0002
icm pull
The icm pull command downloads the specified image to the specified machines. For example, to add an image to the shard master data server in the sharded cluster:
$ icm pull -image iscrepo/csp:stable -role DM
Pulling SHARDING-DM-TEST-0001 image iscrepo/csp:stable
...pulled SHARDING-DM-TEST-0001 image iscrepo/csp:stable

Note that the -image option is not required if the image you want to pull is the one specified by the DockerImage field in the definitions file, for example:
"DockerImage": "iscrepo/iris:stable",
Although the icm run automatically command pulls any images not already present on the host, an explicit icm pull might be desirable for testing, staging, or other purposes.
icm rm
The icm rm command deletes the specified container (or iris by default), but not the image from which it was started, from the specified nodes, or from all nodes if no machine or role is specified. Only a stopped container can be deleted.
icm upgrade
The icm upgrade command replaces the specified container on the specified machines. ICM orchestrates the following sequence of events to carry out an upgrade:
  1. Pull the new image
  2. Create the new container
  3. Stop the existing container
  4. Remove the existing container
  5. Start the new container
By staging the new image in steps 1 and 2, the downtime required between steps 3-5 is kept relatively short.
For example, to upgrade the InterSystems IRIS container on the shard master data server:
$ icm upgrade -image iscrepo/iris:latest -machine ECP-AM-TEST-0003
Pulling ECP-AM-TEST-0003 image iscrepo/iris:latest...
...pulled ECP-AM-TEST-0003 image iscrepo/iris:latest
Stopping container ECP-AM-TEST-0003...
...completed stop of container ECP-AM-TEST-0003
Removing container ECP-AM-TEST-0003...
...removed container ECP-AM-TEST-0003
Running image iscrepo/iris:latest in container ECP-AM-TEST-0003...
...running image iscrepo/iris:latest in container ECP-AM-TEST-0003
The -image option is required for the icm upgrade command. If you are upgrading a container other than iris, you must use the -container option to specify the container name.
Service Management Commands
These commands let you interact with the services running in your deployed containers, including InterSystems IRIS.
A significant feature of ICM is the ability it provides to interact with the nodes of your deployment on several levels — with the node itself, with the container deployed on it, and with the running InterSystems IRIS instance inside the container. The icm ssh (described in Infrastructure Management Commands), which lets you run a command on the specified compute nodes, can be grouped with the first two commands described in this section, icm exec (run a command in the specified containers) and icm terminal (open a Terminal window for an InterSystems IRIS instance in a container) as a set of powerful tools for interacting with your ICM deployment. These multiple levels of interaction are shown in the following illustration.
Interactive ICM Commands
icm exec
The icm exec command runs an arbitrary command in the specified containers, for example
$ icm exec -command "df -k" -machine ECP-DM-TEST-0001
Executing command in container iris on ECP-DM-TEST-0001
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-DM-TEST-0001/docker.out

Filesystem     1K-blocks    Used Available Use% Mounted on
rootfs          10474496 2205468   8269028  22% /
tmpfs            3874116       0   3874116   0% /dev
tmpfs            3874116       0   3874116   0% /sys/fs/cgroup
/dev/xvda2      33542124 3766604  29775520  12% /host
/dev/xvdb       10190100   36888   9612540   1% /intersys/data
/dev/xvdc       10190100   36888   9612540   1% /intersys/wij
/dev/xvdd       10190100   36888   9612540   1% /intersys/journal1
/dev/xvde       10190100   36888   9612540   1% /intersys/journal2
shm                65536     492     65044   1% /dev/shm
Because mixing output from multiple commands would be hard to interpret, when the command is executed on more than one node, the output is written to files and a list of output files provided.
You can use the -options option with icm exec to add Docker options to your command. For example, to set an environment variable before executing a command, add -options "--env var=value".
icm exec -command <command> -options "--env dir=/home/user/bin"
The icm exec command can also be used in interactive mode to execute long-running, blocking, or interactive commands within a container. Unless the command is run on a single-node deployment, the -interactive flag must be accompanied by a -role or -machine option restricting the command to a single node. A good example is running a shell in the container:
$ icm exec -command bash -machine ECP-AM-TEST-0004 -interactive
Executing command 'bash' in container iris on ECP-AM-TEST-0004...
[root@localhost /] $ whoami
root
[root@localhost /] $ hostname
iris-ECP-AM-TEST-0004
[root@localhost /] $ exit
icm terminal
The icm terminal command opens a Terminal for the InterSystems IRIS instance on the specified node. The -namespace option can be used to specify the namespace in which the Terminal opens. For example:
$ icm terminal -machine ECP-AM-TEST-0003 -namespace %SYS

Node: iris-ECP-AM-TEST-0003, Instance: IRIS

Username: _SYSTEM
Password: ********
%SYS> 
You can also use the -command option to provide a routine to be run in the Terminal, for example:
icm terminal -namespace %SYS -command ^MIRROR
icm cp
The icm cp command copies a local file or directory on the specified node into the specified container. The command syntax is as follows:
icm cp -localPath local-path [-remotePath remote-path]
Both localPath and remotePath can be either files or directories. If both are directories, the contents of the local directory are recursively copied; if you want the directory itself to be copied, include it in remotePath.
The remotePath argument is optional and if omitted defaults to /tmp; if remotePath is a directory, it must contain a trailing forward slash (/), or it will be assumed to be a file. You can use the -container option to copy to a container other than the default iris.
Note:
See also the icm scp command, which securely copies a file or directory from the local ICM container to the specified host OS.
icm sql
The icm sql command runs an arbitrary SQL command against the containerized InterSystems IRIS instance on the specified node (or all nodes), for example:
$ icm sql -command "SELECT Name,SMSGateway FROM %SYS.PhoneProviders" -role DM
Executing command in container iris on Acme-DM-TEST-0001...
...output in ./ICM-4780136574/Acme-DM-TEST/Acme-DM-TEST-0001/jdbc.out

Name,SMSGateway
AT&amp;T Wireless,txt.att.net
Alltel,message.alltel.com
Cellular One,mobile.celloneusa.com
Nextel,messaging.nextel.com
Sprint PCS,messaging.sprintpcs.com
T-Mobile,tmomail.net
Verizon,vtext.com
The -namespace option can be used to specify the namespace in which the SQL command runs.
Because mixing output from multiple commands would be hard to interpret, when the command is executed on more than one node, the output is written to files and a list of output files provided.
icm iscos
The icm iscos command runs an arbitrary InterSystems ObjectScript snippet, specified by the -command option, on the specified node (or all nodes). The -namespace option can be used to specify the namespace in which the snippet runs.
Because mixing output from multiple commands would be hard to interpret, when the command is executed on more than one node, the output is written to files and a list of output files provided.
$ icm iscos -command 'W ##class(%File).Exists("foo.txt")' -role AM
Executing command in container iris on ECP-AM-TEST-0003...
Executing command in container iris on ECP-AM-TEST-0004...
Executing command in container iris on ECP-AM-TEST-0005...
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-AM-TEST-0003/ssh.out
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-AM-TEST-0004/ssh.out
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-AM-TEST-0005/ssh.out
icm docker
The icm docker command runs a Docker command on the specified node (or all nodes), for example:
$ icm docker -command "stats --no-stream" -machine ECP-DM-TEST-0002
Executing command 'status --no-stream' on ECP-DM-TEST-0002...
...output in ./ICM-4780136574/ECP-DM-TEST/ECP-DM-TEST-0002/docker.out

CONTAINER     CPU %  MEM USAGE / LIMIT  MEM %  NET I/O     BLOCK I/O        PIDS
3e94c3b20340  0.01%  606.9MiB/7.389GiB  8.02%  5.6B/3.5kB  464.5MB/21.79MB  0
1952342e3b6b  0.10%  22.17MiB/7.389GiB  0.29%  0B/0B       13.72MB/0B       0
d3bb3f9a756c  0.10%  40.54MiB/7.389GiB  0.54%  0B/0B       38.43MB/0B       0
46b263cb3799  0.14%  56.61MiB/7.389GiB  0.75%  0B/0B       19.32MB/231.9kB  0
The Docker command should not be long-running (or block), otherwise control will not return to ICM. For example, if the ---no-stream option in the example is removed, the call will not return until a timeout has expired.
Unprovision the Infrastructure
Because public cloud platform instances continually generate charges and unused instances in private clouds consume resources to no purpose, it is important to unprovision infrastructure in a timely manner.
The icm unprovision command deallocates the provisioned infrastructure based on the state files created during provisioning. The -stateDir option is required. As described in Provision the Infrastructure, destroy refers to the Terraform phase that deallocates the infrastructure. One line is created for each entry in the definitions file, regardless of how many nodes of that type were provisioned. Because ICM runs Terraform in multiple threads, the order in which machines are unprovisioned is not deterministic.
$ icm unprovision -stateDir /Samples/AWS/ICM-2416821167214483124 -cleanUp
Type "yes" to confirm: yes
Starting destroy of ECP-DM-TEST...
Starting destroy of ECP-AM-TEST...
Starting destroy of ECP-AR-TEST...
...completed destroy of ECP-AR-TEST
...completed destroy of ECP-AM-TEST
...completed destroy of ECP-DM-TEST
Starting destroy of ECP-TEST...
...completed destroy of Binstock-TEST
The -cleanUp option deletes the state directory after unprovisioning; by default, the state directory is preserved. The icm unprovision command prompts you to confirm unprovisioning by default; you can use the -force option to avoid this, for example when using a script.