Using the InterSystems Kubernetes Operator
This document explains how to use the InterSystems Kubernetes Operator (IKO) to deploy InterSystems IRIS sharded clusters on Kubernetes platforms.
Why would I use Kubernetes?
Kubernetes is an open-source orchestration engine for automating deployment, scaling, and management of containerized workloads and services, and excels at orchestrating complex SaaS (software as a service) applications. You provision a Kubernetes-enabled cluster and tell Kubernetes the containerized services you want to deploy on it and the policies you want them to be governed by; Kubernetes transparently provides the needed resources in the most efficient way possible, repairs or restores the configuration when problems with those resources cause it to deviate from what you specified, and can scale automatically or on demand. In the simplest terms, Kubernetes deploys a multicontainer application in the configuration and at the scale you specify on any Kubernetes-enabled platform, and keeps the application operating exactly as you described it.
Why do I need the InterSystems Kubernetes Operator?
The InterSystems Kubernetes Operator (IKO) extends the Kubernetes API with a custom resource representing an InterSystems IRIS sharded cluster. This resource, called an iriscluster, can be deployed on any Kubernetes platform on which the IKO is installed. (The IKO can also deploy a stand-alone instance of InterSystems IRIS on Kubernetes.)
The operator isn’t required to deploy InterSystems IRIS under Kubernetes. But because Kubernetes is application-independent, you would need to create custom definitions and scripts to handle all the needed configuration of the InterSystems IRIS instances or other components in the deployed containers along with networking, persistent storage requirements, and so on. Installing the IKO automates these tasks. By putting together a few settings that define the cluster, for example the number of data and compute nodes, whether they should be mirrored, and where the Docker credentials needed to pull the container images are stored, you can easily deploy your InterSystems IRIS cluster exactly as you want it. The operator also adds InterSystems IRIS-specific cluster management capabilities to Kubernetes, enabling tasks like adding data or compute nodes, which you would otherwise have to do manually by interacting directly with the instances.
Start with your use case
Before beginning your work with the IKO, you should confirm that:
The InterSystems IRIS sharding architecture will be beneficial for the anticipated workload.
Containerized deployment of InterSystems IRIS and your application is the best approach.
Kubernetes is the right orchestrator for your containerized InterSystems-IRIS based application, and that you have identified one or more suitable Kubernetes platforms to deploy it on. For example, major public clouds platforms include Google Kubernetes Engine (GKE), Azure Container Service (AKS), Amazon Elastic Container Service for Kubernetes (EKS), and Tencent Kubernetes Engine (TKE), while platforms such as Red Hat OpenShift, Rancher Kubernetes Engine (RKE), and Docker Enterprise can be used on any infrastructure.
Plan your sharded cluster
For the most beneficial results, it is important to fully plan the configuration of your sharded cluster and its data, including:
The number of data nodes in the cluster and their configuration, such as their database cache size, the storage used for their default databases, and so on)
Whether the data nodes are to mirrored for high availability
Whether to include compute nodes for workload separation and increased query throughput
The schema for the sharded and nonsharded data to be loaded onto the cluster.
For detailed information about InterSystems IRIS sharded clusters, see Horizontally Scaling for Data Volume with Sharding in the Scalability Guide.
Learn to speak Kubernetes
While it is possible to use the IKO if you have not already worked with Kubernetes, InterSystems recommends having or gaining a working familiarity with Kubernetes before deploying with the IKO.
Choose a platform and understand the interface
When you have selected the Kubernetes platform you will deploy on, create an account and familiarize yourself with the provided interface(s) to Kubernetes. For example, to use GKE on Google Cloud Platform, you can open a Google Cloud Shell terminal and file editor to use the gcloud and kubectl command line interfaces. Bear in mind that the configuration of your Kubernetes environment should include access to the availability zones in which you want to deploy the sharded cluster.
Deploy a Kubernetes container cluster to host the iricluster
The Kubernetes cluster is the structure on which your containerized services are deployed and through which they are scaled and managed. The procedure for deploying a cluster varies to some degree among platforms. In planning and deploying your Kubernetes cluster, bear in mind the following considerations:
The IKO deploys one InterSystems IRIS or arbiter container (if a mirrored cluster) per Kubernetes pod, and attempts to deploy one pod per Kubernetes cluster node when possible. Ensure that
You are deploying the desired number of nodes to host the pods of your sharded cluster, including the needed distribution across zones if more than one zone is specified (see below).
The required compute and storage resources will be available to those nodes.
If your sharded cluster is to be mirrored and you plan to enforce zone antiaffinity using the preferredZones fields in the iriscluster definition to deploy the members of each failover pair in separate zones and the arbiter in an additional zone, the container cluster must be deployed in three zones. For example, if you plan to use zone antiaffinity and are deploying the cluster using the gcloud command-line interface, you might select zones us-east1-b,c,d and create the container cluster with a command like this:
$ gcloud container clusters create my-iriscluster --node-locations us-east1-b,us-east1-c,us-east1-dCopy code to clipboard
Upgrade Helm if necessary
Helm packages Kubernetes applications as charts, making it easy to install them on any Kubernetes platform. Because the IKO Helm chart requires Helm version 3, you must confirm that this is the version on your platform, which you can do by issuing the command helm version. If you need to upgrade Helm to version 3, you can use the curl script at https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3. For example:
$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6827 100 6827 0 0 74998 0 --:--:-- --:--:-- --:--:-- 75855 Helm v3.2.3 is available. Changing from version . Downloading https://get.helm.sh/helm-v3.2.3-linux-amd64.tar.gz Preparing to install helm into /usr/local/bin helm installed into /usr/local/bin/helm
Download the IKO archive and upload the extracted contents to Kubernetes
Obtain the IKO archive file, for example iris_operator-126.96.36.1993.0-unix.tar.gz, from the InterSystems Worldwide Response Center (WRC) download area and extract its contents, then upload the extracted directory, iris_operator-version (for example iris_operator-188.8.131.523.0) to the Kubernetes platform. This directory contains the following:
The image/ directory contains an archive file of the IKO image.
The chart/iris-operator directory contains the Helm chart for the operator.
The samples/ directory contains template .yaml and .cpf files ,as described later in this procedure.
Load the IKO image
Make the IKO image available on the Kubernetes platform using the docker load command, for example:
$ docker load -i iris_operator-184.108.40.2063.0/image/iris_operator-220.127.116.113.0-docker.tgz 0d1435bd79e4: Loading layer [=============================================>] 3.062MB/3.062MB 478805a2e99f: Loading layer [=============================================>] 75.54MB/75.54MB Loaded image: intersystems/iris-operator:18.104.22.1683.0 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE intersystems/iris-operator v1.0.0 ad083ca291a2 1 minute ago 77.4MB
Update the values.yaml file
In the chart/iris-operator directory, ensure that the values in the operator section of the file values.yaml, which describe the IKO image, match the IKO image you just loaded on the Kubernetes platform, for example:
operator: registry: intersystems repository: iris-operator tag: 22.214.171.1243.0
Install the IKO
Use Helm to install the operator on the Kubernetes cluster. For example, on GKE you would use the following command:
$ helm install intersystems iris_operator-126.96.36.1993.0/chart/iris-operator NAME: intersystems LAST DEPLOYED: Mon Jun 15 16:43:21 2020 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: To verify that InterSystems Kubernetes Operator has started, run: kubectl --namespace=default get deployments -l "release=intersystems, app=iris-operator" bbinstoc@cloudshell:~ (isc-mktplace-dev)$ kubectl get deployments -l "release=intersystems, app=iris-operator" NAME READY UP-TO-DATE AVAILABLE AGE intersystems-iris-operator 0/1 1 0 30s
Choose the default InterSystems IRIS password and generate a hash
InterSystems IRIS is installed with several predefined user accounts, the initial password for which is SYS. For effective security, it is important that this default password be changed immediately upon deployment of your container. For this reason, the IKO requires you to include a cryptographic hash (with salt) of the new default password in the YAML definition of your iriscluster. InterSystems publishes the image for a nanocontainer, iris-passwordhash, that generates the hash for you; for more information, see Authentication and Passwords in Running InterSystems Products in Cotainers.
Create a secret for image pull information
Kubernetes secrets let you safely and flexibly store and manage sensitive information. Identify the registry, repository, and tag of the InterSystems IRIS image and (if the cluster is to be mirrored) arbiter image to be deployed, and create a Kubernetes secret of type docker-registry containing the URL of the registry and the credentials needed to log into Docker on that registry to pull the images from it. For example, suppose the images are:
The following kubectl command is an example of creating a secret named acme-secret for pulling these images (docker-email is optional):
$ kubectl create secret docker-registry acme-pull-secret --docker-server=https://index.docker.acme.com/latest/ --docker-username=***** --docker-password='*****' --docker-email=**********
Create a secret containing the InterSystems IRIS license key
Upload the sharding-enabled license key for the InterSystems IRIS images in your sharded cluster, and create a Kubernetes secret of type generic to contain the key, allowing it to be mounted on a temporary file system within the container, for example:
$ kubectl create secret generic iris-key-secret --from-file=iris.key
Create configuration parameter files and a config map for them.
The configuration parameter file, also called the CPF, defines the configuration of an InterSystems IRIS instance. On startup, InterSystems IRIS reads the CPF to obtain the values for most of its settings. The CPF merge feature allows you to specify a merge file containing configuration that are different from, and overwrite, the settings in the default CPF that comes with an instance when it is deployed. For details, see Introduction to the Configuration Parameter File.
Kubernetes config maps keep your containerized applications portable by separating configuration artifacts, such as CPF merge files, from image content. If you want to use separate CPF merge files to update the configurations of the iriscluster’s data nodes and compute nodes at deployment, you must do two things:
Customize the provided template data.cpf and compute.cpf files (they contain only the SystemMode setting, which displays text on the InterSystems IRIS management portal) with the CPF settings you want to apply to the data and compute nodes. For example, as described in Deploying the Sharded Cluster in the Scalability Guide, the data nodes in a sharded cluster must be configured to allocate a database cache of the appropriate size, and typically the default generic memory heap size should be increased as well. To do this, you would add the [config] section globals and gmheap parameters to the data.cpf file, as shown below, to configure the database cache and generic memory heap of each data node at 200 GB and 256 MB, respectively:
[StartUp] SystemMode=my-iriscluster [config] globals= 0,0,204800,0,0,0 gmheap= 262144Copy code to clipboard
Create a Kubernetes config map specifying the files, using a command like this:
$ kubectl create cm iris-cpf --from-file data.cpf --from-file compute.cpfCopy code to clipboard
Create a storage class for persistent storage
Kubernetes provides the persistent storage needed by containerized programs (in this case, the data nodes) in the form of persistent volumes; a persistent volume claim (pvc) is a request by a user for a persistent volume. To include these in the cluster, you need to specify in the definition a Kubernetes storage class, which determines the type of storage used. Because storage types are platform specific, the storage class must be defined separately from the cluster, which is why multiple template storage class definition files are provided with the IKO. For example, the template storage class definition file for GKE is named iris-ssd-sc-gke.yaml and has the following contents:
kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: iris-ssd-storageclass provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd volumeBindingMode: WaitForFirstConsumer
The volumeBindingMode: WaitForFirstConsumer setting is required for the correct operation of the IKO.
Update the storage class definition as needed and then create the storage class, for example:
$ kubectl create -f iris-ssd-sc-gke.yaml
Using the definition file shown above, this command creates a storage class called iris-ssd-storageclass.
Create the iriscluster definition file
You are now ready to create your iriscluster definition file by customizing the template file provided with the IKO, my-iriscluster-definition.yaml. The contents of this file and suggestions for customizing it are shown in the following sections.
For each section of the definition, there are numerous other Kubernetes fields that can be included; this section discusses only those specific to or required for an iriscluster definition.
apiVersion: intersystems.com/v1alpha1 kind: IrisCluster metadata: name: my-iriscluster spec:
The first three fields are required by Kubernetes, as is the spec: section, which contains nested fields specifying a particular type of object, in this case an iriscluster. Change the value of the name:field in metadata: to the name you want to give the cluster.
The passwordHash field specifies the cryptographic hash of the desired default password for the InterSystems IRIS instances in the cluster; insert the hash you generated as the value.
licenseKeySecret: name: iris-key-secret
The licenseKeySecret is the Kubernetes secret containing the InterSystems IRIS license key; update the name: field with the name of the license key secret you created.
configSource: name: iris-cpf
The configSource is the config map containing the CPF merge files, data.cpf and compute.cpf. Update the name: field with the name of the config map you created; if you did not create one, do not specify a value for this field.
The topology section specifies the details of the cluster’s data nodes, compute nodes (if any), and arbiter node (if mirrored). Only the data: section is required.
data: shards: <1..N>
In the data: definition, the shards: field specifies the number of data nodes to deploy. Data nodes can be added to the deployed cluster by increasing this setting and reapplying the definition, but the setting cannot be decreased.
The shards: field is optional, with a default value of 1. If it is omitted, and the compute: and arbiter: sections of the definition are also omitted, applying the definition results in stand-alone deployment of a single instance of InterSystems IRIS, rather than a sharded cluster. If the mirrored: field, described below, is set to true, two stand-alone instances are deployed as the primary and backup in a mirror (and you must include the arbiter: section, described later, in the definition).
If mirrored: true, data nodes are mirrored, and two instances are deployed for each data node specified by the shards: field. For example, if shards: 4 and mirrored: true, eight data node instances are deployed as four failover pairs.
The image: field specifies the repository, name, and tag of the InterSystems IRIS image from which to deploy data node containers.
preferredZones: - us-east1-b - us-east1-c
Optionally, use the preferredZones: field to specify the zone or zones in which data nodes are to be deployed, as follows:
If mirrored: true and at least two zones are specified, Kubernetes is discouraged from deploying both members of a failover pair in the same zone, which maximizes the chances that at least one is available.Note:
Bear in mind that deploying the members of a failover pair in separate zones is likely to slightly increase latency in the synchronous communication between them. In addition, specifying multiple zones for the data nodes means that all of the primaries might not be deployed in the same zone, resulting incommunication between the data nodes also being affected by this slightly increased latency. Finally, if compute nodes are included in the cluster, specifying multiple zones for data nodes is very likely to result in some compute nodes being deployed in a separate zone from their associated data nodes, increasing latency in those connections as well.
Under most circumstances these interzone latency effects will be neglible, but with some demanding workloads involving high message or query volume, performance may be affected. If after researching the issue of interzone connections on your Kubernetes platform and testing your application thoroughly you are concerned about this performance impact, consider specifying a single zone for your mirrored data nodes.
Regardless of the zones you specify here, you should use the preferredZones: field in the arbiter: definition (as illustrated later in this section) to deploy the arbiter in a separate zone of its own, which also helps optimize mirror availability.
The data nodes of an unmirrored cluster are typically deployed in the same zone to minimized latency. If mirrored: false and your Kubernetes cluster includes multiple zones, you can use preferredZones to follow this practice by specifying a single zone in which to deploy the data nodes.
Kubernetes attempts to deploy in the specified zones, but if this is not possible, deployment proceeds rather than failing.
podTemplate: spec: imagePullSecrets: - name: dockerhub-secret
The imagePullSecrets field identifies the Kubernetes secret containing the credentials and other information to pull the image specified by the image: field; update the name; field with the name of the image pull secret you created.
updateStrategy: type: RollingUpdate
The Kubernetes update strategy type RollingUpdate is required for the iriscluster.
storage: accessModes: - ReadWriteOnce
The storage section defines the persistent volume claim required for each data node. The accessModes: field specifies the access mode for the persistent volume claim, which must be RWO to enforce a one-to-one relationship between each pod and the persistent volume claim used by the InterSystems IRIS container in the pod for persistent storage.
resources: requests: storage: 100G
The storage: field defines the amount of storage (expressed as any unit between kilobytes and exabytes) available to the persistent volume claim. The amount of storage required by data nodes is determined during the cluster planning process and should include a comfortable margin for the future growth of your workload.
The storageClassName: field identifies the storage class used for the persistent volume claims; enter the name of the storage class you created as its value.
compute: image: intersystems/iris:2020.1.0-dev
The compute: definition shares several fields with the data: definition. The values for most are the same, for example the image: field specifying the InterSystems IRIS image spec.
The replica: field specifies the number of compute nodes to deploy, which must be a multiple of the number of data nodes deployed (the shards: field), including shards: x 0 and shards: x 1. Compute nodes can be added to or removed from the deployed cluster by changing this setting and reapplying the definition, but replicas: must always be a multiple of shards:.
preferredZones: - us-east1-b - us-east1-c
The preferredZones: value for compute nodes should match that for data nodes to ensure that the compute nodes are deployed in the same zone or zones as the data nodes. (Be sure to see the note about interzone latency in the description of the data: section, above.)
podTemplate: spec: imagePullSecrets: - name: dockerhub-secret
Like the image: field’s value, the name: field of imagePullSecrets: is the same as in the data: section..
storage: accessModes: - ReadWriteOnce resources: requests: storage: 10G storageClassName: iris-ssd-storageclass
The storage: settings are also the same as those in the data: section, except the amount of storage for compute nodes as specified in the storage: field should be kept to a bare minimum to conserve resources. You may even want to create a separate storage class for compute nodes and specify it in the storageClassName: field.
arbiter: image: intersystems/arbiter:2020.1.0-dev preferredZones: - us-east1-d podTemplate: spec: imagePullSecrets: - name: dockerhub-secret
The arbiter: section contains just three settings:
The image: field specifying the repository, name, and tag of the image from which to deploy the arbiter. This is different from the image for data and compute nodes, as the arbiter is not an InterSystems IRIS instance.
The secret containing the credentials required to pull the image, specified by the name: field in imagePullSecrets:. Typically, the arbiter and Intersystems IRIS images would be in the same repository, so you would specify the same secret here as in the data node and compute node sections. It is possible, however, that the arbiter image could be in a different repository, in which case you would need to create a different image pull secret and specify it for the arbiter.
The preferredZones: setting, which you should use to deploy the arbiter in a separate zone from those specified for the data node primaries and backups, as described in the description of the data: section above, in order to optimize mirror availability.
Deploy the iriscluster
Once the definition file (for example my-iriscluster-definition.yaml) is complete, deploy the iriscluster with the following command:
$ kubectl apply -f my-iriscluster-definition.yaml iriscluster.intersystems.com/my-iriscluster created
Because the IKO extends Kubernetes to add iriscluster as a custom resource, you can apply commands directly to your cluster. For example, if you want to see its status, you can execute the kubectl get command on the iricluster, as in the following:
$ kubectl get irisclusters NAME DATA COMPUTE MIRRORED STATUS AGE my-iriscluster 2 2 true Creating 28s
Follow the progress of cluster creation by displaying the status of the pods that comprise the deployment, as follows:
$ kubectl get pods NAME READY STATUS RESTARTS AGE intersystems-iris-operator-6499fbbf4-s74lk 1/1 Running 1 1h23m my-iricluster-arbiter-0 1/1 Running 0 36s my-iricluster-data-0-0 0/1 Running 0 28s ... $ kubectl get pods NAME READY STATUS RESTARTS AGE intersystems-iris-operator-6499fbbf4-s74lk 1/1 Running 1 1h23m my-iricluster-arbiter-0 1/1 Running 0 49s my-iricluster-data-0-0 0/1 Running 0 41s my-iricluster-data-0-1 0/1 ContainerCreating 0 6s ... $ kubectl get pods NAME READY STATUS RESTARTS AGE intersystems-iris-operator-6499fbbf4-s74lk 1/1 Running 1 1h35m my-iricluster-arbiter-0 1/1 Running 0 10m my-iricluster-compute-0 1/1 Running 0 10m my-iricluster-compute-1 1/1 Running 0 9m my-iricluster-data-0-0 1/1 Running 0 12m my-iricluster-data-0-1 1/1 Running 0 12m my-iricluster-data-1-0 1/1 Running 0 11m my-iricluster-data-1-1 1/1 Running 0 10m
In the event of an error status for a particular pod, you can examine its log, for example:
$ kubectl logs my-iricluster-data-0-1
Open the cluster's Management Portal
To load the cluster’s Management Portal in your browser, first list the service corresponding to node 1, which can be identified by its name ending in data-0-0, as shown in the preceding examples. Use a command like the following:
$ kubectl get svc my-iricluster my-iricluster-data-0-1 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-iriscluster LoadBalancer 10.35.245.6 188.8.131.52 1972:30011/TCP,52773:31887/TCP 46m
Next, load the following URL in your browser, substituting the listed external IP address for the one shown here:
The cluster's external IP address is used with the superserver and webserver ports (1972 and 52773, respectively) for all connections to the cluster for data ingestion, queries, and other purposes.
Remove the iriscluster
To fully remove the cluster, you must delete the cluster, the persistent volume claims associated with it, and typically the service associated with it. For example:
$ kubectl delete -f my-iriscluster-definition.yaml iriscluster.intersystems.com "my-iriscluster" deleted $ kubectl delete pvc —all persistentvolumeclaim "iris-data-my-iriscluster-compute-0" deleted persistentvolumeclaim "iris-data-my-iriscluster-compute-1" deleted persistentvolumeclaim "iris-data-my-iriscluster-data-0-0" deleted persistentvolumeclaim "iris-data-my-iriscluster-data-0-1" deleted persistentvolumeclaim "iris-data-my-iriscluster-data-1-0" deleted persistentvolumeclaim "iris-data-my-iriscluster-data-1-1" deleted $ kubectl delete svc iris-svc service "iris-svc" deleted
You can also fully remove the iriscluster by unprovisioning the Kubernetes cluster on which it is deployed.