Skip to main content

Using the InterSystems Kubernetes Operator

This document explains how to use the InterSystems Kubernetes Operator (IKO) to deploy InterSystems IRIS sharded clusters on Kubernetes platforms.

Why would I use Kubernetes?

Kubernetes is an open-source orchestration engine for automating deployment, scaling, and management of containerized workloads and services, and excels at orchestrating complex SaaS (software as a service) applications. You provision a Kubernetes-enabled cluster and tell Kubernetes the containerized services you want to deploy on it and the policies you want them to be governed by; Kubernetes transparently provides the needed resources in the most efficient way possible, repairs or restores the configuration when problems with those resources cause it to deviate from what you specified, and can scale automatically or on demand. In the simplest terms, Kubernetes deploys a multicontainer application in the configuration and at the scale you specify on any Kubernetes-enabled platform, and keeps the application operating exactly as you described it.

Why do I need the InterSystems Kubernetes Operator?

The InterSystems Kubernetes Operator (IKO) extends the Kubernetes API with a custom resource representing an InterSystems IRIS sharded cluster. This resource, called an iriscluster, can be deployed on any Kubernetes platform on which the IKO is installed. (The IKO can also deploy a stand-alone instance of InterSystems IRIS on Kubernetes.)

The operator isn’t required to deploy InterSystems IRIS under Kubernetes. But because Kubernetes is application-independent, you would need to create custom definitions and scripts to handle all the needed configuration of the InterSystems IRIS instances or other components in the deployed containers along with networking, persistent storage requirements, and so on. Installing the IKO automates these tasks. By putting together a few settings that define the cluster, for example the number of data and compute nodes, whether they should be mirrored, and where the Docker credentials needed to pull the container images are stored, you can easily deploy your InterSystems IRIS cluster exactly as you want it. The operator also adds sharded cluster management capabilities to Kubernetes, enabling tasks like adding data or compute nodes, which you would otherwise have to do manually by interacting directly with the instances.

Is the IKO right for my use case?

Before beginning your work with the IKO, you should confirm that:

Learn to speak Kubernetes

While it is possible to use the IKO if you have not already worked with Kubernetes, InterSystems recommends having or gaining a working familiarity with Kubernetes before deploying with the IKO.

Plan your sharded cluster

For the most beneficial results, it is important to fully plan the configuration of your sharded cluster and its data, including:

For detailed information about InterSystems IRIS sharded clusters, see Horizontally Scaling for Data Volume with Sharding in the Scalability Guide.

Choose a platform and explore the interface

Major public cloud Kubernetes platforms include Google Kubernetes Engine (GKE), Azure Container Service (AKS), Amazon Elastic Container Service for Kubernetes (EKS), and Tencent Kubernetes Engine (TKE), while platforms such as Red Hat OpenShift, Rancher Kubernetes Engine (RKE), and Docker Enterprise can be used on any infrastructure. When you have selected the Kubernetes platform you will deploy on, create an account and familiarize yourself with the provided interface(s) to Kubernetes. For example, to use GKE on Google Cloud Platform, you can open a Google Cloud Shell terminal and file editor to use the gcloud and kubectl command line interfaces. Bear in mind that the configuration of your Kubernetes environment should include access to the availability zones in which you want to deploy the sharded cluster.

Deploy a Kubernetes container cluster to host the iriscluster

The Kubernetes cluster is the structure on which your containerized services are deployed and through which they are scaled and managed. The procedure for deploying a cluster varies to some degree among platforms. In planning and deploying your Kubernetes cluster, bear in mind the following considerations:

  • The IKO deploys one InterSystems IRIS or arbiter container (if a mirrored cluster) per Kubernetes pod, and attempts to deploy one pod per Kubernetes cluster node when possible. Ensure that

    • You are deploying the desired number of nodes to host the pods of your sharded cluster, including the needed distribution across zones if more than one zone is specified (see below).

    • The required compute and storage resources will be available to those nodes.

  • If your sharded cluster is to be mirrored and you plan to enforce zone antiaffinity using the preferredZones fields in the iriscluster definition to deploy the members of each failover pair in separate zones and the arbiter in an additional zone, the container cluster must be deployed in three zones. For example, if you plan to use zone antiaffinity and are deploying the cluster using the gcloud command-line interface, you might select zones us-east1-b,c,d and create the container cluster with a command like this:

    $ gcloud container clusters create my-iriscluster --node-locations us-east1-b,us-east1-c,us-east1-d
    Copy code to clipboard

Upgrade Helm if necessary

Helm packages Kubernetes applications as charts, making it easy to install them on any Kubernetes platform. Because the IKO Helm chart requires Helm version 3, you must confirm that this is the version on your platform, which you can do by issuing the command helm version. If you need to upgrade Helm to version 3, you can use the curl script at https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3. For example:

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6827  100  6827    0     0  74998      0 --:--:-- --:--:-- --:--:-- 75855
Helm v3.2.3 is available. Changing from version .
Downloading https://get.helm.sh/helm-v3.2.3-linux-amd64.tar.gz
Preparing to install helm into /usr/local/bin
helm installed into /usr/local/bin/helm
Copy code to clipboard

Download the IKO archive and upload the extracted contents to Kubernetes

Obtain the IKO archive file, for example iris_operator-1.0.0-unix.tar.gz, from the InterSystems Worldwide Response Center (WRC) download area and extract its contents, then upload the extracted directory, iris_operator-version (for example iris_operator-1.0.0) to the Kubernetes platform. This directory contains the following:

  • The image/ directory contains an archive file of the IKO image.

  • The chart/iris-operator directory contains the Helm chart for the operator.

  • The samples/ directory contains template .yaml and .cpf files, as described later in this procedure.

Load the IKO image

Make the IKO image available on the Kubernetes platform using the docker load command and optionally push or retag the image, as described in Downloading the InterSystems IRIS Docker Image in Running InterSystems Products in Docker Containers.

$ docker load -i iris_operator-1.0.0/image/iris_operator-1.0.0-docker.tgz
0d1435bd79e4: Loading layer [==================================================>]  3.062MB/3.062MB
478805a2e99f: Loading layer [==================================================>]  75.54MB/75.54MB
Loaded image: intersystems/iris-operator:1.0.0 
Copy code to clipboard

Update the values.yaml file

In the chart/iris-operator directory, ensure that the values in the operator section of the file values.yaml, which describe the IKO image, match the IKO image you just loaded on the Kubernetes platform, for example:

operator:
  registry: intersystems
  repository: iris-operator
  tag: 1.0.0
Copy code to clipboard

Install the IKO

Use Helm to install the operator on the Kubernetes cluster. For example, on GKE you would use the following command:

$ helm install intersystems iris_operator-1.0.0/chart/iris-operator
NAME: intersystems
LAST DEPLOYED: Mon Jun 15 16:43:21 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that InterSystems Kubernetes Operator has started, run:
  kubectl --namespace=default get deployments -l "release=intersystems, app=iris-operator"
bbinstoc@cloudshell:~ (isc-mktplace-dev)$ kubectl get deployments -l "release=intersystems, app=iris-operator"
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
intersystems-iris-operator   0/1     1            0           30s
Copy code to clipboard

Choose the default InterSystems IRIS password and generate a hash

InterSystems IRIS is installed with several predefined user accounts, the initial password for which is SYS. For effective security, it is important that this default password be changed immediately upon deployment of your container. For this reason, the IKO requires you to include a cryptographic hash (with salt) of the new default password in the YAML definition of your iriscluster. InterSystems publishes the image for a nanocontainer, iris-passwordhash, that generates the hash for you; for more information, see Authentication and Passwords in Running InterSystems Products in Cotainers.

Create a secret for image pull information

Kubernetes secrets let you safely and flexibly store and manage sensitive information. Identify the registry, repository, and tag of the InterSystems IRIS image and (if the cluster is to be mirrored) arbiter image to be deployed, and create a Kubernetes secret of type docker-registry containing the URL of the registry and the credentials needed to log into Docker on that registry to pull the images from it. For example, suppose the images are:

docker.acme.com/intersystems-iris:2020.3.0.633.0
docker.acme.com/intersystems-arbiter:2020.3.0.633.0
Copy code to clipboard

The following kubectl command is an example of creating a secret named acme-secret for pulling these images (docker-email is optional):

$ kubectl create secret docker-registry acme-pull-secret 
  --docker-server=https://index.docker.acme.com/latest/ --docker-username=*****
  --docker-password='*****' --docker-email=**********
Copy code to clipboard

Create a secret containing the InterSystems IRIS license key

Upload the sharding-enabled license key for the InterSystems IRIS images in your sharded cluster, and create a Kubernetes secret of type generic to contain the key, allowing it to be mounted on a temporary file system within the container, for example:

$ kubectl create secret generic iris-key-secret --from-file=iris.key
Copy code to clipboard

Create configuration parameter files and a config map for them.

The configuration parameter file, also called the CPF, defines the configuration of an InterSystems IRIS instance. On startup, InterSystems IRIS reads the CPF to obtain the values for most of its settings. The CPF merge feature allows you to specify a merge file containing configuration that are different from, and overwrite, the settings in the default CPF that comes with an instance when it is deployed. For details, see Introduction to the Configuration Parameter File.

Kubernetes config maps keep your containerized applications portable by separating configuration artifacts, such as CPF merge files, from image content. If you want to use separate CPF merge files to update the configurations of the iriscluster’s data nodes and compute nodes at deployment, you must do two things:

  • Customize the provided template data.cpf and compute.cpf files (they contain only the SystemMode setting, which displays text on the InterSystems IRIS management portal) with the CPF settings you want to apply to the data and compute nodes. For example, as described in Deploying the Sharded Cluster in the Scalability Guide, the data nodes in a sharded cluster must be configured to allocate a database cache of the appropriate size, and typically the default generic memory heap size should be increased as well. To do this, you would add the [config] section globals and gmheap parameters to the data.cpf file, as shown below, to configure the database cache and generic memory heap of each data node at 200 GB and 256 MB, respectively:

    [StartUp]
    SystemMode=my-iriscluster
    [config]
    globals= 0,0,204800,0,0,0
    gmheap= 262144
    
    Copy code to clipboard
  • Create a Kubernetes config map specifying the files, using a command like this:

    $ kubectl create cm iris-cpf --from-file data.cpf --from-file compute.cpf
    Copy code to clipboard

Create a storage class for persistent storage

Kubernetes provides the persistent storage needed by containerized programs (in this case, the data nodes) in the form of persistent volumes; a persistent volume claim (pvc) is a request by a user for a persistent volume. To include these in the cluster, you need to specify in the definition a Kubernetes storage class, which determines the type of storage used. Because storage types are platform specific, the storage class must be defined separately from the cluster, which is why multiple template storage class definition files are provided with the IKO. For example, the template storage class definition file for GKE is named iris-ssd-sc-gke.yaml and has the following contents:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: iris-ssd-storageclass
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
Copy code to clipboard

Update the storage class definition as needed and then create the storage class, for example:

$ kubectl create -f iris-ssd-sc-gke.yaml
Copy code to clipboard

Using the definition file shown above, this command creates a storage class called iris-ssd-storageclass.

Create the iriscluster definition file

You are now ready to create your iriscluster definition file by customizing the template file provided with the IKO, my-iriscluster-definition.yaml. The contents of this file and suggestions for customizing it are shown in the following sections.

Note:

For each section of the definition, there are numerous other Kubernetes fields that can be included; this section discusses only those specific to or required for an iriscluster definition.

apiVersion: intersystems.com/v1alpha1
kind: IrisCluster
metadata:
  name: my-iriscluster
spec:
Copy code to clipboard

The first three fields are required by Kubernetes, as is the spec: section, which contains nested fields specifying a particular type of object, in this case an iriscluster. Change the value of the name: field in metadata: to the name you want to give the cluster.

  passwordHash: ''
Copy code to clipboard

The passwordHash field specifies the cryptographic hash of the desired default password for the InterSystems IRIS instances in the cluster; insert the hash you generated as the value.

  licenseKeySecret:
    name: iris-key-secret
Copy code to clipboard

The licenseKeySecret is the Kubernetes secret containing the InterSystems IRIS license key; update the name: field with the name of the license key secret you created.

  configSource:
    name: iris-cpf
Copy code to clipboard

The configSource is the config map containing the CPF merge files, data.cpf and compute.cpf. Update the name: field with the name of the config map you created; if you did not create one, do not specify a value for this field.

  topology:
Copy code to clipboard

The topology section specifies the details of the cluster’s data nodes, compute nodes (if any), and arbiter node (if mirrored). Only the data: section is required.

    data:
      shards: <1..N> 
Copy code to clipboard

In the data: definition, the shards: field specifies the number of data nodes to deploy. Data nodes can be added to the deployed cluster by increasing this setting and reapplying the definition, but the setting cannot be decreased.

Note:

The shards: field is optional, with a default value of 1. If it is omitted, and the compute: and arbiter: sections of the definition are also omitted, applying the definition results in stand-alone deployment of a single instance of InterSystems IRIS, rather than a sharded cluster. If the mirrored: field, described below, is set to true, two stand-alone instances are deployed as the primary and backup in a mirror (and you must include the arbiter: section, described later, in the definition).

      mirrored: <true|false>
Copy code to clipboard

If mirrored: true, data nodes are mirrored, and two instances are deployed for each data node specified by the shards: field. For example, if shards: 4 and mirrored: true, eight data node instances are deployed as four failover pairs.

      image: <iris_image>
Copy code to clipboard

The image: field specifies the repository, name, and tag of the InterSystems IRIS image from which to deploy data node containers.

      preferredZones:
        - us-east1-b
        - us-east1-c
Copy code to clipboard

Optionally, use the preferredZones: field to specify the zone or zones in which data nodes are to be deployed, as follows:

  • If mirrored: true and at least two zones are specified, Kubernetes is discouraged from deploying both members of a failover pair in the same zone, which maximizes the chances that at least one is available.

    Note:

    Bear in mind that deploying the members of a failover pair in separate zones is likely to slightly increase latency in the synchronous communication between them. In addition, specifying multiple zones for the data nodes means that all of the primaries might not be deployed in the same zone, resulting incommunication between the data nodes also being affected by this slightly increased latency. Finally, if compute nodes are included in the cluster, specifying multiple zones for data nodes is very likely to result in some compute nodes being deployed in a separate zone from their associated data nodes, increasing latency in those connections as well.

    Under most circumstances these interzone latency effects will be neglible, but with some demanding workloads involving high message or query volume, performance may be affected. If after researching the issue of interzone connections on your Kubernetes platform and testing your application thoroughly you are concerned about this performance impact, consider specifying a single zone for your mirrored data nodes.

    Regardless of the zones you specify here, you should use the preferredZones: field in the arbiter: definition (as illustrated later in this section) to deploy the arbiter in a separate zone of its own, which also helps optimize mirror availability.

  • The data nodes of an unmirrored cluster are typically deployed in the same zone to minimized latency. If mirrored: false and your Kubernetes cluster includes multiple zones, you can use preferredZones to follow this practice by specifying a single zone in which to deploy the data nodes.

Kubernetes attempts to deploy in the specified zones, but if this is not possible, deployment proceeds rather than failing.

      podTemplate:
        spec:
          imagePullSecrets:
            - name: dockerhub-secret
Copy code to clipboard

The imagePullSecrets field identifies the Kubernetes secret containing the credentials and other information to pull the image specified by the image: field; update the name: field with the name of the image pull secret you created.

      updateStrategy:
        type: RollingUpdate
Copy code to clipboard

The Kubernetes update strategy type RollingUpdate is required for the iriscluster.

      storage:
        accessModes:
          - ReadWriteOnce
Copy code to clipboard

The storage section defines the persistent volume claim required for each data node. The accessModes: field specifies the access mode for the persistent volume claim, which must be RWO to enforce a one-to-one relationship between each pod and the persistent volume claim used by the InterSystems IRIS container in the pod for persistent storage.

        resources:
          requests:
            storage: 100G
Copy code to clipboard

The storage: field defines the amount of storage (expressed as any unit between kilobytes and exabytes) available to the persistent volume claim. The amount of storage required by data nodes is determined during the cluster planning process and should include a comfortable margin for the future growth of your workload.

        storageClassName:
Copy code to clipboard

The storageClassName: field identifies the storage class used for the persistent volume claims; enter the name of the storage class you created as its value.

    compute:
      image: intersystems/iris:2020.1.0-dev
Copy code to clipboard

The compute: definition shares several fields with the data: definition. The values for most are the same, for example the image: field specifying the InterSystems IRIS image spec.

      replicas: 2
Copy code to clipboard

The replica: field specifies the number of compute nodes to deploy, which must be a multiple of the number of data nodes deployed (the shards: field), including shards: x 0 and shards: x 1. Compute nodes can be added to or removed from the deployed cluster by changing this setting and reapplying the definition, but replicas: must always be a multiple of shards:.

      preferredZones:
        - us-east1-b
        - us-east1-c
Copy code to clipboard

The preferredZones: value for compute nodes should match that for data nodes to ensure that the compute nodes are deployed in the same zone or zones as the data nodes. (Be sure to see the note about interzone latency in the description of the data: section, above.)

      podTemplate:
        spec:
          imagePullSecrets:
            - name: dockerhub-secret
Copy code to clipboard

Like the image: field’s value, the name: field of imagePullSecrets: is the same as in the data: section..

      storage:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10G
        storageClassName: iris-ssd-storageclass
Copy code to clipboard

The storage: settings are also the same as those in the data: section, except the amount of storage for compute nodes as specified in the storage: field should be kept to a bare minimum to conserve resources. You may even want to create a separate storage class for compute nodes and specify it in the storageClassName: field.

  arbiter:
      image: intersystems/arbiter:2020.1.0-dev
      preferredZones:
        - us-east1-d
      podTemplate:
        spec:
          imagePullSecrets:
            - name: dockerhub-secret
Copy code to clipboard

The arbiter: section contains just three settings:

  • The image: field specifying the repository, name, and tag of the image from which to deploy the arbiter. This is different from the image for data and compute nodes, as the arbiter is not an InterSystems IRIS instance.

  • The secret containing the credentials required to pull the image, specified by the name: field in imagePullSecrets:. Typically, the arbiter and Intersystems IRIS images would be in the same repository, so you would specify the same secret here as in the data node and compute node sections. It is possible, however, that the arbiter image could be in a different repository, in which case you would need to create a different image pull secret and specify it for the arbiter.

  • The preferredZones: setting, which you should use to deploy the arbiter in a separate zone from those specified for the data node primaries and backups, as described in the description of the data: section above, in order to optimize mirror availability.

Deploy the iriscluster

Once the definition file (for example my-iriscluster-definition.yaml) is complete, deploy the iriscluster with the following command:

$ kubectl apply -f my-iriscluster-definition.yaml
iriscluster.intersystems.com/my-iriscluster created
Copy code to clipboard

Because the IKO extends Kubernetes to add iriscluster as a custom resource, you can apply commands directly to your cluster. For example, if you want to see its status, you can execute the kubectl get command on the iricluster, as in the following:

$ kubectl get irisclusters
NAME             DATA   COMPUTE   MIRRORED   STATUS     AGE
my-iriscluster   2      2         true       Creating   28s
Copy code to clipboard

Follow the progress of cluster creation by displaying the status of the pods that comprise the deployment, as follows:

$ kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
intersystems-iris-operator-6499fbbf4-s74lk   1/1     Running   1          1h23m
my-iricluster-arbiter-0                      1/1     Running   0          36s
my-iricluster-data-0-0                       0/1     Running   0          28s

...

$ kubectl get pods
NAME                                         READY   STATUS              RESTARTS   AGE
intersystems-iris-operator-6499fbbf4-s74lk   1/1     Running             1          1h23m
my-iricluster-arbiter-0                      1/1     Running             0          49s
my-iricluster-data-0-0                       0/1     Running             0          41s
my-iricluster-data-0-1                       0/1     ContainerCreating   0          6s

...

$ kubectl get pods
NAME                                         READY   STATUS    RESTARTS   AGE
intersystems-iris-operator-6499fbbf4-s74lk   1/1     Running   1          1h35m
my-iricluster-arbiter-0                      1/1     Running   0          10m
my-iricluster-compute-0                      1/1     Running   0          10m
my-iricluster-compute-1                      1/1     Running   0          9m
my-iricluster-data-0-0                       1/1     Running   0          12m
my-iricluster-data-0-1                       1/1     Running   0          12m
my-iricluster-data-1-0                       1/1     Running   0          11m
my-iricluster-data-1-1                       1/1     Running   0          10m
Copy code to clipboard

In the event of an error status for a particular pod, you can examine its log, for example:

$ kubectl logs my-iricluster-data-0-1
Copy code to clipboard

Open the cluster's Management Portal

To load the cluster’s Management Portal in your browser, first list the service corresponding to node 1, which can be identified by its name ending in data-0-0, as shown in the preceding examples. Use a command like the following:

$ kubectl get svc my-iricluster my-iricluster-data-0-1
NAME            TYPE          CLUSTER-IP   EXTERNAL-IP     PORT(S)                          AGE
my-iriscluster  LoadBalancer  10.35.245.6  35.196.145.234  51773:30011/TCP,52773:31887/TCP  46m
Copy code to clipboard

Next, load the following URL in your browser, substituting the listed external IP address for the one shown here:

http://35.196.145.234:52773/csp/sys/UtilHome.csp
Copy code to clipboard

The cluster's external IP address is used with the superserver and webserver ports (51773 and 52773, respectively) for all connections to the cluster for data ingestion, queries, and other purposes.

Remove the iriscluster

To fully remove the cluster, you must delete the cluster, the persistent volume claims associated with it, and typically the service associated with it. For example:

$ kubectl delete -f my-iriscluster-definition.yaml
iriscluster.intersystems.com "my-iriscluster" deleted
$ kubectl delete pvc —all
persistentvolumeclaim "iris-data-my-iriscluster-compute-0" deleted
persistentvolumeclaim "iris-data-my-iriscluster-compute-1" deleted
persistentvolumeclaim "iris-data-my-iriscluster-data-0-0" deleted
persistentvolumeclaim "iris-data-my-iriscluster-data-0-1" deleted
persistentvolumeclaim "iris-data-my-iriscluster-data-1-0" deleted
persistentvolumeclaim "iris-data-my-iriscluster-data-1-1" deleted
$ kubectl delete svc iris-svc
service "iris-svc" deleted
Copy code to clipboard

You can also fully remove the iriscluster by unprovisioning the Kubernetes cluster on which it is deployed.