Using the InterSystems Kubernetes Operator (Version 3.3)

Prepare the configuration merge files

The configuration parameter file, also called the CPF, defines the configuration of an InterSystems IRIS instance. On startup, InterSystems IRIS reads the CPF to obtain the values for most of its settings. The configuration merge feature allows you to specify a merge file that overwrites the values of one or more settings that in the default CPF that comes with an instance when it is deployed. For details, see Automating Configuration of InterSystems IRIS with Configuration Merge.

To use configuration merge when deploying your IrisCluster, customize the template common.cpf, data.cpf, and compute.cpf files provided in the samples/ directory by adding the CPF settings you want to apply to all InterSystems IRIS nodes, the data nodes, and the compute nodes (if included) respectively. The provided common.cpf template file contains only a sample passwordHash setting, described in the next section; the data.cpfand compute.cpf template files contain only a sample SystemMode setting, which displays text on the InterSystems IRIS Management Portal.

For numerous helpful examples of parameters you might customize for several purposes, see Useful Parameters for Automated Deployment in Automating Configuration of InterSystems IRIS with Configuration Merge. For example, the data nodes in a sharded cluster must be configured to allocate a database cache of the appropriate size; Deploy Sharded Clusters in “Useful Parameters” illustrates how you would add the [config] section globals parameter to the data.cpf file to configure the database caches of the data nodes at the size you calculated. This is shown in the following, with a value of 20 GB (the globals setting is in MB):

[StartUp]
SystemMode=my-IrisCluster
[config]
globals= 0,0,20480,0,0,0

Set the default InterSystems IRIS password

InterSystems IRIS is installed with several predefined user accounts, the initial password for which is SYS. For effective security, it is important that this default password be changed immediately upon deployment of all InterSystems IRIS containers. For this reason, even if you have no other reason to provide configuration merge files (which are optional), you should include at least a common.cpf containing the passwordHash parameter, as illustrated in the provided template common.cpf, to reset the default password on all InterSystems IRIS nodes, for example:

[Startup]
PasswordHash=dd0874dc346d23679ed1b49dd9f48baae82b9062,10000,SHA512

InterSystems publishes through the ICR the intersystems/passwordhash image, from which you can create a container that generates the hash for you; for more information about this, the passwordHash parameter, and the default password, see Authentication and PasswordsOpens in a new tab in Running InterSystems Products in Containers.

Prepare the Web Gateway configuration file

As described in Configure the Web Gateway in the Web Gateway Configuration Guide, the Web Gateway’s configuration is managed using the Web Gateway management pages, but contained in the CSP.ini file (much as an InterSystems IRIS instance’s configuration is contained in the iris.cpf file). The CSP.ini file you include in your config map, or the one generated by the IKO if you do not include one, is automatically installed on every webgateway node deployed. If you have experience with the Web Gateway, you can use a CSP.ini from an existing installation as a template and prepare one to be installed on your webgateway nodes by the IKO. You can also supply your own CSP.conf file, which contains the Web Gateway’s Apache or Nginx-specific configuration. Bear the following points in mind when deciding whether to provide your own file.

• Populating the remote server pool a CSP.ini file

  Because an IrisCluster does not include a Kubernetes-based mechanism for distributing application connections, deployed web server (webgateway) nodes provide the only means of controlling this distribution within the IrisCluster specification. (You can also deploy a custom load balancer on the Kubernetes cluster hosting the IrisCluster or use a third party-tool to distribute connections.) For this reason, ensure that the remote server pool in the configuration file, which specifies the InterSystems IRIS instances to which the Web Gateway directs incoming connections from the web server, is populated according to the relevant best practice for distributing application connections, as follows:

  • If the deployment is a sharded cluster and includes compute nodes, add all data and compute nodes to the pool; if there are no compute nodes, add all data nodes.

  • If the deployment is a distributed cache cluster, add all compute (application server) nodes to the pool.

  • If the deployment is a standalone instance, it should be the only remote server in the pool.

  Important:

  Web Gateway connections to mirrored data nodes are automatically mirror aware, that is, connections are always to twhichever failover member is the current primary.

  It is possible to populate the remote server pool before deployment because the names assigned to IrisCluster nodes always follow the same pattern, which is clustername-nodetype-0. For example, in an IrisCluster called myCluster, data node 1 of a sharded cluster, the data server of a distributed cache cluster, and a standalone instance would all be called myCluster-data-0 unless mirrored, in which case the primary (as deployed) would be myCluster-data-0-0 and the backup myCluster-data-0-1. Other nodes are named in the same format (except that they cannot be mirrored), for example myCluster-compute-0. Therefore, if an IrisCluster of this name was a sharded cluster with four mirrored data nodes, four compute nodes, an arbiter, and two webgateway nodes, they would be named as shown in the following table. The data nodes that would go in the remote server pool in the Web Gateway configuration file — that is, the deployed mirror primaries – are indicated by *; all of the compute nodes would be added to the pool.

  myCluster-data-0-0* myCluster-data-0-1 myCluster-data-1-0* myCluster-data-1-1 myCluster-data-2-0* myCluster-data-2-1 myCluster-data-3-0* myCluster-data-3-1

  myCluster-compute-0 myCluster-compute-1 myCluster-compute-2 myCluster-compute-3

  myCluster-arbiter-0

  myCluster-webgateway-0 myCluster-webgateway-0

  If you do not include a CSP.ini file in your config map, the IKO will generate one appropriate for the cluster and install it on every webgateway node, populating the remote server pool as described in the foregoing. The IKO can also generate a CSP.conf file if you do not provide one.

  Important:

  The CSP.ini file generated by the IKO does not include a SSL/TLS connection; this can be configured on the Web Gateway management pages on the individual nodes.

• Updating the configuration of multiple Web Gateways

  As described in Connect to the IrisCluster, the serviceTemplate field creates one or more Kubernetes servicesOpens in a new tab that expose the first stateful set in the pod for each deployed node type to the network through external IP addresses. This includes webgateway nodes, which means that if you have deployed just one, you can always access it management pages to change its configuration by loading the URL http://external-ip:80/csp/bin/Systems/Module.cxw in your browser. If you want to make configuration changes across multiple webgateway nodes, however, you have three options:

  • If you provided your own CSP.ini file, use the procedure described in Create a config map for the configuration files for modifying the CSP.ini or .cpf files on deployed nodes.

  • Create a similar service for each of the webgateway nodes that exposes it through an external IP address, then use the URL above to make the changes on each node separately (but see the following item regarding the needed management access credentials).

  • Directly modify the CSP.ini file on each pod by using the kubectl exec command to open a shell or edit the file within the container.

• Securing management access to the Web Gateway

  The predefined CSPSystem account on an InterSystems IRIS instance is used for communication between an InterSystems IRIS instance and the local InterSystems Web Gateway instance installed with it, and by default is used for access to the local Web Gateway’s configuration through its management pages. For effective security, you must change the default password for this account (along with the other predefined accounts) on both an InterSystems IRIS instance and its local Web Gateway instance either as part of deployment or immediately after; Authentication and Passwords in Running InterSystems Products in Containers provides details on this topic and instructions for making these changes in an InterSystems IRIS container. In the case of a webgateway node deployed by the IKO, you must independently secure management access, as follows:

  • If you provide your own CSP.ini file, ensure that if CSPSystem (or any other predefined account) is specified as the management access account, the password is not SYS; if it is, change it. You can also change the access credentials to any username and password you choose. Additionally, while the Web Gateway encrypts the management access password in the CSP.ini file on first starting up, including it in plain text could expose it on disk or the network, so the best practice is to make sure it is encrypted in the file you provide; you can do this after you change it, if necessary, by starting up a local test Web Gateway instance with your configuration file in place, which encrypts the password.

  • The CSP.ini generated by the IKO if you do not provide one specifies CSPSystem/SYS as the management access credentials; to change this, you must use either the second method described above for updating multiple Web Gateway configurations (creating services) or the third (using kubectl exec).

• Securing Web Gateway connections to remote InterSystems IRIS servers

  Each entry in a Web Gateway’s remote server pool includes credentials used to authenticate to the specified InterSystems IRIS instance; these are the username and password of an account on the instance. If you provide your own CSP.ini, ensure that the specified account is a secure one that does not use the default password (see Authentication and Passwords). You can then specify this account for each remote server in your CSP.ini; clearly it is simplest to use the same account for all. The CSP.ini generated by the IKO if you do not provide one specifies _System/(blank) as the management access credentials for each remote server; to change this (which you must for effective security), use either the second method described above for updating multiple Web Gateway configurations (creating services) or the third (using kubectl exec).

Considering the points above, for deployments that include multiple webgateway nodes, the flexibility and convenience of providing the CSP.ini file and being able to modify and redistribute it as needed make this the recommended approach in general.

Create a config map for the configuration files

Create a Kubernetes config map specifying the files, using a command like this:

$ kubectl create cm iris-cpf --from-file common.cpf --from-file data.cpf 
    --from-file compute.cpf --from-file CSP.ini --from-file CSP.conf

You can then specify iris-cpf as the value for configSource. (If you did not create a config map, do not specify a value for this field.)

image:

      image: containers.intersystems.com/intersystems/iris:2021.1.0.205.0

Required

The image: field specifies the URL (registry, repository, image name, and tag) of the InterSystems IRIS image from which to deploy data node containers. The example above specifies an InterSystems IRIS image from the InterSystems Container Registry (ICR). The registry credentials in the secret specified by the imagePullSecrets field are used for access to the registry.

updateStrategy:

      updateStrategy:
        type: {RollingUpdate|OnDelete}

Optional

Overrides the top level updateStrategy setting to specify the Kubernetes update strategyOpens in a new tab used for the stateful setsOpens in a new tab representing this node type only. The value can be either RollingUpdate (the default) or OnDelete.

preferredZones:

      preferredZones:
        - zoneN
        - ...

Optional

Specifies the zone or zones in which data nodes should be deployed, and is typically used as follows:

• If mirrored is set to true and at least two zones are specified, Kubernetes is discouraged (but not prevented) from deploying both members of a failover pair in the same zone, which maximizes the chances that at least one is available, and is therefore the best practice for high availability. Bear the following mind, however:

  • Deploying the members of a failover pair in separate zones is likely to slightly increase latency in the synchronous communication between them.

  • Specifying multiple zones for the data nodes means that all of the primaries might not be deployed in the same zone, resulting in slightly increase latency in communication between the data nodes.

  • Specifying multiple zones for data nodes generally makes it impossible to guarantee that nodes of other types (compute, Web Gateway, SAM, IAM) are in the same zone as all of the data node primaries at any given time regardless of your use of preferredZones in their definitions, increasing latency in those connections as well.

  Under most circumstances these interzone latency effects will be negligible, but with some demanding workloads involving high message or query volume, performance may be affected. If after researching the issue of interzone connections on your Kubernetes platform and testing your application thoroughly you are concerned about this performance impact, consider specifying a single zone for your mirrored data nodes.

  Regardless of the zones you specify here, you should use the preferredZones field in the arbiter definition to deploy the arbiter in a separate zone of its own, which also helps optimize mirror availability.

• The data nodes of an unmirrored cluster are typically deployed in the same zone to minimize latency. If mirrored: false and your Kubernetes cluster includes multiple zones, you can use preferredZones to follow this practice by specifying a single zone in which to deploy the data nodes.

• The value of the preferredZones: field in the compute definition, if included, should ensure that the compute nodes are deployed in the same zone or zones as the data nodes to minimize latency (see Plan Compute Nodes in the Scalability Guide).

Kubernetes attempts to deploy in the specified zones, but if this is not possible, deployment proceeds rather than failing.

podTemplate:

      podTemplate:
        spec:
          args:        

Optional

Specifies overrides and additions to the default pod templateOpens in a new tab applied to the data node pods (and to the pods of other node types in their respective definitions). Because containers are run only within a pod on Kubernetes, a pod template can specify the fields that define numerous Kubernetes entities, including the fields defining a containerOpens in a new tab, which makes it useful for many purposes. Several examples of using containers: fields are provided in the following:

• Prevent InterSystems IRIS from starting up with the container.

  You can use the args field in the pod template to specify options to the InterSystems IRIS entrypoint application, iris-main. For example, if there is something wrong with the InterSystems IRIS configuration which prevents startup from succeeding, iris-main exits, causing the pod to go into a restart loop, which makes it difficult or impossible to diagnose the problem. You can prevent the instance from starting by adding the iris-main option --up false as follows:

  podTemplate:
    spec:
      args:
      - --up
      - "false"
  

  

  When you do this, the readiness probeOpens in a new tab will not be satisfied, and the deployment will be paused indefinitely:

  $ kubectl get pods
  NAME                      READY   STATUS    RESTARTS   AGE
  my-IrisCluster-data-0     0/1     Running   0          32s
  

  

  After addressing the problem, you can do one of the following:

  • Manually start the instance with a command like the following:

    $ kubectl exec -it my-IrisCluster-data-0 -- iris start IRIS
    

    

  • Remove the podTemplate override and redeploy the pod.

• Override the default liveness probe and readiness probeOpens in a new tab.

  If you wanted to replace the default liveness probe and readiness probe for data nodes (or another node type), you could specify something like the following in the applicable pod template:

  podTemplate:
    spec:
      livenessProbe:
        exec:
          command:
          - /bin/true
      readinessProbe:
        httpGet:
          path: /csp/user/cache_status.cxw
          port: 52773
  

  

• Limit the number of CPU cores a data or compute node container can use in order to remain within the limits of the InterSystems IRIS license in use; limit the memory containers can use for efficient resource distribution.

  You can use the resources field in the pod template to both request and limit resources to be used by the containers in the pod. In the following example, the pod starts with 256 MB of memory and 2 CPU cores, and is limited to 2 GB of memory and 2 CPU cores:

  podTemplate:
    spec:
      resources:
        requests:
          memory: "256Mi"
          cpu: "2"
        limits:
          memory: "2Gi"
          cpu: "2"
  

  

• Allocate huge pages on cluster nodes.

  You can also use resources to allocate huge pages on InterSystems IRIS or other nodes:

  podTemplate:
    spec:
      resources:
        requests:
          memory: "2Gi"
        limits:
          memory: "2Gi"
          hugepages-2Mi: "2Gi"
  

  

• Set a variable to use non-FQDN hostnames instead of FQDN hostnames.

  By default, the IKO uses FQDN hostnames, for example my-IrisCluster-data-0-0.iris-svc.us-west1.svc.cluster.local, because including the governing service allows the hosts to find each other. Non-FQDN hostnames, however, are easier to read and may be required in certain contexts (for example, when using custom DNS). The IKO provides a variable, ISC_USE_FQDN, which when set to true (as it is by default) causes FQDN hostnames to be used. To use non-FQDN hostnames, you can set it to false using the env field in the podTemplate of each node type you deploy, as follows:

  podTemplate:
    spec:
      env:
      - name: ISC_USE_FQDN
        value: "false"
  

  

  When you do this, the IKO creates a DNSConfigOpens in a new tab which adds the cluster domain to the DNS search space, allowing pods to communicate with one another using non-FQDN hostnames, and configures non-FQDN hostnames for all InterSystems IRIS node types (data, compute, arbiter, webgateway) for which you have set the variable to false.

shards:

      shards: N

Optional

Specifies the number of data nodes to be deployed as a sharded cluster. If the shards field is omitted, a single standalone instance of InterSystems IRIS is deployed, optionally as the data server in a distributed cache cluster; for more information, see Define the IrisCluster topology.

Data nodes can be added to the deployed cluster by increasing this setting and reapplying the definition (as described in Modifying the IrisCluster), but the setting cannot be decreased .

mirrored:

      mirrored: {true|false}

Optional

Determines whether the data nodes in the deployment are mirrored.

If the value of mirrored: is true, two mirrored instances are deployed for each data node specified by the shards field. For example, if shards: 4 and mirrored: true, eight data node instances are deployed as four failover pairs, creating a mirrored sharded cluster with four data nodes. If mirrored: is true when shards is omitted, two mirrored instances are deployed as a standalone InterSystems IRIS instance, which can optionally be the mirrored data server of a distributed cache cluster; for details, see Define the IrisCluster topology.

The default for mirrored is false.

storage*:

      storage{DB|WIJ|Journal1|Journal2}:
        resources:
          requests:
            storage: size
        storageClassName: storage-class-name

Optional

Specify custom characteristics, such as size or storage class, for one or more of the four predefined persistent volumes deployed with each data node, as follows:

• storageDB — The volume on which data is stored, including durable %SYS data.

• storageWIJ — The volume containing the WIJ directory.

• storageJournal1 — The volume containing the primary journal directory.

• storageJournal2 — The volume containing the alternate journal directory.

These predefined volumes are mounted in /irissys inside the container, and are 2 GB by default. In addition to specifying a size override, you can include any other field from the Kubernetes persistent volume claim specOpens in a new tab to override default characteristics. For example, you can add storageClassName to override the deployment’s default storage class, as shown.

When including a size override, the value of the storage field can be specified in any unit between kilobytes and exabytesOpens in a new tab. The amount of data storage to be mounted on sharded cluster data nodes is determined during the cluster planning process and should include a comfortable margin for the future growth of your workload. In addition to overriding the sizes of the predefined volumes, you can use additional persistent volumes defined in the volumeClaimTemplates field and specified in the volumeMounts field to ensure that sufficient storage is available to each data node.

The same four storage* fields can be used to modify the same predefined volumes in the compute node definition in the same ways. These volumes are also 2 GB by default. The data storage for sharded cluster compute nodes or distributed cache cluster application servers should be kept to a bare minimum to conserve resources, as these nodes do not store application data, so size overrides of these volumes on compute nodes may be desirable.

In the webgateway node definition you can override the characteristics of the single predefined volume, storageDB. The default size of this volume is 32 MB. Predefined volumes are also deployed with sam and iam nodes, and you can specify overrides for these in their respective definitions.

volumeMounts:

      volumeMounts:
      - name: volumeClaimTemplateN
        mountPath: pathN
      - ...

Optional

Specifies one or more persistent storage volumesOpens in a new tab, as defined in the volumeClaimTemplates field, and/or ephemeral storage volumesOpens in a new tab, as specified in the volumes field, to be deployed with each data node, in addition to the predefined persistent volumes. Each volume is defined by the name of one of the volume claim templates or volumes and a mountPath, which is a direct reference to a location in the container’s filesystem, visible to the InterSystems IRIS instance, on which to mount the volume.

The volumeMounts field can also be used to specify additional (typically ephemeral) volumes in the compute node definition.

image

image: containers.intersystems.com/intersystems/iris:2021.1.0.205

Required in optional compute: section

Compute nodes are deployed from the same InterSystems IRIS image as data nodes; an example is shown.

replicas:

         replicas: N

Required in optional compute: section

Specifies the number of identical compute nodes to deploy. In a sharded cluster, this should be a multiple of the number of data nodes specified by shards (for more information, see Plan Compute Nodes in the Scalability Guide).

Compute nodes can be added to or removed from the deployed IrisCluster by changing this setting and reapplying the definition (as described in Modifying the IrisCluster).

The replicas field also appears in the webgateway section, where it specifies the number of InterSystems Web Gateway (web server) nodes to deploy.

image

image: containers.intersystems.com/intersystems/iris:2021.1.0.205

Required in optional webgateway: section

Use one of the following InterSystems images to deploy webgateway nodes:

• webgateway (as shown in the example) — Deploys an InterSystems Web Gateway instance and an Apache web server.

• webgateway-nginx — Deploys a Web Gateway instance and an Nginx web server.

For information about these images and the differences between them, see Using the InterSystems Web Gateway Container in Running InterSystems Products in Containers.

type:

      type: {apache|nginx}

Optional

Specifies deployment of an Apache web server or an Nginx web server (as described for the previous field); the default is apache.

replicas:

      replicas: N

Required in optional webgateway: section

Specifies the number of identical webgateway nodes to deploy.

You can add webgateway nodes to, or remove them from, the deployed IrisCluster by changing this setting and reapplying the definition (as described in Modifying the IrisCluster).

The replicas field also appears in the compute section, where it specifies the number of compute nodes to deploy.

applicationPaths:

      applicationPaths:
        - pathN
        - ...

Optional

Provides a list of application paths to configure in the Web Gateway. Application paths should not have a trailing slash, and the path /csp is reserved.

alternativeServers:

      alternativeServers: {FailOver|LoadBalancing}

Optional

Selects the method by which the Web Gateway on each node determines which InterSystems IRIS server (that is, which data node) in its remote server pool to connect to (see Load Balancing and Failover Between Multiple InterSystems IRIS Server Instances in the Web Gateway Guide). Possible values are FailOver and LoadBalancing, with a default of LoadBalancing.