Using the InterSystems Kubernetes Operator (Version 3.1)

Prepare the configuration merge files

The configuration parameter file, also called the CPF, defines the configuration of an InterSystems IRIS instance. On startup, InterSystems IRIS reads the CPF to obtain the values for most of its settings. The configuration merge feature allows you to specify a merge file that overwrites the values of one or more settings that in the default CPF that comes with an instance when it is deployed. For details, see Automating Configuration of InterSystems IRIS with Configuration Merge.

To use configuration merge when deploying your IrisCluster, customize the template common.cpf, data.cpf, and compute.cpf files provided in the samples/ directory by adding the CPF settings you want to apply to all InterSystems IRIS nodes, the data nodes, and the compute nodes (if included) respectively. The provided common.cpf template file contains only a sample passwordHash setting, described in the next section; the data.cpfand compute.cpf template files contain only a sample SystemMode setting, which displays text on the InterSystems IRIS Management Portal.

For numerous helpful examples of parameters you might customize for several purposes, see Useful Parameters for Automated Deployment in Automating Configuration of InterSystems IRIS with Configuration Merge. For example, the data nodes in a sharded cluster must be configured to allocate a database cache of the appropriate size; Deploy Sharded Clusters in “Useful Parameters” illustrates how you would add the [config] section globals parameter to the data.cpf file to configure the database caches of the data nodes at the size you calculated. This is shown in the following, with a value of 20 GB (the globals setting is in MB):

[StartUp]
SystemMode=my-IrisCluster
[config]
globals= 0,0,20480,0,0,0

Set the default InterSystems IRIS password

InterSystems IRIS is installed with several predefined user accounts, the initial password for which is SYS. For effective security, it is important that this default password be changed during or immediately following deployment of all InterSystems IRIS containers. Although the configuration merge files are optional, you should include at least a common.cpf containing the passwordHash parameter, as illustrated in the provided template common.cpf, to reset the default password on all InterSystems IRIS nodes, for example:

[Startup]
PasswordHash=dd0874dc346d23679ed1b49dd9f48baae82b9062,10000,SHA512

InterSystems publishes through the ICR the intersystems/passwordhash image, from which you can create a container that generates the hash for you; for more information about this, the passwordHash parameter, and the default password,, see Authentication and Passwords in Running InterSystems Products in Cotainers.

If you do not provide a common.cpf, you should provide both a data.cpf and a compute.cpf, even if you don’t intend to deploy compute nodes or application servers, as you may add them later.

Prepare the Web Gateway configuration file

As described in Configure the Web Gateway in the Web Gateway Configuration Guide, the Web Gateway’s configuration is managed using the Web Gateway management pages, but contained in the CSP.ini file (much as an InterSystems IRIS instance’s configuration is contained in the iris.cpf file). The CSP.ini file you include in your config map, or the one generated by the IKO if you do not include one, is automatically installed on every Web Gateway node deployed. If you have experience with the Web Gateway, you can use a CSP.ini from an existing installation as a template and prepare one to be installed on your Web Gateway nodes by the IKO. You can also supply your own CSP.conf file, which contains the Web Gateway’s Apache or Nginx-specific configuration. Bear the following points in mind when deciding whether to provide your own file.

• Populating the remote server pool a CSP.ini file

  Because an IrisCluster does not include a Kubernetes-based mechanism for distributing application connections, deployed web server nodes provide the only means of controlling this distribution within the IrisCluster specification. (You can also deploy a custom load balancer on the Kubernetes cluster hosting the IrisCluster or use a third party-tool to distribute connections.) For this reason, ensure that the remote server pool in the configuration file, which specifies the InterSystems IRIS instances to which the Web Gateway directs incoming connections from the web server, is populated according to the relevant best practice for distributing application connections, as follows:

  • If the deployment is a sharded cluster and includes compute nodes, add all data and compute nodes to the pool; if there are no compute nodes, add all data nodes.

  • If the deployment is a distributed cache cluster, add all compute (application server) nodes to the pool.

  • If the deployment is a standalone instance, it should be the only remote server in the pool.

  Important:

  Web Gateway connections to mirrored data nodes are automatically mirror aware, that is, connections are always to twhichever failover member is the current primary.

  It is possible to populate the remote server pool before deployment because the names assigned to IrisCluster nodes always follow the same pattern, which is clustername-nodetype-0. For example, in an IrisCluster called myCluster, data node 1 of a sharded cluster, the data server of a distributed cache cluster, and a standalone instance would all be called myCluster-data-0 unless mirrored, in which case the primary (as deployed) would be myCluster-data-0-0 and the backup myCluster-data-0-1. Other nodes are named in the same format (except that they cannot be mirrored), for example myCluster-compute-0. Therefore, if an IrisCluster of this name was a sharded cluster with four mirrored data nodes, four compute nodes, an arbiter, and two Web Gateway nodes, they would be named as shown in the following table. The data nodes that would go in the remote server pool in the Web Gateway configuration file — that is, the deployed mirror primaries – are indicated by *; all of the compute nodes would be added to the pool.

  myCluster-data-0-0* myCluster-data-0-1 myCluster-data-1-0* myCluster-data-1-1 myCluster-data-2-0* myCluster-data-2-1 myCluster-data-3-0* myCluster-data-3-1

  myCluster-compute-0 myCluster-compute-1 myCluster-compute-2 myCluster-compute-3

  myCluster-arbiter-0

  myCluster-webgateway-0 myCluster-webgateway-0

  If you do not include a CSP.ini file in your config map, the IKO will generate one appropriate for the cluster and install it on every Web Gateway node. The IKO can also generate a CSP.conf file if you do not provide one.

  Important:

  The CSP.ini file generated by the IKO does not include a SSL/TLS connection; this can be configured on the Web Gateway management pages on the individual nodes.

• Updating the configuration of multiple Web Gateways

  As described in Connect to the IrisCluster, the serviceTemplate field creates one or more Kubernetes servicesOpens in a new tab that expose the first stateful set in the pod for each deployed node type to the network through external IP addresses. This includes Web Gateway nodes, which means that if you have deployed just one, you can always access it management pages to change its configuration by loading the URL http://external-ip:80/csp/bin/Systems/Module.cxw in your browser. (For nodes deployed from the webgateway-lockeddown image, load http://external-ip:52773/csp/bin/Systems/Module.cxw.) If you want to make configuration changes across multiple Web Gateway nodes, however, you have three options:

  • If you provided your own CSP.ini file, use the procedure described in Create a config map for the configuration files for modifying the CSP.ini or .cpf files on deployed nodes.

  • Create a similar service for each of the Web Gateway nodes that exposes it through an external IP address, then use the URL above to make the changes on each node separately (but see the following item regarding the needed management access credentials).

  • Directly modify the CSP.ini file on each pod by using the kubectl exec command to open a shell or edit the file within the container.

• Securing management access to the Web Gateway

  The predefined CSPSystem account on an InterSystems IRIS instance is used for communication between an InterSystems IRIS instance and the local InterSystems Web Gateway instance installed with it, and by default is used for access to the local Web Gateway’s configuration through its management pages. For effective security, you must change the default password for this account (along with the other predefined accounts) on both an InterSystems IRIS instance and its local Web Gateway instance either as part of deployment or immediately after; Authentication and Passwords in Running InterSystems Products in Containers provides details on this topic and instructions for making these changes in an InterSystems IRIS container. In the case of a Web Gateway node deployed by the IKO, you must independently secure management access, as follows:

  • If you provide your own CSP.ini file, ensure that if CSPSystem (or any other predefined account) is specified as the management access account, the password is not SYS; if it is, change it. You can also change the access credentials to any username and password you choose. Additionally, while the Web Gateway encrypts the management access password in the CSP.ini file on first starting up, including it in plain text could expose it on disk or the network, so the best practice is to make sure it is encrypted in the file you provide; you can do this after you change it, if necessary, by starting up a local test Web Gateway instance with your configuration file in place, which encrypts the password.

  • The CSP.ini generated by the IKO if you do not provide one specifies CSPSystem/SYS as the management access credentials; to change this, you must use either the second method described above for updating multiple Web Gateway configurations (creating services) or the third (using kubectl exec).

• Securing Web Gateway connections to remote InterSystems IRIS servers

  Each entry in a Web Gateway’s remote server pool includes credentials used to authenticate to the specified InterSystems IRIS instance; these are the username and password of an account on the instance. If you provide your own CSP.ini, ensure that the specified account is a secure one that does not use the default password (see Authentication and Passwords). You can then specify this account for each remote server in your CSP.ini; clearly it is simplest to use the same account for all. The CSP.ini generated by the IKO if you do not provide one specifies _System/(blank) as the management access credentials for each remote server; to change this (which you must for effective security), use either the second method described above for updating multiple Web Gateway configurations (creating services) or the third (using kubectl exec).

Considering the points above, for deployments that include multiple Web Gateway nodes, the flexibility and convenience of providing the CSP.ini file and being able to modify and redistribute it as needed make this the recommended approach in general.

Create a config map for the configuration files

Create a Kubernetes config map specifying the files, using a command like this:

$ kubectl create cm iris-cpf --from-file common.cpf --from-file data.cpf 
    --from-file compute.cpf --from-file CSP.ini

You can then specify iris-cpf as the value for configSource. (If you did not create a config map, do not specify a value for this field.)

image:

      image: containers.intersystems.com/intersystems/iris:2021.1.0.205.0

Required

The image: field specifies the URL (registry, repository, image name, and tag) of the InterSystems IRIS image from which to deploy data node containers. The example above specifies an InterSystems IRIS image from the InterSystems Container Registry (ICR). The registry credentials in the secret specified by the imagePullSecrets field are used for access to the registry.

updateStrategy:

      updateStrategy:
        type: {RollingUpdate|OnDelete}

Optional

Overrides the top level updateStrategy setting to specify the Kubernetes update strategyOpens in a new tab used for the stateful setsOpens in a new tab representing this node type only. The value can be either RollingUpdate (the default) or OnDelete.

preferredZones:

      preferredZones:
        - zoneN
        - ...

Optional

Specifies the zone or zones in which data nodes should be deployed, and is typically used as follows:

• If mirrored is set to true and at least two zones are specified, Kubernetes is discouraged (but not prevented) from deploying both members of a failover pair in the same zone, which maximizes the chances that at least one is available, and is therefore the best practice for high availability. Bear the following mind, however:

  • Deploying the members of a failover pair in separate zones is likely to slightly increase latency in the synchronous communication between them.

  • Specifying multiple zones for the data nodes means that all of the primaries might not be deployed in the same zone, resulting in slightly increase latency in communication between the data nodes.

  • Specifying multiple zones for data nodes generally makes it impossible to guarantee that nodes of other types (compute, Web Gateway, SAM) are in the same zone as all of the data node primaries at any given time regardless of your use of preferredZones in their definitions, increasing latency in those connections as well.

  Under most circumstances these interzone latency effects will be negligible, but with some demanding workloads involving high message or query volume, performance may be affected. If after researching the issue of interzone connections on your Kubernetes platform and testing your application thoroughly you are concerned about this performance impact, consider specifying a single zone for your mirrored data nodes.

  Regardless of the zones you specify here, you should use the preferredZones field in the arbiter definition to deploy the arbiter in a separate zone of its own, which also helps optimize mirror availability.

• The data nodes of an unmirrored cluster are typically deployed in the same zone to minimize latency. If mirrored: false and your Kubernetes cluster includes multiple zones, you can use preferredZones to follow this practice by specifying a single zone in which to deploy the data nodes.

• The value of the preferredZones: field in the compute definition, if included, should ensure that the compute nodes are deployed in the same zone or zones as the data nodes to minimize latency (see Plan Compute Nodes in the Scalability Guide).

Kubernetes attempts to deploy in the specified zones, but if this is not possible, deployment proceeds rather than failing.

volumes: Request ephemeral storage volumes

  volumes:
  - name: nameN
    type:
      typeName: name
  - ...

Optional

Specifies one or more ephemeral storage volumesOpens in a new tab, which are mounted on data or compute nodes using the volumeMounts field. An ephemeral volume stores data that is used by an application but does not need to persist across restarts, for example read-only input such as configuration data and licenses. (For an example of using ephemeral volumes to provide such data, see licenseKeySercret.) Because ephemeral volumes are created and deleted along with pods, pods using them can be stopped and restarted without having to maintain access to a particular persistent volume. There are several types of ephemeral volume, including configMapOpens in a new tab, secretOpens in a new tab, emptyDirOpens in a new tab, and others.

podTemplate:

      podTemplate:
        spec:
          args:        

Optional

Specifies overrides and additions to the default pod templateOpens in a new tab applied to the data node pods. Because containers are run only within a pod on Kubernetes, a pod template can specify the fields that define numerous Kubernetes entities, including the fields defining a containerOpens in a new tab, which makes it useful for many purposes. Three examples of using containers: fields are provided in the following; one is required when deploying from the iris-lockeddown image, the other two are optional solutions for specific situations.

• Specify the required security contextOpens in a new tab for the iris-lockeddown container.

  The iris-lockeddown highly secure InterSystems IRIS image, described in detail in Locked Down InterSystems IRIS Container in Running InterSystems Products in Containers, contains an InterSystems IRIS instance that was installed with Locked Down security by a nonroot user, irisowner (UID 51773), and is owned by that user. To successfully deploy data nodes from this image, you must specify the required security context in the podTemplate spec in the data definition, as follows:

  podTemplate:
    spec:
      securityContext: {
        runAsNonRoot: true,
        runAsUser: 51773,
        runAsGroup: 51773,
        fsGroup: 51773
  

  

  If you also deploy compute nodes from the iris-lockeddown image, you must include this security context in the podTemplate spec in the compute definition as well.

  Important:

  You cannot successfully deploy data or compute nodes from the iris-lockeddown image unless you include this security context.

• Prevent InterSystems IRIS from starting up with the container.

  You can use the args field in the pod template to specify options to the InterSystems IRIS entrypoint application, iris-main. For example, if there is something wrong with the InterSystems IRIS configuration which prevents startup from succeeding, iris-main exits, causing the pod to go into a restart loop, which makes it difficult or impossible to diagnose the problem. You can prevent the instance from starting by adding the iris-main option --up false as follows:

  podTemplate:
    spec:
      args:
      - --up
      - "false"
  

  

  When you do this, the readiness probeOpens in a new tab will not be satisfied, and the deployment will be paused indefinitely:

  $ kubectl get pods
  NAME            READY   STATUS    RESTARTS   AGE
  iris-data-0     0/1     Running   0          32s
  

  

  After addressing the problem, you can either

  • Manually start the instance with a command like the following:

    $ kubectl exec -it iris-data-0 -- iris start IRIS
    

    

  • Remove the podTemplate override and redeploy the pod.

• Override the default liveness probe and readiness probeOpens in a new tab.

  If you wanted to replace the default liveness probe and readiness probe for data nodes (or another node type), you could specify something like the following in the applicable pod template:

  podTemplate:
    spec:
      livenessProbe:
        exec:
          command:
          - /bin/true
      readinessProbe:
        httpGet:
          path: /csp/user/cache_status.cxw
          port: 52773
  

  

shards:

      shards: N

Optional

Specifies the number of data nodes to be deployed as a sharded cluster. If the shards field is omitted, a single standalone instance of InterSystems IRIS is deployed, optionally as the data server in a distributed cache cluster; for more information, see Define the IrisCluster topology.

Data nodes can be added to the deployed cluster by increasing this setting and reapplying the definition (as described in Modifying the IrisCluster), but the setting cannot be decreased .

mirrored:

      mirrored: {true|false}

Optional

Determines whether the data nodes in the deployment are mirrored.

If the value of mirrored: is true, two mirrored instances are deployed for each data node specified by the shards field. For example, if shards: 4 and mirrored: true, eight data node instances are deployed as four failover pairs, creating a mirrored sharded cluster with four data nodes. If mirrored: is true when shards is omitted, two mirrored instances are deployed as the standalone InterSystems IRIS instance, which can optionally be the mirrored data server of a distributed cache cluster; for details, see Define the IrisCluster topology.

The default for mirrored is false.

storage*:

      storage{DB|WIJ|Journal1|Journal2}:
        resources:
          requests:
            storage: size

Optional

Specify a custom size for one or more of the four predefined persistent volumesOpens in a new tab deployed with each data node, as follows:

• storageDB — The volume on which data is stored, including durable %SYS data.

• storageWIJ — The volume containing the WIJ directory.

• storageJournal1 — The volume containing the primary journal directory.

• storageJournal2 — The volume containing the alternate journal directory.

The predefined volumes are mounted in /irissys inside the container, and are 2 GB by default. When including an override as illustrated, size (the value of the storage field) can be specified in any unit between kilobytes and exabytesOpens in a new tab).

The amount of data storage to be mounted on sharded cluster data nodes is determined during the cluster planning process and should include a comfortable margin for the future growth of your workload. Use a storageDB size override, additional volumes specified by the volumeMounts field, or both to ensure sufficient storage is available to each data node.

The same four fields can be used to modify the same predefined volumes in the compute node definition (compute), which have the same default size of 2 GB. The data storage for sharded cluster compute nodes (or distributed cache cluster application servers), as specified in the storageDB field in the compute section, should be kept to a bare minimum to conserve resources, as compute nodes do not store data; you may even want to create a separate storage class for compute nodes and specify it by adding the storageClassName: field to the storageDB storage override.

In the Web Gateway node definition (webgateway), you can override the size of the only predefined volume, storageDB volume; the default size of this volume on Web Gateway nodes is 32 mb.

volumeMounts:

      volumeMounts:
      - name: volumeClaimTemplateN
        mountPath: pathN
      - ...

Optional

Specifies one or more persistent storage volumesOpens in a new tab, as defined in the volumeClaimTemplates field, and/or ephemeral storage volumesOpens in a new tab, as specified in the volumes field, to be deployed with each data node, in addition to the predefined persistent volumes. Each volume is defined by the name of one of the volume claim templates or volumes and a mountPath, which is a direct reference to a location in the container’s filesystem, visible to the InterSystems IRIS instance, on which to mount the volume.

The volumeMounts field can also be used to specify additional volumes in the compute node definition (compute), typically ephemeral volumes, as compute nodes do not store persistent data.

image

image: containers.intersystems.com/intersystems/iris:2021.1.0.205

Required in optional compute: section

Compute nodes are deployed from the same InterSystems IRIS image as data nodes; an example is shown.

replicas:

         replicas: N

Required in optional compute: section

Specifies the number of identical compute nodes to deploy. In a sharded cluster, this should be a multiple of the number of data nodes specified by shards (for more information, see Plan Compute Nodes in the Scalability Guide).

Compute nodes can be added to or removed from the deployed IrisCluster by changing this setting and reapplying the definition (as described in Modifying the IrisCluster).

The replicas field also appears in the webgateway section, where it specifies the number of InterSystems Web Gateway (web server) nodes to deploy.

image

image: containers.intersystems.com/intersystems/iris:2021.1.0.205

Required in optional webgateway: section

Web Gateway nodes are deployed from an InterSystems webgateway image, an example of which is shown.

type:

      type: {apache|apache-lockeddown|nginx}

Optional

Specifies deployment of an Apache web server, an Apache web server with locked-down security, or an Nginx web server; the default is apache.

replicas:

      replicas: N

Required in optional webgateway: section

Specifies the number of identical Web Gateway nodes to deploy.

The replicas field also appears in the compute section, where it specifies the number of compute nodes to deploy.

applicationPaths:

      applicationPaths:
        - pathN
        - ...

Optional

Provides a list of application paths to configure in the Web Gateway. Application paths should not have a trailing slash, and the path /csp is reserved.

alternativeServers:

      alternativeServers: {FailOver|LoadBalancing}

Optional

Selects the method by which the Web Gateway on each node determines which InterSystems IRIS server (that is, which data node) in its remote server pool to connect to (see Load Balancing and Failover Between Multiple InterSystems IRIS Server Instances in the Web Gateway Guide). Possible values are FailOver and LoadBalancing, with a default of LoadBalancing.