Durable %SYS for Persistent Instance Data

This section describes the durable %SYS feature of InterSystems IRIS, which enables persistent storage of instance-specific data and user-created databases when InterSystems IRIS is run within a container, and explains how to use it.

Overview of InterSystems IRIS Durable %SYS

Separation of code and data is one of the primary advantages of containerization; a running container represents "pure code" that can work with any appropriate data source. However, all applications and programs generate and maintain operating and historical data — such as configuration and language settings, user records, and log files — which must be retained beyond the life of a single container to enable upgrades. For this reason, containerization typically must address the need to persist program-related data, including application databases, on durable data storage. This requires a mechanism that identifies the data to be retained and its persistent location, and can direct the new container to it following an upgrade or a failure of the previous container.

The durable %SYS feature does this for an InterSystems IRIS container by cloning the needed instance-specific data (such as log, journal, and WIJ files and system databases such as IRISSYS) to a chosen location on the file system of the container’s host and resetting the instance to use the data in the new (cloned) location. To use durable %SYS, you must mount the chosen location as a volume within the container and identify it in an environment variable specified when the container is started. While the InterSystems IRIS instance remains containerized, its instance-specific data exists outside the container, just like the databases in which application data is stored, persisting it across container and instance restarts and making it available for upgrading the instance. (For more information on upgrading, see Upgrading InterSystems IRIS Containers.)

Important:

To maintain separation of code and data, InterSystems recommends creating all InterSystems IRIS databases on a mounted external volume, either from within InterSystems IRIS or by using the iris merge commandOpens in a new tab to add them when building a custom InterSystems IRIS image from an InterSystems-supplied image, as described in Creating InterSystems IRIS Images, so that they are cloned to durable storage by durable %SYS.

Contents of the Durable %SYS Directory

The durable %SYS directory, as created when a container is first started, contains a subset of the InterSystems IRIS install tree, including but not limited to:

The configuration parameter file (CPF), which is named iris.cpf. Additional versions of the file (older versions and _LastGood_.cpf) are created as with any InterSystems IRIS instance. (Automating Configuration of InterSystems IRIS with Configuration MergeOpens in a new tab, Configuration Parameter File Reference)
The /csp directory, containing the Web Gateway configuration and log files. (Web Gateway GuideOpens in a new tab)

Note:

Web Gateway containers have a similar durable storage feature, and when this is in use (see Options for Running Web Gateway Containers) the directory is located on its own persistent storage.
The file /dist/install/misc/buildver, which contains the instance’s version, for example 2023.2.0.299.0.
The file /httpd/httpd.conf, the configuration file for the instance web server. (See Supported Web ServersOpens in a new tab)
The /mgr directory, containing the following:
- The IRISSYS system database, comprising the IRIS.DAT and iris.lck files and the stream directory, and the iristemp, irisaudit, iris and user directories containing the IRISTEMP, IRISAUDIT, IRIS and USER system databases. (System-Supplied Databases and Custom Items in IRISSYS)
- The write image journaling file, IRIS.WIJ (which may be relocated to achieve file system separation). (Write Image Journaling and Recovery)
- The /journal directory containing journal files (which may be relocated to achieve file system separation). (See Journaling; see also Separating File Systems for Containerized InterSystems IRIS)
- The /temp directory for temporary files.
- Log files including messages.log, journal.log, and SystemMonitor.log. Additional logs may be present initially and some are created as needed, for example backup and mirror journal logs. (Monitoring InterSystems IRIS Logs.)
  
  Note:
  
  Durable %SYS activity is logged in the messages.log file; if you have any problems in using this feature, examine this log for information that may help. For information about how to read this log from outside the container, see The iris-main Program. For information about accessing the Management Portal of a containerized InterSystems IRIS instance, see Web Access Using the Web Gateway Container.
- The InterSystems IRIS license key file, iris.key, either at container start if it is included in the InterSystems IRIS image or when a license is activated while the container is running. (Activating a License Key)
- Several InterSystems IRIS system files.

All databases defined on the instance, beyond the standard InterSystems IRIS databases listed in the first bullet above, that are not read-only. This is to ensure that databases added to the instance in a user-created image based on an InterSystems-supplied image, as described in Creating InterSystems IRIS Images, are included in the durable data. If a database directory is underneath the install directory in the container, for example if it is under /usr/irissys/mgr, it is copied to the corresponding location in the durable %SYS directory. If one or more database directories are not underneath the install directory, a new folder db is created in the install directory and they are copied there.

Locating the Durable %SYS Directory

When selecting the location in which this system-critical instance-specific information is to be stored, bear in mind the following considerations:

The availability of appropriate backup and restore procedures (Backup and Restore).
Any high availability mechanisms you have in place (High Availability Guide) .
Available storage space and room for expansion (Maintaining Local Databases).

There must be at least 200 MB of space available on the specified volume for the durable %SYS directory to initialize. For various reasons, however, including operational files such as journal records and the expansion of system databases, the amount of data in the directory can increase significantly.

InterSystems recommends specifying a subdirectory of a mounted volume, rather than the top level, as the durable %SYS location. For example, if an external file system location is mounted as the volume /external in the container, /external should not be specified as the durable %SYS location, but rather a directory on /external such as /external/durable.

Important:

Because the instances in the iris and iris-lockeddown images were installed and are owned by user irisowner (UID 51773), as described in Security for InterSystems IRIS Containers, the file system location you specify for durable %SYS must first be made writable by irisowner, for example with this command:

chown 51773:51773 root-of-durable-%SYS-directory

(You will probably need root privileges to effect this.

When using durable %SYS on Kubernetes without the InterSystems Kubernetes OperatorOpens in a new tab, you must include the following security contextOpens in a new tab setting in the pod specification:

securityContext:
  fsGroup: 51773

Running an InterSystems IRIS Container with Durable %SYS

To use durable %SYS, include in the docker run command the following options:

--volume /volume-path-on-host:/volume-name-in-container
--env ISC_DATA_DIRECTORY=/volume-name-in-container/durable-directory

where volume-path-on-host is the pathname of the durable storage location to be mounted by the container, volume-name-in-container is the name for the mounted volume inside the container, and durable_directory is the name of the durable %SYS directory to be created on the mounted volume. For example:

docker run --detach 
  --volume /data/dur:/dur 
  --env ISC_DATA_DIRECTORY=/dur/iconfig 
  --name iris21 intersystems/iris:latest-em

In this example, the durable %SYS directory would be /data/dur/iconfig outside the container, and /dur/iconfig inside the container.

Important:

InterSystems strongly recommends using bind mounts, as illustrated in the preceding example, when mounting external volumes for InterSystems IRIS containers on production systems. However, under some circumstances, such as testing and creating demos or anything that you want to be portable to platforms other than Linux, it is preferable to use named volumes, because they eliminate problems related to directory paths, permissions, and so on. For detailed information about each method, see Manage data in DockerOpens in a new tab in the Docker documentation.

InterSystems does not support mounting NFS locations as external volumes in InterSystems IRIS containers, and iris-main issues a warning when you attempt to do so. A similar warning is issued if the specified durable %SYS location is on a mounted volume that has a file system type of cifs or any type containing the string fuse.

Note:

The --publish option publishes the InterSystems IRIS instance’s superserver port (1972) to the host, so that outside entities (including those in other containers) can connect to the instance. To avoid potential problems with the Docker TCP stack, you can replace the --publish option with the --net host option, which lets the container publish its default socket to the host network layer. The --net host option can be a simpler and faster choice when the InterSystems IRIS container you are running will be the only such on the host. The --publish option may be more secure, however, in that it gives you more control over which container ports are exposed on the host.

When you run an InterSystems IRIS container using these options, the following occurs:

The specified external volume is mounted.
The InterSystems IRIS installation directory inside the container is set to read-only.
If the durable %SYS directory specified by the ISC_DATA_DIRECTORY environment variable, iconfig/ in the preceding example, already exists and contains a /mgr subdirectory, all of the instance’s internal pointers are reset to that directory and the instance uses the data it contains. If the InterSystems IRIS version of the data does not match the version of the instance, an upgrade is assumed and the data is upgraded to the instance’s version as needed. (For information on upgrading, see Upgrading InterSystems IRIS Containers.)
If the durable %SYS directory specified by ISC_DATA_DIRECTORY does not exist, or exists and is empty:
- The specified durable %SYS directory is created if necessary.
- The directories and files listed in Contents of the Durable %SYS Directory are copied from their installed locations to the durable %SYS directory (the originals remain in place).
- All of the instance’s internal pointers are reset to the durable %SYS directory and the instance uses the data it contains.
If for any reason the process of copying the durable %SYS data and resetting internal pointers fails, the durable %SYS directory is marked as incomplete; if you try again with the same directory, the data in it is deleted before the durable %SYS process starts.
If the durable %SYS directory specified by the ISC_DATA_DIRECTORY environment variable already exists and contains data (file or subdirectories) but does not contain a /mgr subdirectory, no data is copied; the container does not start, and the reason (data other than durable %SYS in the directory) is logged to standard output by iris-main, as described in The iris-main Program.

In the case of the example provided, the InterSystems IRIS instance running in container iris21 is configured to use the host path /data/dur/iconfig (which is the path /dur/iconfig inside the container) as the directory for persistent storage of all the files listed in Contents of the Durable %SYS Directory. If durable %SYS data does not already exist in the host directory /data/dur/iconfig (container directory /dur/iconfig) it is copied there from the installation directory. Either way, the instance’s internal pointers are set to container directory /dur/iconfig.

See Running InterSystems IRIS Containers for various examples of launching an InterSystems IRIS container with durable %SYS.

The following illustration shows the relationship between the InterSystems IRIS installation directory and user databases within a container and on the mounted external storage to which they were cloned by durable %SYS (as specified by the options shown). Note that the three application databases outside of the install directory have been cloned to /data/dur/iconfig/db, whereas the LOCALDB database, which is within the install directory under /mgr, is cloned to the same location (/data/dur/iconfig/mgr/localdb).

File System Objects Cloned by Durable %SYS

Shows how only a subset of the InterSystems IRIS installation directory in the container is copied to durable %SYS

Identifying the Durable %SYS Directory Location

When you want to manually verify the location of the durable %SYS directory or pass this location programmatically, you have three options, as follows:

Open a shell inside the container, for example with docker exec -it container_name bash, and do either of the following:
```
echo $ISC_DATA_DIRECTORY

iris list
```
Note:

For detailed information on the iris command, see Controlling InterSystems IRIS Instances.
Within InterSystems IRIS, call $SYSTEM.Util.InstallDirectory() or $SYSTEM.Util.GetEnviron(‘ISC_DATA_DIRECTORY’).

Ensuring that Durable %SYS is Specified and Mounted

When a container is run with the ISC_DATA_DIRECTORY environment variable, pointers are set to the durable %SYS files only if the specified volume is successfully mounted.

If ISC_DATA_DIRECTORY is specified but the needed --volume /external_host:/durable_storage option is omitted from the docker run command, the instance fails to start and an error message is generated.
If the --volume option is included but Docker cannot successfully mount the specified volume, it creates the external storage directory and the volume within the container; in this case, the instance data is copied to the durable %SYS directory, as described under "If the durable %SYS directory specified by ISC_DATA_DIRECTORY does not exist" in Running an InterSystems IRIS Container with Durable %SYS.

If ISC_DATA_DIRECTORY is not specified, the InterSystems IRIS instance uses the instance-specific data within the container, and therefore cannot operate as an upgrade of the previous instance.

To use durable %SYS, you must therefore ensure that all methods by which your InterSystems IRIS containers are run incorporate these two options.

Separating File Systems for Containerized InterSystems IRIS

In the interests of performance and recoverability, InterSystems recommends that you locate the primary and secondary journal directories of each InterSystems IRIS instance on two separate file systems, which should also be separate from those hosting InterSystems IRIS executables, the instance’s system databases, and the IRIS.WIJ file, with the latter optionally on a fourth file system. Following InterSystems IRIS installation, however, the primary and secondary journal directories are set to the same path, install-dir/mgr/journal, and thus may both be set to /mgr/journal in the durable %SYS directory when durable %SYS is in use.

You can configure separate file systems in deployment using a configuration merge file, as described in Automating Configuration of InterSystems IRIS with Configuration MergeOpens in a new tab. After the container is started, you can reconfigure the external locations of the primary and secondary directories using the Management Portal (see the following section) or by editing the CPF (iris.cpf), as long as the volumes you relocate them to are always specified when running a new image to upgrade the InterSystems IRIS instance.

Note:

When the durable %SYS directory is in use, the IRIS.WIJ file and some system databases are already separated from the InterSystems IRIS executables, which are inside the container. Under some circumstances, colocating the IRIS.WIJ file with your application databases instead may improve performance.

For more information about separation of file systems for InterSystems IRIS, see File System Separation.