Deploy a System Alerting and Monitoring Image

SAM 1.1 Release Notes

SAM version 1.1 provides performance improvements for the graphs in the Grafana dashboard and the underlying Prometheus queries, especially when displaying metrics over a longer period of time.

If you are currently using SAM 1.0, you can upgrade to SAM 1.1.

SAM 2.0 Release Notes

SAM version 2.0 adds several additional features. These include:

• An easy to use tool for exporting and importing SAM cluster configurations as JSON files

• Support for SSL/TLS connections between SAM and your IRIS instances

• Support for defining instances with URL prefixes, providing added flexibility for systems hosting multiple IRIS instances at the same IP address

• Custom Grafana dashboards, allowing you to toggle between multiple perspectives of instance performance

Allow Unauthenticated Access to the /api/monitor/ Endpoint

Each InterSystems IRIS instance version 2020.1 or higher contains the /api/monitor web application. In order for SAM to collect metrics and alerts from the /api/monitor endpoint, the endpoint must allow for unauthenticated access. To make sure this is the case:

1. Open the Management PortalOpens in a new tab of the InterSystems IRIS instance you would like to add to SAM.

2. Go to the Web Applications page (System Administration > Security > Applications > Web Applications).

3. Select /api/monitor to open the Edit Web Application page.

4. In the Security Settings section, select Unauthenticated.

For more information about the /api/monitor/ web application, see Monitoring InterSystems IRIS Using REST APIOpens in a new tab in Monitoring Guide.

Enable the Built-in Monitoring Tools

InterSystems IRIS instances have built-in monitoring tools, and SAM uses these tools to collect information about the instance. Check that the following tools are enabled on the InterSystems IRIS instance:

• System MonitorOpens in a new tab, which SAM uses when determining the instance state. By default, System Monitor is enabled.

• Log MonitorOpens in a new tab, which enables SAM to see alerts from the instance. By default, Log Monitor is enabled and writes alerts to the alerts log.

  Important:

  SAM is only able to view instance alerts if Log Monitor writes them to the alerts log. If Log Monitor sends alerts by email instead of by writing them to the alerts log, SAM cannot view alerts for the instance.

Ensure that the Instance is Uniquely Identifiable

In order for SAM to monitor an InterSystems IRIS instance, the instance must be uniquely identifiable.

For SAM 1.0 and 1.1, an instance must be uniquely identifiable by a combination of IP (or domain name) and Port.

For SAM 2.0, an instance must be uniquely identifiable by a combination of IP (or domain name), Port, and URL Prefix.

For more information about configuring multiple InterSystems IRIS instances on the same system and URL prefixes, see the Connecting to Remote ServersOpens in a new tab topic in System Administration Guide.

For Interoperability Productions: Enable Collection of Metrics

Beginning with InterSystems IRIS version 2021.1, instances can also record metrics about active interoperability productionsOpens in a new tab and make them available to SAM. The recording of these interoperability metrics is disabled by default, and must be enabled for each interoperability production you want to monitor.

Refer to Interoperability MetricsOpens in a new tab in Monitoring InterSystems IRIS Using REST APIOpens in a new tab for detailed instructions for enabling these metrics, as well as descriptions of the metrics which are available.

Docker Compose

For instructions on installing Docker Compose, see the installation guideOpens in a new tab provided in the Docker documentation. The following versions are required:

• Docker Engine version 19.03.098 or higher

• Docker Compose version 1.25 or higher

Docker Compose needs no further configuration in order to run SAM. However, after you download and extract a SAM image, you may still need to configure the docker-compose.yml file according to the needs of your system.

Podman Compose

Install Podman using the package manager for your distribution according to the instructions provided in the Podman documentationOpens in a new tab. For instructions on installing Podman Compose, see the GitHub pageOpens in a new tab for the project. InterSystems recommends using Podman version 3.4.0 or higher and the most recent stable version of Podman Compose.

The following additional steps are necessary to run SAM using Podman Compose:

1. If you are running SAM on a system secured with SELinux, SAM components require elevated permissions. (You can check whether this is the case by issuing the sestatus command from the command line.) If this is the case, include the following line for each service in the docker-compose.yml file:

   privileged: true
   

   
   Note:

   The docker-compose.yml file for images of SAM version 2.0 or higher includes this line as a comment everywhere it is required; you only need to remove the # character to complete this configuration step.

2. If the file system for your Linux distribution does not feature both an /etc/timezone file and an /etc/localtime symbolic link, you must map paths to the equivalent locations on your system in the volume configuration for the iris container. To do so:

   1. In the iris section of the docker-compose.yml file, locate the following lines:

      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      

   2. As needed, replace the first path in one or both lines (the path which does not feature the :ro option) to the equivalent path on your system.

3. To allow SAM to alias IP addresses, you must ensure that Podman is using one of the following network stacks:

   • Netavark

   • Container Networking Interface (CNI) with the dnsname plug-in

   If you configure one of these network stacks before you start SAM for the first time, no further configuration is necessary. If you attempt to start SAM using CNI without installing the dnsname plug-in first, startup will fail and you must reconfigure the SAM container network manually by completing the following steps:

   1. Stop all SAM containers.

   2. Install the dnsname plug-in for CNI. The podman-plugins package contains dnsname. On Red Hat Enterprise Linux systems, you can install this package by issuing the following command from the command line:

      sudo yum install podman-plugins
      

      

   3. Locate the configuration file for the SAM container network. By default, this file is saved to /etc/cni/net.d/sam_default.conflist.

   4. Using a text editor, edit the SAM network configuration file so that it includes the following block:

      {
           "type": "dnsname",
           "domainName": "dns.podman",
           "capabilities": {
                "aliases": true
           }
      }
      

      

   5. Delete the misconfigured SAM containers and the sam_default network by issuing the following command:

      podman rm -a; podman network rm sam_default
      

      

   6. Restart SAM to rebuild the SAM containers and the sam_default network using the correct configuration.

   Note:

   If you decide to switch to Netavark instead of adding the dnsname plug-in to CNI, it is not necessary to edit the sam_default.conflist network configuration file as described in the preceding instructions. Instead, you must edit the Podman containers.conf file (described on this Github pageOpens in a new tab) to specify the new network_backend. Then, delete the old sam_default.conflist network configuration file as well as the SAM containers and the sam_default network. When you restart SAM, the startup procedure will automatically generate a new sam_default.conflist file configured for the new network stack.

4. If you would like to run the ./start.sh and ./stop.sh scripts to start and stop SAM (instead of invoking the podman commands directly), edit these scripts in a text editor as follows:

   • In the start.sh script:

     1. Change the docker-compose command to an equivalent podman-compose command, such as podman-compose -p sam up -d

     2. On a new line before the podman-compose command, add the following command:

        podman unshare chown 51773:51773 <instanceDir>
        

        

        Where <instanceDir> is the directory which will contain your “durable” storage directory. This gives SAM containers the appropriate user permissions to access the directory when they run.

        Note:

        The file system entities associated with the InterSystems IRIS container are associated with the user account irisowner (UID 51773). Therefore irisowner user must have full permissions to the locations of these entities within the file system, including the durable storage directory. For more information, refer to Ownership and DirectoriesOpens in a new tab in Running InterSystems Products in ContainersOpens in a new tab.

   • In the stop.sh script:

     1. Change the docker-compose command to an equivalent podman-compose command. Because the specific options supported by podman-compose may change, this guide will use the simplified command podman-compose -p sam down

Set Up Your License Key

SAM comes with a free, built-in Community Edition license with the capacity to monitor approximately 40 instances. If you are using the Community Edition license, you can skip this section.

To specify a different SAM license:

1. Locate the iris service in the contents of the docker-compose.yml file.

2. Add a new command directly beneath the image line that specifies the desired license key to use.

   For example, with a key named iris.key located in the SAM /config subdirectory, add:

   [...]
   image:intersystems/SAM:1.0.0.100
   command: ["--key","/config/iris.key"]
   init:true
   [...]
   

   

For Docker Version 20.10.14 and up: Disable Capability Checker

Beginning with Docker Engine version 20.10.14, Docker grants containers a set of Linux capabilities incompatible with the Linux capability checker which the InterSystems IRIS containers for SAM 1.0 and 1.1 run on startup. For SAM 1.0 and 1.1 deployments using one of these later versions of Docker, the iris container will fail to start unless the InterSystems IRIS Linux capability checker is disabled.

To disable the capability checker:

1. Locate the iris service in the contents of the docker-compose.yml file.

2. Add the following line directly beneath the image line:

   command: --check-caps false
   

   

Configure the System Alerting and Monitoring Port

By default, SAM deploys on port 8080 of the host system. On Linux machines, you can check whether the port 8080 is in use by using the netcat command:

$ nc -zv localhost 8080
Connection to localhost 8080 port [tcp/http-alt] succeeded!

If the connection succeeds (like in the preceding example), then an application on your host system is already using port 8080 to listen for traffic. You may also need to configure your firewall to allow traffic on this port.

If necessary, you can change the host port mapping in the nginx section of the docker-compose.yml file. To do so:

1. Locate the nginx service in the contents of the docker-compose.yml file.

2. In the ports section, enter the desired port on your host machine. For example, if you would like to access SAM on port 9999, edit the section to look like:

   [...]
   ports:
      - 9999:8080
   [...]
   

   

For more information, see the “ports” section of the Docker Compose File Reference (https://docs.docker.com/compose/compose-file/#portsOpens in a new tab).

Change the Default Location for Storing Your Monitoring Data

Although containerized, SAM maintains a persistent copy of the data you collect about your IRIS instances by extending the durable %SYSOpens in a new tab feature of InterSystems IRIS. By default, SAM uses a Docker volume which stores data in the /var/lib/docker/volumes/ directory. However, this may be an inadequate storage location, especially if you are monitoring many instances or if this /var/ directory is located on a small partition.

InterSystems recommends implementing a higher volume storage solution for your monitoring data by configuring the location of the volume mounted for the SAM Manager, which corresponds to the iris service defined in the docker-compose.yml file. To do so:

1. Locate the iris service in the docker-compose.yml file.

2. Locate the volumes section. In its default configuration, this section looks like this:

   [...]
   volumes:
     - irisdata:/dur
     - ./config:/config
   [...]
   

   

3. Change the line which reads - irisdata:/dur so that the /dur directory in the IRIS container maps onto the desired location on your host system, instead of the named volume irisdata. To specify a directory elsewhere in the filesystem, replace irisdata with the path to the desired directory.

   Docker volumes can be configured to allow for a variety of storage solutions. For more information, refer to the Docker page about volumesOpens in a new tab and the “volumes” sectionOpens in a new tab of the Docker Compose File Reference page.