Manage Clusters, Instances, and Alert Rules

Add or Edit a Cluster

To add a SAM cluster, do the following:

1. Navigate to the main (Monitor Clusters) page in the SAM web application. Selecting System Alerting and Monitoring from anywhere within the application returns you to the main page.

2. Select + New Cluster to open the Add New Cluster dialog.

3. Fill in the following fields:

   • Cluster name — A unique identifier for the cluster. SAM ignores case when it evaluates cluster names. If you attempt to assign a name to a cluster which differs from the name of an existing cluster in case alone (for example, transactarchive and transactArchive), you will receive an error.

   • Description — An optional description for the cluster.

4. Select Add Cluster. This opens the Edit Cluster dialog, where you can continue to define your new cluster by adding instances and setting alert rules.

   Note:

   You can edit or delete an existing SAM cluster from its corresponding single cluster page.

Add or Edit an Instance within a Cluster

To add or edit an InterSystems IRIS instance within a SAM cluster, do the following:

1. Ensure that you have prepared the instance for monitoring by SAM.

2. Navigate to the main (Monitor Clusters) page in the SAM web application. Clicking System Alerting & Monitoring from anywhere within the application returns you to this page.

3. Select the cluster to which you would like to add the instance. If there are no clusters, you must create one.

4. Click Edit Cluster to open the Edit Cluster dialog.

5. Click the +New button at the top of the Instances table.

   Note:

   To edit or delete an existing instance, select it from the Instances table. You can also edit or delete an existing instance from its corresponding single instance page.

6. Fill in the following fields:

   • IP – The fully qualified domain name or IP address of the machine hosting the target InterSystems IRIS instance.

     InterSystems recommends using domain names whenever possible, as IP addresses may change.

   • Port – The web server port of the target InterSystems IRIS instance.

     Note:

     If the instance you are monitoring is located on the same system as SAM, you may enter host.docker.internal in this field.

     In SAM versions 1.0 and 1.1, each instance must be addressable by a unique combination of IP and Port. SAM version 2.0 adds support for URL prefixes, which provides greater flexibility for systems which host multiple InterSystems IRIS instances at the same IP address.

   • Cluster – The cluster to add the target instance to. When first adding an instance, this defaults to the current cluster.

     Note:

     Moving an instance between clusters may produce temporary irregularities in the reported alerts, dashboard, and state for that instance. These irregularities should resolve within a few hours.

   • Instance name and Description – Optional text descriptors to help you identify the instance.

7. Click Add Instance to begin monitoring the instance with SAM.

Create an Alert Rule for a Cluster

To create a new alert rule for a cluster, do the following:

1. Navigate to the main (Monitor Clusters) page in the SAM web application. Clicking System Alerting & Monitoring from anywhere within the application navigates to this page.

2. Select the cluster for which you would like to create an alert rule. If there are no clusters, you must create one.

3. Click Edit Cluster to open the Edit Cluster dialog.

4. Click the +New button at the top of the Alert Rules table.

   Note:

   You may select an existing alert rule from this table to edit or delete it.

5. Fill in the following fields:

   • Alert rule name – Any name for the alert rule. It is often useful to include the metric the rule uses in the name.

   • Alert severity – Either Critical or Warning. The severity of the alert determines the impact it will have on the instance state (described in a later section on this page)

     Note:

     You can give multiple alert rules the same name, but different severities. If both rules fire at the same time, SAM suppresses the rule with lower severity. This behavior reduces duplicate alerts firing for the same event.

   • Alert expression – An expression that defines when the alert fires, written in Prometheus Query Language.

     The alert expression syntax section contains an overview of the Prometheus Query Language syntax and several examples.

   • Alert message – A text description of the alert rule, which SAM displays when the alert fires.

     Note:

     The Alert message supports the $value variable, which contains the evaluated value of an alert expression. The syntax is:

     {{ $value }}
     

     

     The $value variable only holds one value; as such, you should not use it for alert rules that evaluate to multiple values (such as a rule that uses the and operator).

6. Click Add Alert Rule. SAM validates the alert expression and then adds the alert rule to the cluster.

Below is an example of an alert rule:

Basic Syntax for Alert Expressions

This section provides an overview of how to write alert expressions and some examples.

A simple alert expression compares a metric to a value. For example:

# Greater than 80 percent of InterSystems IRIS licenses are in use:
iris_license_percent_used{cluster="production"}>80

# There are less than 5 active InterSystems IRIS processes:
iris_process_count{cluster="test"}<5

# The disk storing the MYDATA database is over 75% full:
iris_disk_percent_full{cluster="test",id="MYDATA"}>75

# Same as above, but specifying directory instead of database name:
iris_disk_percent_full{cluster="production",dir="/IRIS/mgr/MYDATA"}>75

The basic format for an alert expression based on a single metric is:

metric_name{cluster="cluster_name",label(s)}>value

Alert Examples

The examples which follow demonstrate some of the capabilities of PromQL.

Example 1: Basic Alert Rule

The simplest alert expressions directly compare a single metric to a value.

The following alert expression evaluates iris_cpu_usage, which measures the total percent of CPU in use on the machine running InterSystems IRIS. If the value of iris_cpu_usage exceeds 90 for any InterSystems IRIS instance in the test cluster, the alert fires.

iris_cpu_usage{cluster="test"}>90

Example 2: Arithmetic Operators

PromQL supports the following arithmetic operators, ordered by precedence:

1. ^ (exponentiation)

2. * (multiplication), / (division), % (modulo)

3. + (addition), - (subtraction)

Arithmetic operators are particularly useful when writing an alert expression that contains two or more metrics.

The following expression is triggered when the USER database in the test cluster is greater than 90 percent full. The expression calculates the percent by dividing the database size (iris_db_size_mb) by the database maximum size (iris_db_max_size_mb).

(iris_db_size_mb{cluster="test",id="USER"}/iris_db_max_size_mb{cluster="test",id="USER"})*100>90

Example 3: Logical OR Operator

PromQL supports logical operators for writing more complex rules. When using the or operator, the expression evaluates two conditions and fires if either is true.

One use for the or operator is to check whether a metric falls outside of a certain range. The following alert expression is triggered when either of the following conditions is true:

• There are greater than 20 active ECP connections in the production cluster.

• There is less than one active ECP connection in the production cluster.

iris_ecp_conn{cluster="production"}<1 or iris_ecp_conn{cluster="production"}>20

Example 4: Logical AND Operator

PromQL also supports the and operator. When using the and operator, the expression evaluates two conditions and fires if both are true.

The following example shows an alert rule that fires when both conditions are true:

• There are unread alerts in the test cluster.

• The system health state of an instance in the test cluster is something other than 0.

iris_system_alerts_new{cluster="test"}>=1 and iris_system_monitor_health_state{cluster="test"}!=0