Skip to main content

Monitoring Caché Using SNMP

This document describes the interface between Caché and SNMP (Simple Network Management Protocol). SNMP is a communication protocol that has gained widespread acceptance as a method of managing TCP/IP networks, including individual network devices, and computer devices in general. Its popularity has expanded its use as the underlying structure and protocol for many enterprise management tools. This is its main importance to Caché: a standard way to provide management and monitoring information to a wide variety of management tools.

SNMP is both a standard message format and a standard set of definitions for managed objects. It also provides a standard structure for adding custom-managed objects, a feature that Caché uses to define its management information for use by other applications.

The interface description includes the following topics:

Note:

For a detailed practical look at configuring SNMP to work with Caché by an InterSystems senior technology architect, see InterSystems Data Platforms and performance - Part 5 Monitoring with SNMPOpens in a new tab on InterSystems Developer Community.

Using SNMP with Caché

SNMP defines a client/server relationship where the client (a network management application) connects to a server program (called the SNMP agent) which executes on a remote network device or a computer system. The client requests and receives information from that agent. There are four basic types of SNMP messages:

  • GET – fetch the data for a specific managed object

  • GETNEXT – get data for the next managed object in a hierarchical tree, allowing system managers to walk through all the data for a device

  • SET – set the value for a specific managed object

  • TRAP – an asynchronous alert sent by the managed device or system

The SNMP MIB (Management Information Base) contains definitions of the managed objects. Each device publishes a file, also referred to as its MIB, which defines which portion of the standard MIB it supports, along with any custom definitions of managed objects. For Caché, this is the ISC-CACHE.mib file, located in the install-dir\SNMP directory. In addition, if you are using Ensemble, refer to the ISC-ENSEMBLE.mib file in the same directory.

Caché as a Subagent

The SNMP client connects to the SNMP agent which is listening on a well-known address, port 161. Since the client expects to connect on this particular port, there can only be one SNMP agent on a computer system. To allow access to multiple applications on the system, developers can implement master agents, which may be extended or connected to multiple subagents. InterSystems has implemented the Caché SNMP interface as a subagent, designed to communicate through an SNMP master agent.

Most operating systems that Caché supports provide an SNMP master agent which is extensible in some way to support multiple subagents. Many of these agents, however, implement their extensibility in a proprietary and incompatible manner. Caché implements its subagent using the Agent Extensibility (AgentX) protocol, an IETF-proposed standard as described in RFC 2741Opens in a new tab.

Some of the standard SNMP master agents support AgentX. If the SNMP master agent supplied by an operating system is not AgentX-compatible, you can replace it with the public domain Net-SNMP agent.

Note:

The exception is the Windows standard agent which does not support AgentX and for which the Net-SNMP version may not be adequate. For this exception, InterSystems supplies a Windows extension agent DLL, iscsnmp.dll, which handles the connection between the standard Windows SNMP service extension API and the Caché AgentX server.

Managing SNMP in Caché

Since SNMP is a standard protocol, the management of the Caché subagent is minimal. The most important task is to verify that the SNMP master agent on the system is compatible with the Agent Extensibility (AgentX) protocol (see Caché as a Subagent) and it is active and listening for connections on the standard AgentX TCP port 705. On Windows systems, the system automatically installs a DLL to connect with the standard Windows SNMP service. Verify that the Windows SNMP service is installed and started either automatically or manually.

Important:

Some SNMP master agents, notably Net-SNMP on Linux, do not enable AgentX by default and do not use TCP port 705 by default once they are enabled. For Net-SNMP you must modify the snmpd.conf file to enable communications with the Caché subagent. Recent versions of Net-SNMP also implement VACM (View-based Access Control Model) security and, by default, only allow access to the mib-2.system subtree; the Caché subagent starts and runs without error, but no SNMP requests are forwarded to Caché. You must expand the “views” defined in snmpd.conf to include the Caché MIB subtree.

Next, enable the Caché monitoring service using the following steps:

  1. Navigate to System > Security Management > Services in the Management Portal.

  2. Click the %Service_Monitor service.

  3. Select the Service enabled check box and click Save.

  4. Return to the list of services page and ensure that the %Service_Monitor service is enabled.

Finally, configure the Caché SNMP subagent to start automatically at Caché startup using the following steps:

  1. Navigate to System > Configuration > Monitor Settings in the Management Portal.

  2. Select Yes for the Start SNMP Agent at System Startup setting and click Save.

  3. When you edit this setting, the Caché end of the SNMP interface immediately stops and starts.

You can also start and stop the Caché SNMP subagent manually or programmatically using the ^SNMP routine:

Do start^SNMP(<port>,<timeout>)
Do stop^SNMP

where <port> is the TCP port for the connection (default is 705) and <timeout> is the TCP port read timeout value (default is 20 seconds). Until the <timeout> value is reached, Caché logs any problems encountered while establishing a connection or answering requests in the SNMP.LOG file in the install-dir\Mgr directory.

Note:

When the SNMP master agent is restarted, it may be necessary to manually restart the Caché SNMP subagent using the ^SNMP routine, as described in the foregoing.

SNMP Troubleshooting

The Caché subagent (running the ^SNMP routine) depends on the correct installation and configuration of the SNMP master agent supplied by the operating system. As noted in Caché as a Subagent, there are two main ways in which the ^SNMP routine communicates with this master agent:

  • Primarily, ^SNMP uses the AgentX protocol on TCP port 705

  • On Windows, ^SNMP uses a Windows extension agent DLL installed as iscsnmp.dll.

Detailed instructions for configuring the SNMP agent should be supplied with the operating system, and system managers should take some time to understand how to do this. The following are some basic guidelines and tips for troubleshooting if problems are encountered in getting Caché to communicate with the SNMP agent.

General

On all systems:

  • Make sure the SNMP agent is working independently of Caché and you can at least query the mib-2.system tree for general system information. If this fails, on Windows check the Windows SNMP Service; on UNIX®/Linux see if the SNMP daemon (snmpd) is running.

  • If you can successfully query the SNMP system information but not the Caché MIB, then check for a background process in Caché running the ^SNMP routine. Try starting it using the $$start^SNMP() function. If the routine starts but does not continue running, check for errors in the cconsole.log and SNMP.log log files in the Caché install-dir/mgr directory. On Windows, iscsnmp.dll logs any errors it encounters in %System%\System32\snmpdbg.log (on a 64–bit Windows system, the file is in the SysWOW64 subdirectory).

  • Make sure the Caché %Service_Monitor service is enabled.

  • More information can be logged to the SNMP.log file if you set ^SYS(“MONITOR”,”SNMP”,”DEBUG”)=1 in the %SYS namespace and restart the ^SNMP Caché subagent process. This logs details about each message received and sent.

Windows

On Windows systems:

  • Not all Windows versions install the Windows SNMP service by default. You may need to do this as an additional step. Make sure the Security tab of the Properties dialog for the service has at least a public community with READ rights. To send SNMP traps, you must define a Community Name and Destination on the Trap tab of the properties dialog.

  • Caché expects the SNMP service to be installed before you install Caché, so it can add iscsnmp.dll to the proper Registry keys. Once Caché is installed, the SNMP service must be restarted so that it properly loads iscsnmp.dll and will find and communicate with the new Caché instance.

    Note:

    If Caché is installed before the SNMP service, iscsnmp.dll cannot be properly registered, and you must use the set myStatus=$$Register^SNMP() function to do this after the Windows SNMP service is installed. Once this is done the SNMP service must be restarted.

  • On Windows, the $$start^SNMP() function only signals the SNMP service, and the Caché ^SNMP process is actually started by a callback from the SNMP service into Cache. It may take a few seconds for the process to start, and a few more seconds before it can respond to queries.

UNIX®

Many UNIX operating systems (such as IBM AIX®) do not support the AgentX protocol at this time. If your system does not support AgentX, you must install a separate SNMP agent which supports AgentX, such as Net-SNMP.

Linux/Mac OS X and Net-SNMP

On Linux and Mac OS X systems:

  • AgentX support is not enabled by default, and the default port is not 705. You must modify the snmpd.conf file and add master agentx and agentXSocket TCP:localhost:705, or use snmpd -x TCP:localhost:705 on the command line.

  • Basic system information like syslocation, syscontact and sysservices must be defined in snmpd.conf to enable successful startup of the snmpd daemon.

  • Recent versions of Net-SNMP also implement VACM (View-based Access Control Model) security and, by default, allow access to the mib-2.system subtree only; as a result, the Caché subagent starts and runs without error, but no SNMP requests are forwarded to Caché. You must expand the “views” defined in snmpd.conf to include the Caché MIB subtree.

  • To send SNMP traps, you must define a destination using the trapsink parameter in snmpd.conf, for example trapsink 192.16.61.36 public.

Caché MIB Structure

All of the managed object data available through the Caché SNMP interface is defined in the Caché MIB file, ISC-CACHE.mib, which is located in the install-dir\SNMP directory. Typically an SNMP management application must load the MIB file for the managed application to understand and appropriately display the information. Since this procedure varies among applications, consult your management application documentation for the appropriate way to load the Caché MIB.

The specific data defined in the Caché MIB is documented in the file itself and, therefore, is not repeated here. However, it may be valuable to understand the overall structure of the Caché MIB tree, especially as it relates to multiple instances on the same system.

Note:

The best way to view the MIB tree is to load the MIB into a management application or MIB browser. These tools display the MIB as a tree with object IDs (OIDs), matching text representations of the objects, and descriptions of the objects.

SNMP defines a specific, hierarchical tree structure for all managed objects called the Structure of Management Information (SMI), which is detailed in RFC 1155Opens in a new tab. Each managed object is named by a unique object identifier (OID), which is written as a sequence of integers separated by periods, for example: 1.3.6.1.2.1.1.1. The MIB translates this dotted integer identifier into a text name.

The standard SNMP MIB defines many standard managed objects. To define application-specific extensions to the standard MIB, as Caché does, an application uses the enterprise branch which is defined as:

iso.org.dod.internet.private.enterprises (1.3.6.1.4.1)

The Internet Assigned Numbers Authority (IANA) assigns each organization a private enterprise number as the next level in the hierarchy. For Caché this is 16563 which represents intersystems.

Below this, Caché implements its enterprise private subtree as follows:

  • The level below intersystems is the “product” or application ID level. For Caché this is .1 (iscCache). This serves as the MIB module identity.

  • The next level is the “object” level, which separates data objects from notifications. For Caché, these are .1 (cacheObjects) and .2 (cacheTraps). By convention, the intersystems tree uses a brief lowercase prefix added to all data objects and notification names. For Caché this is cache.

  • The next level is the “table” or group level. All data objects are organized into tables, even if there is only one instance or “row” to the table. This serves to organize the management data objects into groups. This is also necessary to support multiple Caché instances on one machine. All tables use the Caché instance name as the first index of the table. The tables may also have one or more additional indices.

  • The next level is the “conceptual row” for the table (as required by the SNMP SMI). This is always .1.

  • Finally, the individual data objects contained in that table, including any that are designated as indices.

  • The notifications (traps) are defined as individual entries at the same hierarchical level as the “table”. For more information, see Caché SNMP Traps in this appendix.

  • Caché-specific auxiliary objects sent via notifications (traps) are defined as individual entries at the same hierarchical level as the “table”. For more information, see Caché SNMP Traps in this appendix.

For example, encode the size of a database as:

1.3.6.1.4.1.16563.1.1.3.1.6.4.84.69.83.84.1

This translates to:

iso.org.dod.internet.private.enterprises.intersystems.iscCache.cacheObjects
.cacheDBTab.cacheDBRow.cacheDBSize.TEST(instname).1(DBindex)

Ensemble MIB Structure

In addition to the Caché managed object data defined in ISC-CACHE.mib, Ensemble also has the information defined in ISC-ENSEMBLE.mib available. Ensemble uses the same IANA private enterprise number as Caché, 16563 which represents intersystems.

Below this, Ensemble implements its enterprise private subtree as follows:

  • The level below intersystems is the “product” or application ID level. For Ensemble this is .2 (iscEnsemble). This serves as the MIB module identity.

  • The next level is the “object” level, which separates data objects from notifications. For Ensemble, these are: .1 (ensObjects) and .2 (ensTraps). By convention, the intersystems tree uses a brief lowercase prefix added to all data objects and notification names. For Ensemble this is ens.

  • Subsequent levels follow the same implementation as Caché described in the Caché MIB Structure section.

Extending the Caché MIB

Application programmers can add managed object definitions and extend the MIB for which the Caché subagent provides data. This is not intended to be a complete MIB editor or SNMP toolkit; rather, it is a way to add simple application metrics that you can browse or query through SNMP.

Note:

The objects must follow the basic Caché SNMP structure, there is limited support for SNMP table structures (only integer-valued indexes are supported), and SNMP traps are not created (see the %Monitor.AlertOpens in a new tab class). A basic understanding of SNMP structure of management information is helpful.

To create these objects do the following:

  1. Create Caché object definitions in classes that inherit from the %Monitor.AdaptorOpens in a new tab class. See the InterSystems Class Reference for details about adding managed objects to the %Monitor package.

  2. Execute an SNMP class method to enable these managed objects in SNMP and create a MIB definition file for management applications to use. The method to accomplish this is MonitorTools.SNMP.CreateMIB().

See the MonitorTools.SNMPOpens in a new tab class documentation in the InterSystems Class Reference for details of the CreateMIB() method parameters.

The method creates a branch of the private enterprise MIB tree for a specific application defined in the %Monitor database. In addition to creating the actual MIB file for the application, the method also creates an internal outline of the MIB tree. The Caché subagent uses this to register the MIB subtree, walk the tree for GETNEXT requests, and reference specific objects methods for gathering the instance data in GET requests.

All the managed object definitions use the same general organization as the Caché enterprise MIB tree, that is: application.objects.table.row.item.indices. The first index for all tables is the Caché application ID. All applications must register with the IANA to obtain their own private enterprise number, which is one of the parameters in the CreateMIB() method.

To disable the application in SNMP, use the MonitorTools.SNMP.DeleteMIB() method. This deletes the internal outline of the application MIB, so the Caché subagent no longer registers or answers requests for that private enterprise MIB subtree.

For an example of defining a user Monitor class, see Sample User-defined SNMP Monitor Class in this chapter.

Caché SNMP Traps

In addition to the object data and metrics available through SNMP queries, Caché can send asynchronous alerts or SNMP traps. The following table describes the Caché-specific SNMP traps.

Caché SNMP Notification Objects (Traps)
Trap Name (Number) Description
cacheStart (1) The Caché instance has been started.
cacheStop (2) The Caché instance is in the process of shutting down.
cacheDBExpand (3) A Caché database has expanded successfully.
cacheDBOutOfSpace (4) Future expansion of a Caché database may be limited; there is not enough free space on the file system for 10 more expansions or there is less than 50 MB of free space.
cacheDBStatusChange (5) The read/write status of a Caché database has been changed.
cacheDBWriteFail (6) A write to a Caché database has failed. It includes the Caché error code for the failed write.
cacheWDStop (7) The Write Daemon for a Caché instance has stalled.
cacheWDPanic (8) The Write Daemon for a Caché instance has entered “panic” mode; that is, the Write Daemon is out of buffers and must write database blocks directly to disk without first committing them to the Write Image Journal (WIJ) file.
cacheLockTableFull (9) The lock table for a Caché instance is full, which causes subsequent Locks to fail.
cacheProcessFail (10) A process has exited Caché abnormally (due to an access violation). For detailed information, see the cconsole.log file.
cacheECPTroubleDSrv (11) A connection to this ECP Data Server for a Caché database has encountered a serious communication problem. For detailed information, see the cconsole.log file.
cacheECPTroubleASrv (12) A connection from this ECP Application Server to a remote Caché database has encountered a serious communication problem. For detailed information, see the cconsole.log file.
cacheAuditLost (13) Caché has failed to record an Audit event. The most likely cause is a problem with space for the Audit database, which requires operator assistance.
cacheDaemonFail (14) A major Caché system process (daemon) has died because it encountered an unhandled error.
cacheLoggedError (15) A “severe ” error has been logged in the cconsole.log file. This trap includes the error message defined in cacheSysErrorMsg.
cacheLicenseExceed (16) A request for a license has exceeded the number of licenses currently available or allowed.
cacheAppAlert (100) This is a generic trap that can be used by Caché applications to generate alerts via SNMP. For detailed information about how this trap can be used, see the %Monitor.Alert.External class method.

The following table describes the Caché-specific auxiliary objects that can be sent in the traps described in the preceding table.

Caché-specific Auxiliary Objects Sent in Traps
Auxiliary Object Name (Number) Description
cacheDBWriteError (1) The Caché-specific error code for a failed database write. Possible values are: <DATABASE>, <DISKHARD>, <BLOCKNUMBER>, <FILEFULL> or <DATABASE MAP LABEL>.
cacheApp (2) A short text string (maximum of 20 characters) that identifies the application that generated (or was the source of) a cacheAppAlert trap.
cacheAppSeverity (3) A code that indicates the severity of the problem for a cacheAppAlert trap. The code can be 0 (info), 1 (warning), 2 (severe), or 3 (fatal)
cacheApptext (4) A text string description (maximum of 1024 characters) of the problem, error, or event that caused the cacheAppAlert trap.

The following table describes the Ensemble-specific SNMP traps.

Ensemble SNMP Notification Objects (Traps)
Trap Name (Number) Description
ensEvent (1) The Caché implementation of SNMP signals an Ensemble_LogEvent each time an Ensemble business host posts an alert to the Ensemble Event Log.

Sample User-defined SNMP Monitor Class

This section describes an example of how to define a user Application Monitor class (see Caché Application Monitor in the “Using Caché System Monitor” chapter of this guide) that you can query via SNMP. The Application Monitor includes only properties with %Monitor data types in the SNMP data.

Example Sample Class

The following is the sample class for this example:

Class SNMP.Example Extends %Monitor.Adaptor
{

  /// Give the application a name. This allows you to group different 
  /// classes together under the same application level in the SNMP MIB.
  /// The default is the same as the Package name.
  Parameter APPLICATION = "MyApp";

  /// This groups a set of properties together at the "table" level of the
  /// SNMP MIB hierarchy. The default is the Class name.
  Parameter GROUPNAME = "MyTable";

  /// An integer metric counter
  Property Counter1 As %Monitor.Integer;

  /// Another integer metric counter
  Property Counter2 As %Monitor.Integer;

  /// A status indicator as a string data type
  Property Status As %Monitor.String;

  /// The method is REQUIRED. It is where the Application Monitor
  /// calls to collect data samples, which then get picked up by the
  /// ^SNMP server process when requested.
  Method GetSample() As %Status
  {
    set ..Counter1=$r(200)
    set ..Counter2=200+$r(100)
    set n=$r(4)
    set ..Status=$s(n=1:"Crashed",n=2:"Warning",n=3:"Error",1:"Normal")
    Quit $$$OK
  }

}

Before compiling this class in a user namespace, Caché must load supporting classes into the namespace; these classes are required to store the data samples for SNMP.

To load the classes, run ^%SYSMONMGR, as described in Using ^%SYSMONMGR to Manage Application Monitor in the “Using Caché System Monitor”chapter of this guide, and do the following:

  • Select option 2, Manage Monitor Classes

  • Select option 3, Register Monitor System Classes.

When you compile the sample class, it creates the SNMP.Sample.Example class to store the sample data.

Important:

Do not delete generated sample classes explicitly; if you select both the Application Monitor and generated sample classes for deletion, the sample class routines remain although the monitor class routines are deleted, which causes an error. To ensure that all sample class routines are properly removed, delete only the Application Monitor class that generated it; when you delete the monitor class both the monitor class and generated sample class, as well as related routines for both classes, are deleted. For example, to delete a sample class (for example, SNMP.Sample.Example), use the Management Portal to delete the monitor class from which it is generated (that is, SNMP.Example).

Run ^%SYSMONMGR to activate the sample class and start the Application Monitor to collect samples:

  1. Select option 2, Manage Monitor Classes.

  2. Select option 1, Activate/Deactivate a Monitor Class.

  3. To see an numbered list of registered Monitor Classes, enter ?.

  4. Enter the number of Monitor Class you want to activate; for example, to activate a user-defined class named SNMP.Example, enter the number next to the class name.

  5. Select option 6, Exit (to return to the Application Monitor main menu).

  6. Select option 1, Manage Application Monitor.

  7. Select option 1, Start Application Monitor.

  8. Select option 5, Exit (to return to the Application Monitor main menu).

  9. Select option 6, Exit (to exit from Application Monitor main menu).

Note:

For information about configuring and using Caché Application Monitor, see Caché Application Monitor in the “Using the Caché System Monitor” chapter of this guide.

Example of Creating a User MIB

To create the SNMP MIB, run the MonitorTools.SNMP:CreateMIB method from the %SYS namespace. See the MonitorTools.SNMPOpens in a new tab class documentation for details.

The input parameters for the method are similar to the following:

CreateMIB("MyApp","USER",99990,1,"mycorp","myapp","mc","MC-MYAPP","Unknown",1)

Important:

Do not use 99990 as the Enterprise ID for production; each organization should register with the IANA for their own ID.

USER>set $namespace = %SYS"
 
%SYS>Do ##class(MonitorTools.SNMP).CreateMIB("MyApp","USER",99990,1,"mycorp",
"myapp","mc","MC-MYAPP","Unknown",1)
 
 
Create SNMP structure for Application - MyApp
 
     Group - MyTable
          Counter1 = Integer
          Counter2 = Integer
          Status = String
 
 
Create MIB file for MyApp
 
     Generate table MyTable
          Add object Counter1
          Add object Counter2
          Add object Status
 
 
%SYS>

This creates the MC-MYAPP.MIB file in your default directory (install-dir\Mgr\User), which you can load into your SNMP management application.

Note:

You may need to restart the SNMP master agent and the Caché ^SNMP service on your system before each recognizes this MIB.

--
-- MIB file generated for mcMyApp product.
--
-- Sep 16, 2008
--

MC-MYAPP DEFINITIONS ::= BEGIN

IMPORTS

  MODULE-IDENTITY, OBJECT-TYPE, NOTIFICATION-TYPE,
  Counter32, Gauge32, Integer32
  FROM SNMPv2-SMI
  DisplayString
  FROM SNMPv2-TC
  enterprises
  FROM RFC1155-SMI
  cacheSysIndex
  FROM ISC-CACHE;


mcMyApp MODULE-IDENTITY
    LAST-UPDATED "200809161700Z"
    ORGANIZATION "mycorp"
    CONTACT-INFO "
                    Unknown"
    DESCRIPTION ""
    ::= { mycorp 1 }

mycorp OBJECT IDENTIFIER ::= { enterprises 16563 }

myappObjects OBJECT IDENTIFIER ::= { mcMyApp 1 }

--
-- Begin tables
--

-- Table myappMyTable

myappMyTable         OBJECT-TYPE
       SYNTAX        SEQUENCE OF myappMyTableR
       MAX-ACCESS    not-accessible
       STATUS        current
       DESCRIPTION
       ""
   ::= { myappObjects 1 }

myappMyTableR        OBJECT-TYPE
       SYNTAX        myappMyTableR
       MAX-ACCESS    not-accessible
       STATUS        current
       DESCRIPTION
       "Conceptual row for MyTable table."
       INDEX  { cacheSysIndex }
   ::= { myappMyTable 1 }

myappMyTableR ::=
       SEQUENCE {
        myappCounter1   Integer32
        myappCounter2   Integer32
        myappStatus   DisplayString
       }

myappCounter1        OBJECT-TYPE
       SYNTAX        Integer32
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       ""
       ::= { myappMyTableR 1 }

myappCounter2        OBJECT-TYPE
       SYNTAX        Integer32
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       ""
       ::= { myappMyTableR 2 }

myappStatus          OBJECT-TYPE
       SYNTAX        DisplayString
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       "Status"
       ::= { myappMyTableR 3 }

--  End of MyTable table

myappTraps  OBJECT IDENTIFIER ::= { mcMyApp 2 }

-------------------------------------------------------------
 END

FeedbackOpens in a new tab