Caché Monitoring Guide
Monitoring Caché Using SNMP
[Back] [Next]
   
Server:docs1
Instance:LATEST
User:UnknownUser
 
-
Search:    

This document describes the interface between Caché and SNMP (Simple Network Management Protocol). SNMP is a communication protocol that has gained widespread acceptance as a method of managing TCP/IP networks, including individual network devices, and computer devices in general. Its popularity has expanded its use as the underlying structure and protocol for many enterprise management tools. This is its main importance to Caché: a standard way to provide management and monitoring information to a wide variety of management tools.

SNMP is both a standard message format and a standard set of definitions for managed objects. It also provides a standard structure for adding custom-managed objects, a feature that Caché uses to define its management information for use by other applications.
The interface description includes the following topics:
Note:
For a detailed practical look at configuring SNMP to work with Caché by an InterSystems senior technology architect, see InterSystems Data Platforms and performance - Part 5 Monitoring with SNMP on InterSystems Developer Community.
Using SNMP with Caché
SNMP defines a client/server relationship where the client (a network management application) connects to a server program (called the SNMP agent) which executes on a remote network device or a computer system. The client requests and receives information from that agent. There are four basic types of SNMP messages:
The SNMP MIB (Management Information Base) contains definitions of the managed objects. Each device publishes a file, also referred to as its MIB, which defines which portion of the standard MIB it supports, along with any custom definitions of managed objects. For Caché, this is the ISC-CACHE.mib file, located in the install-dir\SNMP directory. In addition, if you are using Ensemble, refer to the ISC-ENSEMBLE.mib file in the same directory.
Caché as a Subagent
The SNMP client connects to the SNMP agent which is listening on a well-known address, port 161. Since the client expects to connect on this particular port, there can only be one SNMP agent on a computer system. To allow access to multiple applications on the system, developers can implement master agents, which may be extended or connected to multiple subagents. InterSystems has implemented the Caché SNMP interface as a subagent, designed to communicate through an SNMP master agent.
Most operating systems that Caché supports provide an SNMP master agent which is extensible in some way to support multiple subagents. Many of these agents, however, implement their extensibility in a proprietary and incompatible manner. Caché implements its subagent using the Agent Extensibility (AgentX) protocol, an IETF-proposed standard as described in RFC 2741.
Some of the standard SNMP master agents support AgentX (most notably OpenVMS). If the SNMP master agent supplied by an operating system is not AgentX-compatible, you can replace it with the public domain Net-SNMP agent.
Note:
The exception is the Windows standard agent which does not support AgentX and for which the Net-SNMP version may not be adequate. For this exception, InterSystems supplies a Windows extension agent DLL, iscsnmp.dll, which handles the connection between the standard Windows SNMP service extension API and the Caché AgentX server.
Managing SNMP in Caché
Since SNMP is a standard protocol, the management of the Caché subagent is minimal. The most important task is to verify that the SNMP master agent on the system is compatible with the Agent Extensibility (AgentX) protocol (see Caché as a Subagent) and it is active and listening for connections on the standard AgentX TCP port 705. On Windows systems, Caché automatically installs a DLL to connect with the standard Windows SNMP service. Verify that the Windows SNMP service is installed and started either automatically or manually.
Important:
Some SNMP master agents, notably Net-SNMP on Linux, do not enable AgentX by default and do not use TCP port 705 by default once they are enabled. For Net-SNMP you must modify the snmpd.conf file to enable communications with the Caché subagent. Recent versions of Net-SNMP also implement VACM (View-based Access Control Model) security and, by default, only allow access to the mib-2.system subtree; the Caché subagent starts and runs without error, but no SNMP requests are forwarded to Caché. You must expand the “views” defined in snmpd.conf to include the Caché MIB subtree.
Next, enable the Caché monitoring service using the following steps:
  1. Navigate to the [Home] > [Security Management] > [Services] page of the Management Portal.
  2. Click the %Service_Monitor service.
  3. Select the Service enabled check box and click Save.
  4. Return to the list of services page and ensure that the %Service_Monitor service is enabled.
Finally, configure the Caché SNMP subagent to start automatically at Caché startup using the following steps:
  1. Navigate to the [Home] > [Configuration] > [Monitor Settings] page of the Management Portal.
  2. Select Yes for the Start SNMP Agent at System Startup setting and click Save.
  3. When you edit this setting, the Caché end of the SNMP interface immediately stops and starts.
You can also start and stop the Caché SNMP subagent manually or programmatically using the ^SNMP routine:
Do start^SNMP(<port>,<timeout>)
Do stop^SNMP
where <port> is the TCP port for the connection (default is 705) and <timeout> is the TCP port read timeout value (default is 20 seconds). Until the <timeout> value is reached, Caché logs any problems encountered while establishing a connection or answering requests in the SNMP.LOG file in the install-dir\Mgr directory.
Note:
When the SNMP master agent is restarted, it may be necessary to manually restart the Caché SNMP subagent using the ^SNMP routine, as described in the foregoing.
SNMP Troubleshooting
The Caché subagent (running the ^SNMP routine) depends on the correct installation and configuration of the SNMP master agent supplied by the operating system. As noted in Caché as a Subagent, there are two main ways in which the ^SNMP routine communicates with this master agent:
Detailed instructions for configuring the SNMP agent should be supplied with the operating system, and system managers should take some time to understand how to do this. The following are some basic guidelines and tips for troubleshooting if problems are encountered in getting Caché to communicate with the SNMP agent.
General
On all systems:
Windows
On Windows systems:
UNIX®
Many UNIX operating systems (HP-UX, IBM AIX®, and Oracle Solaris) do not support the AgentX protocol at this time. If your system does not support AgentX, you must install a separate SNMP agent which supports AgentX, such as Net-SNMP.
Note:
On Oracle Solaris, the System Management Agent (SMA), although a version of NET-SNMP, is not compatible with the Cache AgentX implementation. You will therefore need to disable the SMA agent (and possibly the older snmpdx agent as well) and install a standard version of NET-SNMP to support AgentX.
Linux/Mac OS X and Net-SNMP
On Linux and Mac OS X systems:
Caché MIB Structure
All of the managed object data available through the Caché SNMP interface is defined in the Caché MIB file, ISC-CACHE.mib, which is located in the install-dir\SNMP directory. Typically an SNMP management application must load the MIB file for the managed application to understand and appropriately display the information. Since this procedure varies among applications, consult your management application documentation for the appropriate way to load the Caché MIB.
The specific data defined in the Caché MIB is documented in the file itself and, therefore, is not repeated here. However, it may be valuable to understand the overall structure of the Caché MIB tree, especially as it relates to multiple instances on the same system.
Note:
The best way to view the MIB tree is to load the MIB into a management application or MIB browser. These tools display the MIB as a tree with object IDs (OIDs), matching text representations of the objects, and descriptions of the objects.
SNMP defines a specific, hierarchical tree structure for all managed objects called the Structure of Management Information (SMI), which is detailed in RFC 1155. Each managed object is named by a unique object identifier (OID), which is written as a sequence of integers separated by periods, for example: 1.3.6.1.2.1.1.1. The MIB translates this dotted integer identifier into a text name.
The standard SNMP MIB defines many standard managed objects. To define application-specific extensions to the standard MIB, as Caché does, an application uses the enterprise branch which is defined as:
iso.org.dod.internet.private.enterprises (1.3.6.1.4.1)
The Internet Assigned Numbers Authority (IANA) assigns each organization a private enterprise number as the next level in the hierarchy. For Caché this is 16563 which represents intersystems.
Below this, Caché implements its enterprise private subtree as follows:
For example, encode the size of a database as:
1.3.6.1.4.1.16563.1.1.3.1.6.4.84.69.83.84.1
This translates to:
iso.org.dod.internet.private.enterprises.intersystems.iscCache.cacheObjects
.cacheDBTab.cacheDBRow.cacheDBSize.TEST(instname).1(DBindex)
Ensemble MIB Structure
In addition to the Caché managed object data defined in ISC-CACHE.mib, Ensemble also has the information defined in ISC-ENSEMBLE.mib available. Ensemble uses the same IANA private enterprise number as Caché, 16563 which represents intersystems.
Below this, Ensemble implements its enterprise private subtree as follows:
Extending the Caché MIB
Application programmers can add managed object definitions and extend the MIB for which the Caché subagent provides data. This is not intended to be a complete MIB editor or SNMP toolkit; rather, it is a way to add simple application metrics that you can browse or query through SNMP.
Note:
The objects must follow the basic Caché SNMP structure, there is limited support for SNMP table structures (only integer-valued indexes are supported), and SNMP traps are not created (see the %Monitor.Alert class). A basic understanding of SNMP structure of management information is helpful.
To create these objects do the following:
  1. Create Caché object definitions in classes that inherit from the %Monitor.Adaptor class. See the InterSystems Class Reference for details about adding managed objects to the %Monitor package.
  2. Execute an SNMP class method to enable these managed objects in SNMP and create a MIB definition file for management applications to use. The method to accomplish this is MonitorTools.SNMP.CreateMIB().
See the MonitorTools.SNMP class documentation in the InterSystems Class Reference for details of the CreateMIB() method parameters.
The method creates a branch of the private enterprise MIB tree for a specific application defined in the %Monitor database. In addition to creating the actual MIB file for the application, the method also creates an internal outline of the MIB tree. The Caché subagent uses this to register the MIB subtree, walk the tree for GETNEXT requests, and reference specific objects methods for gathering the instance data in GET requests.
All the managed object definitions use the same general organization as the Caché enterprise MIB tree, that is: application.objects.table.row.item.indices. The first index for all tables is the Caché application ID. All applications must register with the IANA to obtain their own private enterprise number, which is one of the parameters in the CreateMIB() method.
To disable the application in SNMP, use the MonitorTools.SNMP.DeleteMIB() method. This deletes the internal outline of the application MIB, so the Caché subagent no longer registers or answers requests for that private enterprise MIB subtree.
For an example of defining a user Monitor class, see Sample User-defined SNMP Monitor Class in this chapter.
Caché SNMP Traps
In addition to the object data and metrics available through SNMP queries, Caché can send asynchronous alerts or SNMP traps. The following table describes the Caché-specific SNMP traps.
Caché SNMP Notification Objects (Traps)
Trap Name (Number) Description
cacheStart (1) The Caché instance has been started.
cacheStop (2) The Caché instance is in the process of shutting down.
cacheDBExpand (3) A Caché database has expanded successfully.
cacheDBOutOfSpace (4) Future expansion of a Caché database may be limited; there is not enough free space on the file system for 10 more expansions or there is less than 50 MB of free space.
cacheDBStatusChange (5) The read/write status of a Caché database has been changed.
cacheDBWriteFail (6) A write to a Caché database has failed. It includes the Caché error code for the failed write.
cacheWDStop (7) The Write Daemon for a Caché instance has stalled.
cacheWDPanic (8) The Write Daemon for a Caché instance has entered “panic” mode; that is, the Write Daemon is out of buffers and must write database blocks directly to disk without first committing them to the Write Image Journal (WIJ) file.
cacheLockTableFull (9) The lock table for a Caché instance is full, which causes subsequent Locks to fail.
cacheProcessFail (10) A process has exited Caché abnormally (due to an access violation). For detailed information, see the cconsole.log file.
cacheECPTroubleDSrv (11) A connection to this ECP Data Server for a Caché database has encountered a serious communication problem. For detailed information, see the cconsole.log file.
cacheECPTroubleASrv (12) A connection from this ECP Application Server to a remote Caché database has encountered a serious communication problem. For detailed information, see the cconsole.log file.
cacheAuditLost (13) Caché has failed to record an Audit event. The most likely cause is a problem with space for the Audit database, which requires operator assistance.
cacheDaemonFail (14) A major Caché system process (daemon) has died because it encountered an unhandled error.
cacheLoggedError (15) A “severe ” error has been logged in the cconsole.log file. This trap includes the error message defined in cacheSysErrorMsg.
cacheLicenseExceed (16) A request for a license has exceeded the number of licenses currently available or allowed.
cacheAppAlert (100) This is a generic trap that can be used by Caché applications to generate alerts via SNMP. For detailed information about how this trap can be used, see the %Monitor.Alert.External class method.
The following table describes the Caché-specific auxiliary objects that can be sent in the traps described in the preceding table.
Caché-specific Auxiliary Objects Sent in Traps
Auxiliary Object Name (Number) Description
cacheDBWriteError (1) The Caché-specific error code for a failed database write. Possible values are: <DATABASE>, <DISKHARD>, <BLOCKNUMBER>, <FILEFULL> or <DATABASE MAP LABEL>.
cacheApp (2) A short text string (maximum of 20 characters) that identifies the application that generated (or was the source of) a cacheAppAlert trap.
cacheAppSeverity (3) A code that indicates the severity of the problem for a cacheAppAlert trap. The code can be 0 (info), 1 (warning), 2 (severe), or 3 (fatal)
cacheApptext (4) A text string description (maximum of 1024 characters) of the problem, error, or event that caused the cacheAppAlert trap.
The following table describes the Ensemble-specific SNMP traps.
Ensemble SNMP Notification Objects (Traps)
Trap Name (Number) Description
ensEvent (1) The Caché implementation of SNMP signals an Ensemble_LogEvent each time an Ensemble business host posts an alert to the Ensemble Event Log.
Sample User-defined SNMP Monitor Class
This section describes an example of how to define a user Application Monitor class (see Caché Application Monitor in the “Using Caché System Monitor” chapter of this guide) that you can query via SNMP. The Application Monitor includes only properties with %Monitor data types in the SNMP data.
Example Sample Class
The following is the sample class for this example:
Class SNMP.Example Extends %Monitor.Adaptor
{

  /// Give the application a name. This allows you to group different 
  /// classes together under the same application level in the SNMP MIB.
  /// The default is the same as the Package name.
  Parameter APPLICATION = "MyApp";

  /// This groups a set of properties together at the "table" level of the
  /// SNMP MIB hierarchy. The default is the Class name.
  Parameter GROUPNAME = "MyTable";

  /// An integer metric counter
  Property Counter1 As %Monitor.Integer;

  /// Another integer metric counter
  Property Counter2 As %Monitor.Integer;

  /// A status indicator as a string data type
  Property Status As %Monitor.String;

  /// The method is REQUIRED. It is where the Application Monitor
  /// calls to collect data samples, which then get picked up by the
  /// ^SNMP server process when requested.
  Method GetSample() As %Status
  {
    set ..Counter1=$r(200)
    set ..Counter2=200+$r(100)
    set n=$r(4)
    set ..Status=$s(n=1:"Crashed",n=2:"Warning",n=3:"Error",1:"Normal")
    Quit $$$OK
  }

}
Before compiling this class in a user namespace, Caché must load supporting classes into the namespace; these classes are required to store the data samples for SNMP.
To load the classes, run ^%SYSMONMGR, as described in Using ^%SYSMONMGR to Manage Application Monitor in the “Using Caché System Monitor”chapter of this guide, and do the following:
When you compile the sample class, it creates the SNMP.Sample.Example class to store the sample data.
Important:
Do not delete generated sample classes explicitly; if you select both the Application Monitor and generated sample classes for deletion, the sample class routines remain although the monitor class routines are deleted, which causes an error. To ensure that all sample class routines are properly removed, delete only the Application Monitor class that generated it; when you delete the monitor class both the monitor class and generated sample class, as well as related routines for both classes, are deleted. For example, to delete a sample class (for example, SNMP.Sample.Example), use the Management Portal or Studio to delete the monitor class from which it is generated (that is, SNMP.Example).
Run ^%SYSMONMGR to activate the sample class and start the Application Monitor to collect samples:
  1. Select option 2, Manage Monitor Classes.
  2. To see an numbered list of registered Monitor Classes, enter ?.
  3. Enter the number of Monitor Class you want to activate; for example, to activate a user-defined class named SNMP.Example, enter the number next to the class name.
  4. Select option 6, Exit (to return to the Application Monitor main menu).
  5. Select option 1, Manage Application Monitor.
  6. Select option 1, Start Application Monitor.
  7. Select option 5, Exit (to return to the Application Monitor main menu).
  8. Select option 6, Exit (to exit from Application Monitor main menu).
Note:
For information about configuring and using Caché Application Monitor, see Caché Application Monitor in the “Using the Caché System Monitor” chapter of this guide.
Example of Creating a User MIB
To create the SNMP MIB, run the MonitorTools.SNMP:CreateMIB method from the %SYS namespace. See the MonitorTools.SNMP class documentation for details.
The input parameters for the method are similar to the following:
CreateMIB("MyApp","USER",99990,1,"mycorp","myapp","mc","MC-MYAPP","Unknown",1)
Important:
Do not use 99990 as the Enterprise ID for production; each organization should register with the IANA for their own ID.
USER>zn "%SYS"
 
%SYS>Do ##class(MonitorTools.SNMP).CreateMIB("MyApp","USER",99990,1,"mycorp",
"myapp","mc","MC-MYAPP","Unknown",1)
 
 
Create SNMP structure for Application - MyApp
 
     Group - MyTable
          Counter1 = Integer
          Counter2 = Integer
          Status = String
 
 
Create MIB file for MyApp
 
     Generate table MyTable
          Add object Counter1
          Add object Counter2
          Add object Status
 
 
%SYS>
This creates the MC-MYAPP.MIB file in your default directory (install-dir\Mgr\User), which you can load into your SNMP management application.
Note:
You may need to restart the SNMP master agent and the Caché ^SNMP service on your system before each recognizes this MIB.
--
-- MIB file generated for mcMyApp product.
--
-- Sep 16, 2008
--

MC-MYAPP DEFINITIONS ::= BEGIN

IMPORTS

  MODULE-IDENTITY, OBJECT-TYPE, NOTIFICATION-TYPE,
  Counter32, Gauge32, Integer32
  FROM SNMPv2-SMI
  DisplayString
  FROM SNMPv2-TC
  enterprises
  FROM RFC1155-SMI
  cacheSysIndex
  FROM ISC-CACHE;


mcMyApp MODULE-IDENTITY
    LAST-UPDATED "200809161700Z"
    ORGANIZATION "mycorp"
    CONTACT-INFO "
                    Unknown"
    DESCRIPTION ""
    ::= { mycorp 1 }

mycorp OBJECT IDENTIFIER ::= { enterprises 16563 }

myappObjects OBJECT IDENTIFIER ::= { mcMyApp 1 }

--
-- Begin tables
--

-- Table myappMyTable

myappMyTable         OBJECT-TYPE
       SYNTAX        SEQUENCE OF myappMyTableR
       MAX-ACCESS    not-accessible
       STATUS        current
       DESCRIPTION
       ""
   ::= { myappObjects 1 }

myappMyTableR        OBJECT-TYPE
       SYNTAX        myappMyTableR
       MAX-ACCESS    not-accessible
       STATUS        current
       DESCRIPTION
       "Conceptual row for MyTable table."
       INDEX  { cacheSysIndex }
   ::= { myappMyTable 1 }

myappMyTableR ::=
       SEQUENCE {
        myappCounter1   Integer32
        myappCounter2   Integer32
        myappStatus   DisplayString
       }

myappCounter1        OBJECT-TYPE
       SYNTAX        Integer32
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       ""
       ::= { myappMyTableR 1 }

myappCounter2        OBJECT-TYPE
       SYNTAX        Integer32
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       ""
       ::= { myappMyTableR 2 }

myappStatus          OBJECT-TYPE
       SYNTAX        DisplayString
       MAX-ACCESS    read-only
       STATUS        current
       DESCRIPTION
       "Status"
       ::= { myappMyTableR 3 }

--  End of MyTable table

myappTraps  OBJECT IDENTIFIER ::= { mcMyApp 2 }

-------------------------------------------------------------
 END