Monitoring Caché Using the Management Portal
You can monitor many aspects of your Caché instance starting at the System Dashboard of the Management Portal. From the dashboard you can view performance indicators and then, for selected indicators, navigate to more detailed information. This chapter describes the following monitoring tasks:
See Caché System Monitoring Tools in the “Using Caché Monitor” chapter of this guide for an overview of general Caché instance monitoring tools.
Monitoring System Dashboard Indicators
The System Operation > System Dashboard page of the Management Portal groups the status of key system performance indicators into the following categories. Each category is described in one of the tables that follow.
In most cases, you can click an indicator listed in one of these categories to display a description of the indicator in the bottom detail box at the lower left corner of the page.
|Globals/Second||Most recently measured number of global references per second.|
|Global Refs||Number of global references since system startup.|
|Global Sets||Number of global Set and Kill operations since system startup.|
|Routine Refs||Number of routine loads and saves since system startup.|
|Logical Requests||Number of logical block requests since system startup.|
|Disk Reads||Number of physical block read operations since system startup.|
|Disk Writes||Number of physical block write operations since system startup.|
|Cache Efficiency||Most recently measured cache efficiency (Global references / (physical reads + writes)).|
Click the ... more details link in the bottom detail box to display the System Operation > System Usage page. See the Monitoring System Performance section for details.
|Application Servers||Summary status of ECP (Enterprise Cache Protocol) application servers connected to this system.|
|Application Server Traffic||Most recently measured ECP application server traffic in bytes per second.|
|Data Servers||Summary status of ECP data servers to which this system is connected.|
|Data Server Traffic||Most recently measured ECP data server traffic in bytes per second.|
|Shadow Source||Summary status of shadow connections on this data source.|
|Shadow Server||Summary status of shadows configured on this shadow server.|
For more information on the first four indicators, the ECP indicators, see the “Configuring Distributed Systems” chapter of the Caché Distributed Data Management Guide.
Click the ... more details link in the bottom detail box for the two shadow indicators to display the System Operation > Shadow Servers > System as Data Source page or the System Operation > Shadow Servers > System as Shadow Server page. For more information on shadowing, see the “Shadowing” chapter of the Caché Data Integrity Guide.
|System Up Time||Elapsed time since this system was started.|
|Last Backup||Date and time of last system backup.|
You can run backups or view the backup history from the System Operation > Backup page. For more information on developing a backup plan, see the “Backup and Restore” chapter of the Caché Data Integrity Guide.
|Database Space||Indicates whether there is a reasonable amount of disk space available for database files. Clicking ... more details displays the System Operation > Databases page.|
|Journal Space||Indicates whether there is a reasonable amount of disk space available for journal files. Clicking ... more details displays the System Operation > Journals page.|
|Journal Entries||Number of entries written to the system journal. Clicking ... more details displays the System Operation > Journals page.|
|Lock Table||Current status of the system Lock Table. Clicking ... more details displays the System Operation > Locks > Manage Locks page.|
|Write Daemon||Current status of the system Write daemon.|
|Transactions||Current status of open local and remote (ECP) transactions. If there are no open transactions, status is Normal; status may also be Warning (if the duration of the longest open local or remote transaction is greater than 10 minutes) and Troubled (if greater than 20 minutes). Clicking ... more details displays the Transactions page.|
|Processes||Most recent number of running processes. Clicking ... more details displays the Processes page (System Operation > Processes).|
|CSP Sessions||Most recent number of CSP sessions. Clicking ... more details displays the CSP Sessions page (System Operation > CSP Sessions).|
|Most Active Processes||Running processes with highest amount of activity (number of commands executed). Clicking ... more details displays the Processes page (System Operation > Processes).|
For more information on any of these topics, click the Help link on the portal page displayed when you click the ... more details link.
|Serious Alerts||Number of serious alerts that have been raised. Clicking ... more details displays the View Console Log page (System Operation > System Logs > Console Log).|
|Application Errors||Number of application errors that have been logged. Clicking ... more details displays the View Application Error Log page (System Operation > System Logs > Application Error Log).|
See the Monitoring Log Files section in this chapter for more details.
|License Limit||Maximum allowed license units for this system.|
|Current License Use||License usage as a percentage of available license units.|
|Highest License Use||Highest license usage as a percentage of available license units.|
Click the ... more details link in the bottom details box to display the System Operation > License Usage page. For more information on licensing, see the “Managing Caché Licenses” chapter of the Caché System Administration Guide.
|Upcoming Tasks||Lists the next five tasks scheduled to run.|
|Task||Name of the upcoming task.|
|Time||Time the task is scheduled to run.|
|Status||Task status—one of: scheduled, completed, running.|
Click the ... more details link in the bottom details box to display the System Operation > Task Manager > Upcoming Tasks page. For details on the Task Manager, see the Using the Task Manager section of the “Managing Caché” chapter of the Caché System Administration Guide.
Monitoring System Performance
System performance metrics are described in the following tables:
System Usage Table
To view the system usage statistics, navigate to the System Operation > System Usage page.
|Global references (all)||Logical count of accesses to globals, including Sets, Kills, $Data, $Order, $Increment, $Query, and global references in expressions.|
|Global update references||Logical count of global references that are Set, Kill, or $Increment operations.|
|Routine calls||Number of calls to a routine.|
|Routine buffer loads and saves||Total number of routine loads and saves as a result of ZLoad, ZSave, and running routines. (In a well-tuned environment, this number increases slowly, since most routine loads are satisfied by the routine cache memory without accessing the disk. Each routine load or save transfers up to 32 KB of data (64 KB for Unicode).)|
|Logical block requests||Number of database blocks read by the global database code. (In a well-tuned environment, many of these reads are satisfied without disk access.)|
|Block reads||Number of physical database blocks read from disk for both global and routine references.|
|Block writes||Number of physical database blocks written to disk for both global and routine references.|
|WIJ writes||Number of blocks written to the write image journal file.|
|Journal entries||Number of journal records created—one for each database modification (Set , Kill, etc.) or transaction event (TStart, TCommit) or other event that is saved to the journal.|
|Journal block writes||Number of 64-KB journal blocks written to the journal file.|
|Routine lines||Number of routine lines executed since system startup.|
|Last update||Date and time stamp of the displayed statistics.|
See the “Gathering Global Activity Statistics with ^GLOSTAT” chapter for an alternative method of monitoring these statistics.
Generic (Shared) Memory Heap Usage
To view the Caché’s generic memory heap (gmheap) usage, referred to on this page as shared memory heap (SMH) usage, navigate to the System Operation > System Usage page, and click the Shared Memory Heap Usage link.
To change the size of the generic memory heap or gmheap (sometimes known as the shared memory heap or SMH), navigate to the Advanced Memory Setting page (System Administration > Configuration > Additional Settings > Advanced Memory); see Advanced Memory Settings in the “Caché Additional Configuration Settings” chapter of the Caché Additional Configuration Settings Reference for more information.
The column headings in the table on this page refer to the following:
Description — Purpose for which generic memory is allocated.
Allocated SMH/ST — Total generic (SMH) and string table memory allocated to the purpose.
SMH/ST Available — Generic (SMH) and string table memory allocated to the purpose that is still available.
SMH/ST Used — Generic (SMH) and string table memory allocated to the purpose that is in use.
SMT Used — Static memory table memory in use by the purpose.
GST Used — General string table memory in use by the purpose.
All Used — Total combined memory in use by the purpose.
|Miscellaneous||Shared memory allocated for the static memory table (SMT) and general string table (GST).|
|Audit System||Shared memory used for system auditing.|
|Classes Instantiated||Shared memory allocated/available/used for the class hash table and control blocks.|
|Database Encryption Key Change||Shared memory allocated/available/used for database encryption key changes.|
|Semaphore Objects||Shared memory allocated/available/used for semaphore objects.|
|Event System||Shared memory allocated/available/used for the event system.|
|Global Mapping||Shared memory allocated/available/used for global mapping and subscript-level mapping (SLM).|
|License Upgrade||Shared memory allocated/available/used for license upgrades.|
|Lock Table||Shared memory allocated/available/used for the lock system.|
|National Language Support||Shared memory allocated/available/used for National Language Support (NLS) tables.|
|Performance Monitor||Shared memory allocated/available/used for the Caché Performance Monitor (^PERFMON).|
|Process Table||Shared memory allocated/available/used for the Process ID (PID) table.|
|Routine Buffer in Use Table||Shared memory allocated/available/used for routine buffer-in-use tables.|
|Security System||Shared memory allocated/available/used for the security system.|
|Shadowing||Shared memory allocated/available/used for shadowing.|
|Shared Library||Shared memory allocated/available/used for shared libraries.|
|TTY Hash Table||Shared memory allocated/available/used for TTY hash tables.|
|DB Name & Directory||Shared memory allocated/available/used for database names and directories.|
|iKnow Language Model Data||Shared memory allocated/available/used for iKnow language models.|
|ECP||Shared memory allocated/available/used for ECP.|
|Expand Daemon||Shared memory allocated/available/used for expanding daemons.|
Total memory for each column.
Hover over the column headings for a description of each column.
|Available SMT & GST||Available memory in the static memory table (SMT) and general string table (GST).|
|Total SMT & GST Allocated||Total used and available memory in the static memory table (SMT) and general string table (GST).|
|Total SMH Pages Allocated||Total directly allocated shared memory heap (SMH) and string table allocated Shared memory, together with the total used/available memory in the static memory table (SMT) and the general string table (GST); the number of 64-KB pages is displayed parenthetically.|
Caché locks are created when a Caché process issues a LOCK command on a Caché local variable or global variable, as long as the entity is not already locked by another process. Entities need not exist in the database to lock them. The article Locking and Concurrency Control discusses Caché locks in detail.
To display locks system-wide, navigate to the System Operation > Locks > View Locks page. To delete selected locks system-wide, navigate to the System Operation > Locks > Manage Locks page. In both cases, the displayed lock table lists one row for each held lock and for each waiting lock request, identifying the owner. A single row may identify multiple locks held by an owner on the same entity. For example, holding an incremented lock or holding both a Shared lock and an Exclusive lock. If more than one process holds a lock on the same entity, each owner has its own row.
The Lock Table has the following column entries.
|Owner||The process ID of the process holding or waiting for the lock. Contains the client system name if it is a remote lock.|
|ModeCount||Lock mode and lock increment count. If the lock count is 1 the count is not displayed. For a list of ModeCount values, refer to the “Lock Management” chapter of Using Caché ObjectScript.|
|Reference||Lock reference string of the lock item (does not include the database name).|
|Directory||The database location of the lock item.|
|System||The system name of where the lock is located, if it is the local system the column is blank.|
|Routine||The routine line currently being executed by the process holding or waiting for the lock.|
|Remove||Manage Locks only: If this lock is removable, this option along with the Remove all locks for process option (for local locks) or the Remove all locks from remote client option (for remote locks) appears in the row. Click the appropriate option to remove the lock. remove all locks for the process, or remove all locks from the remote client. If a lock you are removing is part of an open transaction, you are warned before confirming the removal.|
In most cases, the only time you need to remove locks is as a result of an application problem.
For a more in-depth description of the LOCK command and its features, see the LOCK entry of the Caché ObjectScript Reference.
You may need to enlarge the size of the lock table if your system uses a large number of locks. You can do this using the Management Portal:
Navigate to the System Administration > Configuration > Additional Settings > Advanced Memory page.
In the locksiz (locksiz) row, click Edit.
In the locksiz box, update the amount of memory allocated on your system for locks (in bytes), and click OK.
The minimum is 65536; the maximum value depends on the value of gmheap (Generic Memory Heap Size). Increase the heap size if you need more room for the lock table. Caché rounds up the value to the next multiple of 64 KB. The default range is from 65536 – 1769472.
Click Save and restart Caché for this information to take effect.
For more detailed information and alternative ways to manage locks, see the “Lock Management” chapter of Using Caché ObjectScript.
Monitoring Caché Logs
Caché provides the following logs for monitoring various aspects of its operation:
Several log files are available in the system manager directory; two can be viewed using the management portal.
You can view the application error log or ODBC error log using the management portal
The contents of the Caché system error log, or syslog, can be reviewed using one of several methods.
System Manager Directory Log Files
The following log files are available in the system manager directory, typically install-dir\mgr. The console log and System Monitor log can be viewed using the management portal.
Caché reports general messages, startup/shutdown, license, and network errors, certain operating system errors, and the success or failure of jobs started remotely from other systems through an operator console facility, which writes them to the console log, install-dir\mgr\cconsole.log by default. Caché System Monitor also writes notifications to the console log.
On Windows-based platforms, all console messages are sent to the console log file, the name of which is configurable. On UNIX®/Linux platforms, you can configure console messages to be sent to the console log file, the console terminal, or both. See the console parameter in the Caché Parameter File Reference and Advanced Memory Settings in the Caché Additional Configuration Setting Reference for information about console log configuration.
The size of the cconsole.log file is monitored by System Monitor. The file grows until it reaches the configured maximum size (default 5 MB), at which point it is renamed to console.log.old_yyyymmdd, any existing console.log.old_yyyymmdd file is deleted, and a new console.log is created. The maximum number of megabytes used by the console log is therefore twice the configured maximum. To configure the maximum console log size, navigate to the System Administration > Configuration > Additional Settings > Startup page of the Management Portal and update the MaxConsoleLogSize setting.
If the console log or System Monitor log is larger than 1 MB, only the most recent 1 MB portion is displayed by the Management Portal. Click the Show entire file link to display the entire file, which may require some time if the file is very large.
If you have trouble starting Caché, use any text editor or text viewer to view the console log.
Caché Monitor scans the console log at regular intervals for entries of the configured minimum severity and generates corresponding notifications, which it writes to the alerts log, install-dir\mgr\alerts.log, by default. Caché Monitor can be configured to send email notifications instead; see the “Using Caché Monitor” chapter of this guide for more information.
Status messages about the functioning of Caché System Monitor (see the chapter “Using Caché System Monitor” in this guide) are written to the System Monitor log, install-dir\mgr\SystemMonitor.log.
The size of the SystemMonitor.log file is monitored by System Monitor. The file grows until it reaches the maximum size of 5 MB, at which point it is renamed to SystemMonitor.log.old, overwriting any existing SystemMonitor.log.old file, and a new SystemMonitor.log is created. The maximum number of megabytes used by the System Monitor log is therefore 10 MB.
If the System Monitor log is larger than 1 MB, only the most recent 1 MB portion is displayed by the Management Portal. Click the Show entire file link to display the entire file, which may require some time if the file is very large.
The initialization log, cboot.log, contains information about the initialization of the Caché instance.
The journal history log, journal.log, contains a list of all journal files maintained by the Caché instance and is used by all journal-related functions, utilities, and APIs to locate journal files. See the “Journaling” chapter of the Caché Data Integrity Guide for information about journaling.
These .log files are plain text files and can be viewed using any text editor or viewer.
To view the console log or the System Monitor log using the Management Portal, select System Operation > System Logs > Console Log or System Operation > System Logs > System Monitor Log.
Application and ODBC Error Logs
To view the application error log, select System Operation > System Logs > Application Error Log.
To view the xDBC error log, select System Operation > System Logs > xDBC Error Log.
Caché System Error Log
Caché sets aside a small portion of its shared memory to log items of interest. This table, which can contain important diagnostic information, is referred to by several different names, including the Caché system error log, errlog, SYSLOG, and the syslog table.
By default, the system error log contains the 500 most recent log items. For information about configuring the number of items in the system error log, see errlog in the “Advanced Memory Settings” section of the Caché Additional Configuration Settings Reference.
The view the system error log, choose one of the following methods:
Open Terminal, enter set $namespace = %SYS" to switch to the %SYS namespace, and enter do ^SYSLOG. You can also enter do FILTER^SYSLOG, which has options to limit the output based on specific error codes or process ID.
Run a diagnostic report, as described in the “Using the Caché Diagnostic Report” chapter of this guide.
Run the cstat command with the –e1 option, as described in Running cstat with Options in the “Monitoring Caché Using the cstat Utility” appendix of this guide.
Run the CacheHung script, as described in CacheHung Script in the “Monitoring Caché Using the cstat Utility” appendix.
Configure Caché to write the system error log to the console log during shutdown by taking the following steps; you can then review the console log as previously described.
Go to the Compatibility Settings page (System Administration -> Configuration -> Additional Settings -> Compatibility).
In the ShutDownLogErrors row, select edit.
On the Edit ShutDownLogErrors page, select the ShutDownLogErrors check box and click Save.
For a guide to using the Caché system error log, including sample output and a detailed description of the output fields, see SYSLOG - what it really is and what it means by a member of the InterSystems Product Specialist group on InterSystems Developer Community.