Journaling Overview
Global journaling records all global update operations performed on a database, and used in conjunction with backup makes it possible to restore a database to its state immediately before a failure or crash.
While backup is the cornerstone of physical recovery, it is not the complete answer. Restoring a database from backup does not recover global updates made since that backup, which may have been created a number of hours before the point at which physical integrity was lost. These post-backup updates can be restored to the database from journal files after the database is restored from backup, bringing the database up to date. Any transactions open at the time of the failure are rolled back to ensure transaction integrity.
Introduction to Journaling
Each instance of InterSystems IRIS® data platform keeps a journal, a set of files that maintains a time-sequenced log of updates that have been made to databases since the last backup. The process is redundant and logical and does not use the InterSystems IRIS write daemon. InterSystems IRIS transaction processing works with journaling to maintain the logical integrity of data following a failure.
Together, backup and journaling allow you to recreate your database. If a failure renders your database corrupt, inaccessible or unusable, you can restore the most recent backup and then apply the changes in the journal to recreate your database to the point of failure. This method of recovering from a loss of physical integrity is known as “roll forward” recovery. The journal is also used for rolling back incomplete transactions.
The journaling state is a property of the database, not individual globals. A database can have only one of two global journaling states: Yes or No. By default, all databases you create are journaled (the Global Journal State is Yes). In newly installed InterSystems IRIS instances, the IRISAUDIT, IRISSYS, and USER databases are journaled; the IRISLIB, IRISTEMP, and IRISLOCALDATA databases are not. Operations to globals in IRISTEMP are never journaled; map temporary globals to the InterSystems IRIS temporary database, IRISTEMP.
Be sure to read Consequences of Not Journaling Databases for important information about limits to the recovery of non-journaled databases.
When InterSystems IRIS starts, it reapplies all journal entries since the last write daemon pass. Since user processes update the journal concurrently, rather than through the write daemon, this approach provides added assurance that updates prior to a crash are preserved.
In addition to recording all updates to journaled databases, the journal contains all updates to nonjournaled databases that are part of transactions (primarily Set and Kill operations). This greatly improves the reliability of the system, avoiding post-recovery inconsistencies due to updates to globals that may or may not be journaled, and that may or may not be involved in transactions. (Set and Kill operations on local and process-private variables are not journaled.)
Journaling global operations in databases mounted on a cluster depends on the database setting. The local InterSystems IRIS instance does not journal transaction operations to globals on remote nodes. In a network configuration, journaling is the responsibility of the node on which the global actually resides, not the one that requests the Set or Kill. Thus, if node B performs a Set at the request of node A, the journal entry appears in the journal on node B, not node A.
Differences Between Journaling and Write Image Journaling
Do not confuse the InterSystems IRIS journal with the write image journal (also known as the WIJ), which is described in Write Image Journaling and Recovery. Journaling and write image journaling have different functions, as follows:
-
Journaling provides a complete record of all database modifications that have already been written to the database. In the event that some modifications are lost, for example because they occurred after the most recent backup of a recovered database, you restore them to the database by restoring the contents of the journal file.
-
Write image journaling provides a record of all database modifications that are not yet written to the database. When a system crash occurs, the system automatically writes the contents of the write image journal to the database when it restarts.
Protecting Database Integrity
The InterSystems recovery process is designed to provide maximal protection:
-
It uses the “roll forward” approach. If a system crash occurs, the recovery mechanism completes the updates that were in progress. By contrast, other systems employ a “roll back” approach, undoing updates to recover. While both approaches protect internal integrity, the roll forward approach used by InterSystems IRIS does so with reduced data loss.
-
It protects the sequence of updates; if an update is present in the database following recovery, all preceding updates are also present. Other systems which do not correctly preserve update sequence may yield a database that is internally consistent but logically invalid.
-
It protects the incremental backup file structures, as well as the database. You can run a valid incremental backup following recovery from a crash.
Automatic Journaling of Transactions
In an InterSystems IRIS application, you can define a unit of work, called a transaction. InterSystems IRIS transaction processing uses the journal to store transactions. InterSystems IRIS journals any global update that is part of a transaction regardless of the global journal state setting for the database in which the affected global resides.
You use commands to:
-
Indicate the beginning of a transaction.
-
Commit the transaction, if the transaction completes normally.
-
Roll back the transaction, if an error is encountered during the transaction.
InterSystems IRIS supports many SQL transaction processing commands. See Transaction Processing for details on these commands.
Rolling Back Incomplete Transactions
If a transaction does not complete, InterSystems IRIS rolls back the transaction using the journal entries, returning the globals involved to their pre-transaction values. As part of updating the database, InterSystems IRIS rolls back incomplete transactions by applying the changes in the journal, that is, by performing a journal restore. This happens in the following situations:
-
During recovery, which occurs as part of InterSystems IRIS startup after a system crash.
-
When you halt your process while transactions are in progress.
-
When you use the Terminate option to terminate a process from the Processes page of the Management Portal (System Operation > Processes). If you terminate a process initiated by the Job command, the system automatically rolls back any incomplete transactions in it. If you terminate a user process, the system sends a message to the user asking whether it should commit or roll back incomplete transactions.
You can write roll back code into your applications. The application itself may detect a problem and request a rollback. Often this is done from an error-handling routine following an application-level error.
See Managing Transactions Within Applications for more information.
Consequences of Not Journaling Databases
Databases that are not journaled do not participate in journal recovery and transaction rollback at InterSystems IRIS startup. As a consequence, the following conditions apply after a failure, backup restore, and restart:
-
The most recent updates to non-journaled databases can be lost; the data in these databases will represent an earlier moment in time than the data in journaled databases.
-
Transactions or portions of transactions in non-journaled databases can be left partially committed. The durability provided by synchronous-commit transactions does not apply to databases that are not journaled.
-
While updates to non-journaled databases that are part of transactions are journaled, these journal records are used only to roll back transactions during normal operation. These journal records do not provide a means to roll back transactions at startup, nor can they be used to recover data at startup or following a backup restore.
The Journal Write Cycle
The operation that writes the contents of the journal buffer to the journal file is called a journal sync. A journal sync is guaranteed to write all operations currently in the journal buffer to the current journal file.
The frequency with which the journal is synced depends on the operating circumstances of the InterSystems IRIS instance involved. A journal sync can be triggered:
-
Once every two (2) seconds if the system is idle.
-
In an ECP-based distributed cache cluster, by the data server when responding to specific requests (for example, $Increment) from the application servers to guarantee ECP semantics.
-
By a TCOMMIT (in synchronous commit mode, which causes the data involved in that transaction to be flushed to disk) if you are using InterSystems IRIS transactions.
-
As part of every database write cycle by the write daemon.
-
When the journal buffer is full.
Journal Files and Journal History Log
Journal files are stored in the primary journal directory (install-dir\Mgr\journal by default) and are logged in the journal history log file, install-dir\Mgr\journal.log, which contains a list of all journal files maintained by the instance. The log is used by all journal-related functions, utilities, and APIs to locate journal files.
The journal history log file is updated as follows:
-
Entries are added to the log when a new journal file is created.
-
Entries are purged periodically, starting from the beginning of the file, if the corresponding journal file identified by the entry no longer exists and the entry is 30 days old or older. Purging stops when an entry is reached that does not satisfy both criteria.
Do not modify the journal.log file. If the file is modified outside of the journal utilities, it may be viewed as being corrupt, which may disable journaling. If the file is corrupt, contact the InterSystems Worldwide Response Center (WRC)Opens in a new tab for guidance. If journaling is disabled (that is, InterSystems IRIS is not able to update the journal.log file), rename the corrupt log file and restart journaling.
InterSystems recommends that you include the journal.log file in your backup strategy to ensure that it is available when needed for a journal restore following a backup restore; for information about backup and restore strategies and procedures, see Backup and Restore.
If the journal.log file is missing (for example, if you renamed the file because it is corrupt), the system creates a new one when a new journal file is created, but information about previous journal files is lost because the log file only lists journal files created since it was created. Unlisted journal files are not available for journal-related functions, utilities, and APIs that use the journal.log file. However, for journal restores, if the journal.log file is missing or you do not want to use the existing log file, you can specify the journal files manually (see Restore Globals from Journal Files Using ^JRNRESTO.
In addition, you can use the journal.log file to migrate/restore journal files to different locations, as follows:
-
Copy the journal files and journal.log file to a location other than the install-dir\Mgr directory on the target InterSystems IRIS instance.
-
On the target system, run the ^JRNRESTO routine, and enter No in response to the prompt regarding journal file original paths.
-
When prompted, specify the locations (on the target system) of the copied journal files and journal.log file; ^JRNRESTO uses the log file to validate the range of journal files you want to migrate/restore to the target system.
-
Complete the process as described in Restore Globals from Journal Files Using ^JRNRESTO.
When an InterSystems IRIS instance becomes a member of a mirror, the following journaling changes to support mirroring occur:
-
On becoming primary, a journal switch is triggered to a new journal file prefixed with MIRROR-mirror_name, for example MIRROR-MIR21-20230921.001. From that point, all journal files are written as mirror journal files and logged to the mirrorjrn-mirror_name.log, for example mirrorjrn-MIR21-20230921.log, as well as to journal.log.
-
On becoming backup or async, mirror journal files received from the primary are written to the configured journal directory along with the local instance’s standard journal files, and a copy of the primary’s mirror journal log (mirrorjrn-mirror_name.log) is created in install-dir\Mgr and continuously updated.
For more information about the role of journal files in mirroring, see Mirror Synchronization.
Using Temporary Globals and IRISTEMP
Nothing mapped to the IRISTEMP database is ever journaled.
Since the globals in a namespace may be mapped to different databases, some may be journaled and some may not be. It is the journal property for the database to which the global is mapped that determines if InterSystems IRIS journals the global operation. The difference between IRISTEMP and a database with the journal property set to No is that nothing in IRISTEMP, not even transactional updates, are journaled.
A database configured with the Journal globals property set to No (see Create Local Databases) continues to journal global Set/Kill operations in journal transactions, which can cause the journal file to become very large. IRISTEMP, however, does not journal Set/Kill operations, even when they are in a journal transaction.
If you need to exclude new z/Z* globals from journaling, map the globals to a database with the journal property set to No. To always exclude z/Z* globals from journaling, you must map them in every namespace to the IRISTEMP database.
Journal Management Classes and Globals
See the class documentation for %SYS.Journal.SystemOpens in a new tab for information on available journaling methods and queries. It is part of the %SYS.Journal package.
Also, InterSystems IRIS uses the ^%SYS(“JOURNAL”) global node to store information about the journal file. For example:
-
^%SYS("JOURNAL","ALTDIR") stores the name of the alternate journal directory.
-
^%SYS("JOURNAL","CURDIR") stores the name of the current journal directory.
-
^%SYS("JOURNAL","CURRENT") stores journal status and the journal file name.
You can view this information from the Globals page of the Management Portal (System Explorer > Globals)