Write Image Journaling and Recovery
InterSystems IRIS® uses write image journaling to maintain the internal integrity of your InterSystems IRIS database. It is the foundation of the database recovery process.
Write Image Journaling
InterSystems IRIS safeguards database updates by using a two-phase technique called write image journaling. In this process, the Write daemon creates the write image journal file (WIJ) during InterSystems IRIS startup. Database updates are initially written from memory to this transitional journal, IRIS.WIJ, and then to the database. If the system crashes during the second phase, the updates can be reapplied upon recovery.
The following topics are covered in greater detail:
Two-Phase Write Protocol
InterSystems IRIS maintains application data in databases whose structure enables fast and efficient searches and updates. Generally, when an application updates data, InterSystems IRIS must modify multiple blocks in the database structure to reflect the change. InterSystems IRIS uses a two-phase write protocol to protect the integrity of the database in the event of a crash that might otherwise lead to a subset of those block writes getting lost. This protocol operates as follows:
In the first phase, InterSystems IRIS writes the updated blocks to the WIJ. Once all of the updates are durably written, InterSystems IRIS sets a flag in the WIJ to indicate that there are blocks to restore. Then the second phase begins.
In the second phase, InterSystems IRIS writes the same set of blocks recorded in the WIJ to the database on disk. When these blocks are durably written, InterSystems IRIS clears the flag to indicate that there are no blocks to restore from the WIJ.
When InterSystems IRIS starts, it automatically checks the WIJ and runs a recovery procedure if it detects that an abnormal shutdown occurred. When the procedure completes successfully, the internal integrity of the database is restored. InterSystems IRIS also runs WIJ recovery following a shutdown as a safety precaution to ensure that database can be safely backed up.
Write Image Journal (WIJ) Settings
By default, the WIJ file is named IRIS.WIJ and resides in the system manager directory, install-dir/mgr, where install-dir is the installation directory. You may specify a different location for this file, as well as a target file size, from the Journal Settings page of the Management Portal. To do so:
Navigate to the Journal Settings page of the Management Portal (System Administration > Configuration > System Configuration > Journal Settings).
Enter the new location of the WIJ in the Write image journal directory box and click Save. The name must identify an existing directory on the system and may be up to 63 characters long. If you edit this setting for a clustered instance, restart InterSystems IRIS to apply the change; no restart is necessary for a standalone instance.
Enter the target size for the WIJ at the Target size for the wij (MB) (0=not set) prompt. The default of zero allows the WIJ to grow as needed but does not reserve space for this; entering a non-zero value reserves the specified space on the storage device.
WIJ recovery is necessary if a system crash or other major system malfunction occurs. When InterSystems IRIS starts, it automatically checks the WIJ. If it detects that an abnormal shutdown occurred, it runs a recovery procedure. Depending on where the WIJ is in the two-phase write protocol process, recovery does the following:
If the crash occurred after the last update to the WIJ was completed but before completion of the corresponding update to the databases—that is, during the second phase of the process—the WIJ is restored as described in WIJ Restore.
If the crash occurred after the last WIJ update was durably written to the databases—that is, after both phases were completed—a block comparison is done between the most recent WIJ updates and the affected databases, as described in WIJ Block Comparison (Windows and UNIX®/Linux only).
If the WIJ is marked as “active,” the Write daemon completed writing modified disk blocks to the WIJ but had not completed writing the blocks back to their respective databases. This indicates that WIJ restoration is needed. The recovery program, iriswdimj, does the following:
Informs the system manager in the messages log (messages.log) file; see Monitoring Log Files in the “Monitoring InterSystems IRIS Using the Management Portal” chapter of the Monitoring Guide.
Performs Dataset Recovery.
Typically, all recovery is performed in a single run of the iriswdimj program.
A dataset is a specific database directory on a specific InterSystems IRIS system. The iriswdimj program restores all datasets configured in the InterSystems IRIS instance being restarted after an abnormal shutdown.
The iriswdimj program can run interactively or non-interactively. The manner in which it runs depends on the platform, as follows:
Windows — Always runs non-interactively.
UNIX®/Linux — Runs non-interactively until encountering an error, then runs interactively if an operator is present to respond to prompts.
When the iris start quietly command is used on UNIX/Linux systems, it always runs noninteractively.
When the recovery procedure is complete, iriswdimj marks the contents of the WIJ as “deleted” and startup continues.
If an error occurred during writing, the WIJ remains active and InterSystems IRIS will not start; recovery is repeated the next time InterSystems IRIS starts unless you override this option (in interactive mode).
If you override the option to restore the WIJ, databases become corrupted or lose data.
The following topics are discussed in more detail:
Interactive Dataset Recovery
The recovery procedure allows you to confirm the recovery on a dataset-by-dataset basis. Normally, you specify all datasets. After each dataset prompt, type either:
Y — to restore that dataset
N — to reject restoration of that dataset
You can also specify a new location for the dataset if the path to it has been lost, but you can still access the dataset. Once a dataset has been recovered, it is removed from the list of datasets requiring recovery; furthermore, it is not recovered during subsequent runs of the iriswdimj program should any be necessary.
Noninteractive Dataset Recovery
When the recovery procedure runs noninteractively, InterSystems IRIS attempts to restore all datasets and mark the WIJ as deleted. On Unix® and Windows platforms, InterSystems IRIS first attempts a fast parallel restore of all datasets; in the event of one or more errors during the fast restore, datasets are restored one at a time so that the databases that were fully recovered can be identified. If at least one dataset cannot be restored:
The iriswdimj program aborts and the system is not started.
Any datasets that were not successfully recovered are still marked as requiring recovery in the WIJ.
WIJ Block Comparison
Typically, a running InterSystems IRIS instance is actively writing to databases only a small fraction of the time. In most crashes, therefore, the blocks last written to the WIJ were confirmed to have been durably written to the databases before the crash; the WIJ is not marked "active", and there is no WIJ restore to be performed. When InterSystems IRIS starts up after such a crash, however, the blocks in the most recent WIJ updates are compared to the corresponding blocks in the affected databases as a form of rapid integrity check, to guard against starting the instance in an uncertain state after a crash that was accompanied by a storage subsystem failure. The comparison runs for a short time to avoid impacting availability and asynchronous I/O is utilized to maximize throughput. If all blocks match, or no mismatch is detected within 10 seconds, startup continues normally. If a mismatch is found within this time, the results are as follows:
The comparison operation continues until all available WIJ blocks have been compared.
The mismatching WIJ blocks are written to a file called MISMATCH.WIJ in the WIJ directory.
Normal startup is aborted and InterSystems IRIS starts in single-user mode with a message like the following:
There exists a MISMATCH.WIJ file. Startup aborted, entering single user mode. Enter IRIS with iris terminal [instancename] -B and D ^STURECOV for help recovering from this error.
If you encounter MISMATCH.WIJ, contact InterSystems Worldwide Response Center (WRC) before proceeding.
This situation has implications for the integrity of your data and calls for immediate attention. However, performing the wrong action with MISMATCH.WIJ can worsen the situation. Unless you have experience with MISMATCH.WIJ, it is safer to revert to a known good backup and proceed from there or to contact the WRC for guidance.
Use the information that follows to determine the appropriate course of action. When your recovery procedures are complete, you must rename the MISMATCH.WIJ file, either using the STURECOV routine or externally, before InterSystems IRIS startup can continue; the file is persistent and prevents normal startup of the instance.
Run the indicated command to perform an emergency login as system administrator (see Administrator Terminal Session in the “Licensing” chapter of the System Administration Guide).
You are now in the manager’s namespace and can run the startup recovery routine with the command Do ^STURECOV. The following WIJ mismatch recovery message and menu appear on a UNIX®/Linux system:
The system crashed and some database blocks do not match what was expected based on the contents of write image journal (the WIJ). The WIJ blocks have been placed in the MISMATCH.WIJ file. If any database files, or the WIJ, were modified or replaced since the crash, you should rename the MISMATCH.WIJ. Otherwise, MISMATCH.WIJ probably contains blocks that were lost due to a disk problem. You can view those blocks and apply them if necessary. When finished, rename the MISMATCH.WIJ in order to continue startup. 1) List Affected Databases and View Blocks 2) Apply mismatched blocks from WIJ to databases 3) Rename MISMATCH.WIJ 4) Dismount a database 5) Mount a database 6) Database Repair Utility 7) Check Database Integrity 8) Bring up the system in multi-user mode 9) Display instructions on how to shut down the system -------------------------------------------------------------- H) Display Help E) Exit this utility --------------------------------------------------------------
On a Windows system, options 8 and 9 are replaced by 8) Bring down the system prior to a normal startup.
Option Rename MISMATCH.WIJ renames the file by appending the date; if there is already a renamed MISMATCH.WIJ with that name, a number (such as _1) is appended.
The appropriate actions in the event of a WIJ mismatch differ based on the needs and policies of your enterprise, and are largely the same as your site's existing practices for responding to events that imply data integrity problems. Considerations include tolerance for risk, criticality of the affected databases, uptime requirements, and suspected root cause.
The following represent some considerations and recommendations specific to the WIJ block comparison process:
Replacing, restoring, or making any changes to the databases or WIJ files after a crash and before recovery can lead to discrepancies that are then found during WIJ comparison and recorded in the MISMATCH.WIJ file. If this has occurred, rename MISMATCH.WIJ.Note:
If a database is to be restored following a crash, ensure that prior to the restore you start the instance without WIJ and journal recovery (see Starting InterSystems IRIS Without Automatic WIJ and Journal Recovery in the “Backup and Restore” chapter of this guide). This avoids both creating discrepancies that will be detected by the WIJ comparison and incorrectly applying WIJ blocks or journal data (see the “Journaling” chapter of this guide) to a version of a database for which they were not intended.
Some storage subsystems, particularly local drives on laptops and workstations, use an unsafe form of write-back caching that is not backed by battery or by non-volatile memory. This defeats the two-phase write protocol that InterSystems IRIS performs and can lead to corruption following a hardware crash or power loss that is detected during WIJ compare. If this applies to your system, it is likely that MISMATCH.WIJ contains more up-to-date data and therefore can be safely applied to the databases (assuming the system is one with which an abundance of caution is not required).
If you have made any changes to the databases following the WIJ comparison, MISMATCH.WIJ is no longer valid and it is not safe to apply the WIJ blocks.
For servers with enterprise-class storage or any storage subsystem that does not utilize an unsafe form of write-back caching, mismatches found during WIJ compare are always unexpected and warrant careful attention, as they may be a sign of a more serious or more widespread problem.
Depending on the root cause of the problem, it may be that the databases are intact and it is the WIJ that is corrupted. An integrity check of the affected databases can help determine whether this is likely.
Because WIJ comparison only covers the blocks written most recently, there may be problems affecting additional blocks that could be detected by a full integrity check (see Verifying Structural Integrity in the “Introduction to Data Integrity” chapter of this guide).
If the databases are small and/or time allows you can follow a procedure similar to the following for optimal safety:
Run a full integrity check on the databases.
If none are corrupt, rename MISMATCH.WIJ and start up.
If one or more databases are corrupt, copy the IRIS.DAT files for all databases, apply all blocks from MISMATCH.WIJ and run a full integrity check again.
If any database is corrupt after applying MISMATCH.WIJ, the databases can be reverted to the previous copy or restored from a previous backup.
Encrypted databases are excluded from the WIJ block comparison.
Limitations of Write Image Journaling
While the two-phase write protocol safeguards structural database integrity, it does not prevent data loss. If the system failure occurs prior to a complete write of an update to the WIJ, InterSystems IRIS does not have all the information it needs to perform a complete update to disk and, therefore, that data is lost. However, data that has been written to a journal file is recovered as described in Recovery in this chapter.
In addition, write image journaling cannot eliminate database degradation in the following cases:
A hardware malfunction that corrupts memory or storage.
An operating system malfunction that corrupts memory, the filesystem, or storage.
The WIJ is deleted.
The loss of write-back cache contents. In the event of a power outage, the write-back cache could be lost, leading to database degradation. To prevent this degradation, ensure that either the storage array uses nonvolatile memory for its write-back cache or the volatile write-back cache has battery backup.
If you believe that one of these situations has occurred, contact the InterSystems Worldwide Response Center (WRC).