Caché Data Integrity Guide
Shadowing
[Back] [Next]
   
Server:docs2
Instance:LATEST
User:UnknownUser
 
-
Go to:
Search:    

Shadowing enables secondary computers to maintain a “shadow” copy of selected databases as they are updated on a primary machine. By continually transferring journal information from the primary machine to the secondary machines, shadowing enables recovery to a system which is typically within only a few transactions of the source database.

You can use shadowing for many purposes, each with its own set of important considerations depending on your system environment. Some of the most common objectives satisfied by shadowing include the following:
This chapter discusses the following topics:
Shadowing Overview
A primary Caché instance may have one or more shadows. Shadowing monitors database activity on a primary system, the source, and causes the same activity to occur on a secondary system, the destination. It does this through a shadow client service running on the destination that continually requests journal file details from a shadow service running on the source. The shadow service responds by sending the details of the actual Set, Kill, and $Bit journal record entries to the destination shadow over a TCP connection.
Shadowing Overview
Important:
The source and destination servers can be of different hardware, operating system, or CPU chipset. To avoid data incompatibility, however, the destination shadow Caché instance must use the same character width (8-bit or Unicode; see Caché Character Width in the Caché Installation Guide) and the same locale (see Using the NLS Settings Page of the Management Portal in the “Configuring Caché” chapter of the Caché System Administration Guide) as the source Caché instance. The one exception to this requirement is that an 8-bit instance using a locale based on the ISO 8859 Latin-1 character set is compatible with a Unicode instance using the corresponding wide character locale. For example, an 8-bit instance using the enu8 locale can be configured as the shadow source database server with a Unicode instance using the enuw locale as the destination shadow.
All shadowing uses a fast transmission method which allows more efficient performance by sending the compacted journal file block by block. The shadow applies all transactions to the local databases. The transmission mode requires the data to be written to the journal file, which may introduce a delay of a few seconds. The shadow establishes a TCP connection to the server and receives the journal file. As the journal file downloads, another shadow process applies the journal entries to the local destination copy of the database.
Upon connecting to the data source server, the destination shadow sends the server the name of the journal file and the starting point. The shadow checks for new records periodically. If it does not have the latest records, the shadow downloads them and updates the databases. During these processes, Caché continually stores checkpoints in a shadow global to facilitate rollback and restart capabilities.
Caché purges the destination shadow copies of source journal files automatically. You can configure how long to keep files that are eligible for purging, that is, ones that have been dejournaled and do not contain any open transactions.
Configuring Shadowing
This section explains how to configure and set up shadowing in Caché and includes the following procedures:
Important:
If you want to configure both mirroring and shadowing for the same databases, bear in mind the following guidelines:
See the Mirroring chapter of the Caché High Availability Guide for information about configuring mirroring.
Configuring the Source Database Server
Before enabling shadowing on a source database server, ensure that the destination system can make a TCP connection to the source system. If you plan to secure this connection using SSL, a %SuperServer SSL/TLS configuration must exist on the source. See Configuring the Caché Superserver to Use SSL/TLS in the “Using SSL/TLS with Caché” chapter of the Caché Security Administration Guide for details.
Important:
A shadow service cannot run on a system with a single-server license.
Use the Management Portal from the Caché instance running on the source system to enable the shadow service, restrict connections, and enable global journaling for the databases you are shadowing. These procedures are described in the following topics:
For information on methods and queries available for interfacing with the data source of a shadow without using the Management Portal, see the SYS.Shadowing.DataSource class documentation in the InterSystems Class Reference.
Also see the Important Journaling Considerations section for issues that may pertain to your environment.
Enable the Shadowing Service
To use shadowing, you must enable the shadowing service using the Security Management portion of the Management Portal. You may also restrict shadowing access by entering the IP addresses of allowed connections:
  1. Navigate to the [System Administration] > [Security] > [Services] page of the Management Portal.
  2. Click %Service_Shadow in the list of service names to edit the shadow service properties.
  3. Select the Service enabled check box. Before clicking Save, you may want to first restrict what IP addresses can connect to this database source. If so, perform the next step, and then click Save.
  4. In the Allowed Incoming Connections box, any previously entered server addresses are displayed in the IP Address list. Click Add to add an IP Address. Repeat this step until you have entered all permissible addresses.
    You may delete any of these addresses individually by clicking Delete in the appropriate row, or click Delete All to remove all addresses, therefore allowing connections from any address.
Enable Journaling
Verify that you are journaling each database that you wish to shadow.
  1. Navigate to the [System Administration] > [Configuration] > [System Configuration] > [Local Databases] page of the Management Portal and view the Journal column for each database you wish to shadow.
  2. To change the journal state from No to Yes, click Edit in the row of the appropriate database to edit the database properties.
  3. In the Global Journal State list, click Yes and then click Save.
By default, the CACHELIB, CACHE, DOCBOOK, and SAMPLES databases are not journaled and, as a result, you cannot shadow them — CACHETEMP is never journaled.
Important Journaling Considerations
Review the following sections for conditions that may affect your system:
Managing Source Journal File Purging
Caché does not do any special handling of journal file purging on the source for shadowing; therefore, it is your responsibility to configure the journal settings on the source to ensure the journal files that the shadow requires remain available until they are transmitted to the destination shadow.
This is usually only a concern if the shadow falls seriously behind the source; for example, if you suspend the shadow or stop it for a prolonged period of time.
Responding to Disabled Source Journaling
If journaling is disabled on the source, you must determine the best course of action to maintain a valid shadow. Most likely, you will have to resynchronize the shadow with the source after you resolve the condition that caused Caché to disable journaling.
See the Journal I/O Errors section of the “Journaling” chapter of this guide for more details.
Shadowing Class Compiles
Caché journals the database that contains the globals used for compiling classes. If you use shadowing and rely on the compile on the source to update the application code on the shadow, ensure that the default qualifier (compile with /journal=1) is not changed, so that each class compile is journaled and the updates transferred to the shadow database. If class compiles are not journaled, you cannot use the shadow for disaster recovery unless you recompile all your classes.
Configuring the Destination Shadow
To configure shadowing on a destination shadow server, first ensure that the destination system can make a TCP connection to the source system.
If you plan to use SSL, an SSL/TLS client configuration must exist on the destination. See A Note on Caché Client Applications Using SSL/TLS in the “Using SSL/TLS with Caché” chapter of the Caché Security Administration Guide for details.
Use the Management Portal from the Caché instance running on the destination system to configure the destination shadow properties, map and synchronize the databases in the shadow, and start shadowing. These procedures are described in the following sections:
For information on methods and queries available for interfacing with the shadow destination without using the Management Portal, see the SYS.Shadowing.Shadow class documentation in the InterSystems Class Reference.
Define the Shadow
Navigate to the [System Administration] > [Configuration] page of the Management Portal and click Shadow Server Settings under the Connectivity column to display the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page. Perform the following steps to define the shadow properties:
  1. Click Create Shadow Server to define a shadow on this destination server.
    Important:
    You can shadow a failover mirror member only if it is the only failover member in the mirror; if a mirror has two failover members, you must shadow an async member instead. See Configuring Shadowing for more information about shadowing mirrored databases..
  2. Enter an identifying name for this shadow in the Name of the shadow box. This value is also referred to as the shadow ID. The system uses this name to distinguish between shadow instances that may be running on the same system.
    Note:
    Do not use the tilde (~) character in the shadow name; it is used in internal shadow processing.
  3. Enter the TCP/IP address or host name (DNS) of the source database server you are shadowing in the DNS name or IP address of the source box.
  4. Enter the superserver port number of the source Caché instance you are shadowing in the Port number of the source box.
    Important:
    If you change the IP address or the port number on a suspended shadow, it is your responsibility to ensure the shadow can resume properly.
  5. Click Advanced to enter the following optional fields:
  6. Click Save to return to the Shadow Server Settings page, where the new shadow is now listed. When you click the Edit link to edit the settings, the Add database mapping link is now included on the Edit Shadow Server page; see Map the Databases for details.
Map the Databases
After you successfully save the configuration settings, you can add or delete database mappings from the source to the shadow:
  1. Next to Database mapping for this shadow click Add to associate the database on the source system with the directory on the destination system using the Add Shadow Mapping dialog box.
  2. In the Source database directory box, enter the physical pathname of the source database file—the CACHE.DAT file. Enter the pathname of its corresponding destination shadow database file in the Shadow database directory box, and then click Save.
  3. Verify any pre-filled mappings and click Delete next to any invalid or unwanted mappings. Shadowing requires at least one database mapping to start.
  4. Click Close to return to the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page.
If the source database server is part of a cluster, the configuration settings for the destination shadow differ slightly. For information on shadowing a clustered system, see the Cluster Shadowing section of the “Cluster Journaling” chapter of this guide.
Note:
You can use the CACHESYS database as a source database of shadowing, provided that the target (shadow database) is not the CACHESYS database on the shadow. Currently the only way to add a database mapping containing the source manager’s directory (CACHESYS) to a shadow configuration is by using the SYS.Shadowing.Shadow class API. For example:
 Set ShadowOref=##class(SYS.Shadowing.Shadow).%OpenId("MyShadow")
 Do ShadowOref.SetDatabaseToShadow("C:\MyCache\Mgr","D:\MyCacheShdw\Shdwsys")
 Set rc=ShadowOref.%Save()
Where C:\MyCache\Mgr is the source manager’s directory for the CACHESYS database and D:\MyCacheShdw\Shdwsys is the directory for a database that is not the CACHESYS database on the destination. See the SYS.Shadowing.Shadow entry in the InterSystems Class Reference for details.
Synchronize the Databases
Before you start shadowing, synchronize the databases on the shadow destination with the source databases. Use an external backup on the source data server and restore the databases on the destination shadow. See the Backup and Restore chapter of this guide for more information.
Important:
If the source and destination instances are on systems of different endianness (see “Platform Endianness” in the “Supported Technologies” section of InterSystems Supported Platforms), the database restored to the destination shadow must be converted to the endianness of the source before being used; see Considering Endianness in the “Backup and Restore” chapter for procedures.
Managing and Monitoring Shadowing
Caché provides an interface to shadow processing through the Management Portal. You can also configure and manage shadow processing using the ^SHADOW utility or the SYS.Shadowing API classes, SYS.Shadowing.DataSource and SYS.Shadowing.Shadow. This document describes the procedures using the Management Portal and gives some examples of using the shadowing APIs.
A shadow can be in one of three states; depending on the state, you can perform different actions on the shadow. The following sections describe each state and action including the interrelationships among them.
Shadow States and Actions
A shadow can be in one of these states at any given time:
There are four types of allowable actions you can perform on a shadow, depending on its current state and your user privileges:
The following diagram shows the permissible actions on a shadow in each state. It indicates the shadow states with circles and shows the actions you can perform on these states with arrows.
Relationships of Shadow States and Permissible Actions
Shadow Processing Considerations
Keep the following conditions in mind when deciding when and how to change the state of a shadow:
There are two places in the Management Portal that you can perform tasks on a defined shadow:
See the individual method descriptions in the SYS.Shadowing.Shadow entry of the InterSystems Class Reference for details on performing these shadowing tasks programmatically.
Shadow Checkpoints
Caché creates checkpoints periodically throughout the shadowing process. A checkpoint for the shadow is a location in the shadow copy of a source journal with the following implications:
  1. All records at and prior to it are presumed to have been applied to the shadow databases.
  2. It is safe, as far as database integrity is concerned, for the shadow to resume from the checkpoint after being suspended.
You can retrieve checkpoint information using the CheckPointInfo method of the SYS.Shadowing.Shadow class.
Shadow Administration Tasks
The [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page lists each defined shadow with the name, status, source name and port, start point, filter, and choices for performing actions on the shadow configuration. Click the following options to perform the indicated task:
Start Shadowing
Once you add a shadow definition it appears in the list of shadows on the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page. You can start a shadow from this page:
  1. Before starting the shadowing process, verify you have synchronized the databases you are shadowing on the source and destination and mapped the source databases to the corresponding destination databases.
  2. Click Start in the row for the shadow name you want to start.
  3. After verifying the location information for the source instance, click Select Source Event to choose where to begin shadowing. A page displays the available source events from the source journal file directory. You must select a source event before you can start shadowing. See Select a Source Event for details.
From the Management Portal, if you attempt to start a shadow that had been processing, you may see the following warning:
*** WARNING ***
There is a checkpoint from previous shadowing session.
You might want to RESTART the shadow from that
checkpoint instead.
If you do START, the checkpoint and any remaining
shadow copies of journal files from previous shadowing
session will be deleted.
Are you sure you want to start shadowing?
As this warning states, if you start a previously processing shadow, Caché clears all checkpoints and stores the start point you select as the first checkpoint. This process also purges any remaining shadow copies of the source journal files and fetches new copies form the source regardless of a possible overlap between the files. Ensure your new start point coincides with the state of the shadow databases.
If you run multiple shadows on an instance of Caché, see the Generic Memory Heap Considerations section for details on adjustments you may have to make.
Important:
Dismounting a database that is part of a running shadow (one that is in the Processing state) does not interrupt the shadow, meaning that other databases continue to be updated as the dismounted database falls behind. A dismounted shadow database causes a severe message to be posted to the console log and must be resynchronized with the shadow after being remounted, as described in Synchronizing or Resynchronizing a Destination Database.
Select a Source Event
While starting (or resuming) a destination shadow, you must select a source event from the journal files on the data source server where shadowing of the journaled databases should begin.
Click Select Source Event to display a list of journal events on the source database that are valid starting points for shadowing. From this list, click the time to specify at which source event shadowing starts. Choose the starting point after which you synchronized the databases on the source and destination.
For example, Caché automatically switches the journal file after a successful backup. Before starting the shadowing process, synchronize the databases by restoring the successful backup file from the source on the destination shadow databases. On the shadow, click Select Source Event from the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page to see events listed similar to those in the following display:
For this example, to start shadowing at the point when the source backup ended successfully (the point of database synchronization), click the Time (2008-05-22 08:30:54), of the Event displaying end of backup ().
Generic Memory Heap Considerations
The journal reader and database update processes on the shadow destination communicate via shared memory allocated from the generic memory heap size (also known as gmheap). Several processes in Caché use this memory and how Caché allocates it involves very complex interactions; therefore, Caché silently increases gmheap allocation during startup when necessary. In most cases, you should not have to manually adjust the allocation.
If you start multiple shadows at or near the same time while Caché is running, you may receive a gmheap allocation error. You can improve the allocation by starting the shadows as a group. If you start (or resume) multiple shadows one by one consecutively, the first shadow to start uses about half of the free gmheap memory; the second, half of what remains; and so on. In contrast, if you start multiple shadows as a group, every shadow in the group uses 1/(N+1) of the free gmheap memory, where N is the number of the shadows in the group. Thus, starting multiple shadows as a group not only avoids the possible error allocating memory from gmheap, but also allocates memory evenly among the shadows.
See the StartGroup method in the SYS.Shadowing.Shadow entry of the InterSystems Class Reference for more information.
You can adjust the gmheap size from the [System Administration] > [Configuration] > [Additional Settings] > [Advanced Memory] page of the Management Portal.
Stop Shadowing
When you stop shadowing you can choose to roll back or not to roll back any open transactions by selecting or clearing the Roll back open transactions check box.
Stop without Rollback
If you choose not to roll back, it is similar to suspending a shadow, but requires more privileges. You may choose this option if you want to maintain the current checkpoint, but you do not want Caché to automatically resume the shadow at restart, which does happen to a suspended shadow. For example, if you have to make configuration changes that require a Caché restart and additional changes after Caché is up, but before the shadow should start, use this option.
Stop with Rollback
This option is mainly for disaster recovery. The rollback sets the shadow databases to a logically consistent state, though out of sync with the source. Avoid restarting a shadow that you stopped and rolled back. You have the option open to you so that you can recover if it was a mistake to choose the rollback option; thus avoiding the need to resynchronize the shadow with the source.
Caution:
Restarting a shadow that you stopped with rollback may leave the shadow in an indeterministic state until the shadow has progressed beyond the point of the pre-rollback state.
Restart Shadowing
If you stopped shadowing, in addition to the choice of starting the shadow, you can restart the shadow. Processing starts from the last checkpoint taken before you stopped the shadow if you chose not to roll back open transactions. If you chose to roll back when you stopped the shadow, shadow processing begins at the checkpoint prior to the earliest open transaction when the shadow stopped. A shadow restart reuses the journal files retained from the last time you stopped the shadow.
Shadow Operations Tasks
You can monitor the shadowing operation status from both the source and destination servers of the shadow. From the Management Portal, navigate to the [System Operation] > [Shadow Servers] page. On this page you choose the appropriate option depending on whether you are monitoring the shadowing process from the shadow side or the data-source side. The following sections detail the contents of each side:
Managing the Destination Shadow
You can monitor and manage the shadow process from the destination shadow. Click System as Shadow Server to display a list of source servers for this shadow machine and associated actions you can perform on each item. The [System Operation] > [Shadow Servers] > [Shadows] page lists each defined shadow with the information described in the following table.
Field Description
Name Name of the shadow.
Status One of three shadowing states described previously in this section: stopped, processing, suspended. You may see trying to connect as status when you initiate processing.
Checkpoint Offset location in the shadow copy of the journal where it is safe to resume processing.
Errors Number of errors reported on the shadow destination.
Open Transactions Indicates whether or not there are open transactions on the shadow and, if so, how many.
Latency Estimated time for the shadow to process the journal records that it copied from the source but has not yet applied to the shadow databases.
Click the following options to perform the indicated task:
Monitoring the Data Source
You can also monitor the shadow process on the source system from the Data Source column of the [System Operation] > [Shadow Servers] page:
You can also obtain this information programmatically. See the SYS.Shadowing.DataSource entry of the InterSystems Class Reference for details.
Synchronizing or Resynchronizing a Destination Database
For a variety of reasons, you may need to resynchronize a database that has fallen behind the other databases in the shadow (for example, because the destination database was dismounted for maintenance, or has been restored from backup). In addition, when you add a database to an existing shadow configuration, you must ensure that it is synchronized with the journal files being dejournaled by shadowing.
There are several options from which to choose depending on your needs.
Synchronizing Using a New Copy of a New or Existing Source Database
The simplest way to synchronize a shadow database is to suspend shadowing before making a backup of the source database that require synchronization. A database synchronized in this way will be in an inconsistent state on the shadow destination until the shadow catches up to the journal location corresponding to the creation of the backup; only the database being synchronized is affected. The disadvantage of this approach is that you may need to wait until an appropriate time to perform the backup of the source databases.
To synchronize a database in this way, use the following procedure:
  1. Suspend (do not stop) the shadow: navigate to the [System Operation] > [Shadow Servers] > [Shadows] page as described in Managing the Destination Shadow and click Suspend.
  2. If you are adding a new database to the shadow, create it on the destination and the source, then add the source database mapping to the shadow configuration, as described in Map the Databases.
    If you are synchronizing an existing source database, create a backup on the source and restore it on the destination. If you are adding an existing database on the source that was not previously included in the shadow, create a backup on the source and restore it on the destination, then add the source database mapping to the shadow configuration, as described in Map the Databases.
    Note:
    If you wish to use an older backup, you must use one of the following procedures.
  3. Resume the shadow.
Synchronizing Using an Existing Copy of a Source Database
Rather than creating a new backup of the source databases as described in the previous procedure, you may want or need to use an existing copy of the database needing synchronization—that is, a version of the database older than the journal files currently being dejournaled by the shadow. For example, you may be restoring a damaged source database using a backup from an earlier time, adding a database on the source to the shadow under circumstances which prevent you from creating a new backup, or catching up a destination database that fell behind after being dismounted.
When synchronizing databases like these, you have several options. The simplest involves restarting the entire shadow from the source event representing the appropriate journal file—either the journal file corresponding to the backup you are restoring or the journal file that was being dejournaled when the destination database was dismounted. If you are catching up only one or a few databases out of many databases in the shadow, however, this option has the following disadvantages:
Even with these disadvantages, if many databases require synchronization or the amount of journal to be applied is relatively modest, restarting the entire shadow from the appropriate journal file is typically preferred for its simplicity.
To synchronize by restarting the entire shadow from the appropriate journal file:
  1. Stop the shadow: navigate to the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page (as described in Shadow Administration Tasks) and stop the shadow without rolling back open transactions (see Stop Shadowing for details).
  2. Restore the backup or mount the dismounted database.
  3. If you restored a backup of a source database that is not yet in the shadow, add the source database mapping to the shadow configuration, as described in Map the Databases.
  4. Start the shadow: navigate to the [System Administration] > [Configuration] > [Connectivity] > [Shadow Server Settings] page and start the shadow (as described in Start Shadowing), choosing as the source event (see Select a Source Event) either the journal file corresponding to the backup you restored or the journal file that was being dejournaled when the destination database was dismounted.
The alternative procedures involve more steps, but because journal data is applied only to the databases being synchronized, the disadvantages listed for the previous procedure are minimized. To catch up specific databases without synchronizing the entire shadow:
  1. Suspend (do not stop) the shadow: navigate to the [System Operation] > [Shadow Servers] > [Shadows] page as described in Managing the Destination Shadow and click Suspend.
  2. Restore the database(s) from backup, or mount the dismounted database(s).
  3. Choose one of the following options:
  4. If you restored a backup of a source database that is not yet in the shadow, add the source database mapping to the shadow configuration, as described in Map the Databases.
  5. Resume the shadow.
Using Shadowing for Disaster Recovery
As described in the Mirroring chapter of the Caché High Availability Guide, Caché mirroring with automatic failover provides an effective and economical high availability solution for planned and unplanned outages. Mirroring includes a full disaster recovery capability.
When mirroring is not in use, however, Caché shadowing is a good low-cost solution for off site disaster recovery, and may be used in conjunction with one of the numerous Caché-compatible high availability failover strategies provided by the makers of operating systems and computer hardware (see the System Failover Strategies chapter of the Caché High Availability Guide). Shadowing’s benefits in disaster recovery include the following:
This section covers the following topics:
Planned Production Transfer to the Shadow Destination
Shadowing is very suitable for planned temporary relocation of the production databases, for example to perform maintenance on the production host system, because it includes built-in mechanisms to allow shadow destinations to catch up when you perform a planned production cutover to a shadow destination.
To perform a planned transfer of production to a shadow destination, use the following procedure:
  1. Halt application activity on the production server (shadow source).
  2. Gracefully shut down the shadow source Caché instance, for example using the ccontrol stop command (see Controlling Caché Instances in the “Using Multiple Instances of Caché” chapter of the Caché System Administration Guide).
    When a shadow source is shutting down, the shutdown process waits for shadow destinations to receive all current journal files from the source before terminating the jobs servicing those shadow destinations. If the source encounters an error attempting to retrieve information about a destination, or does not get confirmation that all journal files have been received by a destination within the waiting period, a message is written to the console log (see Monitoring Log Files in the “Monitoring Caché Using the Management Portal” of the Caché Monitoring Guide). You can, therefore, confirm that the destination you intend to switch to is caught up with the source by checking the source’s console log and confirming that there are no such messages pertaining to it.
  3. Confirm that the shadow destination has finished dejournaling all journal data from the shadow source, then follow the procedure for stopping shadowing on the shadow destination.
  4. Shut down Caché on the destination and do one of the following:
  5. Restart Caché on the destination.
  6. Resume application activity on the new production server (shadow destination).
If you are certain that all journal data was received from the original shadow source and fully dejournaled on the destination (that is, that there was no data loss) in the previous procedure, you can return to the original configuration when your planned outage is complete and the original production instance has been restarted by reversing the original direction of shadowing—that is, configuring the current production instance (former destination) as the shadow source and the former source as a destination—following the instructions in the Configuring Shadowing section as needed. You can then repeat the previous procedure in the opposite direction.
Disaster Recovery Using the Shadow Destination
Shadowing is a good mechanism for recovery from disk failure, database degradation due to hardware or software failure, or destruction of the primary physical plant. (Shadowing, however, cannot recover from malicious deletion of globals.) A Caché shadow server can apply journals from several dissimilar platforms on a small-scale server over any TCP network. Since shadowing conveys only logical updates to the destination, it eliminates the risk of propagating any structural problem. When deciding if Caché shadowing best suits your disaster recovery strategy, however, you should consider the following limitations when shadowing is interrupted by a failure of the production server:
If your production/shadow source system functions as an application server, install identical applications on your disaster recovery shadow destination to speed recovery.
To use a shadow destination for disaster recovery, you can use the procedure in the previous section, assuming that the shadow source was not gracefully shut down but rather failed or became unavailable, and beginning with the step of confirming that the destination has finished dejournaling all journal data it received from the shadow source before the failure before following the procedure for stopping shadowing on the shadow destination. As previously noted, you can choose to roll back open transactions while stopping shadowing, and once you have stopped shadowing you can evaluate the risk of using the destination databases for disaster recovery.
Because of the near certainty of data loss under unplanned outage circumstances, you cannot simply reverse the direction of shadowing once the original shadow source is restored to operation to catch it up and to return to the original configuration, as described in the previous section. In this situation you must also fully resynchronize all databases, as describe in Synchronizing or Resynchronizing a Destination Database.