DataMove.API

abstract class DataMove.API extends %SYSTEM.Help

The DataMove API.

Move globals from one database to another based on a namespace mapping change.

Example:

#Include %syDataMove 
Set Datamove="ABC1"
// Namespace that contains ^A global
Set Namespace="USER"
// Initialize temporary storage area to hold mapping edits 
Set Status = ##class(DataMove.API).MapInitialize(Namespace)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Create mapping for global ^A to database TEST1 in temporary storage area
Kill Properties
Set Properties("Database") = "TEST1"
Set Status = ##class(DataMove.API).MapGlobalsCreate(Namespace, "A", .Properties)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Create mapping for global range ^A(5) up to but not including ^A(10) 
// to database TEST2 in temporary storage area 
Set Properties("Database") = "TEST2"
Set Status = ##class(DataMove.API).MapGlobalsCreate(Namespace, "A(5):(10)", .Properties)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Create new DataMove, based on map edits in the temporary storage area
// Set flag to run user defined activation routine
Kill Properties
Set Properties("Flags")=$$$BitCheckActivate 
Set Status = ##class(DataMove.API).Generate(Datamove,.Properties, .Warnings, .Errors) 
If $data(Warnings) Write  ! Zwrite  Warnings
// Can't proceed if there are errors
If $data(Errors) Write  ! Zwrite  Errors q
// Once Generate() has been called, you can use the ^DATAMOVE utility 
// from a terminal to Monitor the DataMove Operation.
Set Status = ##class(DataMove.API).StartCopy(Datamove)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Monitor the DataMove until we can activate it
F  {
  hang 5
  Set Status = ##Class(DataMove.API).GetProperties(DataMove,.Properties)
  i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
  // Activate will only run if we have 5 MB or less of journal to apply
  If (Properties("State")=$$$DMStateJrnApply){
     If (Properties("JRNMBToApply")<=5){
        Quit
     } Else {
	  Write !,"Journal MB still to apply: "_Properties("JRNMBToApply")
	  Continue
     }
  } Else {
     Write !,"MB still to copy: "_Properties("MBToCopy")
     Write !,"MB copied: "_Properties("MBCopied")
     Write !,"Journal MB still to apply: "_Properties("JRNMBToCopy")
  }
}
i $$$ISERR(Status) q
Set Display=1
Set Status=##Class(DataMove.API).Activate(DataMove,Display)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Delete the source globals that have been copied in the DataMove
Set Status = ##class(DataMove.API).DeleteSourceGlobals(Datamove)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
Set Status = ##class(DataMove.API).Finish(Datamove)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q
// Delete the DataMove and clean up any temporary storage 
Set Status = ##class(DataMove.API).Delete(Datamove)
i $$$ISERR(Status) w !,$System.Status.GetErrorText(Status) q

Method Inventory (Including Private)

Methods

classmethod Activate(Name As %String, Display As %Boolean = 1, Timeout As %Integer = 120, Force As %Boolean = 0) as %Status [ Language = objectscript ]
Finishes the DataMove and activates the namespace mapping changes.

This method stops the DataMove background jobs, finishes processing any journal files, writes the mapping changes to the CPF, and activates the mapping changes. It momentarily sets switch 10 which freezes all users on the system while the systems namespace mappings tables are updated. If the flag $$$BitCheckActivate was set when the DataMove was initialized, the method will call a user-supplied routine $$CheckActivate^ZDATAMOVE() to check the application status before continuing. If $$CheckActivate^ZDATAMOVE() does not return 1, the method will quit with an error status. The user can then call Activate() again at a later point in time. For more information on switch 10, see "Using Switches" in Specialized System Tools and Utilities. Note: Activate() checks the State property of the DataMove to make sure that the initial copy is complete and the journal apply is caught up before proceeding.
Arguments:
Name - Name of the DataMove.
Display - 0/1 - determines if progress messages are written to the screen.
Timeout - Number of seconds to wait for the Copy job to finish running an apply journals operation before proceeding. If the minimum number of MB of journal data remaining to apply does not fall below the threshold (5MB) within this time period the method will return an error status. If this occurs, the Activate() method can be called at a later time when the system is less busy.
Force - 0/1 - Force activation of the namespace map even if the minimum number of MB of journal data remaining to apply does not fall below the threshold (5MB). When set, the Activate process will wait up until the Timeout value for the minimum number of MB of journal data remaining to apply to fall below the threshold (5MB). If it does not fall below the threshold by the end of the timeout period, Activate will set switch 10 to freeze the system, apply the remaining journal, and activate the namespace map.

WARNING: When the Force flag is specified, and there is a lot of journal data to still be applied, this can cause an extended outage of the system. Once started the only way to stop the activation and unfreeze the system is to force down the system.

Example:
%SYS>s x=##Class(DataMove.Data).Activate("ABC1")
 
Preparing to Activate Data Move
Waiting for Data Move jobs to complete...
Stopping Data Move processes in preparation for activation
Locking Globals...
Freezing system...
Applying journals...
Applying journal: c:\iris\mgr\journal\20200924.021
Copying mapping changes to Config classes...
Writing mapping changes to CPF file...
Activating mapping changes...
Activation complete
classmethod Delete(Name As %String) as %Status [ Language = objectscript ]
Deletes the DataMove.

If the Data Move operation you want to delete has been started and then stopped, you should first call the Rollback() method to rollback any of the data which had been moved.
Arguments:
Name - Name of the DataMove.
classmethod DeleteSourceGlobals(Name As %String) as %Status [ Language = objectscript ]
Deletes the globals from the source directory that have been copied by the DataMove.

Arguments:
Name - Name of the DataMove.
This method deletes all globals in the source databases that have been copied to the destination databases. Several processes may be created to delete the source globals.
classmethod Finish(Name As %String) as %Status [ Language = objectscript ]
Finishes the DataMove process.

This method writes a success or failure message to the log file, closes the log file, and sets the State property of the DataMove to $$$DMStateDone. It also copies the log file into the file DataMove.log which is a record of all the DataMoves performed on the system.
Arguments:
Name - Name of the DataMove.
classmethod Force(Name As %String) as %Status [ Language = objectscript ]
Forces a DataMove to stop when StopCopy() fails to stop all DataMove processes.

Arguments:
Name - Name of the DataMove.
classmethod Generate(Name As %String, ByRef Properties As %String, ByRef Warnings As %String, ByRef Errors As %String) as %Status [ Language = objectscript ]
Creates a new DataMove based on map edits in the temporary storage area.

Arguments:
Name - Name of the DataMove to be created.
Properties - Array of optional properties to be used to create the DataMove.
Properties("LogFile") - optionally specifies a log file name, if other than the default. The default logfile name is Name.log and is found in the MGR directory.
Properties("Description") - optionally provides a description of the DataMove to be performed.
Properties("Flags") optionally provides any flags describing the DataMove operation as follows:
Properties("MaxMBPerMin") - Maximum number of MB per minute the DataMove operation is allowed to move to the destination database. Setting this to 0 allows the DataMove to run as fast as it can. If not passed, it uses the system default which is 0.
$$$BitNoSrcJournal - allow copying of non-journaled databases.
$$$BitNoBatchMode - run data copy without batch mode.
$$$BitCheckActivate - call the user-supplied routine $$CheckActivate^ZDATAMOVE() to check the application status before activating the mapping changes.
Warnings - Array returned with conflicts that do not prevent the DataMove from being performed. The Warnings array contains a list of mappings where the data being moved is also mapped from another namespace. This array is subscripted by the name of the global or the global range.
Errors - Array returned with conflicts that prevent the DataMove from being performed.
The Errors array contains a list of mappings where a conflict prevents the data from being moved. This array is subscripted by the name of the global or the global range.

Example:
; Include the .INC file to get the definitions for the DataMove macros
#include %syDataMove
Set Properties("Flags")= $$$BitCheckActivate
Set Status=##Class(DataMove.API).MapGlobalsGenerate("TEST" .Properties,.Warnings,.Errors)
classmethod GetProperties(Name As %String, ByRef Properties As %String) as %Status [ Language = objectscript ]
Returns the current properties of the DataMove.

Arguments:
Name - Name of the DataMove.
Properties - Array of the DataMove properties as follows:
Properties("ExpandedState") - Expanded external format of the state.
Properties("JRNMBToApply") - Current # of MB of Journal file to process.
Properties("MaxMBPerMin") - Maximum # of MB the DataMove is allowed to move per Minute.
Properties("MBCopied") - Amount of MB copied.
Properties("MBToCopy") - Approximate amount of MB to copy.
Properties("State") - Current state of the move, see the table below.
Properties("StateExternal") - External format of the State.
Properties("Status") - %Status value of any errors which occur.
Properties("Stop") - Stop state of the DataMove.
$$$DMStopNone - Stop not signaled.
$$$DMStopNormal - Stop signaled by Activate().
$$$DMStopShutdown - Stop signaled by normal system shutdown. Data Move will resume on system restart.
$$$DMStopUser - Stop signaled by user StopCopy(). Copy can be restarted by StartCopy().
$$$DMStopForce - Stop signaled by Force(). Copy can be restarted by StartCopy().
$$$DMStopErrorRecoverable - Stop signaled by error. Copy can be restarted by StartCopy() Once the error is corrected.
$$$DMStopErrorUnRecoverable - Stop signaled by unrecoverable error. This may be due to several different reasons including journal errors. The only option here is to call Rollback() and then StartCopy() to restart the DataMove or Delete() to delete the DataMove.

Example:
%SYS>s x=##Class(DataMove.API).GetProperties("ABC1",.Properties)

%SYS>zwrite Properties
Properties("ExpandedState")="Copy/Crashed"
Properties("JRNMBToApply")=0
Properties("MaxMBPerMin")=0
Properties("MBCopied")=17030.67
Properties("MBToCopy")=20034.44
Properties("State")=6
Properties("StateExternal")="Copy"
Properties("Status")=1
Properties("Stop")=0
classmethod IsRunning(Name As %String, ByRef DMState As %Integer) as %Boolean [ Language = objectscript ]
Determines if a DataMove is running.

Arguments:
Name - Name of the DataMove.
DMState - Current state of the DataMove. Values are one of the states from the state table.
classmethod MapGlobalsCreate(Namespace As %String, Name As %String, ByRef Properties As %String) as %Status [ Language = objectscript ]
Creates a new global mapping for this namespace in the temporary storage area.
You can call this method one or more times, depending on the number of mappings you plan to include in this DataMove.
Arguments:
Namespace - The name of the namespace on which you want to perform the DataMove. Name - Name of a global to be mapped to a specific database.
Setting the Name argument to "A" specifies that this mapping affects the entire global ^A. Setting this argument to "A(5):A(10)" specifies that this mapping affects the range of the global with subscripts ^A(5) up to, but not including, ^A(10).
Properties - Array of properties needed for this mapping, in particular, the name of the database used for this mapping.
Setting the Properties argument to an array Properties where Properties("Database") is set to "USER2" specifies that the global (or range of the global) is to be mapped to the database USER2.

Examples:
Set Properties("Database")="DSTDB"
;Move ^X(100) up to but not including ^X(200) to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate("SALES","X(100):(200)",.Properties)
;Move the entire ^INFO global to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate("SALES","INFO",.Properties)
;Move all Data from the first subscript in ^BILLING (including the node ^BILLING itself) 
;up to but not including ^BILLING(100) to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate ("SALES","BILLING(BEGIN):(100)",.Properties)
;Move the entire global ^PROSPECT from its currently mapped database to DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsModify ("SALES","PROSPECT",.Properties)
;Move the entire global ^ORDERS from its currently mapped database 
;back to the default database for the namespace SALES
Set Status=##Class(DataMove.API).MapGlobalsDelete ("SALES","ORDERS")
classmethod MapGlobalsDelete(Namespace As %String, Name As %String) as %Status [ Language = objectscript ]
Delete an existing global mapping for this namespace in the temporary storage area.

Arguments:
Namespace - The name of the namespace which you want to delete the mapping from
Name - Name of the global mapping you wish to delete.

Examples:
;Move the entire global ^ORDERS from its currently mapped database 
;back to the default database for the namespace SALES
Set Status=##Class(DataMove.API).MapGlobalsDelete ("SALES","ORDERS")
;Move the global mapping ^REVENUE("AUGUST") from its currently mapped database 
;back to the default database for the namespace SALES
Set Status=##Class(DataMove.API).MapGlobalsDelete("SALES","REVENUE(""AUGUST"")")
classmethod MapGlobalsModify(Namespace As %String, Name As %String, ByRef Properties As %String) as %Status [ Language = objectscript ]
Modifies a global mapping for this namespace in the temporary storage area.

You can call this method one or more times, depending on the number of mappings you plan to include in this DataMove.
Arguments:
Namespace - The name of the namespace on which you want to perform the DataMove. Name - Name of a global to be mapped to a specific database.
Setting the Name argument to "A" specifies that this mapping affects the entire global ^A. Setting this argument to "A(5):A(10)" specifies that this mapping affects the range of the global with subscripts ^A(5) up to, but not including, ^A(10).
Properties - Array of properties needed for this mapping, in particular, the name of the database used for this mapping.
Setting the Properties argument to an array Properties where Properties("Database") is set to "USER2" specifies that the global (or range of the global) is to be mapped to the database USER2.

Examples:
Set Properties("Database")="DSTDB"
;Move the entire global ^PROSPECT from its currently mapped database to DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsModify ("SALES","PROSPECT",.Properties)
classmethod MapInitialize(Namespaces As %String) as %Status [ Language = objectscript ]
Initializes the temporary storage area for a new set of mapping edits.

Arguments:
Namespaces - Is a comma delimited list of namespaces on which you want to perform the DataMove. This method must be called before any edits are made and is valid only for the specified namespaces.

CAUTION: Calling this method deletes any existing edits in the temporary storage area, which is used for all mapping changes in all namespaces, including those made by the DataMove API and those made using the Global Mappings page of the Management Portal. (See Global Mappings in the "Configuring InterSystems IRIS" chapter of the System Administration Guide.) To prevent concurrent mapping changes from causing failures, you must ensure that there are no mapping changes for any namespace in progress when you call MapInitialize(), and that no other mapping changes are initiated until the DataMove process is complete.

Example:
;Initialize one namespace
Set Namespaces = "SALES"
Set Status = ##Class(DataMove.API).MapInitialize(Namespaces)
If '$$$ISOK(Status) Write !,$SYSTEM.Status.GetErrorText(Status)
;Initialize several namespaces
Set Namespaces = "SALES,MARKETING,INFO"
Set Status = ##Class(DataMove.API).MapInitialize(Namespaces)
If '$$$ISOK(Status) Write !,$SYSTEM.Status.GetErrorText(Status)
classmethod Modify(Name As %String, ByRef Properties As %String) as %Status [ Language = objectscript ]
Modify an existing or running DataMove.

Arguments:
Name - Name of the DataMove.
Properties - Array of properties to modify.
Properties("MaxMBPerMin") - Maximum number of MB per minute the DataMove operation is allowed to move to the destination database. Setting this to 0 allows the DataMove to run as fast as it can. This can be called on a running DataMove to change the rate.
classmethod Rollback(Name As %String) as %Status [ Language = objectscript ]
Rollback the DataMove to the $$$DMStateNotStarted State.

This method deletes any globals that have been copied to destination databases by the DataMove copy job. This method can be used to abort the DataMove or recover from an error in the copy process. StopCopy() must be run before calling this method. After doing the rollback, you can start over with StartCopy() or delete the DataMove with Delete().
Arguments:
Name - Name of the DataMove.
classmethod RollbackCopy(Name As %String, ByRef Warnings As %String, ByRef Errors As %String) as %Status [ Language = objectscript ]
Create a new DataMove which will copy the DataMove data back to its original location, and retore the old mappings.

This will create a DataMove called Name-ROLLBACK which when run will move the data copied in the DataMove back to its original source directories. The DataMove must be in the State $$$DMStateDeleteSrcGlobalsDone (all the source globals have been deleted by the DeleteSourceGlobals()method.) After the method finishes, you can then run StartCopy() and Activate() to move the data back to their original directories. The original DataMove will have its State set to $$$DMStateRollbackCopy with a stop state set to "StopFatal" so it can't be used again.
Arguments:
Name - Name of the DataMove which you want to rollback.
Warnings - See Generate() for description of the Warnings array returned.
Errors - See Generate() for description of the Errors array returned.
classmethod RollbackMappings(Name As %String) as %Status [ Language = objectscript ]
Rolls back the mappings to what they were before the new mappings were activated.

This will restore the systems mappings to the state before the DataMove was run. The DataMove must be in the State $$$DMStateNSPActivateDone. An example of using this would be if you started to test the application after the DataMove activates its mappings, and detected there was something wrong. After the mappings are restored to their previous value, the State will be set to $$$DMStateJrnApplyDone. From here you can call Rollback() (recommended), or StartCopy() and retry the activation.
WARNING: If the mappings are rolled back, any data which was created while the new mappings were in effect will not be able to be accessed.
Arguments:
Name - Name of the DataMove.
classmethod StartCopy(Name As %String) as %Status [ Language = objectscript ]
Start (or restart) the DataMove.

This method will start the DataMove, or restart the DataMove if it was previously stopped. As part of starting, it calls the methods Validate() and ValidateSizes(). Arguments:
Name - Name of the DataMove.
classmethod StopCopy(Name As %String) as %Status [ Language = objectscript ]
Stop the DataMove.

This method stops the DataMove copy background jobs, allowing you to gracefully stop the copy after it is in process. You can restart the DataMove with DataMove.Data.StartCopy(). If you wish to delete the DataMove after it is started, you should call Rollback(), and then Delete().
Arguments:
Name - Name of the DataMove.
classmethod Validate(Name As %String) as %Status [ Language = objectscript ]
Validate the DataMove before starting the copy.

Validating the DataMove involves looking at all of the specified mappings, checking the source and destination databases, and making sure the destination globals do not already exist. Any errors are reported in the status. Validate() is called as part of StartCopy(). You can use this method to validate the DataMove before you actually start the copy with StartCopy().
Arguments:
Name - Name of the DataMove.
classmethod ValidateSizes(Name As %String) as %Status [ Language = objectscript ]
Validate the size requirements of the DataMove.

Validating sizes for a DataMove involves determining the amount of data to be copied and ensuring enough space exists in the destination database. Any errors are reported in the status. ValidateSizes() is called as part of StartCopy(). You can use this method to validate the DataMove size requirements before you actually start the copy with StartCopy().
Arguments:
Name - Name of the DataMove.

Queries

query ListDMs(Names As %String = "*", Flags As %Integer = 0)
Selects RowCount As %Integer, Name As %String, State As %String, UpdateTime As %String, Stop As %String, JRNApplied As %String, JRNCUR As %String, JRNMBToApply As %String, JRNMinMB As %String, JRNFirst As %String, MBCopied As %String, MBToCopy As %String, PctCopied As %String, EndTime As %String, Status As %String, UserName As %String, ExpandedState As %String, JRNSize As %Integer, JRNCount As %Integer, SizeStart As %String, SizeEnd As %String, SizeTime As %String, CopyStart As %String, CopyEnd As %String, CopyTime As %String, FirstJRNStart As %String, FirstJRNEnd As %String, FirstJRNTime As %String, JRNCycleStart As %String, JRNCycleEnd As %String, JRNCycleTime As %String, JRNSets As %String, JRNSetsAvoided As %String, JRNSetsCopy As %String, JRNKills As %String, JRNKillsAvoided As %String, JRNKillsCopy As %String, JRNBitSets As %String, JRNBitSetsAvoided As %String, JRNBitSetsCopy As %String, JRNZKills As %String, JRNZKillsAvoided As %String, JRNZKillsCopy As %String, BlocksCopied As %Integer, MBRate As %String, EstFinishTime As %String, PidApplyJournals As %String, PidMoveGlobals As %String, PidCopy As %String, MaxMBPerMin As %String
List the DataMoves defined on the system.
query ListProcesses(Name As %String)
Selects RowCount As %Integer, JobNum As %String, Info As %String
List the DataMove processes currently running on the system.
query ListRanges(Name As %String, SrcDBs As %String = "*", DstDBs As %String = "*", Ranges As %String = "*", Flags=0)
Selects RowCount As %Integer, SrcDB As %String, DstDB As %String, RangeInternal As %String, Range As %String, State As %String, UpdateTimeExternal As %String, JRNCUR As %String, JRNMBToApply As %String, MBCopied As %String, MBToCopy As %String, PctCopied As %String, MBPerMin As %String, Status As %String, DstDir As %String, Flags As %String, PidCopy As %String, PidJrn As %String, SrcDir As %String, JRNSize As %Integer, JRNCount As %Integer, SizeStart As %String, SizeEnd As %String, SizeTime As %String, CopyStart As %String, CopyEnd As %String, CopyTime As %String, FirstJRNStart As %String, FirstJRNEnd As %String, FirstJRNTime As %String, JRNCycleStart As %String, JRNCycleEnd As %String, JRNCycleTime As %String, JRNSets As %String, JRNSetsAvoided As %String, JRNSetsCopy As %String, JRNKills As %String, JRNKillsAvoided As %String, JRNKillsCopy As %String, JRNBitSets As %String, JRNBitSetsAvoided As %String, JRNBitSetsCopy As %String, JRNZKills As %String, JRNZKillsAvoided As %String, JRNZKillsCopy As %String, LastUpdated As %String, LastUpdatedInternal As %String, BlocksCopied As %Integer, StartNode As %String, EndNode As %String
List all of the ranges of a DataMove.

Inherited Members

Inherited Methods (Including Private)

Feedback