Using DataMove with InterSystems IRIS

DataMove.API.MapInitialize(Name as %String, Namespaces As %String) As %Status

Initializes the temporary storage area for a new set of mapping edits.

Argument:

• Name is the name of the DataMove.

• Namespaces is a comma-separated list of the namespaces on which you want to perform the DataMove.

This method must be called before any edits are made, and is only valid for the specified namespaces. You may move data in multiple namespaces as part of the same DataMove.

Example:

Set Namespace = "SALES"
Set Status = ##Class(DataMove.API).MapInitialize("DMName",Namespace)
If '$$$ISOK(Status) Write !,$SYSTEM.Status.GetErrorText(Status)

Example:

Set Namespaces = "SALES,PROSPECTS"
Set Status=##Class(DataMove.API).MapInitialize("DMName",Namespaces)
If $$$ISOK(Status) Write !,$SYSTEM.Status.GetErrorText(Status)

DataMove.API.MapGlobalsCreate(Name as %String, Namespace As %String, GblName As %String, ByRef Properties As %String) As %Status

Creates a new global mapping for this namespace in the temporary storage area. You can call this method one or more times, depending on the number of mappings you plan to include in this DataMove.

Arguments:

• Name is the name of the DataMove.

• Namespace is the namespace on which you want to perform the DataMove.

• GblName is the name of a global to be mapped to a specific database.

• Properties is an array of properties needed for this mapping, in particular, the name of the database used for this mapping.

Setting the GblName argument to “A” specifies that this mapping affects the entire global ^A. Setting this argument to “A(5):A(10)” specifies that this mapping affects the range of the global with subscripts ^A(5) up to, but not including, ^A(10).

Setting the Properties argument to an array Properties where Properties("Database") is set to “USER2” specifies that the global (or range of a global) is to be mapped to the database USER2.

The MapGlobalsModify() and API.MapGlobalsDelete() methods can be used to modify or delete mappings. See the Class Reference for more information.

Examples:

Set Properties("Database")="DSTDB"

;Move ^X(100) up to but not including ^X(200) to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate("DMName","SALES","X(100):(200)",.Properties)

;Move the entire ^INFO global to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate("DMName","SALES","INFO",.Properties)

;Move all Data from the first subscript in ^BILLING 
;(including the node ^BILLING itself) up to but not including ^BILLING(100) 
;to database DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsCreate("DMName","SALES","BILLING(BEGIN):(100)",.Properties)

;Move the entire global ^PROSPECT 
;from its currently mapped database to DSTDB
Set Status = ##Class(DataMove.API).MapGlobalsModify("DMName","SALES","PROSPECT",.Properties)

;Move the entire global ^ORDERS from its currently mapped database 
;back to the default database for the namespace SALES
Set Status=##Class(DataMove.API).MapGlobalsDelete("DMName","SALES","ORDERS")

For more information on defining global ranges, see Global Mappings.

DataMove.API.Generate(Name As %String, ByRef Properties As %String, ByRef Warnings As %String, ByRef Errors As %String) As Status

Creates a new DataMove based on map edits in the temporary storage area.

Arguments:

• Name is the name of the DataMove to be created.

• Properties is an array of optional properties to be used to create the DataMove.

• Warnings is an array returned with conflicts that do not prevent the DataMove from being performed.

• Errors is an array returned with conflicts that do prevent the DataMove from being performed.

Properties is an array passed in as the Properties argument:

• Properties("MaxMBPerMin") optionally specifies the maximum number of MB per minute the DataMove Operation is allowed to move to the destination database. Setting this to 0 allows the DataMove to run as fast as it can. If not passed, it uses the system default.

• Properties("Description") optionally provides a description of the DataMove to be performed.

• Properties("Flags") optionally provides any flags describing the DataMove operation, such as:

  • $$$BitNoSrcJournal, allow copying of non-journaled databases.

  • $$$BitNoBatchMode, run data copy without batch mode.

  • $$$BitCheckActivate, call the user-supplied routine $$CheckActivate^ZDATAMOVE() to check the application status before activating the mapping changes.

• Properties("LogFile") optionally specifies a log file name, if other than the default.

The Warnings array contains a list of mappings where the data being moved is also mapped from another namespace. This array is subscripted by the name of the global or the global range.

The Errors array contains a list of mappings where a conflict prevents the data from being moved. This array is subscripted by the name of the global or the global range.

Example:

Set Properties("Flags")= $$$BitCheckActivate
Set Status=##Class(DataMove.API).Generate("TEST",.Properties,.Warnings,.Errors)

For more information on database journaling, see Journaling.

For more information on running a process in batch mode, see Process Management.

DataMove.API.StartCopy(Name As %String) As %Status

Starts the DataMove copy, which handles the actual moving of data.

Argument:

• Name is the name of the DataMove.

This method starts the initial DataMove copy as a set of background jobs which copy the individual ranges from the source databases to the destination databases while at the same time journal transactions are applied to the destination databases for data which changes as it is being copied.

Before the copy actually starts, StartCopy() calls Validate() and ValidateSizes() to validate the DataMove. If the validation returns an error, the StartCopy() will return the validation error and will not start the DataMove. The user can then correct the error, and call StartCopy() again.

DataMove.API.GetProperties(Name As %String, ByRef Properties as %String) As %Status

Returns the current properties of the DataMove.

Arguments:

• Name is the name of the DataMove.

• Properties is an array of the DataMove properties as follows:

  • Properties("ExpandedState") – Expanded external format of the state.

  • Properties("JRNMBToApply") – Current number of MB of Journal file to process=0.

  • Properties("MaxMBPerMin") – Maximum number of MB per minute the DataMove is allowed to move.

  • Properties("MBCopied") – Amount of MB copied. At the end of a DataMove when all the data has been copied, and before Activate() is called, the amount of MB copied may not equal the amount of MBToCopy on a system where data in the source database is being updated while the DataMove is running.

  • Properties("MBToCopy") – Approximate amount of MB to copy.

  • Properties("State") – Current state of the move; see the table below.

  • Properties("StateExternal") – External format of the State property.

  • Properties("Status") – %Status value of any errors which occur.

  • Properties("Stop") – Stop state of the DataMove:

    • $$$DMStopNone – Stop not signaled.

    • $$$DMStopNormal – Stop signaled when called by the Activate() method.

    • $$$DMStopShutdown – Stop signaled by normal system shutdown. DataMove will resume on system restart if the parameter to restart on system startup is set.

    • $$$DMStopUser – Stop signaled by user calling the method StopCopy(). Copy can be restarted by StartCopy().

    • $$$DMStopForce – Stop signaled by Force(). Copy can be restarted by StartCopy().

    • $$$DMStopErrorRecoverable – Stop signaled by error. Copy can be restarted by StartCopy() once the error is corrected.

    • $$$DMStopErrorUnRecoverable – Stop signaled by unrecoverable error. This may be due to several different reasons including journal errors. The only option here is to call Rollback() and then StartCopy() to restart the DataMove, or Delete() to delete the DataMove.

Example:

%SYS>Set x=##Class(DataMove.API).GetProperties("ABC1",.Properties)

%SYS>Zwrite Properties
Properties("JRNMBToApply")=0
Properties("MaxMBPerMin")=1200
Properties("MBCopied")=17030.67
Properties("MBToCopy")=20034.44
Properties("State")=6
Properties("StateExternal")="Copy"
Properties("Status")=1
Properties("Stop")=0

Setting DataMove Default Parameters

DataMove default system parameters allow the user to specify what happens when the system restarts, and can also limit the amount of data which can be moved per minute. By default, a running DataMove will copy data as fast as it can. However, this can affect other user processes on the system, as well as affect the latency of a mirrored system.

%SYS>d ^DATAMOVE
1) Monitor DataMove
2) Edit DataMove settings
3) Exit
Option? 2

Restart DataMove after normal system shutdown? Yes => Yes
Restart DataMove after system crash? Yes => Yes

Setting the maximum number of MB DataMove is allowed to copy per minute 
to 0 means that DataMove will copy the data as fast as it can. Changing 
this setting will apply the new value to future DataMoves, and optionally 
any currently running DataMoves. You can also change this setting through 
the 'Metrics' display screen for a running DataMove without changing the 
system default.
Maximum number of MB to copy per minute? 0 => 1200
Do you want to apply this change to existing Data Moves? Yes => Yes
Confirm changes? Yes => Yes
DataMove settings updated

1) Monitor DataMove
2) Edit DataMove settings
3) Exit

Monitoring DataMove

Once you have called the Generate() method, you can use ^DATAMOVE to examine the state of the move, and which ranges are set to move.

%SYS>d ^DATAMOVE
1) Monitor DataMove
2) Edit DataMove settings
3) Exit
Option? 1

This is an example of what is displayed after Generate() and StartCopy() have been called. It shows the accumulated statistics for all the ranges in the DataMove.

These fields show the following information:

• Jrn Cycle Time – Amount of time it took for the journal scanner/apply to reach the end of the current journal file for the last time it reached the end of the journal.

• First Jrn Time – Amount of time it took for the journal scanner/apply to reach the end of the current journal file for the first time.

• CopyTime – Amount of time it took to copy the global.

• Size Time– Amount of time it took for the global size calculation.

• Done – Amount of each operation found in the journal file that it applied to the destination database.

• Avoid – Amount of each operation found in the journal file which could be skipped. Operations can be skipped if the Copy has not yet started, or the operation occurs in a piece of the global which has not yet been copied to the destination.

• Copy – Amount of each operation which were applied to the destination database while the Copy was running.

• Journal Start – Journal file and offset where the DataMove started.

• Journal Current – Current journal file and offset.

• Max MB/Min – Maximum MB per minute the DataMove is allowed to copy. This is set for the DataMove in one of three ways:

  • DataMove default from ^DATAMOVE.

  • The Metrics screen shown below.

  • The Modify() API.

• MB/Min – MB per minute the DataMove has copied over the lifetime of the DataMove.

• Blks Copied – Actual number of database blocks copied and the % which were prefetched

• JRN Count/Size – Number of transactions and total size for this DataMove which were found in the current journal file and the previous journal file before the StartCopy() method was called.

• Pid Move/Jrn – Process ID of the two processes handling the copy and dejournal.

• Pid Copy – Process ID of the Master DataMove process.

When you select the range option, you can see that there are 5 ranges of globals which are being moved.

You can examine the detail of each of the ranges in the Range Detail screen. These are described above, but apply only to the selected range.

The Jobs display show which jobs are operating on which pieces of data.

Here you can see that job 84 is currently copying the data in Range X(40000000), job 73 is scanning the journal for the same range and applying transactions it finds in this range. The "Master Move" job 71 controls all processes which are still copying data. The "Master Journal" job 72 controls all the processes which are scanning or applying journals. And the "Master Copy" job accumulates statistics and determines when the Activate() method can be called. The job number, Commands, Globals, State, and PID here correspond to the same information found in JOBEXAM or the Management Portal process display.

The Metrics display shows the activity on the system, and how much of it is related to the running DataMove.

In this example we can see the DataMove is running with the Max allowed MB Copied/Min set to 0 (Unlimited). We also see that it is taking up the majority of global references and journal entries, as well as causing the mirror latency to increase.

We can restrict the DataMove to a slower rate as follows:

And now we see the new rate and much lower metrics.

DataMove.API.Activate(Name As %String, Display as %Boolean, Timeout As %Integer = 120) As %Status

Finishes the DataMove and activates the namespace mapping changes.

Arguments:

• Name is the name of the DataMove.

• Display should be set to 1 if you want to see progress messages.

• Timeout is number of seconds to wait for the DataMove to finish running and apply journals operation before proceeding.

This method stops the DataMove background jobs, finishes processing any journal files, writes the mapping changes to the CPF, and activates the mapping changes. It momentarily sets switch 10 to prevent other processes from interfering with the execution.

If $$$BitCheckActivate is set, the method will call a user-supplied routine $$CheckActivate^ZDATAMOVE(), if it exists, to execute before continuing. If $$CheckActivate^ZDATAMOVE() does not return 1, the method will quit without activating the new mappings, leaving the DataMove running.

For more information on switch 10, see Using Switches.

DataMove.API.DeleteSourceGlobals(Name As %String) As %Status

Deletes the globals from the source directory that have been copied in the DataMove.

Argument:

• Name is the name of the DataMove.

This method deletes all globals in the source database that have been copied to the destination database. Several processes may be created to delete the source globals.

DataMove.API.Finish(Name As %String) As %Status

Completes the DataMove process.

Argument:

• Name is the name of the DataMove.

This method writes a success or failure message to the log file, closes the log file, and sets the State property of the DataMove to $$$DMStateDone. It also copies the log file into the file DataMove.log, which is a record of all the DataMoves performed on the system.

DataMove.API.Delete(Name As %String) As %Status

Cleans up the DataMove process.

Argument:

• Name is the name of the DataMove.

This method deletes the DataMove and cleans up any temporary storage.

If the DataMove operation you want to delete has been started and then stopped, you should first call the Rollback() method to rollback any of the data which had been moved.

DataMove.API.Validate(Name As %String) As %Status

Validates the DataMove.

Argument:

• Name is the name of the DataMove.

Validating the DataMove involves looking at all of the specified mappings, checking the source and destination databases, and making sure the destination globals do not already exist. Any errors are reported in the status.

Validate() is called as part of StartCopy(). You can use this method to validate the DataMove before you actually start the copy with StartCopy()

DataMove.API.Modify(Name as %String, byref Properties as %String) as %Status

Modifies an existing or running DataMove.

Arguments:

• Name is the name of the DataMove.

• Properties is an array of properties to modify, which may include:

  • Properties("MaxMBPerMin") - Maximum number of MB per minute the DataMove operation is allowed to move to the destination database. Setting this to 0 allows the DataMove to run as fast as it can. This can be called on a running DataMove to change the rate.

DataMove.API.ValidateSizes(Name As %String) As %Status

Makes sure sufficient space exists for the data specified by the DataMove to be copied.

Argument:

• Name is the name of the DataMove .

Validating sizes for a DataMove involves determining the amount of data to be copied and ensuring enough space exists in the destination database. Any errors are reported in the status.

ValidateSizes() is called as part of StartCopy(). You can use this method to validate the DataMove size requirements before you actually start the copy with StartCopy()

DataMove.API.StopCopy(Name As %String) As %Status

Stops the DataMove copy job.

Argument:

• Name is the name of the DataMove.

This method stops the DataMove copy background jobs, allowing you to gracefully stop the copy after it is in process. You can restart the DataMove with StartCopy().

DataMove.API.Rollback(Name As %String) As %Status

Rolls back the DataMove.

Argument:

• Name is the name of the DataMove.

This method deletes any globals that have been copied to destination databases by the DataMove copy job. This method can be used to abort the DataMove or recover from an error in the copy process.

StopCopy() must be run before calling this method.

After doing the rollback, you can start over with StartCopy() or delete the DataMove with Delete().

DataMove.API.RollbackMappings(Name As %String)

Rolls back the DataMove mappings.

Argument:

• Name is the name of the DataMove.

This will restore the systems mappings to the state before the DataMove Name was run. The DataMove Name must be in the State $$$DMStateNSPActivateDone. An example of using this would be if you started to test the application after the DataMove activates its mappings, and detected there was something wrong. After the mappings are restored to their previous value, the State will be set to $$$DMStateJrnApplyDone. From here you can call Rollback() (recommended), or StartCopy() and retry the activation.

DataMove.API.RollbackCopy(Name As %String, Warnings as %String, Errors as %String)

Rolls back a copy of the DataMove.

Arguments:

• Name is the name of the DataMove.

• Warnings is an array returned with conflicts that do not prevent the DataMove from being performed.

• Errors is an array returned with conflicts that do prevent the DataMove from being performed.

This will create a DataMove called Name-ROLLBACK which when run will move the data copied in the DataMove Name back to its original source directories. The DataMove Name must be in the State $$$DMStateDeleteSrcGlobalsDone (all the source globals have been deleted by the DeleteSourceGlobals() method. After the method finishes, you can then run StartCopy() and Activate() to move the data back to their original directories. The DataMove Name will have its State set to $$$DMStateRollbackCopy.

Query DataMove.API.ListDMs(Names As %String, Flags as %Integer=0) As %Status

Query which returns a list of all DataMoves and their properties.

Arguments:

• Names is a commas separated list of DataMoves.

• Flags –

  • 0 = Return current information in the DataMove record

  • 1 = Return information summed across all the ranges

Query DataMove.API.ListRanges(Name As %String, SrcDBs As %String = "*", DstDBs As %String = "*", Ranges As %String = "*", Flags = 0) As %Status

Query which returns a list of all the DataMove ranges and their properties.

Arguments:

• Names is the name of the DataMove.

• SrcDBs – comma separated list of source databases

• DstDBs – Comma separated list of destination databases

• Ranges – Comma separated list of ranges

• Flags –

  • 0 = Return formatted times

  • 1 = Return times as $h UTC times

Query DataMove.API.ListProcesses(Name As %String) As %Status

Query which returns a list of all the DataMove processes.

Argument:

• Name is the name of the DataMove.