Skip to main content

Using Cube Synchronization

This topic describes how to use the cube synchronization feature to keep cubes current.

How Cube Synchronization Works

This section describes briefly how cube synchronization works. Internally, this feature uses two globals: ^OBJ.DSTIME and ^DeepSee.Update.

First, it is necessary to perform an initial build of the cube.

When InterSystems IRIS® data platform detects a change within the source table used by a cube, it adds entries to the ^OBJ.DSTIME global. These entries are to indicate which IDs have been added, changed, or deleted.

When you synchronize the cube (via %SynchronizeCube(), described later in this page), InterSystems IRIS first reads the ^OBJ.DSTIME global and uses it to update the ^DeepSee.Update global. After it adds an ID to the ^DeepSee.Update global, InterSystems IRIS removes the same ID from the ^OBJ.DSTIME global. (Note that in previous versions, the cube synchronization feature used only one global; the newer system prevents a race condition.)

Then InterSystems IRIS uses the ^DeepSee.Update global and updates the fact and dimension tables of the cube, thus bringing the cube up to date.

The following figure shows the overall flow:

1. Perform initial build of cube, 2. ^OBJ.DSTIME updated, 3. Data moved to ^DeepSee.Update, 3. Cube tables updated.

The subsections discuss the following details:

When Cube Synchronization Is Possible

You can use the cube synchronization feature in scenarios where all the following items are true:

  • The base class for the cube is a persistent class (but is not a linked table).

  • The changed record is a record in that class.

When Cube Synchronization Is Not Possible

You cannot use the cube synchronization feature in the following scenarios:

  • The base class for the cube is a data connector. (See Defining Data Connectors.)

  • The base class for the cube is a linked table. (See The Link Table Wizard).

  • The changed record is not in the extent of the base class used by the cube. That is, the changed record belongs to another table.

In these scenarios, the cube synchronization feature cannot detect the change, and your application must update the cube manually as described in Updating Cubes Manually.

Also, cube synchronization does not affect age dimensions (that is, dimensions whose Dimension type is age).

Cube Synchronization in a Mirrored Environment

If you use Business Intelligence on a mirror server, note that the ^OBJ.DSTIME global is part of the application data and should be mirrored (if it mapped to a different database, for example, that database should be mirrored). The ^DeepSee.Update global is generated by Business Intelligence code and thus is present only in the database that contains the cube definitions and data.

^OBJ.DSTIME resides in the mirrored application data DB. ^DeepSee.Update resides in the non-mirrored cube definition DB.

Important:

On the mirror server, the databases that store the ^OBJ.DSTIME and ^DeepSee.Update globals must be read/write. Note that you can store both of these globals in the same database, although the above figure shows them in separate databases.

For a discussion of using Business Intelligence on a mirror server, see Recommended Architecture.

Structure of the Cube Synchronization Globals

This section describes the structure of the cube synchronization globals. You do not need this information to use cube synchronization; this information is provided in case you wish to use these globals for other purposes.

^OBJ.DSTIME

The ^OBJ.DSTIME global has a different form depending on whether DSINTERVAL is set.

If DSINTERVAL is not set, this global has nodes like the following:

Node Value
^OBJ.DSTIME(class,increment,ID) where class is the full package and class name of the source class, increment is 0, and ID is the ID of the new, changed, or deleted record in the given class One of the following values:

  • 0 (which means that the record was changed)

  • 1 (which means that the record was added)

  • 2 (which means that the record was deleted)

Note that it is possible to manually delete a fact from a fact table without deleting the corresponding record from the source class by using the %SetDSTimeIndex() method.

If DSINTERVAL is set, this global has nodes like the following:

Node Value
^OBJ.DSTIME(class,timestamp,ID) where class and ID are the same as in the other scenario, and timestamp is the number of seconds since midnight on December 31st, 1840 Same as in the other scenario

The system removes unneeded entries from the ^OBJ.DSTIME global when you synchronize or rebuild a cube.

^DeepSee.Update

The ^DeepSee.Update global has nodes as follows:

Node Value
^DeepSee.Update Integer that indicates the next value of increment to use
^DeepSee.Update(class,increment,ID) where class is the full package and class name of the source class, increment is 0 or a positive integer, and ID is the ID of the new, changed, or deleted record in the given class. Each time you synchronize cubes, the system new nodes to this global, using the next highest integer for increment. See the example. Same as in the ^OBJ.DSTIME global
^DeepSee.Update("cubes",cube,"dstime") where cube is the logical name of a cube Integer that indicates the next value of increment to use when creating nodes in this global to record changes for the given cube.
^DeepSee.Update("cubes",cube,"lastDataUpdate") where cube is the logical name of a cube The date and time (in $H format) when this cube was last synchronized.

Here is an example:

^DeepSee.Update=3
^DeepSee.Update("DeepSee.Study.Patient",0,1)=0
^DeepSee.Update("DeepSee.Study.Patient",0,2)=0
^DeepSee.Update("DeepSee.Study.Patient",0,100)=0
^DeepSee.Update("DeepSee.Study.Patient",1,1)=2
^DeepSee.Update("DeepSee.Study.Patient",1,120)=0
^DeepSee.Update("DeepSee.Study.Patient",2,42)=0
^DeepSee.Update("DeepSee.Study.Patient",2,43)=0
^DeepSee.Update("DeepSee.Study.Patient",2,50)=0
^DeepSee.Update("DeepSee.Study.Patient",2,57)=0
^DeepSee.Update("cubes","PATIENTS","dstime")=3
^DeepSee.Update("cubes","PATIENTS","lastDataUpdate")="64211,63222.68"

The nodes under ^DeepSee.Update("DeepSee.Study.Patient",0) represent the first set of changes, the nodes under ^DeepSee.Update("DeepSee.Study.Patient",1 represent the second set of changes, and so on.

InterSystems IRIS does not automatically remove nodes from ^DeepSee.Update global. For information on purging this global; see Purging DSTIME.

Enabling Cube Synchronization

Before you can synchronize a cube, you must enable the cube synchronization feature for that cube. To do so:

  1. Make sure that cube synchronization is possible in your scenario. See When Cube Synchronization Is Possible, earlier in this page.

  2. Add the DSTIME parameter to the base class used by that cube, as follows:

    Parameter DSTIME = "value";

    The DSTIME parameter accepts one of three strings as its value. When "AUTO" is specified, the ^OBJ.DSTIME global will receive an update for every modification of a record. This means that when you invoke the %SynchronizeCube() method (as described in the section below), all changes will be transcribed to ^DeepSee.Update and then synchronized with the corresponding cube. When DSTIME is set to "MANUAL", automatic journaling of changes to records of that base class will be disabled. The value "CONDITIONAL" allows you to specify the conditions under which ^OBJ.DSTIME will log changes for synchronization dynamically by specifying a DSCONDITION parameter for the class (see below).

  3. If you set the DSTIME parameter to "CONDITIONAL", add the following parameter to the base class:

    Parameter DSCONDITION = expression

    When the expression provided evaluates as TRUE, ^OBJ.DSTIME will receive updates for records of the given class for synchronization. Note that although DSCONDITION is an expression to be executed at runtime, it is not necessary to specify its parameter type as COSEXPRESSION explicitly, as is usually the caseOpens in a new tab.

  4. Optionally, you may also add the following parameter to the base class:

    Parameter DSINTERVAL = 5;

    This parameter primarily affects how entries are stored in the ^OBJ.DSTIME global; see Structure of the Cube Synchronization Globals. The form of the ^OBJ.DSTIME global has no effect on the behavior of the cube synchronization mechanism.

  5. Recompile the base class and all cube classes that use it.

  6. Rebuild these cubes.

Clearing the ^OBJ.DSTIME Global

This section describes how to clear the ^OBJ.DSTIME global. In some cases, you might want to periodically clear the ^OBJ.DSTIME global. For example, if you are not using cubes in Business Intelligence, you may want to clear the ^OBJ.DSTIME global to free up space.

You can set up a task in the Task Manager to periodically clear the ^OBJ.DSTIME global. To do so, create a new task with an OnTask() method such as the following:

Method OnTask() As %Status
{
  set classname=$ORDER(^OBJ.DSTIME(""))
  while (classname="") {
     
    //check to see if this classname is contained in ^DeepSee.Cubes("classes")
    set test=$DATA(^DeepSee.Cubes("classes",classname))
     
    if (test'=1) {
        kill ^OBJ.DSTIME(classname)
     
    }
    set classname=$ORDER(^OBJ.DSTIME(classname))
  
  }
  
  q $$$OK
}

This task clears ^OBJ.DSTIME entries if they aren’t being used by Business Intelligence cubes. Use the Task Schedule Wizard to schedule the task to run as often as necessary.

Using %SynchronizeCube()

Note:

Before you can synchronize a cube, follow the steps in Enabling Cube Synchronization, earlier in this page.

To synchronize a cube programmatically (that is, without the Cube Manager), call the %SynchronizeCube() method of the %DeepSee.UtilsOpens in a new tab class, which has the following signature:

classmethod %SynchronizeCube(pCubeName As %String, pVerbose As %Boolean = 1) as %Status

For the specified cube (pCubeName), this method finds and applies all changes from the source data that have been made since the last call to this method.

If pVerbose is true, the method writes status information to the console. For additional arguments for this method, see the class referenceOpens in a new tab.

You can call %SynchronizeCube() in either of the following ways:

  • Call the method from the part of your code that changes the data in the base class.

    This is the approach used in the Patients sample.

  • Periodically call %SynchronizeCube() as a recurring task.

If %SynchronizeCube() displays the message No changes detected, this can indicate that you had not previously rebuilt the cube.

Purging DSTIME

For historical reasons and for convenience, the phrase purging DSTIME refers to purging the older entries from the ^OBJ.DSTIME global. It is necessary to purge this global periodically because it can become quite large.

To purge DSTIME for a given cube, do the following:

  1. Call the REST API /Data/GetDSTIME. See GET /Data/GetDSTIME. Pass, as an argument, the full name of the source class of the cube.

    This REST call returns the last ^OBJ.DSTIME timestamp processed for that source class on a given server. In the case of an async mirror setup, the timestamp retrieved from this REST service will be the most recent timestamp that can safely be purged on the primary production server.

  2. Using the returned timestamp as an argument, call the %PurgeUpdateBuffer() method of %DeepSee.UtilsOpens in a new tab so that you purge ^OBJ.DSTIME up to but not including the timestamp processed on the remote server. The default behavior for this method is to increment the top node of the local ^OBJ.DSTIME so that every purge will provide a new sync point to be propagated to the Business Intelligence server.

Other Options

This section discusses other options that are more advanced or less common:

Using DSTIME=MANUAL

Instead of letting the system automatically update the ^OBJ.DSTIME global, you can update this global at times that you choose. To do so:

  1. Specify DSTIME as "MANUAL" rather than "AUTO".

  2. Then within your application, call the method %SetDSTimeIndex() of the class %DeepSee.UtilsOpens in a new tab whenever you add, change, or delete objects of the class, or when you want to update the ^OBJ.DSTIME global.

    This method has the following signature:

    ClassMethod %SetDSTimeIndex(pClassName As %String, 
                            pObjectId As %String, 
                            pAction As %Integer,
                            pInterval As %Integer = 0)

    Where:

    • pClassName is the full package and class name of the object that you have added, changed, or deleted.

    • pObjectId is the object ID for that object.

    • pAction is 0 if you updated the object, 1 if you added it, or 2 if you deleted it or want to delete the corresponding fact from the fact table without deleting the object. The value of pAction is used as the value of the resulting ^OBJ.DSTIME node. Note that facts are removed from a cube during synchronization if the corresponding record does not exist in the source class, or if a value of 2 is specified for pAction.

    • pInterval is an optional integer. If you specify this as a positive integer, the system uses time stamp subscripts in the ^OBJ.DSTIME and ^DeepSee.Update globals. See the discussion of the DSINTERVAL parameter in Enable Cube Synchronization.

Then, when you want to update a given cube, call the %SynchronizeCube() method of the %DeepSee.UtilsOpens in a new tab class, as described previously.

Examples

The Patients sample includes utility methods that change data and that use either synchronization or manual updates as appropriate. To try these methods, you can use a dashboard provided with this sample:

  1. Open the User Portal in the namespace where you installed the samples.

  2. Click the dashboard Real Time Updates.

  3. Click the buttons in the upper left area. Each of these executes a KPI action that executes a method to randomly change data in this sample. The action launches the method via JOB, which starts a background process.

    • Add Patients adds patients.

      This action calls a method that adds 100 patients and calls %SynchronizeCube() after adding each patient.

    • Change Patient Groups changes the patient group assignment for some patients.

      This action calls a method that randomly changes the patient group assignment for some percentage of patients and calls %SynchronizeCube() after each change.

    • Delete Some Patients deletes some patients.

      This action calls a method that deletes 1 percent of the patients and calls %SynchronizeCube() after each deletion.

    • Change Favorite Colors changes the favorite color for some patients.

      This action calls a method that randomly changes the favorite color for some percentage of the patients. In this case, the changed data is stored in the BI_Study.PatientDetails table, which is not the base table for the Patients cube. Hence it is necessary to use %ProcessFact() instead of %SynchronizeCube().

      This block of code executes an SQL query to return all patients who are affected by the change to the data. It then iterates through those patients and updates the Patients cube for each of them.

    • Add Encounters adds encounters for some patients.

      This action calls a method that includes logic similar to that for BI.Study.PatientDetails; see the previous item.

    • Change Doctor Groups changes the doctor group assignment for some of the primary care physicians.

      This action calls a method that includes logic similar to that for BI.Study.PatientDetails.

Tip:

These methods write log details to the global ^DeepSee.Study.Log. For example:

^DeepSee.Study.Log(1)="13 May 2011 05:29:37PM Adding patients..."
^DeepSee.Study.Log(2)="13 May 2011 05:29:38PM Current patient count is 10200"

See Also

Next sectionUsing the Cube Manager
Previous sectionKeeping the Cubes Current
FeedbackOpens in a new tab