Keeping the Cubes Current

Cube Updates and Related Cubes

For any kind of update, whenever you have cube-to-cube relationships, it is necessary to update the cubes in a specific order. In particular, update the independent cube first. Then update any cubes that depend on it. To do this, you can use the Cube Manager, which traverses the relationships and determines the correct update order.

Or you can write and use a utility method or routine that builds your DeepSee cubes in the appropriate order.

Cube Updates and the Result Cache

For any cube that uses more than 512,000 records (by default), DeepSee maintains and uses a result cache. For any combination of update techniques and tools, you should also carefully consider the frequency of cube updates, because any update could invalidate parts of the result cache.

For large data sets, DeepSee maintains and uses a result cache for each cube as follows: Each time a user executes a query (via the Analyzer for example), DeepSee caches the results for that query. The next time any user runs that query, DeepSee checks to see if the cache is still valid. If so, DeepSee then uses the cached values. Otherwise, DeepSee re-executes the query, uses the new values, and caches the new values. The net effect is that performance improves over time as more users run more queries.

When you update a cube in any way, parts of the result cache are considered invalid and are cleared. The details depend upon options in the cube definition (see “Cache Buckets and Fact Order,” earlier in this book). Therefore, it is not generally desirable to update constantly.

When Cube Synchronization Is Possible

You can use the cube synchronization feature in scenarios where all the following items are true:

• The base class for the cube is a persistent class (but is not a linked table).

• The changed record is a record in that class.

When Cube Synchronization Is Not Possible

You cannot use the cube synchronization feature in the following scenarios:

• The base class for the cube is a data connector. (See the chapter “Defining Data Connectors.”)

• The base class for the cube is a linked table. (See “Using the Caché SQL Gateway” in Using Caché SQL.)

• The changed record is not in the extent of the base class used by the cube. That is, the changed record belongs to another table.

In these scenarios, the cube synchronization feature cannot detect the change, and your application must update the cube manually as described in “Updating Cubes Manually.”

Also, cube synchronization does not affect age dimensions (that is, dimensions whose Dimension type is age).

Cube Synchronization in a Mirrored Environment

If you use DeepSee on a mirror server, note that the ^OBJ.DSTIME global is part of the application data and should be mirrored (if it mapped to a different database, for example, that database should be mirrored). The ^DeepSee.Update global is generated by DeepSee code and thus is present only in the database that contains the cube definitions and data.

For a discussion of using DeepSee on a mirror server, see “Recommended Architecture” in the first chapter of this book.

Structure of the Cube Synchronization Globals

This section describes the structure of the cube synchronization globals. You do not need this information to use cube synchronization; this information is provided in case you wish to use these globals for other purposes.

^DeepSee.Update

The ^DeepSee.Update global has nodes as follows:

Node	Value
^DeepSee.Update	Integer that indicates the next value of increment to use
^DeepSee.Update(class,increment,ID) where class is the full package and class name of the source class, increment is 0 or a positive integer, and ID is the ID of the new, changed, or deleted record in the given class. Each time you synchronize cubes, DeepSee new nodes to this global, using the next highest integer for increment. See the example.	Same as in the ^OBJ.DSTIME global
^DeepSee.Update("cubes",cube,"dstime") where cube is the logical name of a cube	Integer that indicates the next value of increment to use when creating nodes in this global to record changes for the given cube.
^DeepSee.Update("cubes",cube,"lastDataUpdate") where cube is the logical name of a cube	The date and time (in $H format) when this cube was last synchronized.

Here is an example:

^DeepSee.Update=3
^DeepSee.Update("DeepSee.Study.Patient",0,1)=0
^DeepSee.Update("DeepSee.Study.Patient",0,2)=0
^DeepSee.Update("DeepSee.Study.Patient",0,100)=0
^DeepSee.Update("DeepSee.Study.Patient",1,1)=2
^DeepSee.Update("DeepSee.Study.Patient",1,120)=0
^DeepSee.Update("DeepSee.Study.Patient",2,42)=0
^DeepSee.Update("DeepSee.Study.Patient",2,43)=0
^DeepSee.Update("DeepSee.Study.Patient",2,50)=0
^DeepSee.Update("DeepSee.Study.Patient",2,57)=0
^DeepSee.Update("cubes","PATIENTS","dstime")=3
^DeepSee.Update("cubes","PATIENTS","lastDataUpdate")="64211,63222.68"

The nodes under ^DeepSee.Update("DeepSee.Study.Patient",0) represent the first set of changes, the nodes under ^DeepSee.Update("DeepSee.Study.Patient",1 represent the second set of changes, and so on.

DeepSee does not automatically remove nodes from ^DeepSee.Update global. For information on purging this global; see “Purging DSTIME.”

Introduction to the Cube Manager

The Cube Manager enables you to define the cube registry, which contains information about the cubes in the current namespace. In particular, it contains information about how they are to be built, synchronized, or both.

The cube registry defines a set of cube groups. A cube group is a collection of cubes that need to be updated together, either because they are related or because you have chosen to update them together. When you first access the Cube Manager, it displays an initial set of cube groups. Each initial cube group is either a single cube or a set of cubes that are related to each other (and thus must be updated as a group). You can merge these initial cube groups together as wanted. You cannot, however, break up any of the initial cube groups.

Each cube group is initially unregistered, which means that it is not included in the cube registry. After you register a cube group (thus placing it into the registry), you define an update plan for it. The Cube Manager creates automatic tasks that use these update plans. See the next section for details.

Introduction to Update Plans

The update plan for a cube group specifies how and when the cubes are to be updated. Each group has a default plan, which you can modify. You can also specify different update plans for specific cubes in the group. In both cases, the plan choices are as follows:

• Build and Synch — Rebuild periodically, once a week by default. Also synchronize periodically, once a day by default.

  This option is not supported for a given cube unless that cube supports synchronization (as described earlier in this chapter).

• Build Only — Rebuild periodically, once a week by default.

• Synch Only — Synchronize the cubes periodically, once a day by default.

  This option is not supported for a given cube unless that cube supports synchronization (as described earlier in this chapter).

  Important:

  Before you synchronize cubes from the Cube Manager, it is necessary to build the cubes at least once from the Cube Manager.

• Manual — Do not rebuild or synchronize from the Cube Manager.

  Instead, use any suitable combination of other tools: the Build option in the Architect and the %BuildCube(), %SynchronizeCube(), %ProcessFact(), and %DeleteFact() methods. For details on the latter three methods, see later sections of this chapter.

For each plan (other than Manual), you can customize the schedule details.

For any namespace, the Cube Manager defines two tasks: one performs all requested cube build activity in this namespace, and one performs all requested cube synchronization activity in this namespace. Both of these tasks follow the instructions provided in the cube registry. Both tasks also automatically process cubes in the correct order required by any relationships.

The Cube Manager provides an Exclude check box for each registered group and cube, which you can use to exclude that group or cube from any activity by the Cube Manager. Specifically, the Cube Manager tasks ignore any excluded groups and cubes. Initially these check boxes are selected, because it is generally best to not to perform updates until you are ready to do so. For example, you might want to adjust the DeepSee agent counts before building any cubes.

Accessing the Cube Manager

To access the Cube Manager, do the following in the Management Portal:

1. Switch to the appropriate namespace as follows:

   1. Click Switch.

   2. Click the namespace.

   3. Click OK.

2. Click DeepSee > Admin > Cube Management > Registry.

3. If you have not used the Cube Manager in this namespace, it prompts you for information about the cube registry. In this case, specify the following information:

   • Cube Registry Class Name — Specify a complete class name, including package. This class definition will be the cube registry for this namespace.

   • Disable — Optionally click this to disable the registry. If the registry is disabled, the Cube Manager tasks are suspended. (Because there are no Cube Manager tasks yet, it would be redundant to disable the registry at this point.)

   • Update Groups — Specify how to update groups with respect to each other. If you select Serially, the tasks update one group at a time. If you select In Parallel, the tasks update the groups in parallel.

   • Allow build to start after this time — Specify the earliest possible build time.

   You can change all these details later, apart from the class name.

   Then click OK.

The system displays the Cube Registry page. You can view this page in two modes (via the View buttons). Click the left View button for tree view or click the right View button for table view.

Tree View

In tree view, the left area of the Cube Manager displays a tree of unregistered cube groups. For example:

The middle area displays a table (initially empty) with information for the registered groups. The following example shows what this table looks like after you have registered a group:

This area is color-coded as follows:

• White background — The group or cube is included, which means that the Cube Manager tasks update it. See the Exclude option in “Specifying an Update Plan,” later in this chapter.

• Gray background — The group or cube is excluded, which means that the Cube Manager tasks ignore it.

This area also lists (in italics) any subject areas based on a given cube, for example:

Note that you cannot specify update plans for the subject areas, because updates in a cube are automatically available in any subject area based on that cube. (So there is no need and no way to update a subject area independently from the cube on which it is based.)

In the right area, the Details tab (not shown) displays details for the current selection. You can make edits in this tab. The Tools tab provides links to other tools.

Note:

When the Cube Manager is in tree view, you can expand or collapse the display of all registered groups, which are shown in the middle area. To do so, use the Expand All or Collapse All button, as applicable, at the top of the middle area. These buttons do not affect the left area of the page, which displays the unregistered groups.

Table View

In table view, the Cube Manager lists all cubes in the current namespace, with their update plans. For example:

This table is color-coded as follows:

• White background — The cube is included, which means that the Cube Manager tasks update it. See the Exclude option in “Specifying an Update Plan,” later in this chapter.

• Gray background — The cube is excluded, which means that the Cube Manager tasks ignore it.

• Pink background — The cube is not registered and therefore has no update plan.

The Group Name field indicates the group to which each cube belongs, and the Group Build Order field indicates the order in which each cube is to be built or synchronized within its group. The Cube Manager computes this order only for cubes in registered groups.

In the right area, the Details tab (not shown) displays details for the current selection. You can make edits in this tab. The Tools tab provides links to other tools.

Modifying the Registry Details

When you first access the Cube Manager, it prompts you for initial information. To modify these details later (other than the registry class name, which cannot be changed):

1. Display the Cube Manager in tree view.

2. In the middle area, click the heading that starts Registered Groups.

3. Edit the details on the right.

   For information on the options, see the previous section.

4. Click Save.

Registering a Cube Group

To register a cube group:

1. Display the Cube Manager in tree view.

2. Expand the list of unregistered cubes on the left.

3. Drag the group from that area and drop it onto the Registered Groups heading in the middle area.

Or display the Cube Manager in table view, click the row for any cube in the group, and click Register Group in the right area.

In either case, the change is automatically saved.

Specifying an Update Plan

To specify the update plan for a cube group and its cubes:

1. Display the Cube Manager in tree view.

2. Click the group in the middle area.

3. In the Details pane on the right, specify the following information:

   • Name — Unique name of this group.

   • Exclude — Controls whether the generated tasks perform update activities for cubes in this group. Initially this option is selected, and the group is excluded.

     The Cube Manager displays any excluded groups or cubes with a gray background.

   • Update Plan — Select an update plan.

     Note that the Cube Manager does not permit you to use synchronization unless that cube supports it (as described earlier in this chapter). For example, you can choose the Build and Synch plan for the group, but the Cube Manager automatically sets the update plan to Build for any cube that does not support synchronization.

     Important:

     Before you synchronize cubes from the Cube Manager, it is necessary to build the cubes at least once from the Cube Manager.

   • Build every — Use these fields to specify the schedule for the build task (if applicable).

   • Synch every — Use these fields to specify the schedule for the synchronization task (if applicable).

   • Build Cubes Synchronously — Select this to cause DeepSee to build these cubes synchronously (if applicable). If this option is clear, DeepSee builds them asynchronously.

   Initially, these details apply to all cubes in the group. If you edit details for a specific cube and then later want to reapply the group defaults, click Apply to All Cubes in Group.

4. Optionally click a cube within this group (in the middle area) and edit information for that cube in the Details pane on the right.

   The options are similar to those for the entire group, but include the following additional options, depending on whether the cube supports synchronization:

   • Post-Build Code — Specify a single line of ObjectScript to be executed immediately after building this cube. For example:

     do ##class(MyApp.Utils).MyPostBuildMethod("transactionscube")
     

     

   • Pre-Synchronize Code — Specify a single line of ObjectScript to be executed immediately before synchronizing this cube. For example:

     do ##class(MyApp.Utils).MyPresynchMethod("transactionscube")
     

     

     If needed, to abort the synchronization, do the following in your code:

     set $$$ABORTSYNCH=1
     

     

   • Post-Synchronize Code — Specify a single line of ObjectScript to be executed immediately after synchronizing this cube. For example:

     do ##class(MyApp.Utils).MyPostsynchMethod("transactionscube")
     

     

   In all cases, your code can perform any processing required.

   Modify each cube as needed.

5. Click Save.

   When you do so, the Cube Manager creates or updates the cube registry in this namespace. If the Task Manager does not yet include the necessary tasks, the Cube Manager creates them.

Merging Groups

You can merge one group (group A) into another (group B). Specifically this moves all the cubes from group A into the group B and then removes the now-empty group A.

To merge one group into another, use the following procedure. In this procedure, group A must not yet be registered, and group B must be registered.

1. Display the Cube Manager in tree view.

2. Drag group A (the group that contains the cubes that you want to move) from the left area and drop it into the group heading of group B (the target group) in the middle area.

   The system prompts you to confirm the action.

3. Click OK.

   If group B currently has an update plan that cannot be used for some of the newly moved cubes, the system displays a dialog box to indicate this. Click OK. For any such cubes, the Cube Manager selects an update plan that can be used.

4. Review the update plan for each newly moved cube and modify it as needed.

5. Click Save.

Or use the following alternative procedure. In this procedure, both groups must already be registered.

1. Display the Cube Manager in table view.

2. In the middle area, click the row for any cube in group A (the group that contains the cubes that you want to move).

3. On the right, click Merge to another group and then select group B (the target group) from the drop-down list.

4. Click Merge.

   The system prompts you to confirm the action.

5. Click OK.

   If group B currently has an update plan that cannot be used for some of the newly moved cubes, the system displays a dialog box to indicate this. Click OK. For any such cubes, the Cube Manager selects an update plan that can be used.

6. Review the update plan for each newly moved cube and modify it as needed.

7. Click Save.

Building All the Registered Cubes

DeepSee provides a utility method that you can use to build all the registered cubes, in the correct order. The method is BuildAllRegisteredGroups() in the class %DeepSee.CubeManager.UtilsOpens in a new tab. This method ignores the schedule specified in the registry but uses the build order specified in the registry.

Performing On-Demand Builds

The Cube Manager also provides options to build cubes on demand (that is, ignoring the schedule). In this kind of build, the Cube Manager rebuilds the requested cube as well as any cubes that depend on it.

To perform an on-demand build:

1. Save any changes to the cube registry.

   Important:

   The build options are disabled if there are any unsaved changes.

2. Select a registered cube. To do so, either:

   • Display the Cube Manager in tree view and then click a cube in the middle area.

   • Display the Cube Manager in table view and click a cube that shows Yes in the Registered column.

3. On the right, clear the Exclude option.

4. Click Build Dependency List.

   The Cube Manager then displays the build dialog box.

5. Click Build List.

   The dialog box displays progress of the build.

6. When the build is done, click OK.

There are other ways to perform on-demand builds:

• Display the Cube Manager in tree view. Click the header of the table in the middle area. Then click Build All Registered Groups. Continue as described previously.

• Display the Cube Manager in tree view. Click a cube group in the middle area. Then click Build This Group. Continue as described previously.

Unregistering a Cube Group

To unregister a cube group:

1. Display the Cube Manager in tree view.

2. In the middle area, click the X in the row for the cube group.

3. Click OK.

Viewing Cube Manager Events

For certain events, the Cube Manager writes log entries to a table, which you can query via SQL. The table name is %DeepSee_CubeManager.CubeEvent. The CubeEvent field indicates the type of cube event. Possible logical values for this field include the following:

For information on other fields in this table, see the class reference for %DeepSee.CubeManager.CubeEventOpens in a new tab.

Restricting Access to the Cube Manager

You may want to manage the cube update schedule without allowing users to change that schedule through the Cube Registry page. To restrict access to the Cube Registry page, set the UserUpdatesLocked attribute to "true" in either the RegistryMap or RegistryMapGroup objects within your saved cube registry. For example:

<RegistryMap Disabled="false" IndependentSync="false" SerialUpdates="false" UserUpdatesLocked="true">

When UserUpdatesLocked is set to "true" for a RegistryMap:

• The registry’s Disable setting cannot be changed through the Details tab. For information on accessing this tab, see Modifying the Registry Details.

When UserUpdatesLocked is set to "true" for a RegistryMapGroup:

• Each registered group’s Exclude checkbox is displayed but disabled

• Each registered cube’s Exclude checkbox is hidden

• Each registered group’s Update Plan is hidden

• Each registered cube’s Update Plan is hidden

• The red X button for removing registered groups is removed

• The Build Frequency and Synch Frequency columns are left blank

• The Build Dependency List is available for cubes, but the Build This Group button is disabled.

Using DSTIME=MANUAL

Instead of letting DeepSee automatically update the ^OBJ.DSTIME global, you can update this global at times that you choose. To do so:

1. Specify DSTIME as "MANUAL" rather than "AUTO".

2. Then within your application, call the method %SetDSTimeIndex() of the class %DeepSee.UtilsOpens in a new tab whenever you add, change, or delete objects of the class, or when you want to update the ^OBJ.DSTIME global.

   This method has the following signature:

   ClassMethod %SetDSTimeIndex(pClassName As %String, 
                               pObjectId As %String, 
                               pAction As %Integer,
                               pInterval As %Integer = 0)
   

   

   Where:

   • pClassName is the full package and class name of the object that you have added, changed, or deleted.

   • pObjectId is the object ID for that object.

   • pAction is 0 if you updated the object, 1 if you added it, or 2 if you deleted it or want to delete the corresponding fact from the fact table without deleting the object. The value of pAction is used as the value of the resulting ^OBJ.DSTIME node. Note that facts are removed from a cube during synchronization if the corresponding record does not exist in the source class, or if a value of 2 is specified for pAction.

   • pInterval is an optional integer. If you specify this as a positive integer, DeepSee uses time stamp subscripts in the ^OBJ.DSTIME and ^DeepSee.Update globals. See the discussion of the DSINTERVAL parameter in “Enable Cube Synchronization.”

Then, when you want to update a given cube, call the %SynchronizeCube() method of the %DeepSee.UtilsOpens in a new tab class, as described previously.

Injecting Facts into the Fact Table and Dimension Tables

In rare cases, you might need the fact table to include records that do not correspond to any source records. In such cases, use the %InjectFact() method of the cube class.

This method has the following signature:

classmethod %InjectFact(ByRef pFactId As %String, 
                        ByRef pValues As %String, 
                        pDimensionsOnly As %Boolean = 0)
                        as %Status

Where:

• pFactId is the ID of the fact. Set this to "" for an insert. On return, this argument contains the ID used for the fact.

• pValues is a multidimensional array of fact values. In this array, the subscript is the sourceProperty name (case-sensitive).

• pDimensionsOnly controls whether the method affects both the fact table and dimension tables or just the dimension tables. If this argument is true, the method affects only the dimension tables. You use this argument if you prebuild the dimension tables as described in the next section.

  Caution:

  Do not use this method to update dimension tables for levels that are based on source expressions. To add records to those tables, instead use an SQL UPDATE statement.

  You can use %InjectFact() to update dimension tables for levels that are based on source properties.

Prebuilding Dimension Tables

By default, DeepSee populates the dimension tables at the same time that it builds the fact table. It is possible to prebuild one or more dimension tables so that they are populated before the fact table, if this is necessary for some reason.

To prebuild one or more dimension tables, do the following:

• Implement the %OnBuildCube() callback in the cube definition class. This method has the following signature:

  classmethod %OnBuildCube() as %Status
  

  

  The %BuildCube() method calls this method just after it removes the old cube contents and before it starts processing the new contents.

• In this implementation, invoke the %InjectFact() method of the cube class and specify the pDimensionsOnly argument as true.

  For details on this method, see the previous section.

For example, the following partial implementation predefines the Cities dimension in the HoleFoods sample:

ClassMethod %OnBuildCube() As %Status
{
    // pre-build City dimension
    Set tVar("Outlet.Country.Region.Name") = "N. America"
    Set tVar("Outlet.Country.Name") = "USA"

    Set tVar("Outlet") = 1000
    Set tVar("Outlet.City") = "Cambridge"
    Do ..%InjectFact("",.tVar,1)

    Set tVar("Outlet") = 1001
    Set tVar("Outlet.City") = "Somerville"
    Do ..%InjectFact("",.tVar,1)

    Set tVar("Outlet") = 1002
    Set tVar("Outlet.City") = "Chelsea"
    Do ..%InjectFact("",.tVar,1)

    Quit $$$OK
}

Notes:

• It is necessary to provide a unique ID as well as a name for a member.

• For completeness, this code should also provide the city population, longitude, and latitude, because the corresponding dimension table contains these values.

• It is also necessary to provide values for any higher level members.

Updating a Dimension Table Manually

In some cases, there is no change to your base class, but there is a change to a lookup table that is used as a level. In these cases, you can update the cube in any of the ways described earlier in this chapter. If the only change is to a single dimension table, however, it is quicker to update the level table directly. You can do so via the %UpdateDimensionProperty() method of %DeepSee.UtilsOpens in a new tab.

This method has the following signature:

classmethod %UpdateDimensionProperty(pCubeName As %String, 
                                     pSpec As %String, 
                                     pValue As %String, 
                                     pKey As %String) 
                                     as %Status

Where:

• pCubeName is the name of the cube.

• pSpec is the MDX member expression that refers to the level member to update. You must use the dimension, hierarchy, and level identifiers in this expression. For example: "[docd].[h1].[doctor].&[61]"

  As a variation, pSpec can be a reference to a member property. For example: "[homed].[h1].[city].&[Magnolia].Properties(""Principal Export"")"

  DeepSee uses this argument and the pCubeName argument to determine the table and row to update.

• pValue is the new name for this member, if any.

  Or, if you specified a member property, pValue is used as the new value of the property.

• pKey is the new key for this member, if any.

  Specify this argument only if you specify a member for pSpec.

You can make three kinds of changes with this method:

• Specify a new key for a member. For example:

  Set tSC = ##class(%DeepSee.Utils).%UpdateDimensionProperty("patients","[docd].[h1].[doctor].&[186]",,"100000")
  

  

  By default, the key is also used as the name, so this action might also change the name.

• Specify a new name for a member. For example:

  Set tSC = ##class(%DeepSee.Utils).%UpdateDimensionProperty("patients","[docd].[doctor].&[186]","Psmith, Alvin")
  

  

  By default, the name is the key, so this action might change the key.

• Specify a new value for some other property (both Name and Key are properties). For example:

  Set memberprop="homed.h1.city.Pine.Properties(""Principal Export"")"
   
  Set tSC = ##class(%DeepSee.Utils).%UpdateDimensionProperty("patients",memberprop,"Sandwiches")