Introduction to the Work Queue Manager

Introduction

The Work Queue Manager is a feature of InterSystems IRIS® data platform that enables you to improve performance by distributing work to multiple concurrent processes programmatically. It provides an efficient and straightforward API that enables you to off-load process management.

InterSystems code uses the Work Queue Manager internally in several places. Wherever you have code that meets the requirements, you can use the Work Queue Manager to perform parallel processing.

In addition to reducing the time needed to process large workloads, the Work Queue Manager provides you with a high level of control over how the CPU resources on your system are used. For example, you can create categories of work and define the number of worker jobs assigned to the categories. Additionally, the Work Queue Manager provides work load metrics so that you can monitor the load on your system in real time.

The following pages provide details on using this feature. Also see %SYSTEM.WorkMgrOpens in a new tab and %SYSTEM.ShardWorkMgrOpens in a new tab, and %SYSTEM.AbstractWorkMgrOpens in a new tab in the class reference.

Code Requirements

The Work Queue Manager processes units of work (also called work items), which are ObjectScript class methods or subroutines that meet the following requirements:

• The class method or subroutine can be processed independently. For example, a unit of work cannot rely on output from a different unit of work. Independence is required because units of work may be processed in any order. (However, you can use callbacks to execute work sequentially if needed. For more information, see Using Callbacks.)

• The class method or subroutine is on the order of thousands of lines of ObjectScript code in size. This requirement ensures that the overhead of the framework is not a significant factor.

  Furthermore, it is preferable to use a large number (for example, 100) of smaller units of work rather than a small number of very large units of work (for example, 4). Distributing the work in such a way permits the system to scale up when more CPU cores are available.

• The code returns a %StatusOpens in a new tab value to indicate success or failure so that the Sync() method can return a %StatusOpens in a new tab value to indicate overall success or failure. Alternatively, the unit of work can throw an exception that is trapped, converted to a %StatusOpens in a new tab value, and returned in the master process.

• If the code changes the same global as a different unit of work, you must employ a locking strategy to ensure that one worker job cannot change the global while another worker is reading it.

• The code does not include exclusive NEWs, KILLs, or unLOCKs since these interfere with the framework.

• If the code includes process-private globals for storing data, these process-private globals are not accessed from the master process or from any other chunk. This requirement is necessary since multiple jobs process each chunk.

• Any logic called as part of the class method or subroutine is correctly cleaned up such that no variables, locks, process-private globals, or other artifacts remain in the partition. This requirement is important since the same process will be used to subsequently process completely separate work items.

Basics

This section discusses %SYSTEM.WorkMgrOpens in a new tab specifically. Also see %SYSTEM.ShardWorkMgrOpens in a new tab in the class reference; both classes share a common API.

To use the Work Queue Manager to perform parallel processing:

1. Identify the ObjectScript code that you want to process in parallel. See Code Requirements.

2. Divide your code into units of work.

3. Create a work queue, which is an instance of the %SYSTEM.WorkMgrOpens in a new tab class. To do so, call the %New() method of the %SYSTEM.WorkMgrOpens in a new tab class. The method returns a work queue.

   You can specify the number of parallel worker jobs to use, or you can use the default, which depends on your machine and operating system. Additionally, if you have created categories, you can specify the category that the jobs should be taken from.

   When you create a work queue, the Work Queue Manager creates the following artifacts:

   • A global that contains information about the work queue such as what namespace the work queue runs in

   • A location and an event queue for the serialized units of work that the work queue must process

   • A location and an event queue for completion events that are created as the work queue finishes processing units of work

4. Add units of work (also called work items) to the work queue. To do so, you can call the Queue() or QueueCallback() method. As arguments, you pass the name of a class method (or subroutine) and any corresponding arguments.

   Processing begins immediately on items added to the queue.

   If there are more items in the queue than there are worker jobs available to the queue, then the jobs compete to empty the queue. For example, if there are 100 items and four jobs, each job removes an item from the head of the queue, processes it, and then returns to the head of the queue to remove and process another item. This pattern continues until the queue is empty.

   The Work Queue Manager uses the security context of the caller when running a work item.

   When you queue work items, the Work Queue Manager performs the following tasks:

   • Serializes the arguments, security context, and class method or subroutine that comprises the unit of work, and then inserts the serialized data into the global that lists the units of work associated with the work queue.

   • Signals an event on the work queue.

   • If additional worker jobs are required and available to process the units of work, causes a worker job to attach to the work queue and decrements the number of available worker jobs.

5. Wait for the work to be completed. To do so, you can call the Sync() method of the work queue.

   The Work Queue Manager then performs the following tasks:

   • Waits for a completion event

   • Displays output such as workload metrics to the terminal

   • Collects any errors related to the unit of work

   • If you added units of work to the work queue using the QueueCallback() method, runs the callback code

6. Continue processing as appropriate for your application.

Example

The following example shows these basic steps:

    Set queue=##class(%SYSTEM.WorkMgr).%New() 
    For i = 1:1:filelist.Count() {
        Set sc=queue.Queue("..Load",filelist.GetAt(i))
        If $$$ISERR(sc) { 
            Return sc
        }
    }
    Set sc=queue.Sync()
    If $$$ISERR(sc) { 
        Return sc
    }

The code initializes the Work Queue Manager and then iterates through a list of files. For each file, the code adds a work queue item that loads a file. After adding all the work queue items, the code waits for the work to be completed.

Methods for Creating Work Queues

To create work queues, add items, and check for completion, use the following methods of the %SYSTEM.WorkMgrOpens in a new tab class:

%New()

classmethod %New(qspec As %String = "", numberjobs As %Integer, category) as WorkMgr

Creates, initializes, and returns a work queue, which is an instance of the %SYSTEM.WorkMgrOpens in a new tab class that you can use to perform parallel processing. The method accepts the following arguments:

qspec

A string of compiler flags and qualifiers that affect code running within this work queue. See Flags and Qualifiers.

numberjobs

The maximum number of parallel worker jobs to use in this work queue. The default depends on the characteristics of the machine and operating system.

category

The name of the category that supplies the worker jobs to use in this work queue. For more information, see Managing Categories.

The system does not allocate any worker jobs to the queue upon creation. Worker jobs are allocated only after you add a unit of work to the work queue.

Queue()

method Queue(work As %String, args... As %String) as %Status

Adds a unit of work to a work queue. The method accepts the following arguments:

work

The code to execute. In general, the code should return a %StatusOpens in a new tab value to indicate success or failure.

If the code returns a %StatusOpens in a new tab value, you can use the following syntax:

• ##class(Classname).ClassMethod for a class method, where Classname is the fully qualified name of the class and ClassMethod is the name of the method.

  If the method is in the same class, you can use the syntax ..ClassMethod as shown in the example.

• $$entry^rtn for a subroutine, where entry is the name of the subroutine and rtn is the name of the routine.

If the code does not return a %StatusOpens in a new tab value, use the following syntax instead:

• =##class(Classname).ClassMethod for a class method (or =..ClassMethod if the method is in the same class)

• entry^rtn for a subroutine

See About Units of Work for information about the requirements for units of work.

args

A comma-separated list of arguments for the class method or subroutine. To pass a multidimensional array as an argument, precede that argument with a period as usual so that it is passed by reference.

The size of the data passed in these arguments should be relatively small to make the most of the framework. To pass a large amount of information, use a global instead of an argument.

As you queue units of work, the system allocates worker jobs one at a time up to the numberjobs value that you specified when you created the work queue or up to the default value. Additionally, the security context of the caller is recorded, and each work item runs within that security context.

Sync()

method Sync(qspec As %String, errorlog As %String) as %Status

Waits for the work queue to complete all the items and then returns a %StatusOpens in a new tab value to indicate success or failure. The %StatusOpens in a new tab value contains information from all %StatusOpens in a new tab values returned by the work items. The method accepts the following arguments:

qspec

A string of compiler flags and qualifiers. See Compiler Flags and Qualifiers.

errorlog

A string of any error information, which is returned as output.

Properties of Work Queues

Each work queue (or instance of %SYSTEM.WorkMgrOpens in a new tab) has the following properties:

Returning Information

The work units can return information (other than status), which is helpful especially in shard queue manager situations where communication with the parent process may not be simple. To do this, the work units can write to the public %result multidimensional array. The caller can access this array in either of two ways:

• This array is returned by reference in the WaitOne() method.

• The variable %result is available within the QueueCallback() method.