This class provides an interface to the work queue manager code that allows work to be distributed to multiple
processes in order to improve performance. In order to use this you need to divide the work up into units that
can be processed independently then you initialize the worker jobs, then queue each unit of work and finally you
wait for the work to be completed. The units of work can output to the current device which will be buffered up
and output to the main job's device when that unit of work is signalled as complete. Also all units of work by
default are expected to return a %Status value so it can indicate errors, these are displayed and
returned by the WaitForComplete() method.
A typical calling sequence is:
Set queue=$system.WorkMgr.Initialize("/multicompile=1",.sc) If $$$ISERR(sc) ; Report Error
For i=1:1:100 {
Set sc=queue.Queue("##class(MyClass).ClassMethod",i) If $$$ISERR(sc) ; Report Error
}
Set sc=queue.WaitForComplete() If $$$ISERR(sc) ; Report Error
The call to Initialize() will allocate some worker jobs from the pool to the work group you are
creating, if there are not enough worker jobs in the pool then a daemon process will start additional worker
automatically. The number of worker jobs we start determined by the work queue manager based on current machine
load and characteristics of the CPU the machine is running on. If '/multicompile=0' then we will not use any worker
jobs at all and will do all the processing in the current job in the WaitForComplete() call.
Then you call Queue() to queue a unit of work to be completed, this takes either a class method
call, or a '$$func^rtn' reference and then any arguments you need to pass to this function. As soon as the
first Queue() is called a worker will start processing this item of work. It is important to make
sure that all the units of work are totally independent and they do not rely on other work units. You can not
rely on the order in which the units of work are processed. If the units may be changing a common global you
will need to add locking to ensure one worker can not change a global while another worker is in the middle of
reading this global. When a unit of work is queued the current security context is stored so when the work
is run it will use the current security context. Note that the worker jobs are started by the super server
and so will run as the operating system user that the super server process is setup to use, this may be different
to your current logged in operating system user.
Finally you call WaitForComplete() to wait for all the units of work to be complete, display any
output each unit produced and report any errors reported from the work units. The WaitForComplete()
will use the qualifiers provided in the Initialize().
Worker jobs are owned by the master process while they are performing work in this group, so when the master exits
the worker jobs will be released immediately. When the object returned by Initialize is destroyed this will remove all work associated
with this group automatically, and release any workers.
Note that the work queued should not use exclusive news, kills or unlocks as this will interfere with the framework.
Also if you use process private globals to store data during the processing note that as multiple Cache jobs will
be processing each chunk you can not rely on accessing these from the master process (or even from another chunk).
The size of each chunk should be on the order of thousands of lines of COS code to ensure the overhead of the framework is
not a significant factor, also rather than a few very large chunks (e.g. 4 big chunks) if possible it is better
to have a fairly large number (100 say) of chunks as this allows us to scale with CPU cores. Worker jobs once started
will remain until they time out given a long enough period of inactivity.
property NumActiveWorkers as %Integer [ Calculated ];
Number of active workers attached to this group. If the system is at the limit
then we limit the number of worker jobs so you may need to wait for existing worker
jobs to become free in order to attach to your work group.
After the work group is created the number of workers allocated to this group.
Note this is the number of jobs we requested, not the number actively working for
this group at this moment. The active number is NumActiveWorkers.
Called from a worker job to flush any output produced so far to the master process.
Without this all output from a worker job is buffered until this unit of work is complete and only
then is it displayed in the master process.
Initialize the worker jobs so they are ready to start accepting work items. In the qspec that is passed
you can determine if you want output with the 'd' flag and if you wish to disable multiple jobs and process
the work in this master process with '/multicompile=0'. This returns an instance of the worker queue manager
which you can then queue work against. If you wish to specify how many workers you require then pass
the numberjobs equal to the number you require. You may also pass the number in the
/multicompile=NUM as long as NUM is greater than 1 or for 1 worker specify '-1' i.e. /multicompile=-1.
Pause any work in this work queue, this stops any workers from picking up additional items from
this specific queue, but leaves the work itself so you can call Resume().
Queues a specific unit of work, you pass the entry point to call in 'work' argument. This can be either '##class(Classname).ClassMethod'
or '$$entry^rtn' and it is expected to return a %Status code on completion. If you want to call a function
that does not return any value on completion then prepend the class syntax with '=' such as '=##class(Classname).ClassMethod'
or for calling a function do not include the '$$' for example 'entry^rtn'.
The item being called may also throw exceptions in order to indicate an error happened which is trapped and converted
to a %Status value to be returned in the master process. You can also pass additional
arguments including arrays by reference. Note that the size of the data passed in these arguments should be kept
relatively small, if there is a large amount of information that needs to be passed then put this in a global.
The security context of the caller is also recorded when this function is called so it can be used when the work
is executed.
Similar to Queue() except you can also pass in a 'callback' which is a function or class method that
is called in the master process when this unit of work is complete. This function is called with the same arguments
the original 'work' is called with so it knows which unit of work is complete. Also the callback function can access
the '%job' public variable which is the $job of the process which really did the work, the '%status' public variable
which is the %Status return code from the work unit this is the callback for and '%workqueue' public variable which
is the oref of the work queue instance.
If using the Wait() to wait for the work to be completed the callback can signal that it should
return to the caller rather than waiting for more events by setting the public variable '%exit' to 1.
If this queue needs some setup work before we process the first item in this queue or if switching from processing
items in another queue to one in this queue then specify the function to call here. The arguments are the same
as for Queue(). This must be called before you queue work.
If this queue needs some work done to restore a process to the previous state before we process the first item
in another queue when switching from processing items in this queue then specify the function to call here.
The arguments are the same as for Queue(). This must be called before you queue work.
After work has been queued this will wait for a signal from a callback to exit back to the caller.
This is done by in the callback queued with QueueCallback() setting the public
variable '%exit' to 1. This method returns AtEnd to show if all the work is complete or
if there are still items outstanding. Note that in the function/method called in the QueueCallback() callback
you can reference the public variable '%workqueue' which is the oref of the instance of the work queue class
in order to queue additional work.
After work has been queued this will wait for all the workers to complete. It will display any output the work
writes to the current device and it will also combine all the %Status codes that the units of work report and
return this combined %Status. If there are no workers then this will execute all the work in this main job
during this phase. When this returns all the work queued up to this point has been completed. Also this is the
phase that will run the QueueCallback() callbacks as the workers jobs report that various
units are complete. Note that in the function/method called in the QueueCallback() callback
you can reference the public variable '%workqueue' which is the oref of the instance of the work queue class
in order to queue additional work.