Maintaining First-In First-Out (FIFO) Processing

Introduction

In some scenarios, it is necessary to preserve FIFO processing of messages. The prime example is healthcare applications. Suppose a patient enters a hospital and requires care. System A sends out an admit event, followed by a treatment order, but System B receives the order first. System B cannot process the order without an admit, so upon receiving the order, it produces an error. This may delay patient care or require the information system to execute complex logic to associate admit with the order after the admit finally arrives at System B.

There are two general approaches to preserve FIFO processing in a production, both described on this page:

• Force overall FIFO processing by preventing parallel activities.

• Programmatically define groups of messages that require FIFO processing.

Forcing Overall FIFO by Preventing Parallel Activity

In this approach for maintaining FIFO order, you simply limit each business host to use at most one job at any time.

If all the business hosts in the production have only one job available, only one message can be processed at a time by each host. This gives each message from a given source only one possible path through the production, so each message is guaranteed to arrive at its configured destination in the same order in which it was sent. (In contrast, with a business host that has multiple jobs, a message from a particular source could skip over other messages from the same source by using a faster, parallel job to arrive at its destination sooner.)

In this approach, you set the Pool Size setting to 1 for every business service, business process, and business operation, and you set the Actor Pool Size for the production to 0. Also, to guarantee FIFO for a BPL business process, in addition to setting its Pool Size to 1, do one of the following:

• Make calls using a <code> SendRequestSync() call

  OR

• Do not make calls from conditional branches of your business process logic and only make calls to elements that are themselves FIFO.

Defining FIFO Groups

In this approach for maintaining FIFO order, you identify and use data within the messages to place the messages into groups. For example, you might use the MRN of a patient, available in specific locations in the messages. This approach requires more work than overall FIFO but permits parallel activity (such as processing different patient messages at the same time). In this case, do the following:

1. Gather information:

   • Identify the field or fields that you can use to group messages. Determine how you will use these fields to create a unique identifier for the FIFO group.

   • Determine if the processing of one group is dependent in any way on the completion of another group.

   • Considering the path that the messages will take through the production, identify the business host (called the starting host) where the FIFO group is first needed. For example, many business services may generate production messages in a single-threaded fashion, sending these production messages to a business process for routing. We can use this business process as the starting host, and define FIFO groups for the inbound messages that come to this business host.

   • Also identify the business host or hosts where FIFO processing is no longer required for this group (the completion host). The completion host or hosts can release messages from their FIFO group; internally this removes identifiers from the message that force them to be grouped.

2. Create a DTL transformation whose purpose is to compute the FIFO group identifier. For this DTL, the source object will be the request type (as available at the business host where you will define the FIFO groups). The target must be of type Ens.Queue.FIFOMessageGroup.Data. In this DTL, set the following properties of the target:

   • Identifier—Required. This is the string that identifies the message group.

     If it is not possible to determine a group, use the special value _SingleThreadOverride, which enables this message to be processed but disables parallel processing while the message is being processed. That is, no additional messages will be taken from the queue until this message has been processed.

     Internally if the production encounters an error when using this DTL for a message, the system will use the special value _SingleThreadOverride for that message, for the same reason.

   • Dependencies—Optional. This is a comma-separated list of other possible message groups that this group must wait on, if those groups are in the queue.

   • CompletionHosts—Optional. This is a comma-separated list of hosts responsible for releasing the hold on the particular message group identifier. The first matching host will be the one to release this hold (enabling multiple messages in this group to be processed at the same time). This is used as an alternative to the host setting (see below).

   In this DTL, use new for the target. Also, for this DTL, aux is not defined.

3. Add this DTL to the production.

4. Configure the starting host (identified above) by specifying values in FIFO Message Grouping as follows:

   • Group Calculation (FMGCalculation)—Specify the name of the DTL you created previously.

   • Group Completion Hosts (FMGCompletionHosts)—Optionally specify a comma-separated list of the completion hosts.

Maintaining FIFO Groups in Custom Code

Rather than use a DTL transformation as described above, you can define FIFO groups in custom code. Instead of specify values for the Identifier, Dependencies, and CompletionHosts properties of the target, use syntax like the following, which uses macros defined for this purpose:

 Set $$$EnsThisFIFOMessageGroup = identifier
 Set $$$EnsThisFIFOMessageGroupDependencies = dependencies
 Set $$$EnsThisFMGCompletionHosts = completionhosts

Where identifier is the FIFO group identifier, dependencies is a comma-separated list of other message groups (if needed), and completionhosts is a comma-separated list of completion hosts (if needed).

If needed, you can use SetFIFOMessageGroupData() function.

Also, Ens.Queue.FIFOMessageGroup provides class methods to release a hold by one or more message headers in the queue. These methods can be also invoked as SQL procedures.

ReleaseHoldForHeaderId()

ClassMethod ReleaseHoldForHeaderId(pMsgHeaderId As %String, pSignalQueue As %Boolean=1) as %Status

Clears the hold on the FIFO group associated with the given message header.

ReleaseHoldForAll()

ClassMethod ReleaseHoldForAll(pQueueName As %String, pOnlyCompletedSessions As %Boolean=1) as %Status

Given a message queue, loops through all message headers in the queue that are marked as being active in the FIFO group pipeline and clears them.

The default is to clear only headers that belong to sessions that are considered Complete. If pOnlyCompletedSessions is 0, then all message headers are cleared.