Skip to main content

Handling Batches of Records Efficiently

The RecordMap feature imports a single record at a time, but if you are importing or exporting a large number of records, you can gain substantial efficiency improvement by using RecordMap Batch. The RecordMap Batch feature handles homogeneous records and processes all of the records in a batch at one time. The batch can optionally be preceded by a header record and followed by a trailer record.

To create a RecordMap batch, you implement a class which inherits from %PersistentOpens in a new tab and EnsLib.RecordMap.BatchOpens in a new tab. The Batch class contains methods that handle parsing and writing out any headers and trailers associated with a specific batch. You must provide code that parses and writes your headers. For simple headers and trailers, you can use the EnsLib.RecordMap.SimpleBatchOpens in a new tab class, which inherits from the Batch class and provides code for handling simple headers and trailers. You can extend either of these two batch implementations if you need to process more complex header and trailer data.

Batch processing follows the approach used for other production message formats like X12. This is particularly relevant for the built-in business operations which handle RecordMap batch objects: these business operations accept either Batch objects or RecordMap objects which extend EnsLib.RecordMap.BaseOpens in a new tab, or a request of type BatchRolloverRequest. When records in a particular batch are received, the Batch is opened and the batch header is written to a temporary file, followed by any objects within that batch received by the operation. If the request in synchronous, the classname, Id, and the count of previously written records for the batch will be returned in a EnsLib.RecordMap.BatchResponseOpens in a new tab. Receipt of the batch object (which may be the default batch) will trigger the batch trailer to be written to the temporary file, and this file will then be sent to the desired destination by the adapter for the business operation. If a Batch object is received by itself, then the entire Batch will be written out to a temporary file which will then be transferred to the desired location.

Important:

If ArchiveIO is enabled during this process, a copy of the in bound stream will be saved in the temporary stream location for the namespace (the default location is <install-dir>/mgr/GLOBAL_DB_DIRECTORY/stream/) These streams will be purged automatically and should not be manually removed.

Batch operations also support a default batch option whereby records which do not already belong to a batch are added to a default batch. Output of this batch can be triggered by either sending the batch object to the operation, or sending a BatchRolloverRequest to the operation. The business operation can also be configured to use schedule- or count-based rollover for the default batch. These options are configured on the business operation, and can be used simultaneously.

The options for services primarily concern the way the Visual Trace displays messages within a batch.

The RecordMap Batch operation creates temporary files in the process of generating the final output file. You can control the location of these temporary files by specifying the IntermediateFilePath setting of the RecordMap Batch operation. If the namespace’s database is being mirrored, it is important that all mirror members have access to the temporary file in order to successfully failover during a RecordMap Batch operation. See the High Availability Guide for information on mirroring.

Important:

RecordMap Batch messages use a one-to-many relationship to hold the records and this makes it quite easy to traverse all records and perform the desired transformation. However, the process can consume enough memory that you can receive <STORE> errors. You may need to increase the memory of the process, or split the input files, or implement customized transformations that use SQL instead of the one-to-many relationship.

FeedbackOpens in a new tab