Skip to main content

HS.FHIRServer.Tools.DataLoader

class HS.FHIRServer.Tools.DataLoader extends %Library.RegisteredObject

Utility class to facilitate bulk loading of FHIR resources to a specified FHIR server.

Property Inventory

Method Inventory

Properties

property JobId as %String;
Unique GUID associated with this DataLoader instance
Property methods: JobIdDisplayToLogical(), JobIdGet(), JobIdIsValid(), JobIdLogicalToDisplay(), JobIdLogicalToOdbc(), JobIdNormalize(), JobIdSet()

Methods

method %OnNew(pServiceType As %String, pServiceName As %String, pJobId As %String = $Job) as %Status
Input:
- pServiceType : (required) "FHIRServer" = FHIRServer Service endpoint, or "HTTP" = Service Registry HTTP service.
- pServiceName : (required) Name of FHIRServer Service or Service Registry HTTP service to which to send data.
- pJobId : (optional) If you don't want the $J value you can set to whatever value is desirable. Multi-threaded entrypoint for the class uses a GUID for example so that all workers have the same JobId
classmethod Cancel(pJobId As %String) as %Boolean
@API.Method
Signals to a running DataLoader job to quit out gracefully
May take several seconds for the running job to exit, poll using the Status() method to know when it's done/Canceled status
classmethod CleanUp(pJobId As %String)
@API.Method
Cleans up globals associated with a DataLoader job
classmethod Job(pInputDirectory As %String, pServiceType As %String, pServiceName As %String, pLogGlobal As %String = "", pFileLimit As %String = "", pTranslateTable As %String = "UTF8", pNumWorkers As %Integer = ##class(%SYSTEM.WorkMgr).DefaultNumWorkers(), Output pJobId As %String) as %Status
@API.Method
Method to start a DataLoader background job
See comments for SubmitResourceHelper() for input parameter descriptions
pJobId is an output from this method and is needed in order to interact with the background job via calls to Status() / Cancel() and Cleanup()
method PreFHIRService(pRequest As HS.FHIRServer.API.Data.Request)
@API.Overrideable
Override this method to carry out specific tasks before dispatching the HS.FHIRServer.API.Data.Request to the FHIRServer Service.
method PreHttp(ByRef pURLAndPath)
@API.Overrideable
Override this method to carry out specific tasks before invoking the HTTP request. The HTTP service object and the HTTP request object are accessbile as instance properties ..HttpService and ..HttpRequest. pURLAndPath is passed in here because it is not part of the request object.
classmethod Status(pJobId As %String = "") as %DynamicObject
@API.Method
Get the status and metrics of a DataLoader job
Returns a DynamicObject like :

{
   "status": "Complete",
   "runDuration": 0.176,
   "filesTotal": 5,
   "resourcesTotal": 45,
   "elapsedTotal": 0.143,
   "elapsedAvgPerFile": 0.029,
   "elapsedAvgPerResource": 0.003,
   "errorCount": 2,
   "errors": [
       {
           "file": "TempBundle001.ndjson||7",
           "message": "ERROR <HSFHIRErr>EmptyValue: Property 'id' of Type 'Patient' cannot be null or an empty string."
       },
       {
           "file": "TempBundle002.ndjson||17",
           "message": "ERROR <HSFHIRErr>EmptyValue: Property 'id' of Type 'Patient' cannot be null or an empty string."
       }
   ]
}
classmethod SubmitResourceFiles(pInputDirectory As %String, pServiceType As %String, pServiceName As %String, pDisplayProgress As %Boolean = 1, pLogGlobal As %String = "", pFileLimit As %String = "", pTranslateTable As %String = "UTF8", pNumWorkers As %Integer = ##class(%SYSTEM.WorkMgr).DefaultNumWorkers(), ByRef pJobId As %String = $SYSTEM.Util.CreateGUID()) as %Status
@API.Method
Submit FHIR content files from a specified operating system directory to a FHIR server. Each file may contain an individual resource or a Bundle of resources or bulk format .ndjson (one JSON resource per line, can be a single resource or a bundle). The files in the specified directory can be a mix of files with an individual resource and files with a Bundle of resources or .ndjson files. The files in the specified directory may be all JSON/NDJSON files or all XML files, or a mix of JSON/NDJSON and XML files.

Input:
- pInputDirectory : (required) Operating system directory location of the source input files.
- pServiceType : (required) "FHIRServer" = FHIRServer Service endpoint, or "HTTP" = Service Registry HTTP service.
- pServiceName : (required) Name of FHIRServer Service or Service Registry HTTP service to which to send data.
- pDisplayProgress : 1 (default) = Display progress messages and errors. 0 = Do not display.
- pLogGlobal : Name of global to which to log statistics. If not specified, then no logging is done. The recorded elapsed time is that time spent while being processed in the relevant FHIR server infrastructure. It does not include time spent by this method in examining and re-formatting of resources or in the construction of request messages.
- pFileLimit : Limit processing to the first pFileLimit files in pInputDirectory. If pFileLimit is not specified, then all files are processed.
- pTranslateTable : Translate table to use on all FHIR resource input files. Default is "UTF8". - pNumWorkers : Number of work queue jobs to allocate to this process. If omitted will use system default. - pJobId : Can be passed by reference or can be specified by user
Output - if pLogGlobal is specified:
- @pLogGlobal@(pJobId,"Status") = Status of current run. Possible values: Running, Canceled, Jobbed, Complete, ERROR text
- @pLogGlobal@(pJobId,"RunDuration") = Time from start to finish of the job - @pLogGlobal@(pJobId,"FilesTotal") = Number of files processed.
- @pLogGlobal@(pJobId,"ResourcesTotal") = Total number of resources within the processed files.
- @pLogGlobal@(pJobId,"ElapsedTotal") = Total elapsed time between each processing request and response. Note for multithreaded use this number will be much longer than the RunDuration since it's count time of all threads
- @pLogGlobal@(pJobId,"ElapsedAvgPerFile") = Average elapsed time per file.
- @pLogGlobal@(pJobId,"ElapsedAvgPerResource") = Average elapsed time per resource. - @pLogGlobal@(pJobId,"ErrorCount") = Number of non-fatal errors encountered during the run (generally errors trying to submit resources to the FHIRServer, like validation etc) - @pLogGlobal@(pJobId,"ErrorCount",errorId,"File") = File name that errored (if from an NDJSON file it will have the count appended) - @pLogGlobal@(pJobId,"ErrorCount",errorId,"Message") = error message associated with the File
classmethod SubmitResourceFilesHelper(pInputFile As %String, pFileName As %String, pDisplayProgress As %Boolean = 1, pLogGlobal As %String = "", pTranslateTable As %String = "UTF8") as %Status
classmethod SubmitResourceNDJSONHelper(pId, pFileName As %String, pDisplayProgress As %Boolean = 1, pLogGlobal As %String = "") as %Status
method SubmitResourceStream(pResourceStream As %Stream.Object, pFormatCode As %String = "", Output pSentStreamSize As %Integer, Output pEntryCount As %Integer, Output pElapsed As %Decimal) as %Status
@API.Method
Submit an individual FHIR stream to a FHIRService or to an HTTP Service.

This method returns statistics regarding number of resources processed, plus elapsed time. Please note that the elapsed time statistic is the accumulation of the timing starting from the submission of each FHIR payload to the return of each FHIR payload.

Input:
- pResourceStream : (required) Stream object containing FHIR content. - pFormatCode : FHIR data format - "JSON", "NDJSON" or "XML". If not provided, then this method derives the format by examining the input stream (only works for "JSON" or "XML" content).

Output:
- pSentStreamSize : Size, in bytes, of the FHIR stream that was actually sent.
- pEntryCount : Number of resources in pResourceStream.
- pElapsed : Elapsed processing time for the resources within pResourceStream. When sending directly to FHIRServer Service, elapsed time is that time spent while being processed in the relevant FHIRServer infrastructure. When sending via HTTP, elapsed time is the time between the invocation of the HTTP request and the return of the response over the wire.
classmethod WorkMgrSetup(pServiceType As %String, pServiceName As %String, pJobId As %String) as %Status
classmethod WorkMgrTearDown() as %Status
classmethod WriteOutput(pString)
@API.Overrideable
Override this method to implement an alternate means of display output.

Inherited Members

Inherited Methods

FeedbackOpens in a new tab