Emit Telemetry Data to an OpenTelemetry-Compatible Monitoring Tool

InterSystems IRIS can automatically generate and emit traces for each request that a CSP, SOAP, or REST web application sends. To enable automatic trace emission for a web application, perform the following steps:

1. Access the Management Portal for the InterSystems IRIS instance which hosts your web application.

2. Navigate to System Administration > Security > Applications > Web Applications. If necessary, select Go to access the Web Applications page.

3. From the Web Application table, select the Name of the web application for which you wish to enable tracing.

4. The Enable section of the Edit Web Application page provides an Enable Traces checkbox for both REST and CSP/ZEN applications. Select the Enable Traces checkbox which is appropriate to your web application.

5. Select Save.

If you have created a TracerProvider object for the web application’s namespace at startup using SetTracerProvider()Opens in a new tab as described in the instrumentation instructions, InterSystems IRIS uses that TracerProvider to generate automatic web application traces. Otherwise, InterSystems IRIS uses a TracerProvider with default attributes.

For each request, the TracerProvider instantiates a Tracer object which is named after the web application. This Tracer emits a span which is named after the class that is dispatching the request (for CSP and SOAP requests) or the endpoint URL (for REST requests).

To populate an outbound request’s traceparent and tracestate headers with information about your application’s current active span, employ one of the injection methods that the %Trace.Propogation class provides, as described in the instrumentation instructions.

Within InterSystems IRIS, the %Trace package provides a straightforward API for instrumenting your application code to produce traces in a way that conforms to the OTel specification. Once instruments within your application are producing traces, InterSystems IRIS emits them to the OTLP/HTTP endpoint that you designate.

In a supported IDEOpens in a new tab, edit your application code to produce traces using the %Trace API as follows:

1. Optional: if you want to sampleOpens in a new tab a subset of your application’s traces, create an instance of a sampler class that extends %Trace.AbstractSamplerOpens in a new tab. You can implement your own sampling algorithm for your application by defining a custom ShouldSample() method. Refer to Implement a Custom Sampling Algorithm for further details.

   To help you get started, the %Trace package provides the following sampler classes, which have the following effects:

   • %Trace.AlwaysOffSamplerOpens in a new tab — the application does not record or emit any spans.

   • %Trace.AlwaysOnSamplerOpens in a new tab — the application records and emits every span.

   • %Trace.TraceIdRatioBasedSamplerOpens in a new tab — the application records and emits a fixed percentage of traces as specified by the sampler class’s Ratio property, which can assigned a decimal value between 0 and 1. To decide whether to sample a span, the sampler converts a hexadecimal interpretation of the rightmost characters of the span’s trace ID to a decimal number value and then compares that value against a threshold derived from the specified Ratio.

   • %Trace.ParentBasedSamplerOpens in a new tab — the application conditionally applies logic from different sampler classes to determine whether a span should be recorded and emitted, depending upon the following conditions:

     • whether the span is the root span of a trace or a child span.

     • whether the span’s parent was initialized locally or remotely.

     • whether the span’s parent was sampled or not.

   The following example creates an instance of the %Trace.TraceIdRatioBasedSamplerOpens in a new tab which configures the application to record and emit 10% of the traces that it creates.

   ObjectScript Python

    set sampleratio = 0.1
    set sampler = ##class(%Trace.TraceIdRatioBasedSampler).%New(sampleratio)

   

   sampleratio = 0.1
   sampler = iris.cls('%Trace.TraceIdRatioBasedSampler')._New(sampleratio)
   

   

2. Create an instance of the %Trace.TracerProviderOpens in a new tab class to serve as your application’s Tracer ProviderOpens in a new tab.

   The constructor for this class accepts two optional arguments. The first argument—passed by reference—is an array of key-value pairs which is used to set the TracerProvider object’s ResourceAttributes property. You can use this array to specify global attributes about your application. If you have defined a sampler object for your application, provide it to the constructor as its second argument. Any traces that the application generates using this TracerProvider will be sampled according to the algorithm described by the specified sampler object.

   The following example instantiates a TracerProvider (tracerProv) with a resource array identifying the name and version number of the service that it is tracing. Any traces which tracerProv creates will be sampled according to the logic described by sampler.

   ObjectScript Python

    set attributes("service.name") = "test_service"
    set attributes("service.version") = "2.0"
    set tracerProv = ##class(%Trace.TracerProvider).%New(.attributes, sampler)

   

   attributes = {}
   attributes["service.name"] = "test_service"
   attributes["service.version"] = "2.0"
   attrArray = iris.arrayref(attributes)
   tracerProv = iris.cls('%Trace.TracerProvider')._New(.attrArray, sampler)
   

   
   Note:

   If you have not specified a sampler object when you create a TracerProvider, InterSystems IRIS records and emits every span that was created by Tracer objects which that TracerProvider generated. However, it does not set the TraceFlags or TraceState properties for a span’s context.

3. In most situations, it is preferable to instantiate a single TracerProvider object at startup, for shared use by instrumentation code across the namespace throughout the entire life cycle of the application. To designate a TracerProvider object for use across a namespace, provide it to the SetTracerProvider() method of the %Trace.ProviderOpens in a new tab class, as follows:

   ObjectScript Python

    do ##class(%Trace.Provider).SetTracerProvider(tracerProv)

   

   iris.cls('%Trace.Provider').SetTracerProvider(tracerProv)
   

   

   When you need to access the TracerProvider object within your instrumentation code (as described in later steps), use the complementary GetTracerProvider() method to recall it:

   ObjectScript Python

    set tracerProv = ##class(%Trace.Provider).GetTracerProvider()

   

   tracerProv = iris.cls('%Trace.Provider').GetTracerProvider()
   

   

4. To instrument your application code, first use the TracerProvider object’s GetTracer()Opens in a new tab method to instantiate a Tracer. (The Tracer object generates the actual spans of the trace.)

   GetTracer() accepts two arguments, Name and Version. Use these arguments to uniquely identify the application or application component that you are tracing and specify its version number. For example:

   ObjectScript Python

    set tracer = tracerProv.GetTracer("service.orderprocessor", "2.0.2")

   

    tracer = tracerProv.GetTracer("service.orderprocessor", "2.0.2")
   

   

5. Start a root span using the Tracer object’s StartSpan()Opens in a new tab method. In order, StartSpan() accepts the following arguments, which are used to define the properties of the span:

   1. Name — A string, used to set the span’s name field

   2. Parent — (Optional.) A %Trace.ContextOpens in a new tab object that identifies the span which you wish to specify as the parent span for the new span. If you do not provide a Parent and you have not previously specified an active span (as described in a later step), then the method initializes the span as a root span, with a unique Trace ID.

   3. Spankind — (Optional.) A string, identifying the span as belonging to one of the OpenTelemetry specification’s recognized span kindsOpens in a new tab. If you do not provide a Spankind, the span is classified as Internal by default.

   4. AttributesOpens in a new tab — (Optional.) An array of key-value pairs, passed by reference.

   5. StartTime — (Optional.) A timestamp recording the span’s start time, in $ZTIMESTAMP format. If not provided, StartTime is set to the current time.

   For example, code which initializes a root span for processing a retail transaction may resemble the following:

   ObjectScript Python

    set rootAttr("customer.id") = customer.ID
    set rootAttr("product.id") = product.ID
    set rootSpan = tracer.StartSpan("order", , "Server", .rootAttr)

   

   rootspan = {}
   rootAttr("customer.id") = customer.ID
   rootAttr("product.id") = product.ID
   rootAttrArray = iris.arrayref(rootAttr)
   set rootSpan = tracer.StartSpan("order", , "Server", .rootAttrArray)
   

   
   Note:

   If you want your application to propagate trace context from a web request that it has received into a new span, use the HttpExtract()Opens in a new tab or CSPRequestExtract()Opens in a new tab method of the %Trace.PropagationOpens in a new tab class to enhance the context of the current active span with the context provided by an HTTP header string or the headers of a %CSP.RequestOpens in a new tab object (respectively). You can then specify this context object as the parent of the new span when you invoke StartSpan(). The following example demonstrates this procedure using a hypothetical CSP request, cspRequest:

   ObjectScript Python

    set currentContext = ##class(%Trace.Context).GetCurrentContext()
    set requestParentContext = ##class(%Trace.Propagation).CSPRequestExtract(cspRequest, currentContext)
    set restApiSpan = tracer.StartSpan("rest_function", requestParentContext, "Server")

   

   currentContext = iris.cls('%Trace.Context').GetCurrentContext()
   requestParentContext = iris.cls('%Trace.Propagation').CSPRequestExtract(cspRequest, currentContext)
   restApiSpan = tracer.StartSpan("rest_function", requestParentContext, "Server")
   

   

6. As needed, nest child spans hierarchically within this root span. For simple implementations, you can manually specify the parent span for a new span by creating a %Trace.Context object which identifies the desired parent as the ActiveSpan, and then providing that Context object as the Parent argument of StartSpan().

   However, attempting to manage context across lexical scopes using this manual approach would be impractical. For this reason, the %Trace API provides a dynamic scoping mechanism for managing context.

   To manage the distributed context of your trace dynamically, perform the following steps:

   1. Designate the parent span as the "active" span using the Tracer object’s SetActiveSpan()Opens in a new tab method, as follows:

      ObjectScript Python

       set rootScope = tracer.SetActiveSpan(rootSpan)

      

      rootScope = tracer.SetActiveSpan(rootSpan)
      

      

      SetActiveSpan() returns an instance of the %Trace.ScopeOpens in a new tab class. This Scope object acts as a record that the corresponding span is active.

   2. Start a new span by invoking StartSpan() without specifying a Parent, as follows:

      ObjectScript Python

       set childSpan1 = tracer.StartSpan("order_payproc")

      

      childSpan1 = tracer.StartSpan("order_payproc")
      

      

      As long as the active span’s Scope object remains in memory, StartSpan() initializes new spans as children of the active span by default when no other Parent is specified.

   3. To nest spans further, invoke SetActiveSpan() on a child span to designate it as the new active span and generate a new Scope object. The active span is identified by the newest Scope object which exists in memory at a given time. Therefore, once you have generated a Scope object for a new active span, StartSpan() will initialize new spans as its children by default.

      Continuing the previous example, the following code starts a new span childSpan2 as a child of childSpan1 (which is itself a child of rootSpan):

      ObjectScript Python

       set child1Scope = tracer.SetActiveSpan(childSpan1)
       set childSpan2 = tracer.StartSpan("order_payproc_addnewcard")

      

      child1Scope = tracer.SetActiveSpan(childSpan1)
      childSpan2 = tracer.StartSpan("order_payproc_addnewcard")
      

      

   4. When you destroy the Scope object for the current active span, the span which was previously active becomes the default parent for StartSpan() once again (assuming you have not destroyed its Scope object as well). Continuing the previous examples, the following code starts a new span childSpan3 as a child of rootSpan and a sibling of childSpan1:

      ObjectScript Python

       kill child1Scope
       set childSpan3 = tracer.StartSpan("order_sendconfirm", , "Server")

      

      iris.execute('kill child1Scope)
      childSpan3 = tracer.StartSpan("order_sendconfirm", , "Server")
      

      

7. As needed, define information about your spans. Available methods for this purpose include the following:

   • Use the Span object’s AddEvent()Opens in a new tab method to add a span eventOpens in a new tab.

   • Use the Span object’s AddLink()Opens in a new tab method to add a span linkOpens in a new tab.

   • Modify the TraceFlags and TraceState properties of the Span object’s Context property (an instance of %Trace.SpanContextOpens in a new tab) to modify a span’s trace flags and trace state, respectively.

   Continuing the previous examples, the following code enhances the "order_payproc" span (childSpan1) with a "paymentdeclined" event and a link to a hypothetical span named paymentProcLivenessSpan:

   ObjectScript Python

    set eventAttr("declined.reason")="Unknown error occurred."
    do childSpan1.AddEvent("paymentdeclined", .eventAttr)
    do childSpan1.AddLink(paymentProcLivenessSpan.Context)

   

   eventAttr = {}
   eventAttr("declined.reason") = "Unknown error occurred."
   eventAttrArray = iris.arrayref(eventAttr)
   childSpan1.AddEvent("paymentdeclined", .eventAttrArray)
   childSpan1.AddLink(paymentProcLivenessSpan.Context)
   

   

8. If you are sending an HTTP request from your application, invoke an injection method from the %Trace.Propogation class to propagate context into the request’s headers. If you are constructing the HTTP request as a %Net.HttpRequestOpens in a new tab object, use the HttpRequestInject()Opens in a new tab method to inject traceparent and tracestate headers into the request object automatically. For other use cases, the HttpInject()Opens in a new tab method accepts a string containing the headers for an HTTP request and appends the aforementioned trace context headers to it.

   Assuming a hypothetical %Net.HttpRequestOpens in a new tab object (request), the following code would retrieve the %Trace.ContextOpens in a new tab object for the current active span and inject traceparent and tracestate headers into request based on that context object. It would then dispatch the enhanced request object as an HTTP GET request.

   ObjectScript Python

    set currentcontext = ##class(%Trace.Context).GetCurrentContext()
    do ##class(%Trace.Propagation).HttpRequestInject(request, currentcontext)
    do request.Get("/payment/auth")

   

   currentcontext = iris.cls('%Trace.Context').GetCurrentContext()
   iris.cls('%Trace.Propagation').HttpRequestInject(request, currentcontext)
   request.Get("/payment/auth")
   

   

9. Before you end a span, update the span’s status using the Span object’s SetStatus()Opens in a new tab method, as in the following example:

   ObjectScript Python

    do childSpan1.SetStatus("Ok")

   

   childSpan1.SetStatus("Ok")
   

   

   SetStatus() accepts one argument (a string) which can have three possible values corresponding to the three span statusesOpens in a new tab recognized by the OpenTelemetry specification.

10. End each span using the Span object’s End()Opens in a new tab method. End() accepts an optional argument: a timestamp, in $TIMESTAMP format. If a timestamp is provided, End() records that time as the end time for the span. Otherwise, End() sets the span’s end time to the current time, as in the following example:

    ObjectScript Python

     do childSpan1.End()

    

    childSpan1.End()
    

    

    InterSystems IRIS invokes the OpenTelemetry SDK to export the span when you end it, assuming that the ‘sampled’ bit in the span Context property’s TraceFlags has been set appropriately.

11. If you are using SetActiveSpan() to manage nested spans across lexical scopes (as suggested in a preceding step), destroy the Scope object for each active span after you end the span. Continuing the previous examples, the following code would conclude the trace encompassed by rootSpan and prepare the application to record a new trace:

    ObjectScript Python

     do rootSpan.SetStatus("Ok")
     do rootSpan.End()
     kill rootScope

    

    rootSpan.SetStatus("Ok")
    rootSpan.End()
    iris.execute('kill rootScope')
    

    

12. As needed, import and recompile the code you have edited to enable tracing for your application.

As mentioned by the instrumentation instructions, you can sample a subset of your application’s spans according to a custom sampling algorithm by defining a custom sampler class.

In a supported IDEOpens in a new tab, create a new class within the namespace for your application. This class should extend %Trace.AbstractSamplerOpens in a new tab, and it must override the superclass’s ShouldSample() method with code that evaluates a span’s context according to your desired logic. When you invoke the TracerProvider for your application with a sampler object specified, the StartSpan() method of any Tracer object which the TracerProvider generates will invoke the sampler’s ShouldSample() method to determine whether to sample the new span.

The ShouldSample() method should accept the following arguments:

• Context — a %Trace.SpanContextOpens in a new tab object which contains information about the new span’s parent.

• TraceId — the trace ID associated with the new span’s parent span.

• Name — the name you have specified for the new span.

• SpanKind — the new span’s span kind.

• Attributes — the optional array of key-value pairs which the StartSpan() method received. You must pass this argument to the method by reference.

Given this information, the ShouldSample() method must return an instance of %Trace.SamplingResultOpens in a new tab. The application samples the new span based on the Decision property, which can have one of three values:

• "DROP" instructs the application to discard the span.

• "RECORD_ONLY" instructs the application to record the span, so that your application can still access the span object and its contents. However, InterSystems IRIS will not emit the span to the OTLP/HTTP endpoint when it ends.

• "RECORD_AND_SAMPLE" instructs the application to record the span and emit it to the OTLP/HTTP endpoint when it ends.

The %Trace.SamplingResultOpens in a new tab that your ShouldSpan() method returns should maintain or update the trace state for the span using the TraceState property. You can also use its Attributes property to add attributes to the Attributes array for the new span.

The %Trace package provides several sampler classes to assist you as you implement your desired sampling algorithm. The %Trace.ParentBasedSamplerOpens in a new tab class is especially noteworthy: it allows you to define multiple custom sampler classes and apply them conditionally to evaluate spans depending upon whether they are root spans or child spans, where their parent spans were created, and whether their parent spans were sampled. The reference for the class describes the conditions which are available (these are mapped as properties within the class). The reference for the %OnNew()Opens in a new tab method describes the order in which you should supply these instances to the constructor when you create an instance of the %Trace.ParentBasedSamplerOpens in a new tab within your application code. You would then specify this instance of the %Trace.ParentBasedSamplerOpens in a new tab class as the sampler for your application’s TracerProvider.

To deactivate tracing for your application after you have instrumented it, perform the following steps:

1. Edit your code so that it initializes any Tracer Provider that your application uses as an instance of %Trace.NoopTracerProviderOpens in a new tab (instead of %Trace.TracerProviderOpens in a new tab). Revisiting the example provided in the preceding instrumentation instructions, deactivation would require you to change the line of code which sets the tracerProv object so that it reads as follows:

   ObjectScript Python

    set tracerProv = ##class(%Trace.NoopTracerProvider).%New(.attributes)

   

   tracerProv = iris.cls('%Trace.NoopTracerProvider')._New(.attrArray)
   

   

   Assuming that tracerProv has been set as the Tracer Provider which serves the entire namespace, no further edits would be necessary.

2. As needed, import and recompile the code you have edited to deactivate tracing for your application.