Using JSON in Caché
Creating and Modifying Dynamic Entities
[Home] [Back] [Next]
InterSystems: The power behind what matters   
Class Reference   
Search:    

This chapter provides basic information on how dynamic entities work. The following topics are discussed:

Using JSON Literal Constructors
Dynamic entities are instances of %DynamicObject or %DynamicArray, which are designed to integrate JSON data manipulation seamlessly into Caché. Although you can create instances of these classes with the standard %New() method, dynamic entities support a much more flexible and intuitive set of constructors. JSON literal constructors allow you to create dynamic entities by directly assigning a JSON string to a variable. For example, the following code creates empty instances of %DynamicObject and %DynamicArray:
   set dynamicObject = {}
   set dynamicArray = []
   write dynamicObject,!,dynamicArray

3@%Library.DynamicObject
1@%Library.DynamicArray
Unlike the %New() constructor, literal constructors {} and [] can accept a string in JSON format as an argument. For example, the following code creates a dynamic object with a property named prop1:
   set dynamicObject = {"prop1":"a string value"}
   write dynamicObject.prop1

a string value
In fact, JSON literal constructors {} and [] can be used to specify any valid JSON array or object structure. In simple terms, any valid JSON literal string is also a valid Caché expression that evaluates to a dynamic entity.
Note:
JSON property names must always be quoted
The JSON language specification (see http://json.org/) is a subset of Javascript Object Notation, and enforces stricter rules in some areas. One important difference is that the JSON specification requires all property names to be enclosed in double-quotes. Javascript syntax, on the other hand, permits unquoted names in many cases.
A dynamic entity stores an exact representation of each object property or array element in the JSON string. Any dynamic entity can use the %ToJSON() method to return the stored data as a JSON string. There is no loss or corruption of data when converting to or from a literal string. The following example creates a dynamic array and then calls %ToJSON() to construct and return a new JSON string representing the stored data:
   set dynamicArray = [[1,2,3],{"A":33,"a":"lower case"},1.23456789012345678901234,true,false,null,0,1,""]
   write dynamicArray.%ToJSON()

[[1,2,3],{"A":33,"a":"lower case"},1.23456789012345678901234,true,false,null,0,1,""]
This dynamic array has stored and returned several significant values:
Using Dynamic Expressions and Dot Syntax
There are significant differences between the way values are stored in JSON and the way they are expressed in ObjectScript. JSON data storage would not be very useful if you had to convert an ObjectScript value to or from JSON syntax every time you wanted to use it, so dynamic entities are designed to make this conversion process transparent. You can always store and retrieve an ObjectScript value without worrying about its representation in JSON syntax.
Literal JSON constructors are no exception to this rule. So far, all of our examples have been entirely in JSON syntax, but literal constructors can also accept values defined in dynamic expressions, which are simply ObjectScript expressions enclosed in parentheses.
For example, the following dynamic array constructor stores two Unicode characters. At runtime, the literal constructor evaluates each element and stores the evaluated value. The first element is defined in JSON syntax and the second element is a Caché function call, but the resulting stored values are identical:
   write ["\u00E9",($CHAR(233))].%ToJSON()

["é","é"]
You can think of an ObjectScript expression as the code on the right side of a set statement. Any ObjectScript expression that evaluates to a value rather than an object reference can be serialized to a JSON literal string. The following example stores a $LIST value (which is a delimited string, not an object) in object property obj.list. It then creates array and extracts each list item in obj.list to a separate element:
   set obj = {"list":($LISTFROMSTRING("Deborah Noah Martha Bowie"," "))}
   set array = [($LIST(obj.list,1)),($LIST(obj.list,2)),($LIST(obj.list,3)),($LIST(obj.list,4))]
   write obj.%ToJSON(),!,array.%ToJSON()

{"list":"\t\u0001Deborah\u0006\u0001Noah\b\u0001Martha\u0007\u0001Bowie"}
["Deborah","Noah","Martha","Bowie"]
You cannot use a dynamic expression to define a property name (although there are ways to define property names programmatically. See Using %Set(), %Get(), and %Remove() for details).
Of course, literal constructors are not the only way to manipulate object properties and array elements. For example, the following code creates an empty dynamic object and uses standard object dot syntax to define the contents:
   set dynArray = []
   set dynArray."0" = "02"_"33"
   set dynArray."1" = {}
   set dynArray."1".foo = $CHAR(dynArray."0")
   write dynArray.%ToJSON()

[233,{"foo":"é"}]
In this example, literal constructors are used only to create empty dynamic entities. The assignment statements obey a few simple rules:
As previously mentioned, values are always stored and retrieved in ObjectScript format regardless of how they are represented in JSON syntax. The following examples demonstrate a few more facts that you should be aware of when using dot syntax.
Creating dynamic object properties with dot syntax
This example uses a literal constructor and dot syntax to create dynamic object dynObj, containing properties named A, a, and C quote. In the literal string, all property names must be quoted. In the set statements and the write statement, quotes are not required for property names a or A, but must be used for C quote:
   set dynObj = {"a":"stuff"}
   set dynObj."C quote" = " ""C quote"" contains a space "
   set dynObj.a = " lower case ""a"" "
   set dynObj.A = " upper case ""A"" "
   write !,dynObj.%ToJSON()

{"a":" lower case \"a\" ","C quote":" \"C quote\" contains a space ","A":" upper case \"A\" "}
Dynamic objects are unordered lists, so values will not necessarily be stored in the order they were created. See Iterating over a Dynamic Entity with %GetNext() for examples that demonstrate this.
Creating dynamic array elements with dot syntax
This example assigns a value to array element 3 before defining element 2. Elements do not have to be defined in order, and element 2 could have been left undefined. See Understanding Sparse Arrays and Unassigned Values for detailed information.
   set dynArray = [true,false]
   set dynArray."3" = "three"
   set dynArray."2" = 0
   write dynArray.%ToJSON()

[true,false,0,"three"]
Although the first two elements were defined and stored as JSON boolean values true and false, they are returned as integers 1 and 0, which are the equivalent Caché boolean values:
   write "0=/"_dynArray."0"_"/, 1=/"_dynArray."1"_"/, 2=/"_dynArray."2"_"/, 3=/"_dynArray."3"_"/"

0=/1/, 1=/0/, 2=/0/, 3=/three/
Since stored values are always returned in ObjectScript format, JSON true, false, and null are returned as Caché 0, 1, and "" (empty string). However, the original JSON values are preserved in the dynamic entity and can be recovered if necessary. See Working with Datatypes for information on identifying the original datatype of a stored value.
Note:
Dot syntax should not be used with very long property names
Although dynamic object properties can have names of any length, ObjectScript cannot use property names longer than 180 characters. If a dynamic object property name exceeds this limit, an attempt to use the name in dot syntax will cause Caché to throw a misleading <PROPERTY DOES NOT EXIST> error, even though the property exists and the name is valid. You can avoid this error by using the %Set() and %Get() methods, which accept property names of any length.
Using %Set(), %Get(), and %Remove()
Although literal constructors and dot syntax can be used to create dynamic entity members and manipulate values, they are not adequate for all purposes. Dynamic entities provide %Set(), %Get(), and %Remove() methods for full programmatic control over create, read, update, and delete operations.
One of the most important advantages to these methods is that member identifiers (property names and array index numbers) do not have to be literals. You can use ObjectScript variables and expressions to specify both values and identifiers.
Specifing values and identifiers programmatically with %Set(), %Get(), and %Remove()
The following example creates an object using literal constructor {}, and calls the %Set() method of the new object to add a series of properties named propn with a value of 100+n. Both names and values are defined by ObjectScript expressions:
   set dynObj = {}
   for i=1:1:5 { do dynObj.%Set("prop"_i,100+i) }
   write dynObj.%ToJSON()

{"prop1":101,"prop2":102,"prop3":103,"prop4":104,"prop5":105}
The same variables can be used with %Get() to retrieve the property values:
   for i=1:1:5 { write dynObj.%Get("prop"_i)_" " }

101 102 103 104 105
The %Remove() method deletes the specified member from the dynamic entity and returns the value. This example removes three of the five properties and concatenates the return values to string removedValues. The write statement displays the string of removed values and the current contents of dynObj:
   set removedValues = ""
   for i=2:1:4 { set removedValues = removedValues_dynObj.%Remove("prop"_i)_" " }
   write "Removed values: "_removedValues,!,"Remaining properties: "_dynObj.%ToJSON()

Removed values: 102 103 104
Remaining properties: {"prop1":101,"prop5":105}
Note:
Although a for loop is used in these simple examples, the normal iteration method would be %GetNext() (described later in Iterating over a Dynamic Entity with %GetNext()).
Both %Get() and %Remove() return a ObjectScript value for the specified member, but there is an important difference in how embedded dynamic entities are returned:
Retrieving a nested dynamic entity with %Get() and %Remove()
In the following example, the value of property dynObj.address is a dynamic object. The %Get() statement stores a reference to the property (not the property value) in variable addrPointer. At this point, addrPointer can be used to access the road property of embedded entity address:
   set dynObj = {"name":"greg", "address":{"road":"Old Road"}} 
   set addrPointer = dynObj.%Get("address")
   set dynObj.address.road = "New Road"
   write "Value of "_addrPointer_" is "_addrPointer.road

Value of 2@%Library.DynamicObject is New Road
The %Remove() statement destroys the property and returns a new OREF to the property value.
   set addrRemoved =  dynObj.%Remove("address")
   write "OREF of removed property: "_addrPointer,!,"OREF returned by %Remove(): "_addrRemoved

OREF of removed property: 2@%Library.DynamicObject
OREF returned by %Remove(): 3@%Library.DynamicObject
After the call to %Remove(), addrRemoved contains a valid OREF to the formerly embedded dynamic object, and variable addrPointer contains an invalid OREF to the former location of property dynObj.address:
   write addrRemoved.%ToJSON()

{"road":"New Road"}

   write addrPointer.%ToJSON()

WRITE addrPointer.%ToJSON()
^
<INVALID OREF>
You can use the %Remove() method to remove members in any order. This has different implications for objects and arrays, as demonstrated in the following examples.
Removing an object property
Object properties have no fixed order. This means that properties can be destroyed in any order, but removing a property and adding another may also change the order in which properties are serialized and returned. The following example creates a dynamic object, and defines three properties with three consecutive calls to %Set():
   set dynObject={}.%Set("propA","abc").%Set("PropB","byebye").%Set("propC",999)
   write dynObject.%ToJSON()

{"propA":"abc","PropB":"byebye","propC":999}
Now %Remove() is called to destroy property PropB, after which new property PropD is added. The resulting dynamic object does not serialize its properties in the order they were created:
   do dynObject.%Remove("PropB")
   set dynObject.propD = "added last"
   write dynObject.%ToJSON()

{"propA":"abc","propD":"added last","propC":999}
This also affects the order in which iterator method %GetNext() returns properties. See Iterating over a Dynamic Entity with %GetNext() for a similar example that uses %GetNext().
Removing an array element
An array is an ordered list. When you call %Remove() on an element, all elements after that one will have their array index number decremented by 1. The following example makes three consecutive calls to %Remove(1), removing a different element each time:
   set dynArray = ["a","b","c","d","e"]
   set removedValues = ""
   for i=1:1:3 { set removedValues = removedValues_dynArray.%Remove(1)_" " }
   write "Removed values: "_removedValues,!,"Array size="_dynArray.%Size()_": "_dynArray.%ToJSON()

Removed values: b c d
Array size=2: ["a","e"]
A stack operation is usually implemented with %Push() and %Pop() rather than %Set() and %Remove(), but you can implement a queue by replacing %Pop() with %Remove(0) (see Using %Push and %Pop with Dynamic Arrays).
%Remove() works the same way with all arrays, including those that contain elements with undefined values. See Understanding Sparse Arrays and Unassigned Values for an example demonstrating how %Remove() works with sparse arrays.
Method Chaining
The %Set(), and %Push() methods return a reference to the entity that they have modified. The returned reference can immediately be used to call another method on the same entity, within the same expression.
The dynamic entity that begins a chain can be either a constructor ({}, or []) or an existing entity. Methods %Set() and %Push() return chainable references and can be called from anywhere in the chain. The last item in a chain can be any method available to the entity.
In the following example, a single write statement uses chained calls to %FromJSON(), %Set(), %Push(), and %ToJSON() to create, modify, and display a dynamic array:
   set jstring = "[123]"
   write [].%FromJSON(jstring).%Set(1,"one").%Push("two").%Push("three").%Set(1,"final value").%ToJSON()

[123,"final value","two","three"]
%FromJSON() is only useful as the first method call in a chain, since it does not return a modified version of the calling entity. Instead, it simply ignores the calling entity and returns an entirely new instance deserialized from a JSON string. For more information, see Converting Dynamic Entities to and from JSON.
You could also start a chain by retrieving a nested entity with %Get(), %Pop(), %GetNext(), or %Remove().
Error Handling
Dynamic entities throw exceptions in the case of an error, rather than returning a %Status value. In the following example, the thrown exception includes enough information to conclude that the second character in the method argument is invalid:
   set invalidObject = {}.%FromJSON("{:}")

<THROW>%FromJSON+37^%Library.DynamicAbstractObject.1 *%Exception.General Parsing error 3 Line 1 Offset 2
When dealing with dynamic data, it is always wise to assume that some data will not fit your expectations. Any code that makes use of dynamic objects should be surrounded with a TRY-CATCH block at some level (see The TRY-CATCH Mechanism in Using Caché ObjectScript). For example:
   TRY {
      set invalidObject = {}.%FromJSON("{:}")
   }
   CATCH errobj {
      write errobj.Name_", "_errobj.Location_", error code "_errobj.Code,!
      RETURN
   }

Parsing error, Line 1 Offset 2, error code 3
Converting Dynamic Entities to and from JSON
You can use the %ToJSON() method to serialize a dynamic entity (convert it to a JSON string) and the %FromJSON() method to deserialize (convert JSON to a dynamic entity).
Serializing a dynamic entity to JSON
The following example creates and modifies a dynamic object, and then uses %ToJSON() to serialize it and display the resulting string:
   set dynObject={"prop1":true}.%Set("prop2",123).%Set("prop3","foo")
   set objString = dynObject.%ToJSON()
   write objString

{"prop1":true,"prop2":123,"prop3":"foo"}
A dynamic array is serialized in the same way:
   set dynArray=[].%Push("1st value").%Push("2nd value").%Push("3rd value")
   set arrayString = dynArray.%ToJSON()
   write arrayString

["1st value","2nd value","3rd value"]
Both of these examples use method chaining (see Method Chaining earlier in this chapter).
Deserializing from JSON to a dynamic object
The %FromJSON() method converts a JSON string to a dynamic entity. The following example constructs a dynamic array and serializes it to string jstring. A call to %FromJSON() deserializes jstring to a new dynamic entity named newArray, which is then modified and displayed:
   set jstring=["1st value","2nd value","3rd value"].%ToJSON()
   set newArray={}.%FromJSON(jstring)
   do newArray.%Push("new value")
   write "New entity:"_newArray.%ToJSON()

New entity:["1st value","2nd value","3rd value","new value"]
Notice this example calls %FromJSON() from a dynamic object constructor ({}) even though the returned value is a dynamic array. %FromJSON() is a class method of %DynamicAbstractObject, and can therefore be called from any dynamic entity or constructor.
Cloning with %ToJSON() and %FromJSON()
Since each call to %FromJSON() creates a new dynamic entity, it can be used to duplicate an existing entity or initialize a set of identical entities.
In the following example, the value of property dynObj.address is a dynamic object. The property is referenced by variable addrPointer, and the property value is cloned by calling %FromJSON() to create new dynamic object addrClone:
   set dynObj = {}.%FromJSON({"name":"greg", "address":{"road":"Dexter Ave."}}.%ToJSON())
   set addrPointer = dynObj.address 
   set addrClone = {}.%FromJSON(dynObj.address.%ToJSON())
Variable addrPointer is just a reference to property dynObj.address, but addrClone is an independent instance of %DynamicObject that can be modified without affecting the original value:
   set addrPointer.road = "Wright Ave."
   set addrClone.road = "Sinister Ave."
   write !,"Property = "_dynObj.address.%ToJSON(),!,"Clone = "_addrClone.%ToJSON()

Property = {"road":"Wright Ave."}
Clone = {"road":"Sinister Ave."}
Serializing Large Dynamic Entities to Streams
If a dynamic entity is large enough, the output of %ToJSON() may exceed the maximum possible length for a Caché string (see Long String Limit in the Caché Programming Orientation Guide). The examples in this section use a maximum length Caché string named longStr. The following code fragment demonstrates how longStr is generated:
   set longStr=""
   for i=1:1:$SYSTEM.SYS.MaxLocalLength() { set longStr = longStr_"x" }
   write "Maximum string length = "_$LENGTH(longStr)

Maximum string length = 3641144
Whenever an expression uses the return value of %ToJSON(), Caché builds the string on the program stack, which is subject to the long string limit. For example, a read/write statement such as write dyn.%ToJSON() or an assignment statement such as set x=dyn.%ToJSON() will attempt to put the string on the stack. The following example adds two copies of longStr to a dynamic array and attempts to assign the serialized string to a variable, causing Caché to return a <MAXSTRING> error:
   set longArray = [(longStr),(longStr)]
   set tooBig = longArray.%ToJSON()
SET tooBig = longArray.%ToJSON()
^
<MAXSTRING>
The general solution to this problem is to pass the %ToJSON() output by reference in a DO command, without actually examining the return value. Output is written directly to the current device, and there is no limit on the length of the output. In the following examples, the device is a stream.
Writing to a file stream
This example writes dynamic object longObject to a file and then retrieves it. Variable longStr is the value defined at the beginning of this section:
   set longObject = {"a":(longStr),"b":(longStr)}
   set file=##class(%File).%New("c:\temp\longObjectFile.txt")
   do file.Open("WSN")
   do longObject.%ToJSON(file)
   do file.Close()

   do file.Open("RS")
   set newObject = {}.%FromJSON(file)
   write !,"Property newObject.a is "_$LENGTH(newObject.a)_" characters long."

Property newObject.a is 3641144 characters long.
This solution can also be used to read input from other streams.
Reading and writing global character streams
In this example, we serialize two large dynamic entities (using temporary streams because %ToJSON() can only serialize one entity per stream). Standard stream handling methods are used to store each temporary stream as a separate line in stream bigLines:
   set tmpArray = ##class(%Stream.GlobalCharacter).%New()
   set dyn = [(longStr),(longStr)]
   do dyn.%ToJSON(tmpArray)

   set tmpObject = ##class(%Stream.GlobalCharacter).%New()
   set dyn = {"a":(longStr),"b":(longStr),"c":(longStr)}
   do dyn.%ToJSON(tmpObject)

   set bigLines = ##class(%Stream.GlobalCharacter).%New()
   do bigLines.CopyFrom(tmpArray)
   do bigLines.WriteLine()
   do bigLines.CopyFrom(tmpObject)
Later, we can deserialize each dynamic entity from bigLines:
   do bigLines.Rewind()
   while ('bigLines.AtEnd) { 
      write !,{}.%FromJSON(bigLines.ReadLineIntoStream())
   }

7@%Library.DynamicArray
7@%Library.DynamicObject