Working with Streams
Streams provide a way to store extremely large amounts of data (longer than the string length limit). You can define stream properties in any object class. You can also define standalone stream objects for other purposes, such as for use as an method argument or return value. This topic describes streams and stream properties.
Introduction to Stream Classes
InterSystems IRIS® data platform provides the following stream classes:
-
%Stream.GlobalCharacterOpens in a new tab — use this to store character data in global nodes.
-
%Stream.GlobalBinaryOpens in a new tab — use this to store binary data in global nodes.
-
%Stream.FileCharacterOpens in a new tab — use this to store character data in an external file.
-
%Stream.FileBinaryOpens in a new tab — use to store binary data in an external file.
-
%Stream.TmpCharacterOpens in a new tab — use this when you need a stream to hold character data but do not need to save the data.
-
%Stream.TmpBinaryOpens in a new tab — use this when you need a stream to hold binary data but do not need to save the data.
These classes all inherit from %Stream.ObjectOpens in a new tab, which defines the common stream interface.
The %Library package also includes stream classes, but those are deprecated. The class library includes additional stream classes, but those are not intended for general use.
Note that stream classes are object classes. Thus a stream is an object.
Many of the methods of these classes return status values. In all cases, consult the class reference for details. If a method returns a status value, your code should check that returned value and proceed appropriately. Similarly, for %Stream.FileCharacterOpens in a new tab and %Stream.FileBinaryOpens in a new tab, if you set the Filename property, you should next check for an error by examining %objlasterror.
Defining Stream Properties
InterSystems IRIS supports both binary streams and character streams. Binary streams contain the same sort of data as type %BinaryOpens in a new tab, and are intended for very large binary objects such as pictures. Similarly, character streams contain the same sort of data as type %StringOpens in a new tab, and are intended for storing large amounts of text. Character streams, like strings, may undergo a Unicode translation within client applications.
Stream data may be stored in an external file or an InterSystems IRIS global (or not at all), depending on how the stream property is defined:
-
The %Stream.FileCharacterOpens in a new tab and %Stream.FileBinaryOpens in a new tab classes are used for streams stored as external files.
-
The %Stream.GlobalCharacterOpens in a new tab and %Stream.GlobalBinaryOpens in a new tab classes are used for streams stored as globals.
-
The %Stream.TmpCharacterOpens in a new tab and %Stream.TmpBinaryOpens in a new tab classes are used for streams that do not need to be saved.
The first four classes can use the optional LOCATION parameter to specify a default storage location.
In the following example, the JournalEntry class contains four stream properties (one for each of the first four stream classes), and specifies a default storage location for two of them:
Class testPkg.JournalEntry Extends %Persistent
{
Property DailyText As %Stream.FileCharacter;
Property DailyImage As %Stream.FileBinary(LOCATION = "C:/Images");
Property Text As %Stream.GlobalCharacter(LOCATION = "^MyText");
Property Picture As %Stream.GlobalBinary;
}
In this example, data for DailyImage is stored in a file (with an automatically generated name) in the C:/Images directory, while the data for the Text property is stored in a global named ^MyText.
Using the Stream Interface
All streams inherit a set of methods and properties used to manipulate the data they contain. The next section lists the commonly used methods and properties, and the following sections provide concrete examples using them:
Many of the methods of these classes return status values. In all cases, consult the class reference for details. If a method returns a status value, your code should check that returned value and proceed appropriately. Similarly, for %Stream.FileCharacterOpens in a new tab and %Stream.FileBinaryOpens in a new tab, if you set the Filename property, you should next check for an error by examining %objlasterror.
Commonly Used Stream Methods and Properties
Some commonly used methods include the following:
-
Read() — Read a specified number of characters starting at the current position in the stream.
-
Write() — Append data to the stream, starting at the current position. Overwrites existing data if the position is not set to the end of the stream.
-
Rewind() — Move to the beginning of the stream.
-
MoveTo() — Move to a given position in the stream.
-
MoveToEnd() — Move to the end of the stream.
-
CopyFrom() — Copy the contents of a source stream into this stream.
-
NewFileName() — Specify a filename for a %Stream.FileCharacterOpens in a new tab or %Stream.FileBinaryOpens in a new tab property.
Commonly used properties include the following:
-
AtEnd — Set to true when a Read encounters the end of the data source.
-
Id — The unique identifier for an instance of a stream within the extent specified by %Location.
-
Size — The current size of the stream (in bytes or characters, depending on the type of stream).
For detailed information on individual stream methods and properties, see the InterSystems Class Reference entries for the classes listed at the beginning of this topic.
Instantiating a Stream
When you use a stream class as an object property, the stream is instantiated implicitly when you instantiate the containing object.
When you use a stream class as a standalone object, use the %New() method to instantiate the stream.
Reading and Writing Stream Data
At the core of the stream interface are the methods Read(), Write(), and Rewind() and the properties AtEnd and Size.
The following example reads data from the Person.Memo stream and writes it to the console, 100 characters at a time. The value of len is passed by reference, and is reset to 100 before each Read. The Read method attempts to read the number of characters specified by len, and then sets it to the actual number of characters read:
Do person.Memo.Rewind()
While (person.Memo.AtEnd = 0) {
Set len = 100
Write person.Memo.Read(.len)
}
Similarly, you can write data into the stream:
Do person.Memo.Write("This is some text. ")
Do person.Memo.Write("This is some more text.")
Specifying a Translation Table
If you are reading or writing a stream of type %Stream.FileCharacterOpens in a new tab in any character set other than the native character set of the locale, you must set the TranslateTable property of the stream. See the Translation Tables reference page.
Copying Data between Streams
All streams contain a CopyFrom() method which allows one stream to fill itself from another stream. This can be used, for example, to copy data from a file into a stream property. In this case, one uses the %Library.FileOpens in a new tab class, which is a wrapper around operating system commands and allows you to open a file as a stream. In this case, the code is:
// open a text file using a %Library.File stream
Set file = ##class(%File).%New("\data\textfile.txt")
Do file.Open("RU") // same flags as the OPEN command
// Open a Person object containing a Memo stream
// and copy the file into Memo
Set person = ##class(Person).%New()
Do person.Memo.CopyFrom(file)
Do person.%Save() // save the person object
Set person = "" // close the person object
Set file = "" // close the file
You can also copy data into a stream with the Set command:
Set person2.Memo = person1.Memo
where the Memo property of the Person class holds an OREF for a stream and this command copies the contents of person1.Memo into person2.Memo.
Inserting Stream Data
Apart from the temporary stream classes (whose data cannot be saved), streams have both a temporary and a permanent storage location. All inserts go into the temporary storage area, which is only made permanent when you save the stream. If you start inserting into a stream, then decide that you want to abandon the insert, the data stored in the permanent location is not altered.
If you create a stream, start inserting, then do some reading, you can call MoveToEnd() and then continue appending to the temporary stream data. When you save the stream, the data is moved to the permanent storage location.
For example:
Set test = ##class(Test).%OpenId(5)
Do test.text.MoveToEnd()
Do test.text.Write("append text")
Do test.%Save()
Here, the stream is saved to permanent storage when the test object is saved.
Finding Literal Values in a Stream
The stream interface includes the FindAt() method, which you can use to find the location of a given literal value. This method has the following arguments:
method FindAt(position As %Integer, target, ByRef tmpstr, caseinsensitive As %Boolean = 0) as %Integer
Where:
-
position is the position at which to start searching.
-
target is the literal value to search for.
-
tmpstr, which is passed by reference, returns information that can be used in the next call to FindAt(). Use this when you want to search the same stream repeatedly, starting from the last position where the target was found. In this scenario, specify position as –1 and pass tmpstr by reference in every call. Then each successive call to FindAt() will start where the last call left off.
-
caseinsensitive specifies whether to perform a case-insensitive search. By default, the method does not consider case.
The method returns the position at this match starting at the beginning of the stream. If it does not find a match, it returns -1.
Saving a Stream
When you use a stream class as an object property, the stream data is saved when you save the containing object.
When you use a stream class as a standalone object, use the %Save() method to save the stream data. (Note that for the temporary stream classes — %Stream.TmpCharacterOpens in a new tab and %Stream.TmpBinaryOpens in a new tab — this method returns immediately and does not save any data.)
Using Streams in Object Applications
Stream properties are manipulated via a transient object that is created by the object that owns the stream property. Streams act as literal values (think of them as large strings). Two object instances cannot refer to the same stream.
In the following class definition, the Person class has a Memo property that is a stream property:
Class testPkg.Person Extends %Persistent
{
Property Name As %String;
Property Memo As %Stream.GlobalCharacter;
}
The following ObjectScript fragment creates a new person object, implicitly instantiating the Memo stream. Then it writes some text to the stream.
// create object and stream
Set p = ##class(testPkg.Person).%New()
Set p.Name = "Mo"
Do p.Memo.Write("This is part one of a long memo. ")
Do p.Memo.Write("This is part two of a long memo. ")
Do p.Memo.Write("This is part three of a long memo. ")
Do p.Memo.Write("This is part four of a long memo. ")
Do p.%Save()
Set id = p.%Id() // remember ID for later
Set p = ""
The following code fragment opens the person object and then writes the contents of the stream. Note that when you open an object, the current position of any stream properties is set to the beginning of the stream. This code uses the Rewind() method for illustrative purposes.
// read object and stream
Set p = ##class(testPkg.Person).%OpenId(id)
Do p.Memo.Rewind() // not required first time
// write contents of stream to console, 100 characters at a time
While (p.Memo.AtEnd = 0) {
Set len = 100
Write p.Memo.Read(.len)
}
Set p = ""
If you want to replace the contents of a stream property, rewind the stream (if the current position of the stream is not already at the beginning), and then use the Write() method to write the new data to the stream. Do not use the %New() method to instantiate a new stream object and assign it to the stream property, for example, set p.Memo = ##class(%Stream.GlobalCharacter).%New(), as this leaves the old stream object orphaned in the database.
Stream Classes for Use with gzip Files
The %StreamOpens in a new tab package also defines the specialized stream classes %Stream.FileBinaryGzipOpens in a new tab and %Stream.FileCharacterGzipOpens in a new tab, which you can use to read and write to gzip files. These use the same interface described earlier. Note the following points:
-
For these classes, the Size property returns the uncompressed size. When you access the Size property, InterSystems IRIS reads the data in order to calculate the size of the file and this can be an expensive operation
-
When you access the Size property, InterSystems IRIS rewinds the stream and leaves you at its start.
Projection of Stream Properties to SQL and ODBC
A persistent class is projected as an SQL table. For such classes, character stream properties and binary stream properties are projected to SQL (and to ODBC clients) as BLOBs (binary large objects).
Stream properties are projected with the ODBC type LONG VARCHAR (or LONG VARBINARY). The ODBC driver/server uses a special protocol to read/write BLOBs. Typically you have to write BLOB applications by hand, because the standard reporting tools do not support them.
The following subsections describes how to use stream properties with SQL. It includes the following topics:
Stream fields have the following restrictions within SQL:
-
You cannot use a stream value in a WHERE clause, with a few specific exceptions.
-
You cannot UPDATE/INSERT multiple rows containing a stream; you must do it row by row.
-
The only kind of index you can add is an SQL Search bitmap index. See Indexing Sources for SQL Search.
Also see Stream Data Types in the InterSystems SQL Reference for additional comments on using streams in SQL.
Reading a Stream via Embedded SQL
You can use embedded SQL to read a stream as follows:
-
Use embedded SQL to select the OID (Object ID) of the stream:
&sql(SELECT Memo INTO :memo FROM Person WHERE Person.ID = 12345)
This fetches the OID of the stream and places it into the memo host variable.
-
Then open the stream and process it as usual.
Writing a Stream via Embedded SQL
To write a stream via embedded SQL, you have several options. For the value to insert, you can use an object reference (OREF) of a stream, the string version of such an OREF, or a string literal.
The following examples show all these techniques. For these examples, assume that you have a table named Test.ClassWStream which has a column named Prop1, which expects a stream value.
The following example uses an object reference:
///use an OREF
ClassMethod Insert1()
{
set oref=##class(%Stream.GlobalCharacter).%New()
do oref.Write("Technique 1")
//do the insert; this time use an actual OREF
&sql(INSERT INTO Test.ClassWStreams (Prop1) VALUES (:oref))
}
The next example uses a string version of an object reference:
///use a string version of an OREF
ClassMethod Insert2()
{
set oref=##class(%Stream.GlobalCharacter).%New()
do oref.Write("Technique 2")
//next line converts OREF to a string OREF
set string=oref_""
//do the insert
&sql(INSERT INTO Test.ClassWStreams (Prop1) VALUES (:string))
}
The last example inserts a string literal into the stream Prop1:
///insert a string literal into the stream column
ClassMethod Insert3()
{
set literal="Technique 3"
//do the insert; use a string
&sql(INSERT INTO Test.ClassWStreams (Prop1) VALUES (:literal))
}
The first character of the string literal cannot be a number. If it is a number, then SQL interprets this as an OREF and attempts to file it as such. Because the stream is not an OREF, this results in an SQL -415 error.
Stream Compression
InterSystems IRIS can automatically compress stream data that is stored in a global, saving database space and reducing the size of journal files.
Compression is controlled in stream classes by a class parameter, COMPRESS.
In the classes %Stream.TmpCharacterOpens in a new tab and %Stream.TmpBinaryOpens in a new tab, COMPRESS is set to 0, meaning that the stream data is not compressed.
In the classes %Stream.GlobalCharacterOpens in a new tab and %Stream.GlobalBinaryOpens in a new tab, COMPRESS is set to 1, meaning that new stream data is automatically compressed, except for the following cases, where the data is not suitable for compression:
-
The stream can be stored in a single chunk less than 1024 characters long.
-
The stream is already compressed, that is, the first chunk of data matches a typical compressed file format, such as JPEG, MP3, or ZIP.
-
The first 4KB of the stream cannot be compressed by at least 20 percent.