%SQL.HLL

class %SQL.HLL extends %Library.RegisteredObject

ObjectScript API for building Hyper Log Log estimates of the number of unique elements (cardinality) in a group of data.

The estimates are kept in containers called sketches. The containers are identified by the id of this class.
Lets assume you have 1 million pieces of data and want to know how many of those pieces are unique:
1) Use %New() to instantiate a new HLL object:

   set hll= ##class(%SQL.HLL).%New()

2) Feed one million pieces of data into the sketch with update():

   for i=1:1:1000000 {do hll.update(i)}

3) Get an estimate of the cardinality by calling estimate()

   write hll.estimate()

Notes: We test this class at Intersystems by using murmur hash with a seed of hll.#SEED:
$zcrc(yourdata,9,2059198193) or $zcrc(yourdata,9,hll.#SEED)
The underlying library uses 64 bits of this 128 bit hash.

Estimate Partitioning: pass an existing sketch into %New() to initialize its state
from the standard serialized form (optionally Base64 encoded).
To combine estimates get() and merge() your sketches, if your data is distributed
across many processes.

Method Inventory

%OnClose()
%OnNew()
estimate()
get()
merge()
update()
updateHash()
version()

Parameters

parameter ENCODE = 1;

Whether or not to Base64 encode/decode by default during get() and %New()

parameter SEED = 2059198193;

Murmur hash seed to use for $zcrc(,9,)

Properties

property precision as %Integer [ Calculated , ReadOnly ];

Precision of the estimator

Property methods: precisionDisplayToLogical(), precisionIsValid(), precisionLogicalToDisplay(), precisionNormalize()

property type as %String [ Calculated , ReadOnly ];

Whether the estimator is currently sparse or dense

Property methods: typeDisplayToLogical(), typeIsValid(), typeLogicalToDisplay(), typeLogicalToOdbc(), typeNormalize()

Methods

method %OnClose() as %Status

Inherited description: This callback method is invoked by the %Close() method to provide notification that the current object is being closed.

The return value of this method is ignored.

method %OnNew(sketch As %Binary = "", decode As %Boolean = ..#ENCODE, Output err As %String = "") as %Status

Creates the memory and sets id for a new sketch. If you pass the sketch parameter, the new sketch will be initialized with the serialized sketch you passed in.

method estimate(Output err As %String = "") as %Integer

Returns the current unique value estimate (cardinality) for this sketch.

method get(encode As %Boolean = ..#ENCODE, Output err As %String = "") as %Binary

Returns the serialized form of the current sketch so that multiple sketches can be merged. Potentially you might obtain the sketch from a different process.

method merge(other As %SQL.HLL, Output err As %String = "") as %Status

Merges the supplied sketch object into the current one. This merges the cardinality estimates.

method update(stringdata As %Binary) as %Integer [ Language = cpp ]

Updates this sketch with the $zcrc(,9,) hash of the stringdata. Hash done inside API.

method updateHash(hash As %Binary) as %Integer [ Language = cpp ]

Updates this sketch with the user supplied hash value
Use $zcrc(yourdata,9,2059198193) or $zcrc(yourdata,9,hll.#SEED) to get the hash.

classmethod version() as %Integer

Returns the version of the underlying callout library.