Skip to main content

%SQL.HLL

class %SQL.HLL extends %Library.RegisteredObject

ObjectScript API for building Hyper Log Log estimates of the number of unique elements (cardinality) in a group of data.

The estimates are kept in containers called sketches. The containers are identified by the id of this class.
Lets assume you have 1 million pieces of data and want to know how many of those pieces are unique:
1) Use %New() to instantiate a new HLL object:
   set hll= ##class(%SQL.HLL).%New()

2) Feed one million pieces of data into the sketch with update():
   for i=1:1:1000000 {do hll.update(i)}

3) Get an estimate of the cardinality by calling estimate()
   write hll.estimate()

   996537


Notes: We test this class at Intersystems by using murmur hash with a seed of hll.#SEED:
$zcrc(yourdata,9,2059198193) or $zcrc(yourdata,9,hll.#SEED)
The underlying library uses 64 bits of this 128 bit hash.

Estimate Partitioning: pass an existing sketch into %New() to initialize its state
from the standard serialized form (optionally Base64 encoded).
To combine estimates get() and merge() your sketches, if your data is distributed
across many processes.

Property Inventory

Method Inventory

Parameters

parameter ENCODE = 1;
Whether or not to Base64 encode/decode by default during get() and %New()
parameter SEED = 2059198193;
Murmur hash seed to use for $zcrc(,9,)

Properties

property precision as %Integer [ Calculated , ReadOnly ];
Precision of the estimator
Property methods: precisionDisplayToLogical(), precisionIsValid(), precisionLogicalToDisplay(), precisionNormalize()
property type as %String [ Calculated , ReadOnly ];
Whether the estimator is currently sparse or dense
Property methods: typeDisplayToLogical(), typeIsValid(), typeLogicalToDisplay(), typeLogicalToOdbc(), typeNormalize()

Methods

method %OnClose() as %Status
Inherited description: This callback method is invoked by the %Close() method to provide notification that the current object is being closed.

The return value of this method is ignored.

method %OnNew(sketch As %Binary = "", decode As %Boolean = ..#ENCODE, Output err As %String = "") as %Status
Creates the memory and sets id for a new sketch. If you pass the sketch parameter, the new sketch will be initialized with the serialized sketch you passed in.
method estimate(Output err As %String = "") as %Integer
Returns the current unique value estimate (cardinality) for this sketch.
method get(encode As %Boolean = ..#ENCODE, Output err As %String = "") as %Binary
Returns the serialized form of the current sketch so that multiple sketches can be merged. Potentially you might obtain the sketch from a different process.
method merge(other As %SQL.HLL, Output err As %String = "") as %Status
Merges the supplied sketch object into the current one. This merges the cardinality estimates.
method update(stringdata As %Binary) as %Integer [ Language = cpp ]
Updates this sketch with the $zcrc(,9,) hash of the stringdata. Hash done inside API.
method updateHash(hash As %Binary) as %Integer [ Language = cpp ]
Updates this sketch with the user supplied hash value
Use $zcrc(yourdata,9,2059198193) or $zcrc(yourdata,9,hll.#SEED) to get the hash.
classmethod version() as %Integer
Returns the version of the underlying callout library.

Inherited Members

Inherited Methods

FeedbackOpens in a new tab