HNSWLIBMemoryLimit

Synopsis

[SQL]    HNSWLIBMemoryLimit=n

n is an integer in the range 0—2,147,483,647 (KB). The default value is -1, which sets the parameter to the current value of the maximum memory per process (bbsiz).

Description

HNSWLIBMemoryLimit is the upper limit on memory consumption during HNSW (Hierarchical Navigable Small World) vector index builds using the hnswlib library. When building large HNSW vector indexes using hnswlib, memory consumption can grow significantly depending on the size of the dataset and index parameters; HNSWLIBMemoryLimit allows you to prevent hnswlib index builds from consuming excessive system memory.

When vector indices are built using hnwslib, InterSystems IRIS® data platform estimates the memory required for the index build and only proceeds if the system can sustain it. It checks the following criteria in order: the value of HNSWLIBMemoryLimit, if not set to -1, the value of bbsiz, or 50% of the total system physical memory (if bbsiz=-1). If any of these limits are exceeded by the amount of estimated memory, the vector index is not built using the hnswlib library and InterSystems IRIS will automatically build the vector index using the existing mechanism without the hnswlib library.

Changing This Parameter

On the SQL page of the Management Portal (System Administration > Configuration > SQL and Object Settings > SQL), enter a number of kilobytes in the HNSW index build memory limit (KB) field.

To set the desired value for HNSWLIBMemoryLimit from the Terminal, use the SetOption(“HNSWLIBMemoryLimit”)Opens in a new tab method of the %SYSTEM.SQL.UtilOpens in a new tab class. See the class reference for details.

You can also change HNSWLIBMemoryLimit with the Config.SQLOpens in a new tab class (as described in the class reference) or by editing the CPF in a text editor (as described in Editing the Active CPF).

When you edit this setting, the updated value takes effect immediately for all processes.