This page describes the new and enhanced features in the 2024.1 release of InterSystems IRIS® data platform, which is an extended maintenance (EM) release. Some of these features were also available in the continuous delivery (CD) releases since 2022.1, the previous EM release.
Enhancing Developer Experience
Using vectors in ObjectScript
As part of the new Vector Search capability (see related note in the Enhancing Analytics and AI section), InterSystems is making the $vector ObjectScript language feature available to all developers. InterSystems introduced $vector as an internal-only feature in 2022.2 to address the need for a new datatype and corresponding set of operators to work with large arrays of elements of a single datatype, and its highly efficient encoding format and use of SIMD functions for bulk operations contributed to the order-of-magnitude performance improvements brought by Columnar Storage. The Vector Search capability introduced with this release offers another use case for working with this kind of datatype, notably to capture embeddings and to perform fast bulk operations on the entire vector at once. Given the feature addresses two distinct use cases in modern computing challenges well, InterSystems has decided to make the feature available to all ObjectScript developers.
You can find full documentation of the new syntax and the current set of operators in the ObjectScript reference.
At first sight, you may think $vector has a lot of overlap with $list and wonder which one to use. They are, however, distinct features that server different use cases. In general, if you are dealing with a single datatype and always access and operate on data elements in bulk, you may fit the $vector use case.
Support for JSON_TABLE
InterSystems IRIS 2024.1 introduces support for the JSON_TABLE function according to the SQL standard. This function, intended for use in the FROM clause of a query, maps JSON values to columns in a relational table, enabling you to query JSON data in the context of SQL and providing very powerful capabilities in combination with lateral joins.
SQL Development
The SQL standard includes a WITH clause that helps in breaking complex nested queries down into small blocks, similar to how you’d use SQL views, but restricted to a single statement. In 2024.1, you can now use such Common Table Expressions (CTEs) in SELECT statements to add more structure and transparency to your queries. Many tools that generate SQL statement can also use CTEs.
As with each release, InterSystems IRIS SQL 2024.1 includes a number of enhancements that improve query performance:
-
When executing queries, any intermediate results that can be shared between different worker threads are now cached. This saves significantly on I/O when running complex queries on large datasets.
-
Space efficiency for tables that use columnar storage where all table columns meet the requirements for pure columnar storage has been optimized.
-
The Adaptive Parallel Execution mechanism introduced in prior releases now applies to a broader set of queries, including those that employ complex conditions, and virtually eliminates the cases in which traditional parallel subqueries would be used in the background.
-
The set of cases where filter and other query predicates for statements involving foreign tables can be pushed down to a remote database has been expanded. This makes foreign tables more performant by limiting the amount of data that gets passed back to your instance of InterSystems IRIS.
Flexible Embedded Python Runtime
Flexible Python Runtime allows an administrator to choose the Python runtime that Embedded Python uses. This is perfect for upgrading to a specific version of Python or using a distribution like Anaconda. The Python runtime library can be configured through the Configuration Parameter File. This feature is available only for Linux-based builds of InterSystems IRIS in 2024.1
Embedded Python for AIX
Embedded Python is now available on AIX systems.
Python BPL Editor
The Business Process Language editor now gives developers the option to write their business processes using Python. While most BPL scripts require no coding, there are a few blocks that enable the developer to implement their own logic in ObjectScript, JavaScript, or (new in the 2024.1 release) Python.
WSGI Web Apps
InterSystems IRIS 2024.1 includes, on an experimental basis, the ability to create, secure, and host Web Applications that conform to the Python WSGI standard. With this enabled, you can run WSGI-compliant frameworks, such as flask or Django, inside your instance and take advantage of all the power of Embedded Python for moden Python web development. Even better, authentication is handled by the Web Gateway just like with all web applications. This feature is experimental in the 2024.1 release.
InterSystems welcomes feedback on how well this feature meets your needs.
Enhancing Analytics and AI
Vector Search
Vector Search is a new experimental feature in InterSystems IRIS 2024.1 that provides the core functionality for semantic and vector-powered search of structured data, the cornerstone of generative AI applications, such as those implementing the Retrieval Augmented Generation (RAG) pattern. Vector search in 2024.1 comprises a new VECTOR SQL datatype and a set of related functions, including TO_VECTOR (which converts strings to vectors) and both VECTOR_DOT_PRODUCT and VECTOR_COSINE (which compute similarity between vectors).
In a typical application, specialized statistical language models called “embedding models” (smaller versions of LLMs like ChatGPT) convert sequences of text (or images, if the embedding model has been trained to process images) into embedding vectors, dense numeric arrays that specify a point in high-dimensional “latent embedding” space. Vectors are then stored in a SQL table column and the similarity functions are used to compare them to an input or query vector, the result of applying the same embedding model to the query string. The results of that search are the most similar vectors, and this can be a higher quality search capability than traditional “keyword” search because the embedding models have been trained to capture the “semantics” of the text. Future releases of InterSystems IRIS will increase the speed of search and introduce SQL syntax that abstracts the process of converting stored text as well as queries into vectors thereby providing a convenient way to implement powerful “semantic search” over data in InterSystems IRIS without having to deal with the underlying vectors directly.
Business Intelligence Improvements
InterSystems continues to improve the performance of InterSystems IRIS BI. In this release, the Analyzer user interface’s filter list management has been enhanced. In addition, this release includes:
-
Handling of deeply nested AND/OR conditions in the Advanced filters in Analyzer. This change introduces a means for computing structures consisting of deeply nested CROSSJOIN and %OR functions in MDX slicer clauses.
-
Enhanced APIs to allow accessing pivot table and dashboard metadata to allow for other visualization user interfaces to access that stored data.
Enhancing Speed, Scale, and Security
Multi-Volume Databases
As customer databases grow, so do database files. To avoid those files becoming unmanageably large, or hit hard filesystem limits, InterSystems IRIS now supports splitting your database across multiple physical “volumes” transparently. This new capability is easily configured: for any database, you can now configure a threshold size. When your initial IRIS.DAT file is about to hit this size and new global data needs to be written, InterSystems IRIS will transparently create a new “database volume” file and start writing new data to that file. When that volume hits the threshold, another file is created, and so on.
There is no impact on applications and code accessing the data, as they continue to see the full database's content, no matter how many volumes it might be spread across. Furthermore, you can configure which directory the next volume should be created in, and when needed, you can rearrange database volumes across directories as a maintenance operation. Combined with planned work to increase the overall maximum database size, this will ensure your data remains easy to manage, well into the petabyte range.
Fast Online Backup
InterSystems for a long time has offered two main backup options: External Backup and Online Backup. With External Backup, users can freeze the write daemon for a brief period of time such that external solutions can create a consistent snapshot of all database, journal and other files. Those external solutions may track what changed versus the previous snapshot to produce an incremental snapshot that is much smaller than a full copy. In Online Backup, InterSystems IRIS tracks which blocks changed, and can produce incremental backups while the system remains fully online (with only a very brief pause in writing).
This release introduces a new mechanism for incremental and full backups to the Online Backup feature, writing separate backup files per database using parallel processes. For large systems with many databases, this provides very significant improvements in the overall time it takes to perform a backup. The new Backup.Online class is the entry point for the new capability.
Important:
This change is the first part of a broader major project improving the performance and interface of the Online Backup feature, and covers the parallelism described above. As we intend to make further changes to the API and output format, the new API is currently labeled as experimental.
InterSystems welcomes feedback from customers through the Early Adopter ProgramOpens in a new tab and look forward to incorporating such feedback in an upcoming release.
During this phase, the pre-existing API in Backup.General and other utilities continue to invoke Online Backup’s pre-existing implementation and can continue to be used without any changes.
