New in InterSystems IRIS 2020.1 Contents Continuous Delivery Releases of InterSystems IRIS API Management In-Place Conversion from Caché and Ensemble InterSystems Reports Client Language Enhancements New Look in the Management Portal SQL Enhancements Interoperability Production Enhancements Sharding Enhancements Infrastructure and Cloud Deployment Improvements New Automatic Configuration Customization Analytics Enhancements Natural Language Processing Enhancements Improved Performance and Scalability of the Database Other Enhancements and Efficiency Improvements This document describes the new and enhanced features in the 2020.1 release of InterSystems IRIS®. It addresses the new features in 2020.1 that were not present in the 2019.1.0 version (see New and Enhanced Features for InterSystems IRIS 2019.1). Some of these features were first introduced in a 2019.1 maintenance release or in a 2019.2, 2019.3, or 2019.4 continuous delivery release. These features are identified in the descriptions. The following sections describes the 2020.1 release and its new capabilities and enhancements: Continuous Delivery releases of InterSystems IRIS API Management InterSystems API Manager Open API/Swagger Specification-First REST Development In-Place Conversion from Caché and Ensemble InterSystems Reports Client Language Enhancements InterSystems IRIS Native API for Python InterSystems IRIS Native API for Node.js Relational access for Node.js Java and .NET Gateway Reentrancy Native API for Java and .NET Enhancements Execute TSQL Code via JDBC New look in the Management Portal SQL Enhancements Universal Query Cache Interoperability Production Enhancements New PEX Framework for Coding Production Components in Java and .NET Port Authority for Monitoring Port Usage in Interoperability Productions X12 Validation Enhancements Enhanced DTL Support for X12 Import X12 Schemas from XSD Files MQTT Adapters Sharding Enhancements Simplified Architecture Flexible Sharded Schema Design and Objects Support Unified Shard Queue Manager Infrastructure and Cloud Deployment Improvements New Automatic Configuration Customization Analytics Enhancements Selective Cube Build PowerBI Connector Pivot Table Preview Natural Language Processing Enhancements Improved Performance and Scalability of the Database Other Enhancements and Efficiency Improvements Continuous Delivery Releases of InterSystems IRIS InterSystems IRIS 2020.1 is both an extended maintenance release and a continuous delivery release of InterSystems IRIS. There are now two streams of InterSystems IRIS releases: Continuous delivery releases — These releases provide access to new features and are ideal for developing and deploying applications in the cloud or in local Docker containers. Extended maintenance releases — These releases are less frequent than the continuous delivery releases but provide the increased stability of maintenance releases. These releases are ideal for large enterprise applications where the ease of getting fixes in maintenance releases is more important than getting early access to new features. Continuous delivery releases are provided in container format and are available on Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, Docker Hub, and the InterSystems WRC download site. You can run a continuous delivery release on any of these cloud platforms or a local system using Docker container. InterSystems does not provide maintenance releases for continuous delivery releases, but instead fixes issues in subsequent continuous delivery releases. The initial major extended maintenance release is provided on all InterSystems IRIS Supported Platforms, including UNIX, Windows, the cloud platforms, and the Docker container. Following maintenance releases are provided on all server and cloud platforms in the InterSystems IRIS Supported Platforms, but are not provided on the Docker container. If you are on a Docker container, you can upgrade to a continuous delivery release.If your application runs on a non-container platform, you can only use an extended maintenance release for that application but can consider using the continuous delivery releases for: Evaluating new features and testing your custom code — this will reduce your upgrade costs when you upgrade to the next extended maintenance major release. Using it for new projects that can be deployed in the cloud or in local containers. In addition to providing fully suppported releases, InterSystems provides access to prerelease software for developers who want to get an early look at new features. API Management This release includes two new API Management features: InterSystems API Manager Open API/Swagger Specification-First REST Development InterSystems API Manager This release includes the InterSystems API Manager (IAM) enabling you to monitor and control traffic to and from your web-based APIs. The API Manager was released with the maintenance release 2019.1.1 and the continuous delivery release 2019.2 (an early version of 2019.2 did not include the API Manager).If you are building service-oriented application layers, you are very likely to find the number of APIs you are using quickly rise. The more distributed your environment the more critical it becomes to properly govern and monitor your API traffic. The API Manager enables you easily route all your traffic through a centralized gateway and forward API request to appropriate target nodes. This enables you to: Monitor all your API traffic in a central spot. Plan, document, and update the list of APIs you are using and the servers that provide them. Identify issues before they become critical. Control API traffic by throttling throughput, configuring allowed payload sizes, whitelist and blacklist IP addresses and domains, and quickly taking an endpoint into maintenance mode. Onboard internal and external developers by providing interactive API documentation through a dedicated and customizable developer portal. Secure your API's in a central place. The API Manager is interoperable, reliant, intuitive, and scalable. You can perform all configuration using a simple web-based user interface, but can also configure the API Manager using API calls, which makes it easy to perform remote deployments,The API Manager is released in its own container. You can configure the API Manager as a cluster of multiple nodes, but even a single node can handle the load of multiple tens of thousands of requests per second.For more information, see InterSystems API Manager. Open API/Swagger Specification-First REST Development This release enhances the API Management service so that it can generate the ObjectScript code for REST services from OpenAPI 2.0 specifications. This generated code handles the incoming REST call and you only have to write custom code to perform the specific function performed by the service. If you are implementing a service that is already defined in an OpenAPI 2.0 specification, your work is significantly reduced. Even if there is no existing OpenAPI 2.0 specification, it is much easier to create a new specification than to write the custom code required to define the REST API and the specification also provides documentation and aids anyone developing client code for the service. For details see Creating REST Services. (first released in 2019.2) In-Place Conversion from Caché and Ensemble This release of InterSystems IRIS allows you to convert an existing instance of Caché or Ensemble to InterSystems IRIS. The conversion process may require some changes to application code, configuration scripts, and other procedures, but will be relatively easy for the majority of cases. As with any major upgrade, you should thoroughly test your custom code, including any production business services, processes, and operations, in a test environment before deploying to a live production environment.Before performing an in-place conversion, it is important that you read the IRIS In-Place Conversion Guide and the InterSystems IRIS Adoption Guide for background information on the differences between Caché or Ensemble and InterSystems IRIS. You can download these documents from the InterSystems Worldwide Response Center documents distribution pageOpens in a new tab. Important: Only the HealthShare Health Connect and InterSystems IRIS for Health products support the HL7 and DICOM features that are available in Ensemble. The InterSystems IRIS product does not support these features. Consequently, if your Ensemble productions use the Ensemble health care features, you should not perform an in-place conversion to InterSystems IRIS, but should rather convert to InterSystems IRIS for Health or HealthShare Health Connect. If you are using Ensemble as a general-purpose data platform for health care development or plan to develop health care applications in the future, you should convert to InterSystems IRIS for Health 2020.1 or later. If you are using Ensemble as an integration engine, you should convert to HealthShare Health Connect 2020.1 or later. For details, see the InterSystems IRIS for Health Release Notes or the HealthShare Health Connect Release Notes and the IRIS In-Place Conversion Guide and the InterSystems IRIS Adoption Guide. InterSystems Reports InterSystems Reports can be used with InterSystems IRIS and InterSystems IRIS for Health. InterSystems Reports is a repackaging of Logi Report (formerly named JReport®), a product of Logi Analytics®. We plan to provide documentation on how to use InterSystems Reports with our database products, but for customers who want to start using InterSystems Reports now, you can start with the documentation at https://www.jinfonet.com/documentation/Opens in a new tab. If you have questions about using InterSystems Reports, contact the Worldwide Response CenterOpens in a new tab (WRC). Client Language Enhancements InterSystems IRIS Native API for Python This release introduces the Native API for Python, which is a lightweight Python interface to the native multidimensional storage data structures that underlie the InterSystems IRIS object and SQL interfaces. The Native API allows you to implement your own data structures by providing direct access to global arrays, the tree-based sparse arrays that form the basis of the multidimensional storage model. By providing direct access to global arrays, the Native API for Python allows you to define very efficient storage structures in Python and, consequently, very efficient applications. For details see Using the Native API for Python and Native API for Python ReferenceOpens in a new tab. (first released in 2019.2) InterSystems IRIS Native API for Node.js This release introduces the Native API for Node.js, which is a lightweight Node.js interface to the native multidimensional storage data structures that underlie the InterSystems IRIS® object and SQL interfaces. The Native API allows you to implement your own data structures by providing direct access to global arrays, the tree-based sparse arrays that form the basis of the multidimensional storage model. By providing direct access to global arrays, the Native API for Node.js allows you to define very efficient storage structures in Node.js and, consequently, very efficient applications. For details see Using the Native API for Node.js and Native API for Node.js ReferenceOpens in a new tab. (first released in 2019.2) Relational access for Node.js This release provides ODBC access to InterSystems IRIS databases to Node.js developers. (first released in 2019.2) Java and .NET Gateway Reentrancy This release allows you to use reentry on the Java and .NET Gateway with forward and reverse proxy. Forward and reverse proxy allow freer use of objects. Proxy objects, or object originally created in InterSystems IRIS to represent objects in Java and .NET, will be created in more situations involving method invocation. You can create objects without concern for the proxy objects. (first released in 2019.3) Native API for Java and .NET Enhancements The IRIS Native API, which allows you to access InterSystems IRIS data using globals, has been extended to include $LIST and pass by reference: $LIST allows you to easily iterate through data structures without needing to parse them in detail. This supports scenarios such as developing Java and .NET applications that access existing global structures from existing applications, as well as simplifying development and speeding performance. (first released in 2019.4) Passing parameters by reference allows you to create applications that use less memory, using cleaner code, and have improved performance. Passing by reference provides an effective mechanism for two-way communication between the referencing routine and the function. Any change the function makes to a variable in its formal list is also made to the corresponding by-reference variable in the actual list. Execute TSQL Code via JDBC This release provides the ability to execute Transact-SQL (TSQL) code directly from JDBC. New Look in the Management Portal This release represents the beginning of a new, more modern look for the Management Portal. In this first phase, the menus and buttons have a new look but the functionality is unchanged. This new implementation provides the basis for future streamlining and improvements to the user interface. (first released in 2019.2) SQL Enhancements As with every release, InterSystems IRIS includes a number of enhancements to its SQL engine, based on advancements in the underlying software and continuous benchmarking against industry-standard and customer workloads. Customers are likely to observe measurable increases in query throughput for high-load scenarios compared to the 2019.1 release and are encouraged to share their experiences with InterSystems in case there is an opportunity for extending our benchmarking to include specific new use cases. Note that when you upgrade to a new major version, existing Query Plans are automatically frozen. This ensures that a major software upgrade will never degrade the performance of an existing query. For performance-critical queries, you should test if you can achieve improved performance.This release includes the following enhancements: Universal Query Cache Improvements to our parallelization engine that enable more types of queries and DML to be parallelized (automatically) and make more efficient use of CPU capacity. (first released 2019.4) Sharded queries can now use implicit joins using -> syntax. (first released 2019.4) Queries issued from the SQL explorer page in the System Management Portal are executed in the background. While this enables query cancellation and avoids web request timeouts, it also means certain legacy stored procedures that depend on foreground execution and write to the current device may no longer display this logging information properly. (first released 2019.3) Universal Query Cache This release introduces a Universal Query Cache, which enables every query (including embedded and class queries) to be saved as a cached query. Previously, the use of embedded SQL meant application code needed to be recompiled in order to pick up current table statistics or newly available indexes. Now, all query plans are managed in a single cache and can be purged (per query, table or namespace) when appropriate. This significantly improves the ability for applications to adapt to actual data characteristics when deployed to multiple sites.Also, all query types can now equally take advantage of more efficient data access implemented in generated query code. Interoperability Production Enhancements New PEX Framework for Coding Production Components in Java and .NET This release includes the Production EXtension (PEX) framework that provides you with a choice of implementation languages when you are developing interoperability productions. In this release you can use Java and .NET to develop business services, processes, and operations and, also, inbound and outbound adapters. In previous releases, you could only code business services and operations, could only code in Java, and had to use a special code generator wizards in the Management Portal. The PEX framework provides the flexible plumbing that connects your Java and .NET code to the interoperability production components. You can connect your Java and .NET code using PEX with minimal or no ObjectScript coding. The PEX package includes the following classes: EnsLib.PEX.BusinessServiceOpens in a new tab EnsLib.PEX.BusinessProcessOpens in a new tab EnsLib.PEX.BusinessOperationOpens in a new tab EnsLib.PEX.InboundAdapterOpens in a new tab EnsLib.PEX.OutboundAdapterOpens in a new tab EnsLib.PEX.MessageOpens in a new tab For details, see PEX: Developing Productions with Java and .NET. Port Authority for Monitoring Port Usage in Interoperability Productions The Port Authority utility allows you to monitor how ports are used in your interchange systems. The Port Authority examines the business services and business operations in multiple productions and instances to determine which ports are being used on each system. You can determine which ports are free for new services and operations and reserve ports for specific uses. For details, see Managing Port Usage.(first released in 2019.3) X12 Validation Enhancements This release provides two kinds of enhanced X12 validation: SNIP levels 1 and 2 validation — validates the X12 message according to the standards developed by the Workgroup for Electronic Data Exchange (WEDI) Strategic National Implementation Process (SNIP). X12 element validation — (first released in 2019.1.1 and 2019.2) In previous releases, you could not use SNIP validation and could only validate the overall segment structure. There was no mechanism to validate the contents of the segment. SNIP allow you to validate that: SNIP level 1 — segments are valid , segment order is valid, element attributes are valid, numeric data elements have numeric values, and message conforms to X12 rules. SNIP level 2 — meets HIPAA requirements, such as presence of required elements, non-use of elements marked as not used, and values conforming to the code tables. X12 element validation enables you to validate that: Required fields are present and that all fields are allowed by the schema. Number of fields within a segment and whether they are repeated as allowed by the schema. Data types for fields and components are correct. Field values conform to the code tables specified. Field and components conform to length restrictions. For details, see X12 Validation. Enhanced DTL Support for X12 In this release you can define data transformations for an entire X12 batch including schemas for the interchange envelope, functional groups, and transaction sets. This allows you to process X12 batch messages using a single data transformation without having to use subtransformations. This release also improves the user interfaces and also provides convenience functions that make it easier to handle repeating elements. For details, see Creating an X12 Data Transformation. Import X12 Schemas from XSD Files In previous versions, you could only import X12 schemas from SEF files or InterSystems proprietary XML format. In this release, you can also import X12 schemas from the newer XSD schema files. For details, see Loading X12 Schemas. MQTT Adapters This release includes MQTT adapters that support Message Queuing Telemetry Transport (MQTT) protocol, which is often used in Internet of Things (IoT) applications. For details, see Using MQTT Adapters in Productions. Sharding Enhancements Simplified Architecture This release introduces a simple and straightforward blueprint for sharded clusters, the node-level architecture, which can be configured through the new %SYSTEM.Cluster API. In this cluster architecture, we’ve implemented some best practices on laying out the different foundational elements introduced with sharding in the initial InterSystems IRIS 2018.1 release that will make it significantly easier to deploy and expand your cluster. As the node-level architecture is essentially a smart way of leveraging the existing infrastructure, it is fully transparent to application code and does not require any changes to existing deployments. For details, see Elements of Sharding and Sharding APIs in the Scalability Guide and see the %SYSTEM.Cluster class documentation in the InterSystems Class Reference. (first released in 2019.2) Flexible Sharded Schema Design and Objects Support This release introduces further improvements to how sharding supports designing your application’s schema in the following ways: You can now coshard any two sharded tables. Before, only tables with a common user-defined shard key could be cosharded (that is, explicitly defining the shard key for an Order and OrderLine table to be OrderID, the field on which they are joined). With this release, you can use COSHARD WITH syntax in DDL or the CoShardWith index keyword to coshard a new table with an existing table that has a system-assigned shard key. This significantly increases the flexibility for designing your application’s schema, preserving the operational benefits of using system-assigned shard keys. For details, see Create the Tables in the Scalability Guide and Defining a Sharded Table. Where previously sharded schema design could only happen through DDL, you can now mark a persistent class (table) as sharded through its class definition, using the new “Sharded” class keyword. The class compiler has been extended to warn against using class definition features incompatible with sharding, such as customized storage definitions, at compile time. For details, see Defining a Sharded Table by Creating a Persistent Class. You can use the object paradigm to interact with sharded classes. This means you can create new or open existing sharded class instances with %New() or %OpenId() methods and save them with %Save(). The sharding infrastructure will make sure new instances are appropriately distributed across the cluster, fully transparent to the application. Note that this object code will still be executed on the machine your client is connected to. These sharding enhancements were first released in 2019.2. Unified Shard Queue Manager In this release, the Unified Shard Queue Manager improves sharding efficiency when a sharded cluster is being queried by large numbers of clients by queuing up sharded query work rather than spawn individual processes for each shard-local query and potentially flood the system. An efficient algorithm then ensures an appropriate number of worker processes is used to process work from the queue, based on available hardware and system load. Infrastructure and Cloud Deployment Improvements This release contains improvements to the infrastructure and cloud deployment, including the following: Tencent Cloud Support — InterSystems Cloud Manager (ICM) now provides end-to-end cloud provisioning and deployment for applications based on InterSystems IRIS and running on Tencent Cloud. (first released in 2019.4) Support for Docker named volumes in addition to bind mounts. (first released in 2019.4) InterSystems Cloud Manager (ICM) support for elastic scaling — Existing configurations can now be scaled, that is, reprovisioned and redeployed with more or fewer nodes. For details, see Reprovisioning the Infrastructure and Redeploying Services in theInterSystems Cloud Manager Guide. (first released in 2019.2; scale out DATA nodes and scale in/out COMPUTE nodes in node-level architecture first released in 2019.4) Improved user experience when packaging your own container. (first released in 2019.3) InterSystems Cloud Manager (ICM) support for node-level sharding. (first released in 2019.3) Containers use non-root default user, which is a container best practice and improves security. (first released in 2019.3) ICM support for creating and deploying on a private network, in which bastion servers connect your private network to the public network and provide improved security protection from denial of service attacks. (first released in 2019.3) Support for service discovery with secure RPC communication. (first released in 2019.3) ICM support for multi-region deployments, which can provide high availability even if an entire region stops functioning. (first released in 2019.3) Ability to upgrade ICM and retain knowledge of deployed systems. (first released in 2019.3) Containerless Mode — The following were previously restricted but can now be performed: deploying sharded configurations on Google Cloud Platform using containerless mode and deploying the Web Gateway on Ubuntu or SUSE nodes using containerless mode. (first released in 2019.2) New Automatic Configuration Customization A new InterSystems IRIS configuration feature enables customization of the configuration parameter file (CPF) of an InterSystems IRIS instance prior to startup, upon which the custom configuration is automatically implemented. This feature greatly simplifies automation and supports the use of configuration management tools such as Kubernetes with InterSystems IRIS, and is also included in ICM in this version. Automatic configuration customization is an important new capability that will be expanded in future versions. (first released in 2019.4) Analytics Enhancements This release contains the following analytics enhancements: Selective Cube Build PowerBI Connector Pivot Table Preview Selective Cube Build This release provides Selective Cube Build, a feature of InterSystems IRIS Business Intelligence, that allows you to select the measures and dimensions to be built individually. You can make changes and selectively rebuild without taking the full cube out of service. The user interface also automatically flags the dimensions or measures that have been added or changed so that you know what to rebuild. PowerBI Connector InterSystems customers can now use Microsoft Power BI to access tabular and cube data stored on InterSystems IRIS. This allows combining the data visualization capabilities offered by Power BI with the high-performance data management and querying capabilities offered by InterSystems IRIS. While the connector leverages ODBC, it will also allow customers to access InterSystems IRIS BI cubes directly from Power BI when connecting to InterSystems 2019.2 or above. The connector ships as part of Power BI starting with its April 2019 release. For details, see InterSystems IRIS Connector for Power BI . Pivot Table Preview This release contains the Analytics Pivot Table Preview, a new mode for the Analyzer that presents a representative pivot table based on a truncated data set. This will allow previewing a pivot table much more quickly than analyzing the complete result set. A Show All button is also presented when in Preview mode to indicate that the result set is not complete. Selecting the Show All button automatically turns off Preview mode. (first released in 2019.2) Natural Language Processing Enhancements This release contains the following natural language processing enhancements. InterSystems IRIS Natural Language Processing (NLP) will now extract value and unit information for measurement attributes in the English language model. This information can be visualized through highlighting or retrieved through the query and REST APIs. (first released in 2019.3) This release introduces support for the Czech language with InterSystems IRIS Natural Language Processing. The embedded NLP engine will now also identify concepts and their context for natural language text written in Czech as it does for the 10 other languages supported previously. (first released in 2019.2) Improved Performance and Scalability of the Database This release has significant optimizations in the database engine. This is especially important for very large systems and significantly increases the ability to scale systems to handle very heavy loads. One of the efficiency changes in this release improves efficiency when traversing globals. For a database block in memory that is accessed frequently but not modified often, the system may automatically build an optimization structure, called a node table, to speed up searches for nodes within the block. This speeds up global accesses, particularly when access to nodes are sparsely or randomly distributed, or for patterns that access the nodes in reverse collation order (including reverse $order / $query). The memory for this comes from the database cache itself, a small fraction typically less than one percent. Other Enhancements and Efficiency Improvements In each release, InterSystems makes many efficiency improvements and minor enhancements. In this release these improvements include: Journal performance enhancements. Easier configuration for mirrored environments. Support for new versions of Apache Spark version 2.3 and 2.4. Support for WebSocket client in addition to the existing WebSocket server support in the Web Gateway. Source Control for Productions — source control hooks have been added to allow check-in and check-out of a production as an entity, simplifying change tracking and configuration management. (first released in 2019.4) Whitelists to Support Penetration Testing — customers performing their own security penetration testing can reduce or eliminate false positives related to CSP, Zen, and REST. (first released in 2019.4) Upgrades .NET support to .NET Core 2.1. (first released in 2019.3) Improved efficiency for ODBC database access. (first released in 2019.3) Structured logging to improve access to log messages. (first released in 2019.3) Improved API to fetch alerts. (first released in 2019.3) Compiler now tests for global names that are too long and reports an error. Previously, these global names were silently truncated. (first released in 2019.3) Maintenance release 2019.1.1, continuous delivery release 2019.3, and subsequent releases include the set of changes identified as JournalingGroup2019, which corrects issues associated with journaling and mirroring. The changes associated with these issues are SML2776, SML2781, SML2782, SML2783, SML2785, JO2990, JO3117, JO3137, JO3140, JO3141, RJF391, RJF392, HYY2362, HYY2364, and HYY2373.