Streaming Data (2.9)

You can configure the solution to receive streaming data, written directly into tables that you can use within recipes; this refers specifically to streaming data from the Apache Kafka streaming platform. The process starts with configuring and testing one or more brokers, including credentials as needed for authentication. Once the connections are tested, you configure one or more topics for each broker. Each topic represents a stream of data that follows a specific schema, and you specify the table into which this data is written. This table is then available as a schema within the Data Catalog.

Configuring a Broker

To configure a broker:

Click the Streaming icon in the application menu.
The system then displays any existing brokers.
Click New Broker.
Enter values as follows:
- Broker Name — A unique name for the broker, for use within this system.
- Broker Server URL—Host and port to use when establishing the initial connection to the broker cluster. Use a string of the following form:
```
host:port
```
- Broker’s Security Protocol—Select the option that describes how to connect to this broker. These protocols describe both whether the connection is encrypted as well as the form of authentication to use. Select one of the following and specify details as follows:
  - Unauthenticated, unencrypted channel (PLAINTEXT)—With this protocol, the connection is not encrypted and no credentials are used.
  - SASL authenticated, unencrypted channel (SASL_PLAINTEXT)—With this protocol, the connection is not encrypted and SASL credentials are used.
    If you select this protocol, also specify Basic Auth Credentials; for this value, select an existing credential.
  - SASL authenticated, SSL channel (SAL_SSL)—With this protocol, the connection is encrypted and SASL credentials are used.
    If you select this protocol, also do the following:
    1. Click Select files and upload the key store (.jks) file that contains the private key to use when encrypting the channel.
    2. For SSL Trust Store’s Password, enter the password for the key store file.
    3. For Confirm SSL Trust Store’s Password, enter the same password.
    4. For Basic Auth Credentials, specify the credentials to use; select an existing credential.
  - SSL channel (SSL)—With this protocol, the connection is encrypted and no credentials are used.
    If you select this protocol, also do the following:
    1. For SSL Trust Store, click Select files and upload the trust store file containing certificates from certificate authorities (CAs).
    2. For SSL Trust Store’s Password, enter the password for the trust store file, if needed.
    3. For SSL Key Store, click Select files and upload the key store file containing a private key.
    4. For SSL Key Store’s Password, enter the password for the key store file, if needed.
    5. For SSL Key Password, enter the password for the private key held in the key store file.
    6. For SSL Provider, optionally enter the name of the security provider used for the SSL connection.
    7. For SSL Cipher Suites, optionally enter a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate security settings.
    8. For SSL Enabled Protocols, optionally enter a list of protocols enabled for the SSL connection.
    9. For SSL Trust Store Type, optionally specify the file format of the trust store.
    10. For SSL Key Store Type, optionally specify the file format of the key store.
- Schema Registry URL (optional) — See Configuring a Schema Registry.
Click Submit.

Configuring a Schema Registry

When you configure a broker, you can configure an optional schema registry for it, which will simplify the process of defining topics. A schema registryOpens in a new tab provides data governance. It acts as a central location for managing and validating the schemas of topic message data. In addition, the schema registry offers serializers and deserializers for converting events to and from binary strings for transit.

To configure a schema registry for a broker, specify the following values while configuring or reconfiguring the broker:

Schema Registry URL (optional) — Specify the full URL for the registry, including either http: or https:, depending on whether the connection is encrypted.
Schema Registry’s Security Protocol — Select the option that describes how to connect to the registry. These protocols describe both whether the connection is encrypted as well as the form of authentication to use. The choices are the same as for the connections to the broker; see the previous section. The default is SASL_SSL.

Testing a Broker

To test the configuration for a broker, right click the broker and select Test connection. The system then tests the connection and displays a dialog box indicating whether the test was successful.

Configuring a Topic

To configure a topic:

Obtain a copy of the schema of the messages for this topic, in JSON format.
Click the Streaming icon in the application menu.
The system then displays the brokers.
Click the broker with which the topic is associated.
Click New Topic in the upper right.
Specify the following details:
- Topic Name—Name of the topic (as defined in the broker).
- Message Schema—Schema of the messages for this topic, as obtained earlier.
- Class Name—Name of the InterSystems IRIS persistent class to create, to contain the data received for this topic. This class definition does not have to exist. Specify a fully qualified class name in the form Package.Class (or Package.Package.Class and so on). Note that any package name must either be new or must exactly match an existing package name, including case.
- Polling Time—Specifies the intervals of time (in milliseconds) at which the stream is polled before all collected messages are written to the database.
- Notification Interval—Specifies the intervals of time (in milliseconds) at which the streaming process collects statistics and checks for a stop signal.
- Consumer Group—ID for the consumer group that will process messages from this topic in parallel.
- Message Key Deserializer—Deserializer to use when converting the message key back into a readable object.
- Message Value Deserializer—Deserializer to use when converting the message back into a readable object.
- Retrieve message schema from schema registry at runtime using subject and version—Optionally select this check box if you want to retrieve the message schema from the schema registry at runtime.
  If you select this option, also specify:
  - Schema Subject—Name of the message’s data model as stored in the schema registry. A schema is a particular version of a subject. The message schema does not need to be provided directly if the schema’s subject name and version are supplied instead. The schema can then be accessed via the schema registry’s API.
  - Schema Version—Schema version to use.
Click Submit.