Adding Data to Your Knowledge Store

To make information available to assistants, you must add data to a Knowledge Store. Data Studio supports ingestion of both structured and unstructured data sources.

Use Narratives to define how data is processed and added to a Knowledge Store. For structured data, such as SQL tables, a Narrative connects the source table to a Knowledge Store and defines a Narrative Template that expresses each row in natural language. As the source table changes, the Narrative processes new or updated rows and keeps the stored embeddings up to date.

For unstructured data, such as PDFs and text documents, files are first uploaded or synchronized to a Document Store. When creating a Narrative associated with a Document-type Knowledge Store, you select a Document Store as the source. The Narrative then processes those documents and adds their embeddings to the Knowledge Store.

Narrative

There are two narrative types: SQL Narratives, which work with structured tables, and Document Narratives, which work with documents imported through Document Stores.

Data Studio may also include system-provided Narratives, such as Data Catalog, News, SEC, Text-to-SQL, and Text-to-SQL Column Value Index. These Narratives are available by default and can be used as-is.

Understanding the Narrative Interface

Navigate to Management > Semantic Search >Narratives, and you will see a table of your existing Narratives. If you have not made any Narratives yet, you still will see Narratives with the type System. These are automatically derived from your existing data. The rest of the details contained in the table are as follows:

Name

The narrative name.

Status

Can be Ready or Paused.

Type

SQL Based (Collection) or out-of-the-box (System).

Rows in Table

The count of the rows in the source table.

Processed / s

Displays the number of records that have been processed by the Narrative and the current processing rate (records per second). This value helps you monitor ingestion progress and performance.

Embed. Up-to-Date

Indicates whether all processed records have up-to-date embeddings in the associated Knowledge Store. If this value shows that embeddings are not current, additional processing is still required.

Est. Time Left

Provides an estimated amount of time remaining to complete processing of pending records. This estimate updates as processing continues.

Failed

Displays the number of records that failed during processing. If this number exceeds the configured failure threshold, the Narrative may enter an error state.

Insert Queue

The number of new records waiting to be processed and added to the Knowledge Store.

Update Queue

The number of existing records that have changed and are waiting to be reprocessed so their embeddings remain current.

Delete Queue

The number of records scheduled for removal from the Knowledge Store because they were deleted from the source data.

Filtering the Display

To filter the display, use the dropdown menus and type-in boxes above the table.

Creating a New Narrative

Click the Management icon in the application menu.
Under Semantic Search, click Narratives.
Click +New Narrative.
Provide the following information on the New Narratives screen. Complete the following information:

Narrative Name

Required— Unique Identifier of Narrative.

Knowledge Store

Required— The Knowledge Store that determines how content generated by the Narrative is vectorized and stored.

Description

Optional— A description of the Narrative.

Max Failed Records

Required— Maximum number of records that can fail processing before the Narrative enters an error state.

(Document Only) Document Store

Required— Select the document store that will be the source of documents for this narrative.

(Document Only) Tag Filter

Required— Select a tag to be used to filter the documents on the Document Store that should be indexed by this Narrative.

(Document Only) Summarization Prompt

Optional— Provide an optional prompt to guide the system when generating the document summary.

(Document Only) Summarization Chunk Limit

Required— Specifies the maximum number of document chunks to use when generating the summary. A value of 0 uses all available chunks.

(Document Only) Parser

Required— Select the parser to be used to process the documents. IntelliSense is best suited for mixed media, StructuredRead is best suited for text-heavy documents.

(SQL Only) Table Name

Required— Name of the structured SQL table that the Narrative should be creating vectors for.

(SQL Only) SQL Query

Required— SQL Query used to get information to populate Narrative Template.

(SQL Only) Narrative Template

Required— A text template that utilizes the fields returned from a user's query. The template will be populated with data and then vectorized into embeddings.
Click Submit.

If you created a document based narrative, you will have to upload your documents into a Document Store before configuring any assistants. For SQL based assistants, proceed to Configuring your Assistants.

Editing an Existing Narrative

Click the Management icon in the application menu.
Under Semantic Search, click Narratives.
Click the Edit menu. A drop-down menu appears.
Click Edit Narrative.
Modify the fields you would like to change.
Click Submit.

Deleting a Narrative

Click the Management icon in the application menu.
Under Semantic Search, click Narratives.
Click the Edit menu in the row of the narrative you would like to delete. A drop-down menu appears.
Click Delete Narrative. A confirmation box appears.
Click Delete.

Deleting a Narrative is a permanent action and cannot be reversed.