[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally #191043

sorenlouv · 2024-08-22T04:22:33Z

The Problem
The LLM decides the identifier (both _id and doc_id) for knowledge base entries. The _id must be globally unique in Elasticsearch but the LLM can easily pick the same id for different users thereby overwriting one users learning with another users learning.

Solution
The LLM should not pick the _id. With this PR a UUID is generated for new entries. This means the LLM will only be able to create new KB entries - it will not be able to update existing ones.

doc_id has been removed, and replaced with a title property. Title is simply a human readable string - it is not used to identify KB entries.
To retain backwards compatability, we will display the doc_id if title is not available

obltmachine · 2024-08-22T04:22:45Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

…iting accidentally

dgieselaar · 2024-08-27T18:57:30Z

I've not looked through the code so maybe you took this into account, but we also have the documents that we pre-load into the knowledge base. Those should not have dynamically generated uuids, but predetermined IDs.

sorenlouv · 2024-08-27T20:28:32Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

@@ -79,9 +79,10 @@ export type ConversationUpdateRequest = ConversationRequestBase & {

 export interface KnowledgeBaseEntry {
  '@timestamp': string;
-  id: string;
+  id: string; // unique ID
+  doc_id?: string; // human readable ID generated by the LLM and used by the LLM to lookup and update existing entries. TODO: rename `doc_id` to `lookup_id`


id is globally unique, doc_id is only unique per user. Multiple entries can be assigned the same doc_id if they are created for different users.

sorenlouv · 2024-08-27T20:37:41Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

-  doc_id?: string;
+  id?: string;


doc_id can be used by the LLM to lookup entries. I see no reason to expand that concept to instructions. instructions can still have pre-determined id's - they do not have to be UUIDs. See the lens docs for an example of this

sorenlouv · 2024-08-27T20:46:32Z

x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/chat/route.ts

@@ -42,7 +42,7 @@ const chatCompleteBaseRt = t.type({
      ]),
      instructions: t.array(
        t.intersection([
-          t.partial({ doc_id: t.string }),
+          t.partial({ id: t.string }),


It's still possible to overwrite existing instructions by specifying the id

sorenlouv · 2024-08-27T20:58:22Z

...ns/observability_solution/observability_ai_assistant/server/service/kb_component_template.ts

+          keyword: {
+            type: 'keyword',
+            ignore_above: 256,
+          },


Adding nested keyword in order to be able to sort on it. Using nested keyword is recommended over fielddata as it is more performant (should have been used for doc_id as well).

sorenlouv · 2024-08-27T21:01:20Z

...rvability_solution/observability_ai_assistant/server/service/knowledge_base_service/index.ts

+    this.dependencies.logger.debug(
+      `Adding ${operations.length} operations to queue. Queue size now: ${this._queue.length})`
+    );
+    this._queue.push(...operations);


Afaict we had a bug here before: By calling this._queue.push conditionally we were not adding operations to the queue when isModelReady=true. This meant that anything imported after the model had been setup was being dropped 😱

In general I hope we can get rid of the queue, or separate the queuing logic from the knowledge base. Having the queue embedded makes it more complex to work with the KB than it needs to be.

...s/observability_solution/observability_ai_assistant/server/utils/recall/score_suggestions.ts

sorenlouv · 2024-08-28T07:55:34Z

I've not looked through the code so maybe you took this into account, but we also have the documents that we pre-load into the knowledge base. Those should not have dynamically generated uuids, but predetermined IDs.

@dgieselaar Perhaps see this comment #191043 (comment)

sorenlouv · 2024-11-06T23:29:32Z

x-pack/test/plugin_api_integration/test_suites/task_manager/check_registered_task_types.ts

@@ -151,7 +151,6 @@ export default function ({ getService }: FtrProviderContext) {
        'fleet:update_agent_tags:retry',
        'fleet:upgrade_action:retry',
        'logs-data-telemetry',
-        'observabilityAIAssistant:indexQueuedDocumentsTaskType',


The task was removed. It is not longer needed

elasticmachine · 2024-11-07T08:19:18Z

⏳ Build in-progress

Buildkite Build
Commit: 732f4ae
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-191043-732f4ae5430d

History

kibanamachine · 2024-11-07T08:55:52Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11719652062

…iting accidentally (elastic#191043) Closes elastic#184069 **The Problem** The LLM decides the identifier (both `_id` and `doc_id`) for knowledge base entries. The `_id` must be globally unique in Elasticsearch but the LLM can easily pick the same id for different users thereby overwriting one users learning with another users learning. **Solution** The LLM should not pick the `_id`. With this PR a UUID is generated for new entries. This means the LLM will only be able to create new KB entries - it will not be able to update existing ones. `doc_id` has been removed, and replaced with a `title` property. Title is simply a human readable string - it is not used to identify KB entries. To retain backwards compatability, we will display the `doc_id` if `title` is not available --------- Co-authored-by: Sandra G <neptunian@users.noreply.github.com> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 7c92a10)

kibanamachine · 2024-11-07T09:00:36Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…overwriting accidentally (#191043) (#199263) # Backport This will backport the following commits from `main` to `8.x`: - [[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally (#191043)](#191043)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Søren Louv-Jansen <soren.louv@elastic.co>

sorenlouv requested a review from a team as a code owner August 22, 2024 04:22

botelastic bot added ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Observability AI Assistant labels Aug 22, 2024

[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwr…

e627b85

…iting accidentally

sorenlouv force-pushed the add-uuid-to-kb-entries-to-avoid-overwriting branch from 614ee57 to e627b85 Compare August 22, 2024 04:26

sorenlouv added release_note:fix v8.16.0 and removed ci:project-deploy-observability Create an Observability project labels Aug 22, 2024

Fix issues

ede9593

botelastic bot added the ci:project-deploy-observability Create an Observability project label Aug 22, 2024

sorenlouv and others added 2 commits August 26, 2024 12:24

Fix user instruction test

4f5cb94

Merge branch 'main' into add-uuid-to-kb-entries-to-avoid-overwriting

bafa18b

sorenlouv force-pushed the add-uuid-to-kb-entries-to-avoid-overwriting branch from 0cc07e8 to a9ed9e9 Compare August 27, 2024 18:41

sorenlouv commented Aug 27, 2024

View reviewed changes

Fix test

14854d2

sorenlouv force-pushed the add-uuid-to-kb-entries-to-avoid-overwriting branch from a9ed9e9 to 14854d2 Compare August 27, 2024 20:44

sorenlouv commented Aug 27, 2024

View reviewed changes

Cleanup

f6b8303

sorenlouv commented Aug 27, 2024

View reviewed changes

...s/observability_solution/observability_ai_assistant/server/utils/recall/score_suggestions.ts Show resolved Hide resolved

sorenlouv added 2 commits August 28, 2024 00:05

Improve types

f88d826

i18n

f42f1ec

sorenlouv added 3 commits August 28, 2024 10:56

Remove unused imports

b718ae5

Re-add ShortIdTable

59a9362

Change recall to not return entries as nested object

51cb688

sorenlouv added 8 commits November 5, 2024 21:27

Fix lint issues

993745e

Fix serverless test

4022dab

Fix broken api test

3671085

fix tests

5708359

Fix test

6e448b8

Improve type

3114e7d

Fix functional test

094f94c

Merge branch 'main' into add-uuid-to-kb-entries-to-avoid-overwriting

744a279

sorenlouv mentioned this pull request Nov 6, 2024

Enable summarisation spec #199134

Merged

dgieselaar approved these changes Nov 6, 2024

View reviewed changes

Remove queue for task manager types

55e5800

sorenlouv requested a review from a team as a code owner November 6, 2024 23:25

Merge branch 'main' into add-uuid-to-kb-entries-to-avoid-overwriting

3f715be

sorenlouv enabled auto-merge (squash) November 6, 2024 23:26

sorenlouv commented Nov 6, 2024

View reviewed changes

editorUser -> editor

efa7bf2

mikecote approved these changes Nov 7, 2024

View reviewed changes

Fix summarisation test

732f4ae

sorenlouv merged commit 7c92a10 into elastic:main Nov 7, 2024
26 checks passed

kibanamachine added the v9.0.0 label Nov 7, 2024

kibanamachine mentioned this pull request Nov 7, 2024

[8.x] [Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally (#191043) #199263

Merged

sorenlouv deleted the add-uuid-to-kb-entries-to-avoid-overwriting branch November 7, 2024 09:55

kibanamachine added the v8.17.0 label Nov 7, 2024

This was referenced Nov 29, 2024

[Obs AI Assistant] Knowledge base entries are not consistently updated correctly by LLM due to changing identifiers #191085

Closed

Extract queue from Knowledge base #191640

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally #191043

[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally #191043

sorenlouv commented Aug 22, 2024 •

edited by kibanamachine

Loading

obltmachine commented Aug 22, 2024

dgieselaar commented Aug 27, 2024

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv commented Aug 28, 2024

sorenlouv Nov 6, 2024

elasticmachine commented Nov 7, 2024

kibanamachine commented Nov 7, 2024

kibanamachine commented Nov 7, 2024

[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally #191043

[Obs AI Assistant] Add uuid to knowledge base entries to avoid overwriting accidentally #191043

Conversation

sorenlouv commented Aug 22, 2024 • edited by kibanamachine Loading

obltmachine commented Aug 22, 2024

🤖 GitHub comments

dgieselaar commented Aug 27, 2024

sorenlouv Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

sorenlouv Aug 27, 2024 • edited Loading

Choose a reason for hiding this comment

sorenlouv commented Aug 28, 2024

sorenlouv Nov 6, 2024

Choose a reason for hiding this comment

elasticmachine commented Nov 7, 2024

⏳ Build in-progress

History

kibanamachine commented Nov 7, 2024

kibanamachine commented Nov 7, 2024

💚 All backports created successfully

Questions ?

sorenlouv commented Aug 22, 2024 •

edited by kibanamachine

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading

sorenlouv Aug 27, 2024 •

edited

Loading