[Obs AI Assistant] Implement contextual actions #178405

dgieselaar · 2024-03-11T16:00:12Z

Implements contextual actions. With this change, consumers of the Observability AI Assistant can add contextual actions to screen context, which are functions that are executed on the client. Its results are then pushed back to the API and the LLM to continue the conversation. This allows consumers to do things like:

have the LLM navigate the app
push component/browser state to the LLM
provide functions that make the most sense in a specific UI context

Additionally, this also opens up the possibility for consumers of the complete API to implement their own functions.

apmmachine · 2024-03-11T16:00:30Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
/oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…t-screen-actions

dgieselaar · 2024-03-13T15:40:27Z

/ci

elasticmachine · 2024-03-13T15:42:21Z

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

…fix'

…ar/kibana into obs-ai-assistant-screen-actions

miltonhultgren

Cool stuff!

miltonhultgren · 2024-03-14T09:40:54Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

@@ -4,6 +4,8 @@
 * 2.0; you may not use this file except in compliance with the Elastic License
 * 2.0.
 */
+import type { ObservabilityAIAssistantChatService } from '../public';


Is it allowed for common to import from public?

for types, yes

miltonhultgren · 2024-03-14T09:41:59Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

@@ -77,11 +79,37 @@ export interface KnowledgeBaseEntry {
  role: KnowledgeBaseEntryRole;
 }

+export interface ObservabilityAIAssistantScreenContextRequest {


Off topic: Is it needed to prefix so many types with ObservabilityAIAssistant?

auto-imports search across all projects so in the end I think it's better to be more verbose... e.g. try to import KibanaReactProvider and see what comes up 😄

miltonhultgren · 2024-03-14T09:44:28Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

+  actions?: Array<{ name: string; description: string; parameters?: CompatibleJSONSchema }>;
+}
+
+export type ScreenContextActionRespondFunction<TArguments extends unknown> = ({}: {


Nit: Would it be wrong to just call this ScreenContextAction? or ScreenContextFunction?

it's an action and then it's the respond property of the action. I do understand it's verbose and I'm not too happy with it but I think it's better than those alternatives

miltonhultgren · 2024-03-14T09:46:23Z

x-pack/plugins/observability_solution/observability_ai_assistant/common/types.ts

+  name: string;
+  description: string;
+  parameters?: CompatibleJSONSchema;
+  respond: ScreenContextActionRespondFunction<TArguments>;


Off topic: It's interesting to see how the language gets mixed between it being an "action" (function) and a "part of conversation" (respond).

miltonhultgren · 2024-03-14T09:58:42Z

...ck/plugins/observability_solution/observability_ai_assistant/public/service/complete.test.ts

+    });
+
+    it('the observable errors out', async () => {
+      await expect(async () => await lastValueFrom(callComplete())).rejects.toThrowError(


Is it needed to make the call 3 times to check each thing?

mostly an artifact of expect and rejects. open to alternatives but couldn't easily figure out a better way.

miltonhultgren · 2024-03-14T13:13:41Z

x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts

+            if (functionClient.hasAction(functionCallName)) {
+              this.dependencies.logger.debug(`Executing client-side action: ${functionCallName}`);
+
+              functionClient.validate(


This throws an error if it fails? And if it fails, then what happens with the subscriber? Does it not complete?

the error falls through and eventually subscriber.error is called. but that shouldn't actually happen here, we should inject a function error message and send it back to the LLM. will fix and add a test, thanks.

miltonhultgren · 2024-03-14T13:17:31Z

...ility_solution/observability_ai_assistant_app/public/hooks/use_nav_control_screen_context.ts

+    const originalPushState = window.history.pushState.bind(window.history);
+    const originalReplaceState = window.history.replaceState.bind(window.history);


Do we need to revert these as part of a clean up for this hook?

it's hard to clean up because something else might have overridden window.history.pushState at that point. I'll add a flag for the setState calls and clean up the window listeners.

miltonhultgren · 2024-03-14T13:17:52Z

...ility_solution/observability_ai_assistant_app/public/hooks/use_nav_control_screen_context.ts

+    window.addEventListener('popstate', () => {
+      setHref(window.location.href);
+    });
+
+    window.addEventListener('hashchange', () => {
+      setHref(window.location.href);
+    });


And clean these up maybe as well?

miltonhultgren · 2024-03-14T13:18:58Z

...ility_solution/observability_ai_assistant_app/public/hooks/use_nav_control_screen_context.ts

+        createScreenContextAction(
+          {
+            name: 'foo',
+            description: 'foo',
+            parameters: {
+              type: 'object',
+              properties: {
+                foo: {
+                  type: 'string',
+                },
+              },
+            },
+          },
+          async ({}) => {
+            return {
+              content: 'Action succesfully executed',
+            };
+          }
+        ),


I guess this is to clean up?

yes (thanks :) )

miltonhultgren · 2024-03-14T13:25:01Z

x-pack/test/observability_ai_assistant_api_integration/tests/complete/complete.spec.ts

+
+        const conversations = await observabilityAIAssistantAPIClient
+          .writeUser({
+            endpoint: 'POST /internal/observability_ai_assistant/conversations',


Why is this POST rather than GET?

it's a search request, with optionally a body with filters (right now just a query string).

…t-screen-actions

kibana-ci · 2024-03-15T16:01:22Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`observabilityAIAssistant`	80	85	+5
`observabilityAIAssistantApp`	211	212	+1
total			+6

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`observabilityAIAssistant`	216	225	+9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`observabilityAIAssistant`	13.7KB	15.7KB	+1.9KB
`observabilityAIAssistantApp`	139.4KB	140.5KB	+1.1KB
total			+3.0KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`observabilityAIAssistant`	21	24	+3

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`observabilityAIAssistant`	42.3KB	43.4KB	+1.1KB
`observabilityAIAssistantApp`	12.3KB	12.3KB	+51.0B
total			+1.2KB

Unknown metric groups

API count

id	before	after	diff
`observabilityAIAssistant`	218	227	+9

History

💔 Build #197856 failed 2198889
💚 Build #197618 succeeded 3d6c949
💔 Build #197482 failed 19bd361

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

#178405 made some changes to how the chat service passes data to the API, while doing this we missed to pass along the `responseLanguage` setting, breaking the language selection feature. This PR just wires those up again and adds a test to catch this for the future.

dgieselaar added 4 commits March 11, 2024 16:46

Implement contextual actions

ae91174

Execute contextual actions

507bb36

Execute contextual actions

8837408

Fix tests

dc4d2f8

dgieselaar added 4 commits March 13, 2024 16:10

Unit tests

f7665f8

API tests

e7abb08

Revert APM changes

6c00691

Merge branch 'main' of github.com:elastic/kibana into obs-ai-assistan…

342a546

…t-screen-actions

dgieselaar marked this pull request as ready for review March 13, 2024 15:40

dgieselaar requested a review from a team as a code owner March 13, 2024 15:40

dgieselaar added release_note:enhancement v8.14.0 Team:obs-knowledge Observability Experience Knowledge team labels Mar 13, 2024

kibanamachine and others added 3 commits March 13, 2024 15:47

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

19bd361

…fix'

Fix functional tests

605f18a

Merge branch 'obs-ai-assistant-screen-actions' of github.com:dgiesela…

3d6c949

…ar/kibana into obs-ai-assistant-screen-actions

miltonhultgren self-requested a review March 14, 2024 09:37

miltonhultgren approved these changes Mar 14, 2024

View reviewed changes

dgieselaar added 4 commits March 15, 2024 13:30

Review feedback

d05d2dd

More review feedback

b032a47

Merge branch 'main' of github.com:elastic/kibana into obs-ai-assistan…

2198889

…t-screen-actions

Fix tests

ff3b039

dgieselaar merged commit acc5beb into elastic:main Mar 16, 2024
19 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Mar 16, 2024

dgieselaar deleted the obs-ai-assistant-screen-actions branch March 17, 2024 08:25

miltonhultgren mentioned this pull request Mar 20, 2024

[Obs AI Assistant] Fix passing of response language #179064

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Obs AI Assistant] Implement contextual actions #178405

[Obs AI Assistant] Implement contextual actions #178405

dgieselaar commented Mar 11, 2024 •

edited

Loading

apmmachine commented Mar 11, 2024

dgieselaar commented Mar 13, 2024

elasticmachine commented Mar 13, 2024

miltonhultgren left a comment

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

miltonhultgren Mar 14, 2024

dgieselaar Mar 14, 2024

kibana-ci commented Mar 15, 2024

API count

		const originalPushState = window.history.pushState.bind(window.history);
		const originalReplaceState = window.history.replaceState.bind(window.history);

[Obs AI Assistant] Implement contextual actions #178405

[Obs AI Assistant] Implement contextual actions #178405

Conversation

dgieselaar commented Mar 11, 2024 • edited Loading

apmmachine commented Mar 11, 2024

🤖 GitHub comments

dgieselaar commented Mar 13, 2024

elasticmachine commented Mar 13, 2024

miltonhultgren left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Mar 15, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

API count

History

dgieselaar commented Mar 11, 2024 •

edited

Loading