feat: Theia AI integration

[sdirix]

What it does

Implements support for creating assistants ("agents") for Theia-based applications via integrating AI (more specifically "LLM") support as optionally consumable Theia extensions, a.k.a "Theia AI"

The base functionality is provided by the following extensions:

• @theia/ai-core
• @theia/ai-chat
• @theia/ai-chat-ui

ai-core contains the basic LLM integration and defines the core concepts for interacting with LLM via agents, prompts and variables. ai-chat builts on top to define a model for chat like conversations. ai-chat-ui provides the actual Chat UI.

The AI integration was built from the ground up to be flexible, inspectible, customizable and configurable. More precisely, we want to follow these core principles:

1. Transparency of Communication: Theia AI provides full visibility into the history of data exchanged between the IDE and the Large Language Models (LLMs).
2. Transparency of Data Interactions: Users gain full transparency over data accessed and modified by the LLM, including any automatic changes like inserted code snippets.
3. Openness of Prompts: Prompts used by AI agents are visible and customizable, allowing developers to optimize LLM requests for their specific environments and innovate in designing efficient prompts for developer tasks.
4. Flexibility in Choosing LLMs: Theia AI supports integrating arbitrary LLMs, enabling developers to use their preferred models and hosting environments, whether cloud-based, self-hosted, or local solutions, providing greater independence from proprietary constraints.

This feature is still highly experimental. We will actively continue to develop this feature. However, we want to provide early access to potential adopters and users of the Theia IDE. Therefore, even when the AI extensions are included in a Theia based application, they are turned off by default and need to be enabled in the preferences. The preferences include a convenient turn all AI features off setting.

Additional features and integrations are offered by the remaining
extensions:

• @theia/ai-history
• @theia/ai-code-completion
• @theia/ai-terminal
• @theia/ai-workspace-agent
• @theia/ai-openai

ai-history offers a service to record requests and responses. The recordings can be inspected via the 'AI History View'. ai-code-completion offers AI based code completion via completion items and inline suggestions. ai-terminal offers a specialized AI for the Theia terminal which will suggest commands to execute. ai-workspace-agent is a specialized agent which is able to inspect the current workspace content for context specific questions. ai-openai integrates the LLM offerings of Open AI into Theia as an LLM provider

How to test

• Start either the browser of Electron example application. Note that the only available LLM integration via Open AI is running in the backend. So while most of the features conceptually work in browser-only, they can't be tested without integrating a browser-only LLM
• Enable the AI features within the preferences
• If you don't have the OPENAI_API_KEY environment variable set, then configure your API key in the preferences
• Play around with the Chat, Terminal, Code Completion, History, Prompt Editing etc. etc.

Demo examples

Chat View

Terminal

Architecture

The following diagram gives a rough overview over the architecture and the intended communication paths

Chat Flow

• The user invokes a new session via the Chat UI which in turn uses the ChatService
• The ChatService resolves variables, if there are any in the request, and determines the agent to use. It invokes the ChatAgentService to do so
• The invoked agent is responsible to collect all information it requires and then invoke the desired LLM. The base implemention will
  • use the PromptService to receive its basic prompt. The PromptService may return the originally registered prompt or a user modified variant
  • resolve variables and tools specified within the prompt (or request). The user is able to inspect all variables and tools the agent will use via an "AI Configuration" view.
  • ask for an LLM with specific capabilities from the LanguageModelRegistry. The user is able to manually specify which specific LLM shall be used within the "AI Configuration" view
  • finally invoke the LLM
• The response of the LLM is converted by the agent into ChatResponseParts
• A registry of ChatResponsePartRenderers within the UI take over the rendering of the parts. The registry can be contributed to by developers.

Chat Flow Considerations

• The chat flow is similar to the implementation of VS Code, however it's much more generic and customizable
• This should allow Theia to support VS Code AI extensions in the future
• We proxy requests to LLMs to log the requests and responses automatically to output channels
• Agents are given tools for a more high-level logging capability with the CommunicatonRecordingService which brings its own "AI History" view

Delegating Agent

• The default agent is the DelegatingChatAgent. It will invoke an LLM itself to determine which agent shall handle the current request
• This can be bypassed by manually specifying the agent to be used using @AgentName within the chat request

Other Flows and Considerations

Contrary to VS Code we don't force all LLM integrations through the ChatFlow. Agents are not necessarily ChatAgents and can therefore provide customized API tailored to specific use cases. In this first contribution we implemented a CodeCompletionAgent as well as a TerminalAgent which are specialized for their use cases and can't be used in Chat.
Still all agents can participate in global services like the AI History, whether they are chat related or not.

All of the features and concepts are optional. If some developer only wants to use the LLM base layer to invoke their own specialized LLM without going through chats, prompts etc., then this of course possible.

Frontend / Backend Split

The diagram above gives a rough overview over the frontend/backend split.

• All UI related components are, of course, located in the frontend
• All generic AI services have a common base implementation and can therefore be bound and used in both the frontend and the backend. This is marked in orange in the diagram
• We expect many LLMs to be integrated in the backend, either because they are actually running locally, the API is only available on server side or secrets need to be hidden. Therefore we implemented an automatic frontend delegation of all backend registered LLMs. As long as the LLM in question follows the provided interfaces, no additional work is necessary to bridge the RCP gap, including response streaming.
• Most services in the frontend don't use the common implementation but a specialized one, integrating features only available in the frontend. For example as preferences and the workspace are frontend concepts, the respective services only respect these customization in the frontend.
• Conceptually the AI integration is therefore also ready for browser-only. However in this initial contribution we only include Open AI bindings for the backend.
• Generally speaking we expect most agents and functionality to be developed within the frontend. However the concept of basic common implementations should be kept so that headless Theia applications can also use the backend Theia AI integration if they want to (grey box)

Follow-ups

As noted above, this contribution is highly experimental. Therefore there are a lot of known issues and potential followups. We still think that this contribution is already valuable enough to be included in Theia.

Incomplete list of follow up topics:

• Provide LLM integration with Ollama / Llamafile
• Serve VS Code AI API
• Refine all default prompts to deliver better results
• All variables and tools specified in prompts shall be listed per agent in the AI Configuration View
• Make sure UI contributions of enabled/disabled agents are properly included/excluded on the fly
• Allow to set hard and soft limits for tool functions
• Allow no-code registration and configuration of new chat agents
• Stabilize response format and error handling for unexpected LLM answers
• Delegating Chat Agent should log its own requests and responses
• Improve AI History view (sort, clear, search etc.)
• Implement a code-diff response part
• Make editor completion configurable (enabled/disable, different prompts per file type)
• Invoke chat from different contexts, e.g. from editor, from terminal, etc.

Many more features and improvements can be made. We will continue to actively develop this feature, however we think the current code is ready to be evaluated by the community.

We already implemented POCs for an Llamafile integration, RAG based variable resolving as well specialized agents like a "PR Finalization Agent". However these are not yet stable enough to be part of the initial contribution.

We are excited what the future will bring and which concepts and ideas will be contributed by the community.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

[sdirix]

Superseded by the "real" PR: eclipse-theia#14048