Theia AI LLM Support [Experimental]

[sdirix]

What it does

Implements support for creating assistants ("agents") for Theia-based applications via integrating AI (more specifically "LLM") support as optionally consumable Theia extensions, a.k.a "Theia AI"

The base functionality is provided by the following extensions:

• @theia/ai-core
• @theia/ai-chat
• @theia/ai-chat-ui

ai-core contains the basic LLM integration and defines the core concepts for interacting with LLM via agents, prompts and variables. ai-chat builts on top to define a model for chat like conversations. ai-chat-ui provides the actual Chat UI.

The AI integration was built from the ground up to be flexible, inspectible, customizable and configurable. More precisely, we want to follow these core principles:

1. Transparency of Communication: Theia AI provides full visibility into the history of data exchanged between the IDE and the Large Language Models (LLMs).
2. Transparency of Data Interactions: Users gain full transparency over data accessed and modified by the LLM, including any automatic changes like inserted code snippets.
3. Openness of Prompts: Prompts used by AI agents are visible and customizable, allowing developers to optimize LLM requests for their specific environments and innovate in designing efficient prompts for developer tasks.
4. Flexibility in Choosing LLMs: Theia AI supports integrating arbitrary LLMs, enabling developers to use their preferred models and hosting environments, whether cloud-based, self-hosted, or local solutions, providing greater independence from proprietary constraints.

This feature is still highly experimental. We will actively continue to develop this feature. However, we want to provide early access to potential adopters and users of the Theia IDE. Therefore, even when the AI extensions are included in a Theia based application, they are turned off by default and need to be enabled in the preferences. The preferences include a convenient turn all AI features off setting.

Additional features and integrations are offered by the remaining
extensions:

• @theia/ai-history
• @theia/ai-code-completion
• @theia/ai-terminal
• @theia/ai-workspace-agent
• @theia/ai-openai

ai-history offers a service to record requests and responses. The recordings can be inspected via the 'AI History View'. ai-code-completion offers AI based code completion via completion items and inline suggestions. ai-terminal offers a specialized AI for the Theia terminal which will suggest commands to execute. ai-workspace-agent is a specialized agent which is able to inspect the current workspace content for context specific questions. ai-openai integrates the LLM offerings of Open AI into Theia as an LLM provider

How to test

• Start either the browser of Electron example application. Note that the only available LLM integration via Open AI is running in the backend. So while most of the features conceptually work in browser-only, they can't be tested without integrating a browser-only LLM
• Enable the AI features within the preferences
• If you don't have the OPENAI_API_KEY environment variable set, then configure your API key in the preferences
• Play around with the Chat, Terminal, Code Completion, History, Prompt Editing etc. etc.

Demo examples

Chat View

Prompt Editing

Terminal

Architecture

The following diagram gives a rough overview over the architecture and the intended communication paths

Chat Flow

• The user invokes a new session via the Chat UI which in turn uses the ChatService
• The ChatService resolves variables, if there are any in the request, and determines the agent to use. It invokes the ChatAgentService to do so
• The invoked agent is responsible to collect all information it requires and then invoke the desired LLM. The base implemention will
  • use the PromptService to receive its basic prompt. The PromptService may return the originally registered prompt or a user modified variant
  • resolve variables and tools specified within the prompt (or request). The user is able to inspect all variables and tools the agent will use via an "AI Configuration" view.
  • ask for an LLM with specific capabilities from the LanguageModelRegistry. The user is able to manually specify which specific LLM shall be used within the "AI Configuration" view
  • finally invoke the LLM
• The response of the LLM is converted by the agent into ChatResponseParts
• A registry of ChatResponsePartRenderers within the UI take over the rendering of the parts. The registry can be contributed to by developers.

Chat Flow Considerations

• The chat flow is similar to the implementation of VS Code, however it's much more generic and customizable
• This should allow Theia to support VS Code AI extensions in the future
• We proxy requests to LLMs to log the requests and responses automatically to output channels
• Agents are given tools for a more high-level logging capability with the CommunicatonRecordingService which brings its own "AI History" view

Orchestrator Agent

• The default agent is the Orchestrator. It will invoke an LLM itself to determine which agent shall handle the current request
• This can be bypassed by manually specifying the agent to be used using @AgentName within the chat request or by specifying another default agent in the preferences

Other Flows and Considerations

Contrary to VS Code we don't force all LLM integrations through the ChatFlow. Agents are not necessarily ChatAgents and can therefore provide customized API tailored to specific use cases. In this first contribution we implemented a CodeCompletionAgent as well as a TerminalAgent which are specialized for their use cases and can't be used in Chat.
Still all agents can participate in global services like the AI History, whether they are chat related or not.

All of the features and concepts are optional. If some developer only wants to use the LLM base layer to invoke their own specialized LLM without going through chats, prompts etc., then this of course possible.

Frontend / Backend Split

The diagram above gives a rough overview over the frontend/backend split.

• All UI related components are, of course, located in the frontend
• All generic AI services have a common base implementation and can therefore be bound and used in both the frontend and the backend. This is marked in orange in the diagram
• We expect many LLMs to be integrated in the backend, either because they are actually running locally, the API is only available on server side or secrets need to be hidden. Therefore we implemented an automatic frontend delegation of all backend registered LLMs. As long as the LLM in question follows the provided interfaces, no additional work is necessary to bridge the RCP gap, including response streaming.
• Most services in the frontend don't use the common implementation but a specialized one, integrating features only available in the frontend. For example as preferences and the workspace are frontend concepts, the respective services only respect these customization in the frontend.
• Conceptually the AI integration is therefore also ready for browser-only. However in this initial contribution we only include Open AI bindings for the backend.
• Generally speaking we expect most agents and functionality to be developed within the frontend. However the concept of basic common implementations should be kept so that headless Theia applications can also use the backend Theia AI integration if they want to (grey box)

Follow-ups

As noted above, this contribution is highly experimental. Therefore there are a lot of known issues and potential followups. We still think that this contribution is already valuable enough to be included in Theia.

Incomplete list of follow up topics:

• Provide LLM integration with Ollama / Llamafile
• Serve VS Code AI API
• Refine all default prompts to deliver better results
• All variables and tools specified in prompts shall be listed per agent in the AI Configuration View
• Make sure UI contributions of enabled/disabled agents are properly included/excluded on the fly
• Allow to set hard and soft limits for tool functions
• Allow no-code registration and configuration of new chat agents
• Stabilize response format and error handling for unexpected LLM answers
• Orchestrator Chat Agent should log its own requests and responses
• Improve AI History view (sort, clear, search etc.)
• Implement a code-diff response part
• Make editor completion configurable (enabled/disable, different prompts per file type)
• Invoke chat from different contexts, e.g. from editor, from terminal, etc.

Many more features and improvements can be made. We will continue to actively develop this feature, however we think the current code is ready to be evaluated by the community.

We already implemented POCs for an Llamafile integration, RAG based variable resolving as well specialized agents like a "PR Finalization Agent". However these are not yet stable enough to be part of the initial contribution.

We are excited what the future will bring and which concepts and ideas will be contributed by the community.

Review checklist

• As an author, I have thoroughly tested my changes and carefully followed the review guidelines

Reminder for reviewers

• As a reviewer, I agree to behave in accordance with the review guidelines

Addendum

@dhuebner contributed an initial Ollama integration which allows to use locally executed models. See here for more information.

[dhuebner]

Q: Some vendors explicitly support code-completion/generation/fill-in-the-middle functionality. Is it planned to have a dedicated LanguageModelRequest type or maybe a dedicated function here?

[sdirix]

Good question! From a technical standpoint both approaches are feasible, covering all use cases in request with flexible input and output parameters or implementing different functions.

For now all use cases were similar enough that they did not warrant a separate function however that might change with new use cases or other architectural considerations.

[sdirix]

Update: @dhuebner contributed an initial Ollama integration to this contribution. The ollama integration allows to specify the URL on which to contact ollama and the respective model id. This is very nice as it allows to run and test the AI integration locally.

Note that this is a barebones integration, for example the "tool" integration is missing which is used by the workspace agent. Therefore some of the features are restricted when using Ollama.

These are the suggested follow ups:

• Tools request handling is not implemented yet.
• Check for locally available Ollama models and suggest them in the settings.
• Automatically pull Ollama model on user request.
• Use Ollama specific prompts (PREFIX/SUFFIX) for code completion requests

[JonasHelming]

@tsmaeder @msujew Now that the release is done, could you have a look at this, as discussed?

[sdirix]

Hi @tsmaeder Thank you so much for the great and very detailed review ❤️ !

I tackled the major part of the smaller to medium review comments in commit 566b447 and marked them as resolved. I answered on all review questions.

For everything which is not yet covered I suggest to move the topic to this follow up #14143. I posted a link to the follow up on all relevant review comments.

Merging the current state allows for easier contributions and smaller PRs going forward. So if you are fine with the current state and the suggested follow ups, then please approve. I'll squash to two commits then, the initial EclipseSource contribution and @dhuebner's Ollama support.

[sdirix]

@tsmaeder Thank you for your detailed answers. I applied further changes in commit 4d96c19 and suggested some topics for the follow up ticket #14143. Please let me know if you have further comments or concerns and what is required to get your approval stamp ;)

[sdirix]

@tsmaeder Thanks for the further comments. I added all remaining topics to the follow up #14143.

[tsmaeder]

I think we can merge the PR as an experimental feature and address the follow-ups at a later stage.

[sdirix]

I squashed the contribution into two separate commits:

• The initial Theia LLM contribution
• The additional Ollama integration contributed by @dhuebner

No further changes were done in the force push.

@tsmaeder Can you rebase-merge? Thank you!

Let me know in case you would like to see different commit messages

[sdirix]

I forced pushed once more as the license check failed. There were minor yarn.lock changes because I ran yarn once manually during the rebase/squash process. I restored the previously approved yarn.lock now.