Add API to summarize text

[ChristophWurst]

How to use GitHub

• Please use the 👍 reaction to show that you are interested into the same feature.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

Is your feature request related to a problem? Please describe.

As a Nextcloud developer I want to be able to summarize text for the users of the app. E.g. to get the gist of an email thread.

Describe the solution you'd like

Provide an OCP API and optionally an OCS API where an app can send long text and receive a short summary text. The API has to be optional because not every installation will have a backend for language processing.

Describe alternatives you've considered

N/a

Additional context

cc @DaphneMuller

[marcelklehr]

Word of caution: The new llm app will be rather slow at least for now (ie. ~10+ minutes for a summary), so it would behoove us to avoid allowing users to generate summaries on demand but restrict use cases to scenarios where we prepare the summary in advance.

[ChristophWurst]

In today's discussion with @AndyScherzinger and @marcoambrosini we identified some open questions

1. How does the runtime complexity scale with the size of the input? Is it static? In other words, would a thread of three messages take roughly as much time to summarize as a thread of 25 messages?
2. With uncertain quality of the output text we wonder if the output format can be influence, e.g. to not just dump a few sentences but bulletpoints with important infos or "hard facts".
   1. If the format output format is not just sentences without formatting but paragraphs, lists or similar, would it make sense to specify if e.g. markdown or HTML is returned to make processing and rendering deterministic?
3. How will the language model work with localization? Does it detect the source language? Are languages other than English supported? Do we need to specify the source text language?
4. Do we want users to be able to opt-out of AI features, e.g. when they are not happy with the results?
5. Where does the LLM processing happen and could it block the execution of any other background jobs?

[marcelklehr]

1. On CPU it's linear or worse, because there's no parallel processing. On GPU I believe it can be constant, but the length of the output affects runtime negatively as well. In the current iteration we're only targeting CPUs.
2. I think this is unlikely to work consistently. The current model especially cannot produce well-formed markup.
3. The current model supports English only
4. I would recommend this.
5. LLM processing currently happens on the same machine as nextcloud on the CPU in sequential background tasks (no two LLM tasks are allowed to run in parallel)

If you'd like to try the model yourself (which I recommend to suss out what it's capable of and what not), you can install https://gpt4all.io on your computer. Currently I'm using the GPT4All-v1.3-groovy model. better models will become available in the coming weeks.

[marcoambrosini]

4. If these are the current limitations I'd suggest to flip it into an "opt in at your own risk" feature rather than just giving the ability to opt out, or wait until the technology is more reliable.

[DaphneMuller]

but the features will only be available / visible to the user anyway if the LLM app is enabled, right? Wouldn't a warning in the readme of the LLM app then be enough?

[AndyScherzinger]

I also wouldn't do opt-in since that lowers the adoption rate and rather have opt-out. I think for the moment like Daphne said, it can be "managed" via having the app enabled or disabled. If the results are bad they will be across the user base and if they are good they will be good across the user base (very simplified, I know). So at some point the app will be activated/deactivated by admins. So yeah - I also think info in the readme should be enough. maybe app description in the info.xml since that is shown in the app store.

[marcoambrosini]

How about default active if LLM app is enable but still give ability to opt out by just hiding the component in the front end? This could be very simply saved in the browser storage.

[marcelklehr]

Also worth noting: integration_openai and perhaps replicate might implement this API as well. Since they also have other uses opt out == disable app might not be the way to go.

[DaphneMuller]

ok, then it is up to product management to decide. @jancborchardt @AndyScherzinger

[AndyScherzinger]

That would be @jancborchardt @karlitschek call. I still think we shouldn't provide a user-level opt-in/-out option.

[jancborchardt]

Since we have more and more "Assistant" features, it might be nice to have a dedicated section in the settings on that.

There we can:

• Group all settings related to assistance, AI and such in one easily findable section
• Talk about Ethical AI and mark the relevant features accordingly
• Offer people to opt out of individual Assistant features (agree with @AndyScherzinger it should be opt-out in this case)

[DaphneMuller]

@jancborchardt can you make sure this is aligned where necessary with Frank so the devs can just implement it, and provide the necessary mock-ups / details / requirements for the devs? We can then see who can do the work.

[ChristophWurst]

To fulfill the updated requirements from nextcloud/mail#8508 (comment) the new API has to provide rich text as a result, not just plain text. We could go with Markdown or HTML. @marcelklehr would that be doable with the available LLMs?

[marcelklehr]

As mentioned in the kick-off meeting, rich-text/HTML is kinda unreliable with the current model. I think as a first iteration we can only rely on plain text.