Document how to get started without paying for API access

[pfmoore]

The getting started documentation suggests getting an API key in order to start using the tool. While that may be the best approach for significant levels of usage, it's off-putting for someone who wants to "try out" the tool, maybe as a first step in trying to understand what LLM tools can do. To make things worse, my experience is that the various AI companies are not particularly clear on what is available for free, and how much you can expect from "free tier" access. That's not surprising, as they are clearly looking to make a profit on the tools they have developed, but in the wider context of "AI assistants" there's a general sense of "look at what AI can do" rather than "look at whay you can get if you pay for it" - and trying things out via a tool like this is one way of trying to separate the hype from the reality in a tool-agnostic way.

While I appreciate that it's not the place of the documentation here to explain pricing and benefits of various LLM offerings, it would be extremely useful if there was a section in the documentation explaining how to get started without spending money on an API key. In addition, clearly distinguishing where features or capabilities rely on a paid API key and are unavailable, or impractical, with current free access, would make it a lot easier for someone interested in finding out more about LLM tools to get an idea of what's possible.

[woog97]

Some platforms give free credits, for example deepinfra gives $1.80 is one of the cheaper platforms for API inference. I think in all my personal usage over the past few months, I've used a total of $0.25 of that fund. If you copy in entire codebases at a time with llama-405b you may go through that allowance faster, but this is a reasonable starting place.

[pfmoore]

Thanks, but that's sort of my point. Finding which services provide free tiers, how much credit such tiers give, and how good the engine behind the free tier is, isn't easy. And it usually involves creating a login to a service, as well as possibly (I haven't checked on this) providing credit card details. I don't want to do any of that - I'm trying to establish whether LLMs are worth my time, without any up-front commitment¹. My experiments with the free web-based services (ChatGPT and the like) have been pretty disappointing, but I'm willing to consider that there may be better options.

What I'm asking for is some form of "how to get started for free" guide in the manual, that takes the user step by step through something that results in being able to try out a model that, in the view of the project maintainers, is a good starting point for an interested beginner who doesn't want to sign up for a (potentially) paid service.

Footnotes

1. In particular, I'm very uncomfortable with the ethical basis of the training data, and I'm reluctant to give my details to organisations that appear to have such a fast-and-loose attitude towards personal data. ↩

[hardik1408]

I guess the project can be extended to open LLMs, so that people can atleast try out the tool, without actually needing to buy an API key.

[simonw]

Google Gemini offers a generous free tier, which you can use with the https://github.com/simonw/llm-gemini plugin.

llm install llm-gemini

Get an API key from here: https://aistudio.google.com/app/apikey

Then you can set it like so:

llm keys set gemini
# Paste key here

And use it like this:

llm -m gemini-1.5-flash-latest 'a story about a duck and a walrus'