Library Interface Considerations for FFI (passing plain structs)

[julmb]

I am using llama.cpp as a dynamic library from Haskell via FFI. Haskell's FFI system only supports basic types like char, int, float, bool, and pointers to be passed to and from foreign functions. In order to work around this, I currently have a C wrapper that wraps those functions from the llama.h interface that use plain structs (like llama_load_model_from_file or llama_batch_init) to use pointers to structs instead.

There seem to be only a handful of those cases so it feels feasible to adjust the llama.h interface to restrict itself to functions passing only basic types, which would make my life a little easier. However, it would either be a breaking change or introduce redundant functions. This might not be worth it unless other people would also benefit from this change.

So my questions are:

• Are other people (maybe using llama.cpp via FFI from other languages) also affected by this?
• What is the policy on changes to the llama.h interface? How big of a benefit does a change need to have to justify a breaking change or a potentially redundant addition?

[ggerganov]

I think it is unlikely to change the existing functions, unless it turns out to be a common limitation for other FFI systems. Adding "overloads" might be more realistic.

What about functions that return structs - do you need/have a workaround?

    LLAMA_API struct llama_model_params llama_model_default_params(void);
    LLAMA_API struct llama_context_params llama_context_default_params(void);
    LLAMA_API struct llama_model_quantize_params llama_model_quantize_default_params(void);

[julmb]

That is okay, I would not expect to make breaking changes in the library interface just because of a single user being inconvenienced.

For functions returning structs, I currently have the following wrappers:

size_t model_params_size()
{
	return sizeof(struct llama_model_params);
}
void model_params_default(struct llama_model_params * params)
{
	*params = llama_model_default_params();
}

Something like this would work too:

struct llama_model_params * model_params_init()
{
	struct llama_model_params * params = malloc(sizeof(struct llama_model_params));
	*params = llama_model_default_params();
	return params;
}
void model_params_free(struct llama_model_params * params)
{
	free(params);
}

The second approach is a bit more convenient from the perspective of the FFI user, since all the memory allocation and associated size measurements happen on the C side.

I have similar wrappers for llama_context_params and llama_batch.

If this is something that would in principle be acceptable to have in the library, I could draft up a PR, that would make it easier to talk about the interface design, unless you already have some opinions or ideas on how to do this.

I am mostly worried about making the library interface redundant/ambiguous/confusing just for my singular use case and I am not sure how to get to a long-term clean interface that is non-redundant and works for everyone.

[ggerganov]

I guess we can extend the API with a few extra functions that accept/return pointers. It is a bit redundant, but it seems the implementation of these functions will be very trivial. We can put it at the end of the header/source and add comments to avoid using these unless the FFI system does not accept structs. The new API should share a common suffix, for example: struct llama_model_params * llama_model_default_params_ptr(); or something more descriptive. Feel free to open a PR and unless we think of some significant drawbacks that I am not seeing atm, we could merge it.

[julmb]

I have had some time to think about this and I realized that there is another problem. It is not enough to be able to create/free the structs, they also need to be filled. One option would be to add Java-style setter functions like this:

void context_params_set_n_ctx(struct llama_context_params * params, uint32_t value) { params->n_ctx = value; }

This feels a little out of place in a C library, and I am not sure if it is a good idea unless a significant number of other people would also find something like this useful.

In principle, I could also calculate the offset of n_ctx within the llama_context_params struct and set it directly in memory, but this is very fragile and since the offset calculation happens at runtime, the compiler cannot tell me if the struct layout has changed and my code no longer does the right thing. Haskell has some tool support do calculate these offsets automatically (hsc2hs and c2hs), but they come with their own drawbacks and additional complexity.

[bullno1]

C alignment is fairly well-defined.
You "just" need to be able to parse the header to get the struct layout.
I know of one way to do it: https://floooh.github.io/2020/08/23/sokol-bindgen.html.
Basically, parse clang's JSON AST output and simplify it down to a JSON that describes the struct and functions.

To be FFI friendly, the struct/signature definition file could be automatically generated.

[julmb]

Yeah, hsc2hs and c2hs do just that, they read C header files and generate struct access code from that. I considered using them, but since I needed a C wrapper anyways (due to functions passing structs as values), it didn't really seem like it was worth the extra complexity.