CLBlast: q5_0, q5_1, q8_0 dequant kernels

[0cc4m]

I had or still have an issue with q5_0 that I can't figure out. On Nvidia trying to transfer the quantized weights to the device leads to a CL_OUT_OF_RESOURCES error. On AMD and on POCL it leads to a segfault. It seems to have a problem with 22 byte structs, while 20 or 24 bytes are alright. I am not sure why this is the case.

As a workaround I copy the weights into a new struct and do the FP16 to FP32 conversion on CPU. This seems to have little overhead and works, but it should not be needed. If anyone knows what's up here please let me know.

I also moved the .cl file into the opencl.c as requested.

[LostRuins]

lgtm

[ggerganov]

For Q5_0 you have to copy the qh bytes into a uint32_t instead of casting a pointer as we usually do:

llama.cpp/ggml-cuda.cu

Lines 134 to 137 in c3ca7a5

	uint32_t qh;
	memcpy(&qh, x[i].qh, sizeof(qh));

[0cc4m]

@ggerganov I did, that happens during the transfer to the GPU.

https://github.com/0cc4m/koboldcpp/blob/369d903edabd5c0acd866337542cbb4150485940/ggml-opencl.c#L74-L79

It works just fine for Q5_1. Q5_0 has a different problem that I can only suspect is related to memory alignment.

[ggerganov]

Use 2x memcpy - one for qh and one for qs

[0cc4m]

Done

[LostRuins]

Btw might want to add a #include <stdlib.h> as it appears to prevent compilation on Termux without it.

[SlyEcho]

It was an alignment / padding issue. the half and uint were together 8 bytes, not 6 as expeted which meant the whole struct was too big and the memory was accessed out of bounds.

In #1422 I added an __attribute__((packed)) to the CL struct. It's not an extension or anything, it's part of the OpenCL language.