Prompt caching

[StefanKarlsson321]

Hi! As far as I understand prompt caching is not supported on the invoke_model for chat completion and the converse calls. Is there an ongoing initiative to support it?

[jakob-keller]

Thanks for reaching out! Can you elaborate a bit? Does botocore support "prompt caching"? Is there any documentation available?

[StefanKarlsson321]

Yes, bedrock has implemented support for prompt caching: https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html.

It is only available to a sub-set of customers for now, but will be available to all customers at one point.

It seems botocore does NOT support prompt caching. When I add a cache point to test it with boto3 the response is:

botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in messages[0].content[1]: "cachePoint", must be one of: text, image, document, video, toolUse, toolResult, guardContent

Just wondering if this will be supported by aiobotocore anytime soon, but perhaps it is better to ask in the botocore project?

[jakob-keller]

aiobotocore is basically just a wrapper around functionality provided by botocore. Unless they support that feature, we most likely won't be able to. Please ask the botocore project. Presumably, they will provide support as soon as the feature nears GA.