Add Support for Prompt Caching in SpiceMessages Class #107
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![mentatai[bot]](https://avatars.githubusercontent.com/in/850097?s=60&v=4){kind=small}
This update introduces support for Anthropic's prompt caching feature to the `SpiceMessages` class in the `spice` library. The changes will enable faster and more cost-efficient API calls by reusing cached prompt prefixes. Additionally, cache performance metrics are tracked to verify cache hits and the number of input tokens cached.
### Key Changes:
1. **Update `SpiceMessages` Class:**
- Added a `cache` argument to the message creation methods.
- Set the `cache_control` parameter based on the `cache` argument.
2. **Modify Message Creation Functions:**
- Updated the message creation functions to handle the `cache` argument.
3. **Track Cache Performance Metrics:**
- Updated the `get_response` method in the `Spice` class to handle the new API response fields related to caching.
- Logged the number of input tokens cached and verified cache hits using the `client.extract_text_and_tokens` method.
### Example Usage:
```python
from spice import Spice
from spice.spice_message import SpiceMessages
client = Spice()
messages = SpiceMessages(client)
messages.add_system_message("You are an AI assistant tasked with analyzing literary works.", cache=True)
messages.add_user_message("Analyze the major themes in 'Pride and Prejudice'.", cache=True)
response = await client.get_response(messages=messages, model="claude-3-5-sonnet-20240620")
print(response.text)
```
### Acceptance Criteria:
- The `SpiceMessages` class should support the `cache` argument.
- The `get_response` method should log cache performance metrics.
- The implementation should be backward compatible and ignore the `cache` argument for non-Anthropic clients.
This commit addresses the CI failure encountered in the previous pull request by reformatting the `spice_message.py` file to comply with the Ruff formatting standards. The changes ensure that the code adheres to the required style guidelines, allowing the CI pipeline to pass successfully. ### Key Changes: 1. **Reformatted Function Definitions:** - Adjusted the formatting of function definitions to ensure proper line breaks and indentation. 2. **Reformatted Error Messages:** - Updated the formatting of error messages to improve readability and comply with style guidelines. 3. **Reformatted Class Methods:** - Reformatted class methods to ensure consistent indentation and line breaks. These changes do not alter the functionality of the code but ensure that it meets the required formatting standards.
This commit addresses the CI failure encountered in the previous pull request by reformatting the `spice_message.py` file to comply with the Ruff formatting standards. The reformatting ensures that all code adheres to the required style guidelines, allowing the CI pipeline to pass successfully.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This update introduces support for Anthropic's prompt caching feature to the
SpiceMessagesclass in thespicelibrary. The changes will enable faster and more cost-efficient API calls by reusing cached prompt prefixes. Additionally, cache performance metrics are tracked to verify cache hits and the number of input tokens cached.Key Changes:
Update
SpiceMessagesClass:cacheargument to the message creation methods.cache_controlparameter based on thecacheargument.Modify Message Creation Functions:
cacheargument.Track Cache Performance Metrics:
get_responsemethod in theSpiceclass to handle the new API response fields related to caching.client.extract_text_and_tokensmethod.Example Usage:
Acceptance Criteria:
SpiceMessagesclass should support thecacheargument.get_responsemethod should log cache performance metrics.cacheargument for non-Anthropic clients.Closes #106
Thanks for using MentatBot. Give comments a 👍 or 👎 to help me improve!