Absence of Proper Response Handling for Code Blocks in JSON Outputs

[kangsuhyun-yanolja]

The Gemma model is trained to include code blocks when returning JSON responses, but our service does not consider this behavior. As a result, these responses are often treated as bad responses. Currently, our system retries when it encounters this issue, and if it still fails, the response is returned as is, leading users to think the output is incorrect even when it could be valid.

To address this, we should improve response handling by detecting and trimming code blocks around JSON outputs automatically. This approach would prevent the system from misclassifying these responses as failures and reduce unnecessary retries.